All accounts should now be back online. We are continuing to monitor this.
Posted over 1 year ago. Nov 02, 2017 - 17:09 CDT
Customer search is back for most accounts, with a few remaining.
Posted over 1 year ago. Nov 02, 2017 - 14:33 CDT
Just an update on what is happening:
We use Elasticsearch for our customer search and profile pages. This allows us to scale search out to millions of records, and still deliver very fast search results.
Yesterday elasticsearch crashed, due to hitting the JVM 32gb heap limit. However, we did not notice this error as it was buried in the logs around other errors. We attempted to restore the cluster, which hit the limit again a few hours later.
Right now, we are scaling up more machines, and rebalancing the elasticsearch cluster to resolve this issue. Sadly the restore time is slow, but we are making progress.
Posted over 1 year ago. Nov 01, 2017 - 09:31 CDT
We have identified the issue, but it may take a while for this to recover. We will keep you updated on the progress.
Posted over 1 year ago. Oct 31, 2017 - 06:50 CDT
The customer search section of the dashboard and API are currently unavailable