All accounts should now be back online. We are continuing to monitor this.
Posted Nov 02, 2017 - 17:09 CDT
Update
Customer search is back for most accounts, with a few remaining.
Posted Nov 02, 2017 - 14:33 CDT
Update
Just an update on what is happening:
We use Elasticsearch for our customer search and profile pages. This allows us to scale search out to millions of records, and still deliver very fast search results.
Yesterday elasticsearch crashed, due to hitting the JVM 32gb heap limit. However, we did not notice this error as it was buried in the logs around other errors. We attempted to restore the cluster, which hit the limit again a few hours later.
Right now, we are scaling up more machines, and rebalancing the elasticsearch cluster to resolve this issue. Sadly the restore time is slow, but we are making progress.
Posted Nov 01, 2017 - 09:31 CDT
Identified
We have identified the issue, but it may take a while for this to recover. We will keep you updated on the progress.
Posted Oct 31, 2017 - 06:50 CDT
Investigating
The customer search section of the dashboard and API are currently unavailable