Remaining accounts will be caught up shortly. A post mortem report is on the way as well.
Posted 2 months ago. Oct 09, 2017 - 13:56 CDT
95% of accounts are now fully caught up. The rest will take a bit more time.
Posted 2 months ago. Oct 08, 2017 - 13:27 CDT
Most accounts are caught back up. Still working through the last few and monitoring overall stability.
Posted 2 months ago. Oct 06, 2017 - 13:59 CDT
During the night we managed to get most accounts caught up, but we have had to slow things down again this morning to do some more work on the DB cluster. We should see most accounts caught up today.
Posted 2 months ago. Oct 06, 2017 - 06:10 CDT
We believe we’ve found the issue and are working on rolling out a solution now.
Posted 2 months ago. Oct 05, 2017 - 13:37 CDT
We’re still trying to get to the bottom of the segmentation faults in the database cluster. Working directly with our database provider to create a new binary that gives us more insight in to the cause of the faults.
Posted 2 months ago. Oct 05, 2017 - 10:03 CDT
We are experiencing segmentation faults within our database cluster, this is causing nodes in the cluster to restart, which then delays our metric processing anymore. We are working as fast as we can to figure out a solution to this.
Posted 2 months ago. Oct 05, 2017 - 07:03 CDT
We are still working through these issues, but accounts are processing (at a slower rate than normal) and should be caught up soon.
Posted 2 months ago. Oct 05, 2017 - 02:25 CDT
The issue has been identified and a fix is being implemented.