Remaining accounts will be caught up shortly. A post mortem report is on the way as well.
Posted 13 days ago. Oct 09, 2017 - 13:56 CDT
95% of accounts are now fully caught up. The rest will take a bit more time.
Posted 14 days ago. Oct 08, 2017 - 13:27 CDT
Most accounts are caught back up. Still working through the last few and monitoring overall stability.
Posted 16 days ago. Oct 06, 2017 - 13:59 CDT
During the night we managed to get most accounts caught up, but we have had to slow things down again this morning to do some more work on the DB cluster. We should see most accounts caught up today.
Posted 16 days ago. Oct 06, 2017 - 06:10 CDT
We believe we’ve found the issue and are working on rolling out a solution now.
Posted 17 days ago. Oct 05, 2017 - 13:37 CDT
We’re still trying to get to the bottom of the segmentation faults in the database cluster. Working directly with our database provider to create a new binary that gives us more insight in to the cause of the faults.
Posted 17 days ago. Oct 05, 2017 - 10:03 CDT
We are experiencing segmentation faults within our database cluster, this is causing nodes in the cluster to restart, which then delays our metric processing anymore. We are working as fast as we can to figure out a solution to this.
Posted 17 days ago. Oct 05, 2017 - 07:03 CDT
We are still working through these issues, but accounts are processing (at a slower rate than normal) and should be caught up soon.
Posted 17 days ago. Oct 05, 2017 - 02:25 CDT
The issue has been identified and a fix is being implemented.