Remaining accounts will be caught up shortly. A post mortem report is on the way as well.
Oct 9, 13:56 CDT
95% of accounts are now fully caught up. The rest will take a bit more time.
Oct 8, 13:27 CDT
Most accounts are caught back up. Still working through the last few and monitoring overall stability.
Oct 6, 13:59 CDT
During the night we managed to get most accounts caught up, but we have had to slow things down again this morning to do some more work on the DB cluster. We should see most accounts caught up today.
Oct 6, 06:10 CDT
We believe we’ve found the issue and are working on rolling out a solution now.
Oct 5, 13:37 CDT
We’re still trying to get to the bottom of the segmentation faults in the database cluster. Working directly with our database provider to create a new binary that gives us more insight in to the cause of the faults.
Oct 5, 10:03 CDT
We are experiencing segmentation faults within our database cluster, this is causing nodes in the cluster to restart, which then delays our metric processing anymore. We are working as fast as we can to figure out a solution to this.
Oct 5, 07:03 CDT
We are still working through these issues, but accounts are processing (at a slower rate than normal) and should be caught up soon.
Oct 5, 02:25 CDT
The issue has been identified and a fix is being implemented.
Oct 4, 11:08 CDT
We are currently investigating this issue.
Oct 4, 09:22 CDT