Database Outage
Incident Report for Baremetrics
Resolved
Remaining accounts will be caught up shortly. A post mortem report is on the way as well.
Posted Oct 09, 2017 - 13:56 CDT
Monitoring
95% of accounts are now fully caught up. The rest will take a bit more time.
Posted Oct 08, 2017 - 13:27 CDT
Update
Most accounts are caught back up. Still working through the last few and monitoring overall stability.
Posted Oct 06, 2017 - 13:59 CDT
Update
During the night we managed to get most accounts caught up, but we have had to slow things down again this morning to do some more work on the DB cluster. We should see most accounts caught up today.
Posted Oct 06, 2017 - 06:10 CDT
Update
We believe we’ve found the issue and are working on rolling out a solution now.
Posted Oct 05, 2017 - 13:37 CDT
Update
We’re still trying to get to the bottom of the segmentation faults in the database cluster. Working directly with our database provider to create a new binary that gives us more insight in to the cause of the faults.
Posted Oct 05, 2017 - 10:03 CDT
Update
We are experiencing segmentation faults within our database cluster, this is causing nodes in the cluster to restart, which then delays our metric processing anymore. We are working as fast as we can to figure out a solution to this.
Posted Oct 05, 2017 - 07:03 CDT
Update
We are still working through these issues, but accounts are processing (at a slower rate than normal) and should be caught up soon.
Posted Oct 05, 2017 - 02:25 CDT
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 04, 2017 - 11:08 CDT
Investigating
We are currently investigating this issue.
Posted Oct 04, 2017 - 09:22 CDT