tag:status.baremetrics.com,2005:/historyBaremetrics Status - Incident History2024-03-29T04:49:43-05:00Baremetricstag:status.baremetrics.com,2005:Incident/191305312023-11-17T06:30:08-06:002023-11-17T06:30:08-06:00Database maintenance<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>06:30</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>05:30</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>14:14</var> CST</small><br><strong>Scheduled</strong> - We will be performing maintenance to one of our databases. This will cause some disruption and may result in ~20 minutes of downtime to the Baremetrics dashboard. We will post updates when the maintenance has been completed.</p>tag:status.baremetrics.com,2005:Incident/191080562023-11-14T06:31:24-06:002023-11-14T06:31:24-06:00Database maintenance<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>06:31</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>06:30</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>08:37</var> CST</small><br><strong>Scheduled</strong> - We will be performing maintenance to one of our databases. This will cause some disruption and may result in ~20 minutes of downtime to the Baremetrics dashboard. We will post updates when the maintenance has been completed.</p>tag:status.baremetrics.com,2005:Incident/169305122023-04-24T01:00:35-05:002023-04-24T01:00:35-05:00Database Maintenance<p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>01:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>00:02</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>08:09</var> CDT</small><br><strong>Scheduled</strong> - We will be performing maintenance to one of our databases. This will cause some disruption and may result in ~20 minutes of downtime to the Baremetrics dashboard. We will post updates when the maintenance has been completed.</p>tag:status.baremetrics.com,2005:Incident/139336492022-11-29T09:16:01-06:002022-11-29T09:16:01-06:00Database maintenance<p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>09:16</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>09:15</var> CST</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>09:00</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>06:37</var> CST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.baremetrics.com,2005:Incident/96532592022-03-29T01:39:26-05:002024-02-22T07:56:20-06:00500 Errors<p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>01:39</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>19:58</var> CDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>19:43</var> CDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>19:19</var> CDT</small><br><strong>Investigating</strong> - We're having some issues. We're working on it! 🔨</p>tag:status.baremetrics.com,2005:Incident/94003262022-02-24T11:16:44-06:002022-02-24T11:16:44-06:00Hosting provider outage<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>11:16</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>10:53</var> CST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>10:53</var> CST</small><br><strong>Investigating</strong> - Looking to resolve</p>tag:status.baremetrics.com,2005:Incident/68582412021-04-24T10:30:00-05:002021-04-27T19:59:03-05:00Dashboards Returning 500 Error Code For Some Customers<p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>10:30</var> CDT</small><br><strong>Resolved</strong> - Some customers have reported seeing 500 error code on their dashboards. The impact for those customers is 'Major', while other customers do have have issues.</p>tag:status.baremetrics.com,2005:Incident/60897942021-01-25T11:51:52-06:002021-01-25T11:51:52-06:00Intermittent Timeouts<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>11:51</var> CST</small><br><strong>Resolved</strong> - We identified the issue and accounts should be back to updating normally.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>15:59</var> CST</small><br><strong>Investigating</strong> - We are currently investigating why some accounts are experiencing timeouts and slow updates.</p>tag:status.baremetrics.com,2005:Incident/59592032021-01-06T10:30:31-06:002021-01-06T10:30:31-06:00Database Maintenance<p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>10:30</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>09:30</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>02:48</var> CST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this timeframe. Some parts of the dashboard may become temporarily unavailable.</p>tag:status.baremetrics.com,2005:Incident/58984782020-12-28T07:21:54-06:002020-12-28T07:21:54-06:00Customers & Segmentation downtime<p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>07:21</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>07:18</var> CST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>01:44</var> CST</small><br><strong>Investigating</strong> - Our Elasticsearch servers are having some trouble keeping up. This affects some Customers and Segmentation. Other parts of the app will keep working as usual.</p>tag:status.baremetrics.com,2005:Incident/58290102020-12-18T11:31:50-06:002020-12-18T11:31:50-06:00Hosting instability<p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>11:31</var> CST</small><br><strong>Resolved</strong> - Everything is back to normal. We're continuing to monitor as our systems are picking up where they left off, but don't expect any further issues.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>08:01</var> CST</small><br><strong>Update</strong> - We're back up and running on reduced capacity. Our top priority right now is getting everything back to full capacity, while ensuring stability of the whole system.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>07:39</var> CST</small><br><strong>Monitoring</strong> - We've rebuilt a large proportion of our infrastructure and we're slowly migrating traffic to it. Access should be resumed shortly.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>02:43</var> CST</small><br><strong>Update</strong> - We're still working hard to bring our app back up after networking issues took everything offline. We will post updates as we have them.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>17:26</var> CST</small><br><strong>Investigating</strong> - We are currently aware of issues occurring at our hosting partner's datacenter which has resulted in us going offline.<br /><br />We are working with them to identify a timeline for resolution.</p>tag:status.baremetrics.com,2005:Incident/57644022020-12-09T18:59:35-06:002020-12-09T18:59:35-06:00Account Update Delays<p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>18:59</var> CST</small><br><strong>Resolved</strong> - Most accounts are now caught up.</p><p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>14:56</var> CST</small><br><strong>Monitoring</strong> - Some accounts are running behind on updates due to a large import. Accounts are in the process of catching up, but it will take time for every account to update. Thanks for your patience!</p>tag:status.baremetrics.com,2005:Incident/57061662020-12-02T10:30:32-06:002020-12-02T10:30:32-06:00Database Maintenance<p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>10:30</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>08:44</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>08:44</var> CST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>08:38</var> CST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this timeframe. Some parts of the dashboard may become temporarily unavailable.</p>tag:status.baremetrics.com,2005:Incident/53881352020-10-22T10:08:15-05:002020-10-22T10:08:18-05:00Failing Stripe webhooks<p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>10:08</var> CDT</small><br><strong>Resolved</strong> - Everything seems to be back in order and we've confirmed all webhooks are properly processing now.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>09:43</var> CDT</small><br><strong>Monitoring</strong> - CloudFlare reports they've resolved the issue on their end. We're monitoring their fix now, but otherwise webhooks should be processing normally now and we're working our way through the backlog of previously failed webhooks.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>09:24</var> CDT</small><br><strong>Update</strong> - We've deployed a temporary fix that should allow Stripe API calls to process while we work with CloudFlare.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>09:13</var> CDT</small><br><strong>Identified</strong> - We've identified the issue is indeed with CloudFlare's rate-limiting incorrectly blocking IP addresses that we've whitelisted.<br /><br />Working with CloudFlare now to address this.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>08:48</var> CDT</small><br><strong>Investigating</strong> - We've detected that roughly half of all webhooks from Stripe are getting blocked. This appears to be a new issue with the way CloudFlare handles rate-limiting. In this case, they're incorrectly limiting calls from Stripe's IP addresses. We're working with them now to resolve this.<br /><br />All failing webhooks will automatically be reprocessed once this is resolved.<br /><br />We'll provide more updates as we have them.</p>tag:status.baremetrics.com,2005:Incident/53342692020-10-15T13:33:22-05:002020-10-15T13:33:22-05:00Recover emails not sending<p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>13:33</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>13:30</var> CDT</small><br><strong>Monitoring</strong> - We believe the outage has been resolved and email sending has been resumed. We'll keep monitoring the incident to make sure there are no further hiccups.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>12:34</var> CDT</small><br><strong>Identified</strong> - Our email service provider has identified the issue and is working on a fix now.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>12:29</var> CDT</small><br><strong>Update</strong> - Our email service provider appears to be having a widespread outage. Working on a way to potentially reroute emails in the meantime.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>12:16</var> CDT</small><br><strong>Investigating</strong> - Due to an issue with our email service provider, Recover emails are not sending. We're working with them now to resolve this. Any emails not sent are queued up and will be sent out once the issue is resolved.<br /><br />The caveat here is if you have a custom DNS setup for use with Recover, it will not be affected by this.<br /><br />We'll update as we have more info.</p>tag:status.baremetrics.com,2005:Incident/49913962020-09-15T18:00:29-05:002020-09-15T18:00:29-05:00Data Centre Maintenance<p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>18:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>17:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>03:15</var> CDT</small><br><strong>Scheduled</strong> - Our data centre is performing some network maintenance on September 16th between 18:00 and 19:00 EST (22:00 - 23:00 UTC). This will cause some disruption and may result in ~20 minutes of downtime to the Baremetrics dashboard. We will post updates when the maintenance has been completed.</p>tag:status.baremetrics.com,2005:Incident/48823462020-09-02T18:00:30-05:002020-09-02T18:00:30-05:00Data Centre Maintenance<p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>18:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>17:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>10:25</var> CDT</small><br><strong>Scheduled</strong> - Our data centre is performing some network maintenance on September 2nd between 18:00 and 19:00 EST (22:00 - 23:00 UTC). This will cause some disruption and may result in ~20 minutes of downtime to the Baremetrics dashboard. We will post updates when the maintenance has been completed.</p>tag:status.baremetrics.com,2005:Incident/47682072020-08-16T02:05:30-05:002020-08-16T02:05:30-05:00Data centre network upgrade<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>02:05</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>23:05</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>02:35</var> CDT</small><br><strong>Scheduled</strong> - Our data centre is performing some major network upgrades on July 19th between 12:05 AM and 03:05 AM EST (0405 - 0705 UTC). This will cause some disruption and may result in degraded performance to Baremetrics over the course of the window. We will post updates when the upgrades have been completed.</p>tag:status.baremetrics.com,2005:Incident/47261912020-07-29T12:00:32-05:002020-07-29T12:00:32-05:00Shopify account maintenance<p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>12:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>06:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>11:04</var> CDT</small><br><strong>Scheduled</strong> - We will be performing some upgrades to our Shopify implementation in the morning. During this time your account may be unavailable.<br /><br />If you haven't connected Shopify, you will not be affected.</p>tag:status.baremetrics.com,2005:Incident/45623992020-07-19T02:05:30-05:002020-07-19T02:05:31-05:00Data centre network upgrade<p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>02:05</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>23:05</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>06:05</var> CDT</small><br><strong>Scheduled</strong> - Our data centre is performing some major network upgrades on July 19th between 12:05 AM and 03:05 AM EST (0405 - 0705 UTC). This will cause some disruption and may result in degraded performance to Baremetrics over the course of the window. We will post updates when the upgrades have been completed.</p>tag:status.baremetrics.com,2005:Incident/46440902020-07-17T16:46:23-05:002020-07-17T16:46:23-05:00Intermittent availability<p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>16:46</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>16:27</var> CDT</small><br><strong>Monitoring</strong> - Our DNS provider appears to be having temporary issues preventing some users from accessing Baremetrics. We are monitoring for any changes and will update once resolved.</p>tag:status.baremetrics.com,2005:Incident/42682152020-06-21T05:05:31-05:002020-06-21T05:05:31-05:00Data centre network upgrade<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>05:05</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>23:06</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>10:19</var> CDT</small><br><strong>Scheduled</strong> - Our data centre is performing some major network upgrades between 23:05 and 05:05 CDT (04:05 and 10:05 UTC) on 20th June 2020 (21st June 2020 UTC). This will cause some disruption and may result in degraded performance to Baremetrics over the course of the window. We will post updates when the upgrades have been completed.</p>tag:status.baremetrics.com,2005:Incident/42034702020-05-27T03:26:54-05:002020-05-27T03:26:54-05:00Processing Delays<p><small>May <var data-var='date'>27</var>, <var data-var='time'>03:26</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>21:42</var> CDT</small><br><strong>Update</strong> - Background processing has been resumed and workers are catching up with the backlog of tasks.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>21:42</var> CDT</small><br><strong>Monitoring</strong> - Background processing has been resumed and workers are catching up with the backlog of tasks.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>20:33</var> CDT</small><br><strong>Update</strong> - As part of our ongoing maintenance, plan breakouts have been temporarily disabled.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>19:16</var> CDT</small><br><strong>Identified</strong> - Our background processing is currently paused. This will prevent your metrics from updating, and some other actions from updating across the API and dashboard.</p>tag:status.baremetrics.com,2005:Incident/41998192020-05-26T10:58:30-05:002020-05-26T10:58:30-05:00Database Maintenance<p><small>May <var data-var='date'>26</var>, <var data-var='time'>10:58</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>09:58</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>09:57</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.baremetrics.com,2005:Incident/41309162020-05-17T06:06:59-05:002020-05-17T06:06:59-05:00General Maintenance<p><small>May <var data-var='date'>17</var>, <var data-var='time'>06:06</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>05:30</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>05:20</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing some required maintenance during which time Baremetrics will be temporarily unavailable.</p>