Everything seems to be back in order and we've confirmed all webhooks are properly processing now.
Posted Oct 22, 2020 - 10:08 CDT
Monitoring
CloudFlare reports they've resolved the issue on their end. We're monitoring their fix now, but otherwise webhooks should be processing normally now and we're working our way through the backlog of previously failed webhooks.
Posted Oct 22, 2020 - 09:43 CDT
Update
We've deployed a temporary fix that should allow Stripe API calls to process while we work with CloudFlare.
Posted Oct 22, 2020 - 09:24 CDT
Identified
We've identified the issue is indeed with CloudFlare's rate-limiting incorrectly blocking IP addresses that we've whitelisted.
Working with CloudFlare now to address this.
Posted Oct 22, 2020 - 09:13 CDT
Investigating
We've detected that roughly half of all webhooks from Stripe are getting blocked. This appears to be a new issue with the way CloudFlare handles rate-limiting. In this case, they're incorrectly limiting calls from Stripe's IP addresses. We're working with them now to resolve this.
All failing webhooks will automatically be reprocessed once this is resolved.
We'll provide more updates as we have them.
Posted Oct 22, 2020 - 08:48 CDT
This incident affected: 3rd Party Services (Stripe API).