Webhook delivery delayed

All metrics are healthy, this incident is closed.

Upscaled webhook infrastructure is in place and healthy. New events are forwarded as they come in with minimal delay. The delayed webhooks are not lost, but are being processed as well. We continue to monitor.

The new replacement infrastructure is rotated into PROD, causing delays to decrease. Current events are processed immediately, events still in backlog are now processed as well.

Several potential issue causes have been eliminated, however we are still experiencing high load in the queue system. Additional infrastructure has been scaled up and will now be rotated into production.

We are balancing capacity to identify the root cause.

Webhook delivery is currently delayed by up to 2 hours. The team is investigating with highest priority.