Webhook delivery delayed

Minor incident Outbound channels Webhook
2023-07-07 18:08 CEST · 3 hours, 17 minutes



All metrics are healthy, this incident is closed.

July 7, 2023 · 21:25 CEST

Upscaled webhook infrastructure is in place and healthy. New events are forwarded as they come in with minimal delay. The delayed webhooks are not lost, but are being processed as well. We continue to monitor.

July 7, 2023 · 20:27 CEST

The new replacement infrastructure is rotated into PROD, causing delays to decrease. Current events are processed immediately, events still in backlog are now processed as well.

July 7, 2023 · 20:03 CEST

Several potential issue causes have been eliminated, however we are still experiencing high load in the queue system. Additional infrastructure has been scaled up and will now be rotated into production.

July 7, 2023 · 19:22 CEST

We are balancing capacity to identify the root cause.

July 7, 2023 · 18:28 CEST

Webhook delivery is currently delayed by up to 2 hours. The team is investigating with highest priority.

July 7, 2023 · 18:08 CEST

← Back