Service API seems to be down

Major incident API
2021-12-22 17:46 CEST · 16 minutes

Updates

Post-mortem

Root cause was identified to be a dependency that removed support for Python 2 that was used to orchestrate the deployment pipeline. Through that change, automatic scaling and deployment of API instances could no longer be performed, which led to the API instance CPU load to max out. The deployment pipeline has been updated accordingly.

December 22, 2021 · 21:50 CEST
Update

Instances recovered through the rollback and are back up as of 18:02 CET. We are investigating the root cause.

December 22, 2021 · 18:11 CEST
Resolved

Service is back up.

December 22, 2021 · 18:02 CEST
Investigating

We are attempting a rollback of a Postgres connection scaling change that was deployed earlier today.

December 22, 2021 · 17:58 CEST
Investigating

We are investigating the issue.

December 22, 2021 · 17:49 CEST
Issue

According to our monitoring system this service has become unresponsive, we’re investigating.

December 22, 2021 · 17:46 CEST

← Back