API service degradation (increased error rate)

All system operational and healthy.

API services are 100% available.

Replacement resources for the caches are scaling up and taking production load, API is recovering.

A cache service failed after an enforced forced security update. This caused a minor share (~10%) of API requests on the EU API to timeout or fail, and larger share (~50%) of API requests on the US API to timeout or fail. The service degradation lasted for two hours between 17.00 and 19.00 UTC. We are offloading traffic onto additional resources.