Reporting Dashboards unavailable

The root cause was identified as: a dormant event emitting concurrency bug on the frontend client/ JavaScript side surfaced due to unknown reasons (no changes on parcelLab side, change in underlying framework possible). This caused a reload loop in Chrome/Chromium (but Safari/Firefox bot affected). This reload loop caused a pattern of a DoS against the PowerBI Service, where the increased load took down the service.

Data has been refreshed and the Reporting Dashboards are available again.

We rolled out an update on the frontend embedding framework and confirmed it to be running on Chrome.

Currently, we are running a refresh load for the reporting data.

After investigation on both the backend reporting infrastructure and the frontend embedding framework, a likely root cause has been identified:

Due to a change in the frontend embedding frameworks, a missing de-registration of an event handler during the initialization sequence caused the Reporting UI to be stuck in a reload-loop.

This loop generated so many requests against the backend, that the backend used up its capacity and therefore made debugging difficult. We rolled out an update on the portal to fix the init sequence and are monitoring results.

We are currently experiencing a global outage of the Reporting Dashboards in the portal. All users are affected. The rest of the portal like trackings, communication and configurations are not affected. Also, no end users/ communication recipients experience any outage,.

We are investigating the root cause and confirmed the portal to be operating correctly, and also the Reporting infrastructure is healthy and operating as expected. The issue has been identified to be in Microsoft Azure’s PowerBI Embedded Handler. An incident with Microsoft has been raised.