2020-November-25 Service Incident
Postmortem

Dates:

Wednesday, November 25th 2:50 - 5:00 PST

What happened:

Customers were unable to reach REST API endpoints and access the application UI.

Why it happened:

All requests were mistakenly marked as blocked by our Web Application Firewall (WAF).

How we fixed it:

We disabled and then redeployed our Web Application Firewall configuration.

What we are doing to prevent it from happening again:

We’re introducing automated monitoring and alerting for WAF. Additional alerting and monitoring has also been put in place to alert immediately if and when all requests receive a 406 in the future.

Posted Dec 22, 2020 - 08:36 PST

Resolved
This incident has been resolved.
Posted Nov 25, 2020 - 05:31 PST
Monitoring
We have taken remedial action and our service is recovering. We are monitoring.
Posted Nov 25, 2020 - 05:07 PST
Investigating
We are experiencing a service incident. The UI is unavailable and returns an HTTP 406 error. Our REST API is also affected. Tests are running fine. We are investigating.
Posted Nov 25, 2020 - 04:13 PST