2020-November-19 Resolved Service Incident
Postmortem

Dates:

Thursday, November 19th 07:26 - 12:03 PST

What happened:

Customers weren't able to reach REST API endpoints that satisfied certain conditions. Those requests were mistakenly directed to a service that wasn't able to handle them.

Why it happened:

We merged a change that contained erroneous regular expressions and broke the main API router.

How we fixed it:

The breaking change was rolled back and we merged a fixed version at a later point.

What we are doing to prevent it from happening again:

We’re introducing thorough post-deploy tests that will cover the API routing.

Posted Nov 24, 2020 - 09:45 PST

Resolved
We detected errors on our REST API, Sauce Connect and UI for users with a sso prefix on their username. We took remedial action which resolved the issue. All services are fully operational.
Posted Nov 19, 2020 - 12:43 PST