2020-November-03 Resolved Service Incident
Postmortem

Dates:

Tuesday, November 3 04:39 AM PT - 05:49 AM PT

What happened:

Customers using Legacy RDC were not able to start Sauce Connect tunnels. 

Why it happened:

In the process of upgrading to a newer version of Sauce Connect API Service, an old version of the service was decommissioned. One of our subsystems was pointing to the old implementation instead of an API of the new service. As a result, the Sauce Connect infrastructure was unreachable from Legacy RDC.

How we fixed it:

We rolled back the change as soon as the issue was detected.

What we are doing to prevent it from happening again:

The misconfiguration has been removed and additional testing has been added.

Posted Nov 10, 2020 - 10:41 PST

Resolved
Automated and Live RDC tests using Sauce Connect tunnels failed to start today between 4:39am and 5:49am PST. We quickly identified and fixed the problem. All services are fully operational.
Posted Nov 03, 2020 - 06:00 PST