2020-October-02 Service Incident
Postmortem

Dates:

Friday, October 2 02:26 AM - 04:15 AM PDT 

What happened:

Sauce Labs customers on the US-West data center were unable to start tests, Sauce Connect tunnels, use the REST API or access dashboards.

Why it happened:

A switch failure in the US-West data center caused a critical system component to enter into an inconsistent state.

How we fixed it:

The switch was power cycled to restore the inconsistent component. The switch was then replaced on Monday, October 5th.

What we are doing to prevent it from happening again:

We’re fixing the condition in the critical control plane component that caused it to become inconsistent. We’ve also replaced the failed switch hardware.

Posted Oct 06, 2020 - 14:10 PDT

Resolved
This incident has been resolved and all services have returned to normal operation.
Posted Oct 02, 2020 - 04:59 PDT
Monitoring
We have taken steps to return service to normal levels, but we are continuing to monitor the situation.
Posted Oct 02, 2020 - 04:30 PDT
Identified
Automated and Manual PC, Mac, Emulator/Simulator and RDC via Unified Platform testing, REST API and Sauce Connect tunnels are currently unavailable. In addition, the US-West-1 Sauce Labs dashboard at https://app.saucelabs.com is unavailable. We have identified the issue and are working to restore normal service.
Posted Oct 02, 2020 - 02:55 PDT