will be having a planned emergency maintenance window on Monday, August 24th from 8pm to 10pm PDT.
You may see some errors during that time.
2020-March-12 Service Incident
Incident Report for Sauce Labs US West Data Center


Thursday March 12 12:50 AM PT - Thursday March 12 2:00 AM PT

What happened:

Tests using Sauce Connect on the U.S. Real Device Cloud were not available for one hour.

Why it happened:

An internal database migration took longer than expected and we exceeded the scheduled one-hour deployment window. We evaluated rolling back vs. continuing the migration and determined this would present a greater risk. Changes in the new environment that had not been reflected on the U.S. Real Device Sauce Connect implementation caused the deployment to fail repeatedly.

How we fixed it:

We corrected the erroneous configuration rebuilt, and deployed the affected services, enabling Sauce Connect tunnels to start again..

What we are doing to prevent it from happening again:

  • Require changes in the dependencies and configurations to be rolled out to all services in production within a week after implementation..
  • Finish deployments for the scheduled window within 30 minutes. Otherwise changes will be rolled back automatically.
  • Schedule all larger changes in a dedicated maintenance window, not the weekly deployment window.
Posted Mar 20, 2020 - 14:44 PDT

This incident has been resolved.
Posted Mar 12, 2020 - 02:19 PDT
A fix has been implemented and Sauce Connect tunnels for RDC are now starting. We are monitoring.
Posted Mar 12, 2020 - 02:00 PDT
Sauce Connect Tunnels for Real Device testing are not starting. RDC tests that rely on a Sauce Connect tunnel will fail. All other services are unaffected. We are investigating.
Posted Mar 12, 2020 - 01:10 PDT
This incident affected: Sauce Connect (Sauce Connect RDC).