Wait times are intermittently spiking to 100 or more seconds on all clouds. These spikes only last about three minutes and occur about once every two hours. We are investigating the cause.
2020-March-12 Service Incident
Incident Report for Sauce Labs US West Data Center
Postmortem

Dates:

Thursday March 12 12:50 AM PT - Thursday March 12 2:00 AM PT

What happened:

Tests using Sauce Connect on the U.S. Real Device Cloud were not available for one hour.

Why it happened:

An internal database migration took longer than expected and we exceeded the scheduled one-hour deployment window. We evaluated rolling back vs. continuing the migration and determined this would present a greater risk. Changes in the new environment that had not been reflected on the U.S. Real Device Sauce Connect implementation caused the deployment to fail repeatedly.

How we fixed it:

We corrected the erroneous configuration rebuilt, and deployed the affected services, enabling Sauce Connect tunnels to start again..

What we are doing to prevent it from happening again:

  • Require changes in the dependencies and configurations to be rolled out to all services in production within a week after implementation..
  • Finish deployments for the scheduled window within 30 minutes. Otherwise changes will be rolled back automatically.
  • Schedule all larger changes in a dedicated maintenance window, not the weekly deployment window.
Posted Mar 20, 2020 - 14:44 PDT

Resolved
This incident has been resolved.
Posted Mar 12, 2020 - 02:19 PDT
Monitoring
A fix has been implemented and Sauce Connect tunnels for RDC are now starting. We are monitoring.
Posted Mar 12, 2020 - 02:00 PDT
Investigating
Sauce Connect Tunnels for Real Device testing are not starting. RDC tests that rely on a Sauce Connect tunnel will fail. All other services are unaffected. We are investigating.
Posted Mar 12, 2020 - 01:10 PDT
This incident affected: Sauce Connect (Sauce Connect RDC).