A SSL certificate in the chain of the certificate used for TLS 1.2 on https://app.testobject.com has expired.
Any client (web browser, REST API client) using TLS 1.2 and making requests to https://app.testobject.com may see “certificate has expired” errors. Any client using TLS 1.3 is unaffected. We are working to rectify this issue.
2020-March-12 Service Incident
Incident Report for Sauce Labs US West Data Center
Postmortem

Dates:

Thursday March 12 12:50 AM PT - Thursday March 12 2:00 AM PT

What happened:

Tests using Sauce Connect on the U.S. Real Device Cloud were not available for one hour.

Why it happened:

An internal database migration took longer than expected and we exceeded the scheduled one-hour deployment window. We evaluated rolling back vs. continuing the migration and determined this would present a greater risk. Changes in the new environment that had not been reflected on the U.S. Real Device Sauce Connect implementation caused the deployment to fail repeatedly.

How we fixed it:

We corrected the erroneous configuration rebuilt, and deployed the affected services, enabling Sauce Connect tunnels to start again..

What we are doing to prevent it from happening again:

  • Require changes in the dependencies and configurations to be rolled out to all services in production within a week after implementation..
  • Finish deployments for the scheduled window within 30 minutes. Otherwise changes will be rolled back automatically.
  • Schedule all larger changes in a dedicated maintenance window, not the weekly deployment window.
Posted Mar 20, 2020 - 14:44 PDT

Resolved
This incident has been resolved.
Posted Mar 12, 2020 - 02:19 PDT
Monitoring
A fix has been implemented and Sauce Connect tunnels for RDC are now starting. We are monitoring.
Posted Mar 12, 2020 - 02:00 PDT
Investigating
Sauce Connect Tunnels for Real Device testing are not starting. RDC tests that rely on a Sauce Connect tunnel will fail. All other services are unaffected. We are investigating.
Posted Mar 12, 2020 - 01:10 PDT
This incident affected: Sauce Connect (Sauce Connect RDC).