Wait times are intermittently spiking to 100 or more seconds on all clouds. These spikes only last about three minutes and occur about once every two hours. We are investigating the cause.
2019-Jul-08 (PST) Major Service Incident
Incident Report for Sauce Labs US West Data Center
Postmortem

Dates:

July 8th, 2019 7:00 PM - 8:20 PM PDT

What happened:

New Sauce Connect Tunnel requests were unable to start for a short period of time. Afterwards, some tunnels could start but others would fail as we throttled new requests to allow our US-West Tunnels cloud to catch up with the backlog of requests and stabilize.

Why it happened:

While deploying to our US-West region a subsystem's configuration was changed to an incorrect value that rendered our Tunnels Cloud unable to accept new tunnel requests. This change was quickly reverted, but not before our Tunnels Cloud became overwhelmed working through the backlog of new tunnel requests that had quickly accumulated.

How we fixed it:

We corrected the original issue by returning the configuration to its proper value and restarting the related subsystem. We also throttled new tunnel requests until our Tunnel Cloud was able to stabilize.

What we are doing to prevent it from happening again:

We’ve begun auditing all configuration settings across both regions and documenting known variances. We’ve added additional logic to our Tunnels Cloud services to better handle this edge case and prevent this error from recurring. We’ve introduced a new module and method to ensure service health status as part of our deploy process.

Posted about 2 months ago. Jul 29, 2019 - 14:14 PDT

Resolved
All services are fully operational.

If you continue to experience problems, please contact support@saucelabs.com.
Posted 2 months ago. Jul 08, 2019 - 20:30 PDT
Investigating
Customers will see some attempts to start new Sauce Connect tunnels fail. Existing tunnels are not impacted. We are investigating.
Posted 2 months ago. Jul 08, 2019 - 19:54 PDT
This incident affected: Sauce Connect (Sauce Connect VM).