will be having a planned emergency maintenance window on Monday, August 24th from 8pm to 10pm PDT.
You may see some errors during that time.
2020-August-05 Service Incident
Incident Report for Sauce Labs US West Data Center


Monday, August 3, 09:04 AM - Wednesday, August 5 4:31 PM PDT 

What happened:

Customers experienced increased job execution times and test latency beginning Monday, August 3rd.

Why it happened:

In the process of upgrading to a newer version of HAProxy, we observed intermittent performance issues. We rolled back the upgrade but kept one load balancer out of rotation to further investigate, causing the other load balancers to become intermittently saturated with traffic.

How we fixed it:

We added the load balancer back into the rotation pool on Wednesday, August 5.

What we are doing to prevent it from happening again:

The fleet of production load balancers has been increased and can now sustain the removal of a single load balancer before any degradation in performance. We are in the process of deprecating services that are blocking the HAProxy upgrades.

Posted Aug 18, 2020 - 00:52 PDT

We've continued to observe a return to normal operations and performance after taking remedial action. All services have been fully operational for the last 16hrs
Posted Aug 06, 2020 - 08:29 PDT
We've taken remedial action and have seen test times get back to normal thresholds. We are monitoring the state of tests.
Posted Aug 05, 2020 - 16:01 PDT
We've ruled out a few parts of our service being the culprit as we continue to test to find root cause.
Posted Aug 05, 2020 - 14:30 PDT
Slowness of tests is still seen across our clouds. We continue to investigate.
Posted Aug 05, 2020 - 12:54 PDT
We have received multiple reports of slowness while running automated tests.
We are investigating.
Posted Aug 05, 2020 - 12:16 PDT
This incident affected: Automated VM Testing (Automated PC Testing, Automated Mac Testing, Automated iOS Simulator Testing, Automated Android Emulator Testing).