Wait times are intermittently spiking to 100 or more seconds on all clouds. These spikes only last about three minutes and occur about once every two hours. We are investigating the cause.
2019-May-10 Service Incident
Incident Report for Sauce Labs US West Data Center
Postmortem

Dates:

May 10th, 10:29 PM - May 10th, 2019 11:31 PM

What happened:

Some automated tests using Appium in our real device cloud would not start between 10:29 PM and 11:31 PM (PDT.)

Why it happened:

One of our users started an extremely high number of simultaneous tests, significantly surpassing their allotted concurrency level. Our concurrency limit enforcement mechanisms did not work as expected and all the tests were allowed to run at once. This overloaded our system and most other tests were not able to start.

How we fixed it:

We suspended the offending account.

What we are doing to prevent it from happening again:

We made changes to our system in order to be more aggressive about checking and enforcing concurrency limits.

Posted May 20, 2019 - 02:36 PDT

Resolved
We have identified the root cause and have fully recovered. All services are fully operational.
Posted May 10, 2019 - 23:31 PDT
Update
We are taking remedial action. Tests are still not starting.
Posted May 10, 2019 - 23:05 PDT
Investigating
Automated tests are not starting in our US Real Device Cloud. We are investigating.
Posted May 10, 2019 - 22:29 PDT
This incident affected: Automated RDC Testing (Automated RDC Testing).