Wait times are intermittently spiking to 100 or more seconds on all clouds. These spikes only last about three minutes and occur about once every two hours. We are investigating the cause.
2019-September-12 Service Incident
Incident Report for Sauce Labs US West Data Center
Postmortem

IM-218, Customer Facing Post Mortem September 12th, 2019

Dates:

September 12th, 2019 5:52 PM - September 12th, 2019 7:16 PM CEST.

What happened:

Appium tests failed to start if our service could not allocate a device for the test within 3 minutes. This was true on both the European and US data centers.

Why it happened:

We changed the behavior of how pending (scheduled) Appium sessions work. Pending sessions were being cleaned up before the Appium client got a response from the server signaling successful session creation.

How we fixed it:

We realized this was an issue introduced in the last release, so we rolled back to the last stable release.

What we are doing to prevent it from happening again:

We are adding test cases in our staging environment in order to prevent us from deploying broken releases. We are setting up alerts to notify us when the Appium session creation success rate decreases significantly.

Posted Oct 03, 2019 - 08:34 PDT

Resolved
All services are now fully operational.
Posted Sep 12, 2019 - 10:16 PDT
Update
We are continuing to take remedial action for this problem which only affects our real device cloud.
Posted Sep 12, 2019 - 09:19 PDT
Identified
We are experiencing a high failure rate for Appium tests. We are taking remedial action that will stop all running Appium tests for a short period of time.
Posted Sep 12, 2019 - 08:51 PDT
This incident affected: Automated RDC Testing (Automated RDC Testing).