Wait times are intermittently spiking to 100 or more seconds on all clouds. These spikes only last about three minutes and occur about once every two hours. We are investigating the cause.
2019-March-28 Service Incident
Incident Report for Sauce Labs US West Data Center
Postmortem

Dates:

March 28th, 2019 07:00 AM - 08:30 AM PDT

What happened:

Customers of the Real Device Cloud in the EU and the US may have experienced failures with starting Appium tests due to timeouts and increased wait times for initialization of Live Testing sessions.

Why it happened:

Performance of real devices in the EU and the US was negatively impacted by a software change causing excessive usage of system resources.

How we fixed it:

We rolled back the change as soon as we noticed performance degradations per region.

What we are doing to prevent it from happening again:

We are resolving the described performance degradation and upgrading the affected services while also introducing better and safer mechanisms for rolling out this change to production. In addition, we are enriching our application monitoring system so that such issues could be detected faster and in most cases prevented in the future.

Posted 7 months ago. Apr 11, 2019 - 16:41 PDT

Resolved
Automated and manual tests are running normally again. All services are fully operational.
Posted 7 months ago. Mar 28, 2019 - 09:34 PDT
Investigating
Appium tests are not starting in our Real Device Cloud in our US West data center. Manual tests are starting, but slowly. We are investigating.
Posted 7 months ago. Mar 28, 2019 - 08:57 PDT
This incident affected: Manual Testing (Manual RDC Testing) and Automated RDC Testing (Automated RDC Testing).