Wait times are intermittently spiking to 100 or more seconds on all clouds. These spikes only last about three minutes and occur about once every two hours. We are investigating the cause.
2019-October-4 Service Incident
Incident Report for Sauce Labs US West Data Center
Postmortem

Dates:

October 4th, 2019 18:47 -19:30 PDT

What happened:

Some customers experienced high wait times running tests in our PC, Mac, & Android clouds in our US West datacenter

Why it happened:

We ran an urgent batch job that overloaded our database. Consequently, current tests were unable to complete and new tests were unable to start.

How we fixed it:

We killed the job, which returned our service to its normal state.

What we are doing to prevent it from happening again:

Future jobs of this scope will be divided into smaller batches that can be run sequentially, using a single thread and an increased database cache size. We also intend to monitor the database and core systems more closely during any similar updates.

Posted Oct 14, 2019 - 09:20 PDT

Resolved
Wait times are back to normal. All services are now fully operational.
Posted Oct 04, 2019 - 19:30 PDT
Monitoring
We have taken remedial action. We are monitoring closely.
Posted Oct 04, 2019 - 19:11 PDT
Investigating
Wait times on our PC, Mac and Android Cloud are high. We are investigating.
Posted Oct 04, 2019 - 18:47 PDT
This incident affected: Automated VM Testing (Automated PC Testing, Automated Mac Testing, Automated iOS Simulator Testing, Automated Android Emulator Testing) and Manual Testing (Manual VM Testing).