We have implemented changes to address intermittent timeouts or hang ups some customers experienced connecting to Automated VM testing services. We are monitoring our services to ensure full functionality; Please reach out to support@saucelabs.com if you need assistance.
2019-November-19 Service Incident
Incident Report for Sauce Labs US West Data Center
Postmortem

Dates:

November 19th, 2019 18:47 -19:30 PDT

What happened:

Some customers experienced high wait times running tests in our PC, Mac, & Android clouds in our US-West datacenter.

Why it happened:

A hypervisor rebooted which caused a load balancer running on it to lose some configuration changes and become overloaded

How we fixed it:

We restored the configuration values to the proper setting and it recovered.

What we are doing to prevent it from happening again:

We have reviewed all systems with high connections loads and verified that their configurations are correct. We have also verified that these changes are in the proper deployment configurations and that they are persistent in all required systems.

Posted Dec 18, 2019 - 13:10 PST

Resolved
Wait times are back to normal. All services are now fully operational.
Posted Nov 19, 2019 - 15:19 PST
Monitoring
We have taken remedial action and tests should be starting. We are monitoring closely.
Posted Nov 19, 2019 - 14:56 PST
Investigating
Wait times on our PC Cloud are high. There may be errors while starting PC jobs. We are investigating.
Posted Nov 19, 2019 - 14:30 PST
This incident affected: Automated VM Testing (Automated PC Testing).