2021-February-02 Service Incident

Incident Report for ZZ [Archive] Sauce Labs US West Data Center

Postmortem

Dates:

Tuesday, February 2nd, 2021 09:40 - 12:00 PST

What happened:

Customers experienced increased wait times in the macOS/iOS cloud in our US West Datacenter between 9:40am and 11:40am PST.

Why it happened:

We experienced a spike in service utilization while an image synchronization process was underway, increasing network traffic and depleting the pool of available MacOS VMs.

How we fixed it:

The system recovered once the image synchronization process was stopped and the amount of network traffic was reduced.

What we are doing to prevent it from happening again:

We are doubling our network bandwidth to better prepare for traffic peaks and accelerating our plans to make the image distribution process more efficient.

Posted Feb 09, 2021 - 01:45 PST

Resolved

Mac Cloud is fully operational. All services have returned to normal.

Posted Feb 02, 2021 - 13:03 PST

Update

We are taking remedial action. Mac Desktop and iOS Simulator cloud should be stabilizing but you may see errors.

Posted Feb 02, 2021 - 12:25 PST

Investigating

iOS simulators and Mac Desktop tests may be slower to start and may have high error rates. We are investigating.

Posted Feb 02, 2021 - 11:50 PST