2021-February-02 Service Incident
Postmortem

Dates:

Tuesday, February 2nd, 2021 09:40 - 12:00 PST

What happened:

Customers experienced increased wait times in the macOS/iOS cloud in our US West Datacenter between 9:40am and 11:40am PST.

Why it happened:

We experienced a spike in service utilization while an image synchronization process was underway, increasing network traffic and depleting the pool of available MacOS VMs.

How we fixed it:

The system recovered once the image synchronization process was stopped and the amount of network traffic was reduced. 

What we are doing to prevent it from happening again:

We are doubling our network bandwidth to better prepare for traffic peaks and accelerating our plans to make the image distribution process more efficient.

Posted Feb 09, 2021 - 01:45 PST

Resolved
Mac Cloud is fully operational. All services have returned to normal.
Posted Feb 02, 2021 - 13:03 PST
Update
We are taking remedial action. Mac Desktop and iOS Simulator cloud should be stabilizing but you may see errors.
Posted Feb 02, 2021 - 12:25 PST
Investigating
iOS simulators and Mac Desktop tests may be slower to start and may have high error rates. We are investigating.
Posted Feb 02, 2021 - 11:50 PST