2021-01-15 Resolved Service Incident
Postmortem

Dates:

Thursday, January 14th, 23:30 - Friday, January 14th, 06:00 PST

What happened:

During the time period above, some files uploaded to Sauce Storage became corrupted, making them unusable for testing.

Why it happened:

A version update to a third party library used by our app storage service introduced a race condition where under some circumstances files were not uploaded properly even after returning a success message.

How we fixed it:

The App Storage Service deployment was reverted to an earlier version containing the older snapshot of the third party library component. Any affected files uploaded during the time period above need to be re-uploaded.

What we are doing to prevent it from happening again:

We improved the alerting notifications to expose more warnings to be able to detect potential problems faster in our staging environment before hitting production.

Posted Jan 21, 2021 - 00:42 PST

Resolved
Between 2300 and 0600 PST, some apps uploaded to the new Sauce Application Storage service were corrupted. This caused app installation errors on Automated and Live Real Device tests and Automated Emulator/Simulator tests using the new Sauce Application Storage service (using the prefix "storage:" in the "app" capability). The problem has been fixed, but any app uploaded between these times that is exhibiting app installation errors should be uploaded again.
Posted Jan 15, 2021 - 06:00 PST