2021-February-03 Incident
Postmortem

Dates: 

February 3, 2021, 11:00am - 2:30pm PST

What happened:

Test archives in our US-West-1 data center were not loading.

Why it happened:

Sauce Labs uses an ElasticSearch cluster to store archive data, and the cluster ran out of disk space during the affected period. We did not have sufficient monitoring enabled to send a low disk space alert.

How we fixed it:

Disk space for the affected cluster was increased.

What we are doing to prevent it from happening again:

The Insights team has hardened monitoring and data retention management for the ES cluster.

Posted Feb 11, 2021 - 03:54 PST

Resolved
app.saucelabs.com/archives is operational.
Posted Feb 03, 2021 - 14:58 PST
Investigating
The archives page is not loading: https://app.saucelabs.com/archives. We are aware and investigating.
Posted Feb 03, 2021 - 13:41 PST