Dates: 

February 3, 2021, 11:00am - 2:30pm PST

What happened:

Test archives in our US-West-1 data center were not loading.

Why it happened:

Sauce Labs uses an ElasticSearch cluster to store archive data, and the cluster ran out of disk space during the affected period. We did not have sufficient monitoring enabled to send a low disk space alert.

How we fixed it:

Disk space for the affected cluster was increased.

What we are doing to prevent it from happening again:

The Insights team has hardened monitoring and data retention management for the ES cluster.