2020-August-30 Service Incident
Postmortem

Dates:

Sunday August 30 05:00 PDT to Sunday August 30 10:00 PDT

What happened?

Some Sauce Labs customers experienced intermittent connectivity issues.

Why it happened?

CenturyLink, a global Internet backbone network, experienced a network outage that affected the Internet globally, particularly in Europe.

How we fixed it?

Internet Service Providers began disconnecting from CenturyLink while they worked on resolving their outage. As CenturyLink’s outage began to resolve, we started recovering Real Devices that did not recover automatically, and restarted some services.

What we are doing to prevent it from happening again?

We’re adding additional monitoring of external endpoints to provide better insight into issues experienced on the Internet and enable more rapid response.

Posted Sep 16, 2020 - 12:25 PDT

Resolved
The Real Device Cloud has recovered and all services are fully operational.
Posted Aug 30, 2020 - 10:16 PDT
Monitoring
All services are recovering and are fully operational. We are monitoring.
Posted Aug 30, 2020 - 08:14 PDT
Update
US Real Device availability continues to recover with over 90% of devices now available for testing.
Posted Aug 30, 2020 - 07:18 PDT
Update
We have made some changes to resolve the degraded performance of the Sauce Labs dashboard, but issues affecting the availability of real devices and access to https://app.testobject.com are ongoing.
Posted Aug 30, 2020 - 07:04 PDT
Identified
We are seeing some improvements, but the resolution of this issue is very much dependent on Level3/CenturyLink. https://app.testobject.com is available from some locations, but not others, and our Real Device availability is gradually recovering.
Posted Aug 30, 2020 - 05:55 PDT
Update
We are continuing to investigate this issue.
Posted Aug 30, 2020 - 05:05 PDT
Investigating
A major global internet provider (Level3/CenturyLink) is experiencing issues which are causing parts of our service to be unavailable or degraded. Currently Real Device testing, the legacy RDC UI at app.testobject.com and the Sauce Labs wiki are unavailable, and the Sauce Labs dashboard is experiencing intermittent issues.
Posted Aug 30, 2020 - 05:04 PDT