Tuesday, November 17, 01:00 - 11:00 PT
The number of available devices decreased over a period of 10 hours to a point where 20% were unavailable. This impacted mostly customers with private devices or customers querying for very specific public device models.
We deployed an update to our real device cloud which caused a small portion of live tests to not free up the device under use after the session was closed.
We rolled back the changes which were causing the problem and released all affected devices so that they became available for use again.
We enhanced our monitoring to catch situations where we see a reduction of available devices, no matter the reason availability is reduced. In addition, we are going to add alerting for long-running sessions.