October 4th, 2019 18:47 -19:30 PDT
Some customers experienced high wait times running tests in our PC, Mac, & Android clouds in our US West datacenter
We ran an urgent batch job that overloaded our database. Consequently, current tests were unable to complete and new tests were unable to start.
We killed the job, which returned our service to its normal state.
Future jobs of this scope will be divided into smaller batches that can be run sequentially, using a single thread and an increased database cache size. We also intend to monitor the database and core systems more closely during any similar updates.