May 10th, 10:29 PM - May 10th, 2019 11:31 PM
Some automated tests using Appium in our real device cloud would not start between 10:29 PM and 11:31 PM (PDT.)
One of our users started an extremely high number of simultaneous tests, significantly surpassing their allotted concurrency level. Our concurrency limit enforcement mechanisms did not work as expected and all the tests were allowed to run at once. This overloaded our system and most other tests were not able to start.
We suspended the offending account.
We made changes to our system in order to be more aggressive about checking and enforcing concurrency limits.