Wednesday, September 9 08:30 - 12:40 PDT
Some customers received erroneous messages stating they were running at their maximum concurrency.
During a routine deploy, a serialization library was updated. This update contained a hidden dependency with a non-compatible, breaking change which made the read and write values of this service to not work.
The deployment was rolled back.
We are improving our endpoint monitoring to detect stale values and other anomalies and improving dependency monitoring and alerts when a cross-dependency gets updated. We are also enhancing the deploy process of this service to ensure changes are properly validated before getting deployed.