Monday, August 3, 09:04 AM - Wednesday, August 5 4:31 PM PDT
Customers experienced increased job execution times and test latency beginning Monday, August 3rd.
In the process of upgrading to a newer version of HAProxy, we observed intermittent performance issues. We rolled back the upgrade but kept one load balancer out of rotation to further investigate, causing the other load balancers to become intermittently saturated with traffic.
We added the load balancer back into the rotation pool on Wednesday, August 5.
The fleet of production load balancers has been increased and can now sustain the removal of a single load balancer before any degradation in performance. We are in the process of deprecating services that are blocking the HAProxy upgrades.