Previous incidents

Pushed a fix here, things should be smoother now.

We've pushed a scaling fix and traffic is being balanced now. Will continue to monitor for any outliers.

These specific issues are resolved.

We have pushed up some further mitigations and are working with our infra provider to ensure a complete resolution to this problem. We will be posting some more updates here when that is in place.

We've pushed a fix here and have been monitoring it for the last 20 minutes or so -- it seems like traffic is properly routing for now. We'll share any additional updates here.

We have patched the majority of the issues here. There are still lingering network related issues here (especially networks where NAT traversal is limited).

We'll push some updates there soon.

Things are stabilizing and seem to be back on track across the board -- we're still keeping an eye on things and will notify if things change.

We've been monitoring the situation for quite a while now and it seems like session acquisition + launching is back to being stable. We'll follow up with an incident report with more details.

Last artifact has been cleaned up! We should be smooth sailing from now on.

Overview

The Steel Sessions API entered a degraded state with a higher error rate due to intermittent failures in the session acquisition and reservation logic. The switching logic responsible for managing session lifecycles exhibited unpredictable behavior, resulting in users being unable to establish or maintain browser sessions. This was made prevalent to us due to a large amount of requests/backpressure which revealed this specific issue.

Timeline

• 6:03 PM UTC - Initial increase...

This should be resolved.

We have pushed changes to reflect the situation in Mumbai and we're no longer routing there. All services should be back online.