Incidents | Steel

Live streaming issues

Mon, 20 Oct 2025 18:40:00 -0000

We have patched the majority of the issues here. There are still lingering network related issues here (especially networks where NAT traversal is limited). We'll push some updates there soon.

Live streaming issues

Sun, 19 Oct 2025 19:52:00 -0000

We're facing some issues on our streaming service but pushing a patch soon -- in the meantime; you can set `headless: true` in the session creation field and it should be patched

Some sessions are not successfully being created/released under maintenance

Fri, 17 Oct 2025 19:26:00 -0000

Things are stabilizing and seem to be back on track across the board -- we're still keeping an eye on things and will notify if things change.

Some sessions are not successfully being created/released under maintenance

Fri, 17 Oct 2025 14:59:00 -0000

We are working on a fix here and all things should be resolved soon

Last Minute Maintenance for Sessions API

Fri, 17 Oct 2025 14:43:44 -0000

Maintenance completed

Last Minute Maintenance for Sessions API

Fri, 17 Oct 2025 13:43:44 -0000

Requests will fail for some users for <10 mins as we ship some crucial infrastructure changes. Apologies in advance, we will be back shortly.

Elevated error rates from high usage

Sat, 20 Sep 2025 01:22:00 -0000

We've been monitoring the situation for quite a while now and it seems like session acquisition + launching is back to being stable. We'll follow up with an incident report with more details.

Elevated error rates from high usage

Fri, 19 Sep 2025 23:44:00 -0000

Our scaling services are slowly healing from the backlash of earlier requests. We're continuing to monitor the situation and will post an update once everything is all clear.

Elevated error rates from high usage

Fri, 19 Sep 2025 21:46:00 -0000

We're looking into this right now -- we'll be back with updates shortly

Facing some blowback from earlier

Wed, 17 Sep 2025 01:55:00 -0000

Last artifact has been cleaned up! We should be smooth sailing from now on.

Facing some blowback from earlier

Wed, 17 Sep 2025 01:18:00 -0000

Some instances are still awaiting new changes -- manually expediting this currently.

Elevated error rates due to some backpressure

Tue, 16 Sep 2025 20:48:00 -0000

## Overview The Steel Sessions API entered a degraded state with a higher error rate due to intermittent failures in the session acquisition and reservation logic. The switching logic responsible for managing session lifecycles exhibited unpredictable behavior, resulting in users being unable to establish or maintain browser sessions. This was made prevalent to us due to a large amount of requests/backpressure which revealed this specific issue. ## Timeline - 6:03 PM UTC - Initial increase in number of requests to the Sessions API - 6:53 PM UTC - Engineering team identified intermittent failures in session management + some early user reports - 7:48 PM UTC - Root cause traced to switching logic in session acquisition/reservation system - 8:12 PM UTC - Issue isolated and fix implemented - 8:48 PM UTC - Service fully restored and monitoring confirmed stability ## Root Cause The session acquisition and reservation system contained flawed switching logic that intermittently failed to properly: - Allocate available browser sessions to incoming requests - Release reserved sessions back to the available pool - Handle concurrent session requests during high load periods This resulted in a cascading failure where the session pool became exhausted due to sessions being incorrectly marked as reserved but not properly allocated or released. ## Impact - **User Impact:** Slowly increasing unavailability for browser automation tasks - **Duration:** 2h46m - **Affected Users:** Many users attempting to create new browser sessions during the incident window - **Business Impact:** Service interruption affecting customer workflows and API integrations ## Resolution The engineering team identified and corrected the faulty switching logic in the session management system. The fix involved: - Refactoring the session state transitions to ensure atomic operations - Implementing proper error handling for edge cases in session allocation - Adding additional validation checks for session pool consistency ## Current Status ✅ **Issue Resolved:** The switching logic has been fixed and deployed ✅ **Service Restored:** All session acquisition functionality is operating normally ✅ **Monitoring Active:** Enhanced monitoring is in place to detect similar issues ## Next Steps and Preventive Measures ### Immediate Actions (Next 7 Days) 1. **Enhanced Monitoring Implementation** - Deploy additional alerting for session pool health metrics 2. **Load Testing** - Conduct comprehensive load testing of session management under various scenarios - Validate fix effectiveness under simulated high-concurrency conditions ### Medium-term Actions (Next 30 Days) 1. **Code Review and Testing Enhancement** - Comprehensive audit of session management codebase - Implement additional unit and integration tests for session lifecycle edge cases - Establish chaos engineering practices for session management resilience 2. **Infrastructure Improvements** - Evaluate session pool sizing and auto-scaling mechanisms and test with methods in #1 - Design graceful degradation strategies for session pool exhaustion 3. **Documentation and Runbooks** - Create detailed runbooks for session management incidents ## Lessons Learned - High-concurrency scenarios expose edge cases not apparent under normal load - Proactive monitoring of internal system states (session pools) is critical for early detection - Automated testing should include concurrent access patterns and resource exhaustion scenarios ## Post-Incident Review A detailed post-incident review meeting will be scheduled within 48 hours to discuss: - Technical deep-dive into the root cause - Evaluation of response time and communication - Assessment of proposed preventive measures - Assignment of follow-up action items

Elevated error rates due to some backpressure

Tue, 16 Sep 2025 19:00:00 -0000

Several services are being affected, but we're working on some upgrades

Region Selection Failures

Tue, 09 Sep 2025 17:08:00 -0000

This should be resolved.

Region Selection Failures

Tue, 09 Sep 2025 14:38:00 -0000

Region selection is failing due to an upstream issue with our global routing provider. We are looking into this.

Network issues in BOM

Mon, 08 Sep 2025 13:05:00 -0000

We have pushed changes to reflect the situation in Mumbai and we're no longer routing there. All services should be back online.

Network issues in BOM

Mon, 08 Sep 2025 12:39:00 -0000

We are continuing to work with our external connectivity providers to address the service degradation caused by multiple submarine cable outages. Traffic between India and Europe/US East is impacted. Since this is an external incident, some degradation may still occur. We'll provide another update as soon as we have more information.

We are facing turbulence with our captcha solving services

Fri, 18 Jul 2025 18:37:00 -0000

We increased our scale capabilities and we are back to smooth sailing!

We are facing turbulence with our captcha solving services

Tue, 15 Jul 2025 17:02:00 -0000

Currently facing a larger number of requests to solve Cloudflare captchas -- we are working on a fix and we'll update this shortly

Some session creations are facing issues

Mon, 23 Jun 2025 15:47:00 -0000

Things are starting to look okay, we're monitoring the situation as we get back to peak performance

Some session creations are facing issues

Mon, 23 Jun 2025 13:57:00 -0000

We're investigating an error that is showing up for a lot of users when creating a new session, specifically: ``` Maximum call stack size exceeded ``` We'll be back with more updates

Our Proxies are Facing Some Connection Issues

Fri, 20 Jun 2025 15:11:00 -0000

A fix has been deployed -- everything looks good on initial testing but we'll continue to monitor the situation.

Our Proxies are Facing Some Connection Issues

Fri, 20 Jun 2025 14:33:00 -0000

We're looking into this right now and will have an update shortly.

We're facing a higher number of requests than usual

Thu, 29 May 2025 22:27:00 -0000

Okay, after some long investigations; we've found some temporary fixes to help with some of the scale problems we're dealing with. We'll continue to improve on this with a more thoughtful long-term solution -- but in the meantime we'll be monitoring our service closely. Thank you all for your patience!

We're facing a higher number of requests than usual

Tue, 27 May 2025 19:15:00 -0000

We’re investigating an issue with our sessions API that is impacting some users due to a larger number of requests than usual. We’re working to fix the problem as quickly as we can. We’ll share another update shortly.

Service is degraded

Wed, 30 Apr 2025 02:31:00 -0000

API was slow to respond for a little bit but the issue has been resolved!

Service is degraded

Wed, 30 Apr 2025 02:11:00 -0000

Running into issues

Burst usage from surf.new

Sat, 08 Feb 2025 21:28:00 -0000

Everything has been resolved here!

Burst usage from surf.new

Sat, 08 Feb 2025 21:18:00 -0000

The API is slow to create sessions + some users are experiencing errors

Session creation is slower than usual (and browser performance is hindered)

Sat, 25 Jan 2025 08:23:00 -0000

Fixed!

Session creation is slower than usual (and browser performance is hindered)

Sat, 25 Jan 2025 00:21:00 -0000

We're taking a look at a fix -- should be available soon!

Session store is down

Fri, 24 Jan 2025 01:55:00 -0000

We are back.

Session store is down

Fri, 24 Jan 2025 01:54:00 -0000

This issue is related to our internal service. We're working on fixing the issue and service will be available again very soon.

Session creation is slower than usual

Wed, 22 Jan 2025 22:59:00 -0000

We have identified the cause and pushed a fix! Things look somewhat stable now.

Session creation is slower than usual

Wed, 22 Jan 2025 14:32:00 -0000

We’re looking into it but session creation times have gone up a bit (P99 of 3.7 seconds) We will update this page once a fix is up!

Session Live Viewer

Tue, 14 Jan 2025 00:54:00 -0000

Live session views were down for about ~30 minutes due to an error with our session event persistence layer, everything should be good now!

Our routing between services is experiencing some latency

Tue, 17 Dec 2024 13:00:00 -0000

We have finally pushed the fixes for the session creation bugs -- sessions and other requests should be back to being quick and snappy!

Our routing between services is experiencing some latency

Mon, 16 Dec 2024 22:28:00 -0000

Some requests are taking longer than normal -- we are on it, we'll have an update shortly

Session creation times are taking longer than expected

Sat, 14 Dec 2024 00:15:00 -0000

We have pushed a temporary fix while we continue to work on a permanent solution.

Session creation times are taking longer than expected

Fri, 13 Dec 2024 15:06:00 -0000

We’re having some trouble with our machine locks during session creation. We’re working to fix the problem as quickly as we can. We’ll share another update shortly.

API was down briefly

Fri, 18 Oct 2024 19:51:00 -0000

Our API was briefly down for ~3 minutes as we scaled our cache. Everything should be back to normal.