# AutoSmoke > AI-powered smoke testing platform. Runs a real browser through your critical user flows (signup, login, checkout, onboarding) after every deploy. Tests are written in plain English and self-heal when the UI changes — no selectors, no scripts, no maintenance. AutoSmoke is a testing automation tool that uses AI to understand and interact with web applications the way humans do. Instead of relying on fragile element selectors that break when UI changes, AutoSmoke uses computer vision and contextual understanding to find and interact with elements. ## Core Features - **AI-Powered Test Generation**: Describe your tests in plain English. Our AI understands your intent and generates robust, maintainable test flows automatically. - **Self-Healing Tests**: No more broken tests when your UI changes. AutoSmoke adapts to DOM changes, keeping your tests green without manual updates. - **Visual Verification**: Every test run captures screenshots and visual diffs. Catch visual regressions before your users do. - **Instant Setup**: No complex configuration or infrastructure. Paste your URL, describe your test, and you're running. - **Scheduled Monitoring**: Run your tests on a schedule to continuously monitor your application. Get notified when something breaks. - **Cross-Browser Support**: Test across Chromium, Firefox, and WebKit. - **CI/CD Integration**: GitHub Action and GitLab CI templates for PR-based testing. ## Pricing **Free — $0/mo**: 1 project, 20 runs/month, basic email alerts. No credit card required. **Pro — $29/mo**: 5 projects, 200 runs/month, scheduled runs, GitHub Actions integration, full run recordings. **Team — $79/mo**: Unlimited projects, 1,500+ runs/month, team member access, priority support. --- # Documentation # Getting Started Welcome to AutoSmoke! This guide will help you create and run your first AI-powered smoke test. ## Quick Start ### Step 1: Enter Your URL Navigate to the AutoSmoke dashboard and paste the URL of the page you want to test. Our AI will analyze the page and suggest relevant test cases. ### Step 2: Review Test Suggestions Based on your page content, we'll suggest test cases that cover common user flows: - Form submissions - Navigation flows - Authentication - Critical business processes You can accept, modify, or create your own tests. ### Step 3: Write Your Test in Plain English Describe what you want to test using natural language: ``` Navigate to the login page Enter "user@example.com" in the email field Enter "password123" in the password field Click the "Sign In" button Verify that the dashboard is displayed ``` ### Step 4: Run Your Test Click "Run Test" and watch AutoSmoke execute your test in a real browser. You'll see: - Live execution progress - Screenshots at each step - Pass/fail status - Error details if something goes wrong ## Understanding Test Results After a test run, you'll see: - **Status**: Whether the test passed or failed - **Duration**: How long the test took to run - **Screenshots**: Visual proof of each step - **Logs**: Detailed execution logs for debugging ## Scheduling Tests Set up recurring test runs to monitor your application: 1. Open your test 2. Click "Schedule" 3. Choose your frequency (hourly, daily, weekly) 4. Configure notifications You'll be alerted immediately if a scheduled test fails. ## Next Steps - [Writing Effective Tests](/docs/writing-tests) - Best practices for reliable tests - [Understanding Selectors](/docs/selectors) - How AutoSmoke finds elements - [Handling Authentication](/docs/authentication) - Testing behind login - [Notifications & Alerts](/docs/notifications) - Get notified via email, Slack, or webhook --- # Writing Effective Tests Well-written tests are the foundation of reliable monitoring. Follow these guidelines to create tests that are robust and easy to maintain. ## Be Specific and Clear Write test steps that clearly describe the intended action: **Good:** ``` Click the "Add to Cart" button next to the product ``` **Avoid:** ``` Click the button ``` ## Use Descriptive Verifications When verifying results, be explicit about what you expect: **Good:** ``` Verify that the success message "Your order has been placed" is displayed ``` **Avoid:** ``` Check if it worked ``` ## Test One Flow at a Time Keep tests focused on a single user journey: - Login test - Checkout test - Search test Don't combine unrelated flows in a single test. ## Handle Dynamic Content For content that changes (like dates or generated IDs), use flexible verifications: ``` Verify that a confirmation number is displayed ``` Instead of: ``` Verify that "Order #12345" is displayed ``` ## Account for Loading States Web applications often have loading states. Our AI handles most waits automatically, but you can be explicit when needed: ``` Wait for the product list to load Click the first product ``` ## Organize Your Tests Group related tests together: - **Smoke Tests**: Critical paths that must always work - **Feature Tests**: Specific feature functionality - **Regression Tests**: Previously broken functionality ## Common Patterns ### Form Submission ``` Navigate to the contact form Enter "John Doe" in the name field Enter "john@example.com" in the email field Enter "Hello, I have a question" in the message field Click the "Send" button Verify that "Thank you for your message" is displayed ``` ### Navigation ``` Navigate to the homepage Click "Products" in the navigation menu Verify that the products page is displayed Click the first product Verify that the product details are shown ``` ### Search ``` Navigate to the homepage Enter "blue shoes" in the search box Press Enter or click the search button Verify that search results are displayed Verify that results contain "blue" or "shoes" ``` --- # Understanding Selectors AutoSmoke uses AI to intelligently find elements on your page, so you don't need to write CSS selectors or XPath expressions. ## How It Works When you write a step like: ``` Click the "Sign Up" button ``` Our AI: 1. Analyzes the page structure 2. Identifies all potential matches 3. Uses context to select the correct element 4. Falls back to alternative selectors if needed ## Self-Healing Selectors Traditional tests break when: - CSS classes change - Element IDs are updated - DOM structure is reorganized AutoSmoke's self-healing technology adapts to these changes automatically by: - Using multiple selector strategies - Prioritizing stable attributes (text, labels, roles) - Learning from successful past executions ## Providing Hints While AutoSmoke usually finds elements correctly, you can provide hints for precision: ### Using Text Content ``` Click the button with text "Submit Order" ``` ### Using Position ``` Click the first "Add to Cart" button Click the "Delete" button next to "Product Name" ``` ### Using Section Context ``` In the header, click the "Login" link In the sidebar, click "Settings" ``` ## Handling Ambiguous Elements If multiple elements match your description: 1. AutoSmoke will attempt to identify the most likely match 2. If uncertain, it will ask for clarification 3. You can provide more context to disambiguate ### Example: Multiple Buttons ``` ❌ Click the "Submit" button (ambiguous if multiple exist) ✅ Click the "Submit" button in the payment form ✅ Click the primary "Submit" button ``` ## Accessibility and Testing AutoSmoke works best with accessible websites because: - Proper labels help identify form fields - ARIA roles indicate element purposes - Semantic HTML provides context This is a win-win: accessible sites are easier to test AND better for users. --- # Handling Authentication Many tests require authenticated access. Here's how to handle authentication with AutoSmoke. ## Login as Part of the Test The simplest approach is to include login steps in your test: ``` Navigate to the login page Enter "test@example.com" in the email field Enter "testpassword" in the password field Click the "Sign In" button Verify that the dashboard is displayed # Now continue with your actual test Click "Settings" ... ``` ## Reusable Login Flow If multiple tests need authentication, create a dedicated login test that other tests can reference: 1. Create a test called "Login" 2. In other tests, start with "Run the Login test" ## Test Account Best Practices - Use dedicated test accounts, not real user accounts - Create accounts with predictable, stable data - Don't use production credentials in tests - Reset test data periodically if needed ## Testing Different User Roles For applications with multiple user types: ``` # Admin Test Navigate to login Enter admin credentials Verify admin dashboard features # Regular User Test Navigate to login Enter user credentials Verify user dashboard (no admin features) ``` ## Handling MFA/2FA For applications with multi-factor authentication: - Use test accounts with MFA disabled - Or use a test environment that bypasses MFA - Contact us for advanced MFA handling options ## Session Management AutoSmoke maintains browser sessions within a test run. This means: - Cookies persist between steps - Local storage is maintained - You stay logged in throughout the test Each new test run starts with a fresh browser session. --- # Setting Up Google OAuth Authentication This guide walks you through configuring Google OAuth in AutoSmoke using your existing Google OAuth 2.0 client credentials. By the end, your smoke tests will be able to sign in automatically before running. ## Prerequisites - A Google OAuth 2.0 client with a **Client ID** and **Client Secret** (created in [Google Cloud Console](https://console.cloud.google.com/) under **APIs & Services > Credentials**) - An AutoSmoke project with a site configured ## Step 1: Add the AutoSmoke Redirect URI to Your Google Client Before configuring AutoSmoke, your Google OAuth client must allow AutoSmoke's callback URL. 1. Open [Google Cloud Console](https://console.cloud.google.com/) and navigate to **APIs & Services > Credentials** 2. Click on your OAuth 2.0 client to edit it 3. Under **Authorized redirect URIs**, add: ``` https://autosmoke.dev/api/auth/google-oauth/callback ``` 4. Click **Save** ## Step 2: Check Your OAuth Consent Screen Settings Your consent screen configuration determines which accounts can authorize and what permissions are requested. 1. In Google Cloud Console, go to **APIs & Services > OAuth consent screen** 2. Verify the following: - **Scopes** include at least `openid`, `email`, and `profile` — these are required for AutoSmoke to identify the authenticated user - If your consent screen is in **Testing** publishing status, the Google account you plan to test with must be listed under **Test users**. Only explicitly added test users can complete the OAuth flow while in testing mode. > **Tip:** If you need any Google account to authenticate (not just test users), set your consent screen to **Production** publishing status. This may require Google verification if you use sensitive scopes. ## Step 3: Open the Authentication Configuration in AutoSmoke 1. Log in to [AutoSmoke](https://autosmoke.dev) and open your project 2. Navigate to the **Dashboard** for the site you want to configure 3. Find the **Authentication** section — it shows the current configuration status with a shield icon 4. Click **Configure Authentication** (or **Edit** if authentication is already set up) This opens the Authentication Configuration dialog. ## Step 4: Enter Your Credentials In the dialog, fill in the following fields: ### Client ID *(required)* Paste your Google OAuth Client ID. It looks like: ``` 123456789-abcdefg.apps.googleusercontent.com ``` ### Client Secret *(required)* Paste your Google OAuth Client Secret. It looks like: ``` GOCSPX-aBcDeFgHiJkLmNoPqRsTuVwXyZ ``` ### Scopes *(optional)* Comma-separated list of OAuth scopes. Defaults to: ``` openid,email,profile ``` Add additional scopes if your application requires them (e.g. `https://www.googleapis.com/auth/gmail.readonly` for Gmail access). ## Step 5: Generate a Refresh Token The refresh token allows AutoSmoke to obtain fresh access tokens for each test run without requiring manual sign-in. 1. Click the **Generate** button next to the Refresh Token field 2. A Google sign-in popup will appear 3. Sign in with the Google account you want your tests to authenticate as 4. Review and grant the requested permissions 5. The popup will close and the **Refresh Token** field will be filled in automatically > **Important:** If no refresh token is returned, Google may have already issued one for this app previously. To fix this: > > 1. Go to [myaccount.google.com/permissions](https://myaccount.google.com/permissions) > 2. Find your app and click **Remove Access** > 3. Click **Generate** again in AutoSmoke ## Step 6: Configure Optional Settings ### Token Exchange URL If your application has a backend endpoint that converts Google access tokens into session cookies or JWT tokens, enter its full URL here. During test setup, AutoSmoke will call this endpoint with the access token to establish an authenticated session in the browser. **Leave this blank** if you want AutoSmoke to inject the tokens as flow variables instead. ### Cookie Domain If your application uses cookies scoped to a specific domain (e.g. `.example.com`), enter that domain here. This ensures authentication cookies are set on the correct domain during tests. **Leave this blank** if you're unsure — the default behavior works for most setups. ## Step 7: Save and Test 1. Click **Save** to store your configuration — all credentials are encrypted before storage 2. Re-open the Authentication Configuration dialog 3. Click **Test Connection** 4. A successful test shows a green confirmation with the email address of the authenticated Google account If the test fails, see the troubleshooting section below. ## Troubleshooting ### "No refresh token returned" Google only issues a refresh token on the **first** authorization. If you've previously authorized this app: 1. Go to [myaccount.google.com/permissions](https://myaccount.google.com/permissions) 2. Find the app and click **Remove Access** 3. Try generating the refresh token again in AutoSmoke ### "redirect_uri_mismatch" error The redirect URI in your Google Cloud credentials must exactly match: ``` https://autosmoke.dev/api/auth/google-oauth/callback ``` Check for trailing slashes, `http` vs `https`, and typos. ### "Access blocked: app has not been verified" Your consent screen is in **Testing** mode and the Google account you're using is not listed as a test user. Add it under **APIs & Services > OAuth consent screen > Test users** in Google Cloud Console. ### Token expires or stops working Refresh tokens can be invalidated if: - The user revokes access at [myaccount.google.com/permissions](https://myaccount.google.com/permissions) - The OAuth client ID or secret is changed or deleted - The token has been unused for 6 months - Google's security policies detect suspicious activity Re-generate the refresh token using the **Generate** button to fix this. ### Test Connection shows wrong email The test displays the email of the Google account that authorized the refresh token. If it shows an unexpected account, click **Generate** again and sign in with the correct account. --- # CI/CD Integration AutoSmoke integrates with your existing CI/CD pipeline so tests run automatically on every pull request. No browser infrastructure to manage — just add a config file and start catching regressions before they ship. ## GitHub Actions Add the following workflow file to your repository at `.github/workflows/smoke-tests.yml`: ```yaml name: Smoke Tests on: pull_request: branches: [main] push: branches: [main] jobs: smoke-test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Run AutoSmoke Tests uses: autosmoke/action@v1 with: api-key: ${{ secrets.AUTOSMOKE_API_KEY }} url: ${{ vars.STAGING_URL }} ``` The action will: 1. Trigger all tests associated with your project 2. Wait for them to complete 3. Post a status check with pass/fail results 4. Comment a summary with screenshots on the PR ## GitLab CI Add to your `.gitlab-ci.yml`: ```yaml smoke-tests: stage: test image: node:20-slim script: - npx @autosmoke/cli run --api-key $AUTOSMOKE_API_KEY --url $STAGING_URL rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" ``` ## Environment Variables | Variable | Required | Description | | ------------------- | -------- | ------------------------------------------------- | | `AUTOSMOKE_API_KEY` | Yes | Your project API key from the AutoSmoke dashboard | | `STAGING_URL` | Yes | The URL of the environment to test against | Store secrets using your CI provider's secret management — never commit API keys to the repository. ## Running Against Preview Deployments If you use Vercel, Netlify, or similar platforms that create preview URLs per PR, pass the dynamic preview URL instead of a fixed staging URL: ```yaml # Vercel example - name: Run AutoSmoke Tests uses: autosmoke/action@v1 with: api-key: ${{ secrets.AUTOSMOKE_API_KEY }} url: ${{ steps.deploy.outputs.preview-url }} ``` ## Parallel Execution Tests run in parallel by default. For large test suites, you can configure concurrency in your project settings to control how many tests execute simultaneously. ## Failure Handling When a test fails in CI: - The pipeline is marked as failed - A detailed report is generated with screenshots at each step - The team is notified via your configured [notification channels](/docs/notifications) (email, Slack, or webhook) Review the screenshots to quickly identify whether the failure is a real bug or a flaky environment issue. ## Next Steps - [Getting Started](/docs/getting-started) — Set up your first test - [Writing Tests](/docs/writing-tests) — Best practices for reliable tests - [Troubleshooting](/docs/troubleshooting) — Common issues and fixes --- # Notifications & Alerts AutoSmoke can notify you whenever your scheduled tests complete. You can set up multiple alerts per project, each with its own delivery channel and trigger conditions. ## Alert Types ### Email Send test results to any email address via [Resend](https://resend.com). Each email includes a summary of total, passed, and failed tests with a link to view full results. **Setup:** Enter the recipient email address — no additional configuration needed. ### Slack Post test results to a Slack channel using an [incoming webhook](https://api.slack.com/messaging/webhooks). Messages include a color-coded attachment (green for all passing, red for failures), test stats, and a button linking to the results page. **Setup:** 1. In Slack, go to **Apps > Incoming Webhooks** (or create a new Slack app) 2. Add a new webhook to your desired channel 3. Copy the webhook URL (starts with `https://hooks.slack.com/services/...`) 4. Paste it into the Webhook URL field when creating the alert ### Webhook Send test results as a JSON payload to any HTTPS endpoint. This is useful for integrating with custom dashboards, incident management tools, or automation pipelines. **Setup:** Enter any HTTPS URL. AutoSmoke will POST a JSON payload to this URL whenever alert conditions are met. **Payload format:** ```json { "event": "tests_completed", "timestamp": "2026-03-30T12:00:00.000Z", "project": { "domain": "example.com", "url": "https://autosmoke.dev/dashboard/sites/abc123" }, "results": { "status": "failed", "total": 5, "passed": 3, "failed": 2 } } ``` | Field | Description | | --- | --- | | `event` | Always `"tests_completed"` | | `timestamp` | ISO 8601 timestamp of when the notification was sent | | `project.domain` | The domain being tested | | `project.url` | Link to the project dashboard | | `results.status` | `"passed"` if all tests passed, `"failed"` if any failed | | `results.total` | Total number of tests in the batch | | `results.passed` | Number of passing tests | | `results.failed` | Number of failing tests | Your endpoint should return a `2xx` status code. Non-2xx responses are treated as delivery failures. ## Trigger Conditions Each alert supports three trigger conditions that control when notifications are sent: | Condition | Fires when | | --- | --- | | **On failure** | One or more tests in the batch failed | | **On recovery** | All tests passed, but the previous batch had failures | | **Every run** | Every time a batch of tests completes, regardless of outcome | You can enable any combination of these. For example, enabling both "On failure" and "On recovery" notifies you when things break and when they're fixed, but stays quiet during normal passing runs. ## Creating an Alert ### From the Alerts page 1. Go to **Dashboard > Alerts** 2. Click **New Alert** 3. Select the site, choose the alert type (Email, Slack, or Webhook) 4. Enter the destination (email address, Slack webhook URL, or webhook URL) 5. Choose your trigger conditions 6. Click **Save Alert** ### From a site detail page 1. Open any site in the dashboard 2. Expand the **Alerts** section 3. Click **Add Alert** 4. Fill in the same fields as above ## Managing Alerts - **Enable/disable** an alert with the toggle switch — disabled alerts are not sent - **Delete** an alert with the delete button — this is permanent ## Next Steps - [Getting Started](/docs/getting-started) — Set up your first test - [CI/CD Integration](/docs/ci-cd-integration) — Run tests automatically on every PR - [Troubleshooting](/docs/troubleshooting) — Common issues and fixes --- # Troubleshooting This guide covers the most common issues teams run into when setting up and running AutoSmoke tests, along with step-by-step fixes. ## Test Timing Out **Symptom:** Test fails with "Timeout waiting for page to load" or "Step timed out." **Causes and fixes:** - **Slow staging environment** — Increase the step timeout in your test settings. The default is 30 seconds per step; set it higher for pages that load large datasets. - **Waiting for an element that never appears** — Check that the element text or description in your test step matches what's actually on the page. AutoSmoke uses AI vision, but a completely wrong description won't match. - **Network issues in CI** — If timeouts only happen in CI, check that your CI runner has network access to the staging URL. Firewalled environments may need allowlisting. ## Authentication Failures **Symptom:** Test fails at the login step or lands on an unexpected page after login. **Fixes:** - Verify that the test credentials are still valid and the account isn't locked - If using SSO or OAuth, consider using a direct email/password login for test accounts - For MFA-protected flows, either disable MFA on the test account in staging or use the self-hosted runner where you control the environment - See [Handling Authentication](/docs/authentication) for detailed setup instructions ## Element Not Found **Symptom:** "Could not find element matching: ..." error. AutoSmoke uses computer vision and contextual understanding to find elements. If it can't find one: - **Check your description** — Use the visible label text, not internal IDs. For example, write "Click the Sign In button" rather than "Click #btn-signin." - **Check for overlays** — Cookie banners, modals, or onboarding tooltips can obscure elements. Dismiss them in an earlier test step. - **Dynamic content** — If content loads asynchronously, add a "Wait for" step before interacting with the element. - See [Understanding Selectors](/docs/selectors) for how AutoSmoke locates elements. ## Tests Pass Locally but Fail in CI **Common causes:** - **Different viewport size** — CI runners default to a smaller viewport. Set an explicit viewport in your test configuration to match what you expect. - **Missing environment variables** — Double-check that `AUTOSMOKE_API_KEY` and `STAGING_URL` are set in your CI secrets. - **DNS resolution** — If your staging URL is on a private network, make sure the CI runner can resolve it. - See [CI/CD Integration](/docs/ci-cd-integration) for setup details. ## Flaky Tests **Symptom:** Test passes sometimes and fails other times with no code changes. **Strategies to reduce flakiness:** - Add explicit "Wait for" steps before assertions on dynamically loaded content - Avoid testing against production if data changes frequently — use staging with seed data instead - Check for race conditions in your app (e.g., a dashboard that briefly shows empty state before data loads) - Use the test run history in the dashboard to identify patterns in failures ## Screenshots Show Wrong Page **Symptom:** Screenshots in the report don't match what you expect. - **Redirect** — The URL you provided may redirect. Check the actual URL in the screenshot metadata. - **Auth redirect** — You may have been redirected to a login page. Add an authentication step at the beginning of your test. - **A/B testing or feature flags** — Different users may see different content. Use a deterministic test account. ## Rate Limits and Concurrency If you see "Rate limit exceeded" errors: - Check your plan's concurrency limits on the [pricing page](/pricing) - Stagger test runs in CI if multiple pipelines trigger simultaneously - Use the dashboard to review current usage ## Getting Help If none of the above resolves your issue: - Check the [AutoSmoke documentation](/docs) for updated guides - Reach out via the [contact page](/contact) with your project ID and test run ID - Include screenshots and error messages for faster resolution --- # Blog # After TanStack: A Practical Defense Against the Next npm Supply Chain Attack Published: 2026-05-13 Author: AutoSmoke Team URL: https://autosmoke.dev/blog/surviving-npm-supply-chain-attacks-after-tanstack Summary: The TanStack compromise was the latest in a long run of 2025–2026 npm supply chain attacks. The pattern is consistent, and so is the playbook for surviving the next one. Here's what actually works. A recurring story has been working its way through engineering Slack channels for the better part of a year: a popular JavaScript package compromised, a malicious version briefly live on npm, developer machines and CI runners pillaged before the package is yanked. TanStack — the family of widely-used React libraries behind Query, Router, Table, and Form — became the latest name on that list. By the time the bad versions were pulled, every CI job that had run an unpinned `npm install` in the affected window had already executed attacker code. It was not the first time. It will not be the last. ![A close-up of a chain of metal links rendered in cool blue light, evoking a software supply chain](https://images.unsplash.com/photo-1614064548237-02f0cee06fbf?w=1600&q=80) ## The pattern that keeps working The specifics differ between incidents. The shape does not. - **September 2025 — Shai-Hulud.** A self-replicating worm published trojanized versions of dozens of npm packages, hopping from one maintainer's stolen token to the next. It infected hundreds of packages before npm staff caught up. - **August 2025 — Nx CLI.** Malicious versions of `nx` shipped a postinstall script that scanned for cloud credentials and SSH keys and exfiltrated them. The bad versions were live for hours; the rotation work for affected teams ran for weeks. - **The maintainer-phishing wave.** Through the second half of 2025, attackers sent convincing "your npm account is suspended" emails to maintainers of high-traffic packages. A surprising number of tokens were given up. Each one became a publishing key. - **TanStack.** The same shape. A maintainer compromise, a malicious version, an install-time payload, a window measured in hours. What unifies these is not the cleverness of the payload. It is how mundane the attack surface is. A maintainer's npm token gets phished, exfiltrated by malware on their laptop, or lifted from a leaked CI log. A malicious version goes up. Every project with a floating range — `"^1.2.3"`, `"~2.0.0"`, even a fresh `npm install` against an old lockfile that has since been deleted — pulls it. A `postinstall` or `prepare` script runs arbitrary code as the current user. It reads `~/.aws/credentials`, `~/.npmrc`, `~/.docker/config.json`, environment variables, GitHub tokens, anything readable. It posts to a URL. It exits cleanly. The build succeeds. By the time the registry pulls the version, the secrets are already gone. What follows is rotation, audit, and the awkward conversation about why a CI runner had production credentials in the first place. ## Why npm is the soft target The Node ecosystem has three structural traits that make this kind of attack disproportionately effective. **Install-time code execution is the default.** A `postinstall` script in any transitive dependency runs whenever you `npm install`. Most teams don't audit their own direct dependencies' install scripts, let alone the install scripts of the 1,400 packages in their lockfile. `npm` added `--ignore-scripts`, but almost nobody uses it because almost everyone has at least one dependency (native bindings, Husky, Playwright browsers) that genuinely needs it. **Floating versions are the default.** `npm init` produces a `package.json` with `^` ranges. `npm install ` adds them. The lockfile pins the resolved version *until something changes it* — and CI environments that delete `node_modules` between runs, or use `npm install` instead of `npm ci`, can quietly upgrade across patch versions. **Maintainer tokens publish anything.** An npm token authorized for `@scope/*` can publish any package in that scope, at any version, at any time. There is no second signer, no protected version line. The maintainer's laptop is the perimeter. That last point is the one most teams underestimate. **When you depend on a package, you are extending trust to every device, browser session, and credential store its maintainer has ever logged into.** The 2025–2026 attacks are the natural outcome of an ecosystem where that trust is implicit and untested. ## The playbook There is no single control that prevents this. There is a stack of mundane ones that, applied together, narrow the blast radius to something survivable. None of them are exotic. Most teams apply two or three. ### 1. Use `npm ci`, never `npm install`, in CI `npm ci` installs exactly what is in the lockfile and fails if `package.json` and the lockfile disagree. `npm install` resolves ranges fresh and can quietly pull a new version. The difference is whether a compromise that happens between yesterday's commit and today's build can reach your runner. Pin to the lockfile or do not pin at all. ### 2. Turn off install scripts where you can — and isolate where you can't `npm ci --ignore-scripts` is the single highest-leverage control. The teams who use it survived the recent waves with no rotation work because the malicious code never ran. The cost is real: anything with a genuine install step (Husky, Playwright, native modules) needs to be allowlisted explicitly. Tools like `@lavamoat/allow-scripts` and `pnpm`'s `onlyBuiltDependencies` field formalize this — you maintain a short list of packages permitted to run scripts, and everything else is silenced. If you can't turn scripts off, run installs in an isolated environment that has nothing worth stealing: no cloud credentials, no production tokens, no SSH keys, no long-lived OAuth sessions. A throwaway container with read-only access to the registry is dramatically less interesting to a postinstall payload than a developer's laptop. ### 3. Pin your direct dependencies, and audit before you upgrade Floating ranges in `package.json` were a 2010s convenience. In 2026 they are a foot-gun. Pin to exact versions (`"1.2.3"`, no caret) for everything you list as a direct dependency. Use Renovate or Dependabot to propose upgrades as PRs — and treat the PR as a real review, with a glance at the changelog, the release author, and the publish date. A version published 30 minutes ago by a maintainer who hasn't published in six months is a signal worth pausing on. ### 4. Require npm 2FA and provenance, then verify it Enable **mandatory 2FA for publish** on your own packages — `npm access set 2fa=publish` — and for the registry-wide org policies you control. For dependencies you consume, prefer packages that ship with **npm provenance attestations**: cryptographic proof that the version on the registry came from a specific GitHub Actions workflow on a specific commit. Provenance is opt-in for publishers, and adoption is uneven, but for the packages that have it (`@vercel/*`, `@sentry/*`, `react`, and a growing list), it converts "trust the maintainer's laptop" into "trust the published workflow." You can enforce this on consumption with `npm audit signatures` in CI. It fails the build if a package in your tree lacks a valid signature or provenance attestation from the registry. ### 5. Scope CI tokens like they will leak — because they will A CI job that runs untrusted dependency code has the privileges of whatever environment variables and OIDC tokens you handed it. The defense is not "make sure no dependency is malicious." It is "assume one will be, and limit what it can reach." - Use **GitHub Actions OIDC** to mint short-lived cloud credentials per job, not long-lived `AWS_ACCESS_KEY_ID` secrets in the repo. - Scope npm publish tokens to a single package or scope, with 2FA, with an expiry. - Separate the job that runs tests from the job that publishes. The test job sees source code; the publish job sees the token. They should not be the same shell session. - Treat `GITHUB_TOKEN` as a credential too. `permissions:` blocks default to `contents: read` — let them. ### 6. Watch for the IOCs after a compromise When a compromise is disclosed, the public advisories usually include the malicious package versions, the exfiltration domains, and sometimes the hashes of the dropped binaries. Pipe those into whatever endpoint or DNS monitoring you have and search backward. If a developer laptop or runner reached one of those domains, the rotation list is whatever was reachable from that machine — not whatever the advisory says was "exposed." ### 7. Lock down the developer laptop, not just the cloud The Shai-Hulud and Lumma Stealer-class attacks succeed because they harvest from local disk: `.npmrc`, `.aws/credentials`, browser cookie stores, SSH keys, password manager exports. Hardware-backed credential storage (1Password CLI with biometric unlock, `gh auth` with device flow, AWS SSO with short-lived sessions) makes a malicious postinstall measurably less productive. The goal is that an install-time payload running as your user finds nothing worth taking. ## The thing your defenses won't catch Even with all of the above, something will get through eventually. The interesting question is what you do in the **30 minutes after** an advisory drops. You rotate. Loudly, urgently, under time pressure, often across dozens of services. And then — this is the part everyone forgets — you have to confirm that the application still works. A worker that picks up a stale key and starts 500-ing isn't visible in the rotation PR. A webhook verified with the old signing secret silently drops legitimate traffic. A build succeeds because the env var loads, but the integration it gated is broken. The failure isn't the rotation. It's the delta between "rotated" and "verified in production." That gap is where supply chain incidents quietly become customer incidents. ## The summary worth keeping Supply chain attacks don't get prevented by a single brilliant tool. They get prevented by a stack of small, boring controls — `npm ci`, ignored scripts, pinned versions, provenance checks, scoped tokens, isolated runners — applied consistently before the advisory hits your inbox. The teams that survived TanStack, Nx, and Shai-Hulud without rotation marathons are not the ones with the best detection. They are the ones whose runners had nothing worth stealing in the first place. The next compromise is already published. The question is whether your build was set up assuming it. --- *At AutoSmoke, we run agentic smoke tests against your production deployment after every change — including the chaotic minutes after a forced credential rotation. If a supply chain advisory has your team rotating secrets right now, the next thing worth confirming is that your critical user flows still actually work. [Get started free](/new).* --- # Beyond Playwright: Why Agentic AI Is Eating End-to-End Testing in 2026 Published: 2026-04-27 Author: AutoSmoke Team URL: https://autosmoke.dev/blog/agentic-ai-testing-beyond-playwright-2026 Summary: Playwright just overtook Selenium for the first time — and a new wave of agentic AI testing tools is already making script-based frameworks feel like the next legacy. Here's what changed in 2026, and why post-deploy smoke testing will never look the same. Sometime in late 2025, Playwright quietly overtook Selenium to become the most-used end-to-end testing framework on the planet. The latest QA tooling surveys put it at **45.1% adoption**, with Selenium at 22.1% and Cypress at 14.4%. For a project that started as a Microsoft side experiment in 2020, that is a generational shift. It is also, almost certainly, a peak. Because while teams were busy migrating from `selenium-webdriver` to `@playwright/test`, a different conversation was happening one floor up: **agentic AI testing**. By April 2026, every major QA-trends report — Tricentis, Applitools, ThinkSys — points at the same thing. The question stopped being "which framework writes my selectors better." It became "why am I writing selectors at all." ![A blurred long-exposure photo of streaks of light on a circuit board, evoking automation and machine speed](https://images.unsplash.com/photo-1518770660439-4636190af475?w=1600&q=80) ## What "agentic testing" actually means The phrase gets used loosely. Stripped to fundamentals, agentic testing is a small **perception–action loop** that an LLM-driven agent runs against a real browser: 1. **Snapshot** the current page — DOM, accessibility tree, screenshot. 2. **Plan** the next action — the model picks a click, fill, scroll, or wait. 3. **Act** in the browser — execute the action via CDP or an equivalent driver. 4. **Observe** the result — toast text, URL change, new element, console error. Then it does it again. And again. Until either the goal is observably met — and the agent quotes the evidence (a confirmation message, a redirect URL, a row in a table) — or it gives up. ![Diagram of the four-step agentic testing loop: Snapshot, Plan, Act, Observe, with a retry arrow looping back and a "Done" terminal that requires quoted evidence](/blog-images/agentic-testing-loop.svg) That is the whole architecture. There are no selectors to write, no `await page.locator('[data-testid="checkout-button"]').click()`, no Page Object Models to maintain. The test is a sentence: *"Sign up with a fresh email, complete checkout for one item, confirm the order summary shows 'Order confirmed.'"* The agent works out the rest at runtime. The more sophisticated implementations split that loop across **multiple specialized agents** — a planner that decomposes the goal, a generator that proposes the next step, a runner that executes, and an analyzer that decides whether to retry, advance, or fail with evidence. The split keeps each agent small and predictable, even as the test surface grows. ## Why script-based testing hit a wall Playwright winning the framework war did not solve the actual problem teams face with end-to-end testing. It just made a familiar problem 60% less painful: *flaky tests caused by timing*. The other failure modes are still there. **Selectors rot.** Every refactor, every design-system upgrade, every A/B test that renames a CSS class breaks tests that depended on it. In a healthy codebase that ships daily, the maintenance tax compounds quietly. Tricentis put the average team's E2E maintenance time at **50–70% of total QA effort** — the rest split between writing new tests and triaging real failures. Their customers using AI agents instead reported an **85% reduction in manual effort** and a 60% productivity bump, almost entirely from not maintaining selectors. **Multi-framework reality.** The same survey found that **74.6% of QA teams now run two or more automation frameworks**. Playwright for the modern web app, Selenium for the one legacy admin panel that nobody wants to touch, a separate tool for mobile. Each one is a different DSL, a different CI configuration, a different mental model. None of them know about each other. **The "did the user actually succeed" problem.** A green test suite tells you that 487 assertions passed. It does not tell you whether a real user, hitting your real production deployment, can sign up. That gap is exactly where outages live. The CrowdStrike, Snowflake, and Vercel Dubai incidents of the last 18 months all had passing CI pipelines minutes before the production failure surfaced. ![A laptop on a wooden desk showing colorful analytics dashboards and trendlines](https://images.unsplash.com/photo-1551288049-bebda4e38f71?w=1600&q=80) The deeper issue is what Applitools called the **signal-to-noise problem** in their 2026 outlook: as test suites grow, the cost stops being execution time and starts being human attention. A flaky test that fails twice a week trains the team to ignore failures. Once that habit sets in, the suite has become decorative. Agentic testing does not fix this by adding more tests. It fixes it by **removing the layer that produces most of the noise** — the brittle selector code itself. ## What changes in practice Here is the same smoke check, written first as a Playwright test and then as an agentic step. **Playwright:** ```typescript test('user can sign up and reach dashboard', async ({ page }) => { await page.goto('https://app.example.com/signup'); await page.getByLabel('Email').fill(`qa+${Date.now()}@example.com`); await page.getByLabel('Password').fill('Test1234!'); await page.getByRole('button', { name: 'Create account' }).click(); await expect(page).toHaveURL(/\/dashboard/); await expect(page.getByText('Welcome')).toBeVisible(); }); ``` **Agentic:** ```yaml - goto: https://app.example.com/signup - step: Sign up with a fresh email and any valid password. - verify: The dashboard loads and shows a welcome message. ``` The Playwright version breaks the day someone renames the **Create account** button to **Sign up**. The agentic version does not. It re-reads the page, sees a button labeled "Sign up" that visually does the same job, clicks it, and continues. What you trade for that resilience: - **Determinism.** A scripted test does the same thing every time. An agentic test does *the goal* every time, but the path it takes can vary. - **Speed per run.** A snapshot-and-LLM-call cycle takes a couple of seconds. A scripted click takes milliseconds. Agentic tests are slower per execution — usually fine for post-deploy smoke, sometimes painful for tight inner loops. - **Verifiability.** This is where the better implementations earn their keep: requiring the agent to **quote observable evidence** when it declares done — the actual confirmation text, the actual redirect URL — instead of just self-certifying. Without that, agents will happily report success on a page that quietly 500'd. The new bottleneck, as Applitools put it, is **trust**. Not whether the test ran, but whether you can believe the result. ## What this means for your QA stack The pragmatic move in 2026 is not to delete Playwright. It is to recognize that the two approaches are good at different things, and to layer them. - **Keep Playwright (or Cypress) for the inner loop.** Stable, fast, deterministic checks against components and APIs you control. Things you run on every commit. The script tax is bearable when the surface is small. - **Use agentic testing for the outer loop.** The post-deploy smoke. The critical user journeys — signup, checkout, login, the one "money flow" your business actually depends on. The tests that you most need to *not* go stale, and that you most need to run against real production. In other words: **scripts for what you control, agents for what you ship to users.** The teams getting the most out of 2026's tooling are not picking sides — they are letting each tool do the job it is actually built for. The reason this matters is the same reason the framework numbers shifted in the first place. Software is being shipped faster than it can be tested by hand. AI is generating more code than humans can review. Every deploy is a chance for something invisible to break. Whatever you call your testing strategy, it has to keep up. Playwright winning was not the end of that story. It was the prologue. --- *At AutoSmoke, we run agentic smoke tests in real Chrome against your production deployments — no scripts, no selectors, evidence-backed pass/fail on the user journeys that matter. [Get started free](/new) and watch your critical flows after every deploy.* --- # The Vercel Breach: When Your AI Assistant Is the Supply Chain Published: 2026-04-21 Author: AutoSmoke Team URL: https://autosmoke.dev/blog/vercel-april-2026-security-breach Summary: On April 20, 2026, Vercel disclosed that attackers accessed internal systems through a compromised AI tool's OAuth connection to Google Workspace. The blast radius started with a Roblox cheat script. Here's the chain—and what it means for anyone wiring AI tools into their SaaS stack. Trace the chain backwards and it gets absurd fast: a user downloaded a Roblox exploit script. That script carried Lumma Stealer. The stealer harvested credentials that let attackers pivot into an AI startup called Context.ai. Context.ai had a Google Workspace OAuth app with broad scopes. A Vercel employee had granted that app access. From there, the attackers were inside Vercel's internal systems. This is the April 2026 Vercel breach—and it's a story about OAuth trust boundaries as much as it is about Vercel. ![A laptop in a darkened room with cascading green Matrix-style code glowing on its screen](https://images.unsplash.com/photo-1510511459019-5dda7724fd87?w=1600&q=80) ## What Happened On **April 19–20, 2026**, Vercel disclosed that attackers had gained unauthorized access to certain internal systems and contacted "a limited subset of customers" directly. The data exposed in that subset was narrow in one respect and alarming in another: attackers could read **non-sensitive environment variables** in plaintext. Sensitive environment variables—those stored with read-prevention—were not exposed, and Vercel confirmed that its npm packages and the broader software supply chain were not compromised. The threat actor claiming the intrusion operates under the **ShinyHunters** banner, reportedly seeking $2 million for the stolen data. Vercel has engaged **Mandiant** and additional incident-response firms, notified law enforcement, and shipped product changes in response (more on those below). ## The Attack Chain ![Diagram of the five-stage Vercel breach attack chain: Roblox exploit script, credential theft, Context.ai OAuth app compromise, pivot into Vercel employee Workspace, lateral movement into Vercel internal systems](/blog-images/vercel-breach-attack-chain.svg) The intrusion is a textbook supply-chain story, but with an AI-era twist. Each link in the chain was probably considered "low-risk" on its own. Together, they formed a path straight to production. - **February 2026 — Patient zero, far from Vercel.** A user (unaffiliated with Vercel) downloaded a Roblox exploit script. The script bundled **Lumma Stealer**, which scraped browser sessions, tokens, and credentials from the infected machine. - **Lateral drift into Context.ai.** The stolen credentials gave attackers a foothold inside **Context.ai**, a third-party AI tool. Specifically, they compromised Context.ai's **Google Workspace OAuth app**—the integration that Context.ai customers use to authorize the product against their Workspace accounts. - **OAuth pivot into a Vercel employee's Workspace.** A Vercel employee had authorized Context.ai against their enterprise Google Workspace with broad ("Allow All") scopes. Because OAuth tokens ride on the vendor's OAuth app, compromising Context.ai was equivalent to compromising every token that app had ever been granted. - **Lateral movement into Vercel internals.** With a Vercel employee's Workspace account effectively taken over, attackers moved laterally into certain internal Vercel systems where they could read environment variables for a subset of customer projects. For defenders, the indicator of compromise Vercel published is the Context.ai OAuth client ID: `110671459871-30f1spbu0hptbs60cb4vsmv79i7bbvqj.apps.googleusercontent.com` If your Workspace logs show that client ID, the exposure is potentially much broader than Vercel—reporting suggests the Context.ai compromise affected hundreds of downstream customers. ## Why "Allow All" OAuth Is the Real Story Every time someone clicks **Allow** on an OAuth prompt that says "This app will be able to read and manage your email, files, and calendars," they are not granting a permission to a person. They are granting it to that vendor's infrastructure, employees, build pipeline, and incident-response posture—forever, until revoked. When that vendor is a fast-moving AI startup that has existed for twelve months, you are effectively extending your corporate perimeter to include theirs. Your security is the minimum of your security and their security. This is not a new idea—CI vendors, logging vendors, and Slack apps have lived with the same math for years. What changed in 2026 is **how many AI tools have quietly become OAuth-privileged participants in production workflows**, often granted the broadest possible scopes because narrowing them would break the product. A meeting-notes bot that "reads your calendar, email, drive, and chats" is indistinguishable from adversary-grade surveillance the moment its vendor is breached. The Vercel incident is a clean proof of that failure mode: the breach didn't go through Vercel's code, Vercel's pipelines, or Vercel's customers. It went through a vendor that a single Vercel employee had authorized. ## What Was (and Wasn't) Exposed The distinction Vercel is drawing between **non-sensitive** and **sensitive** environment variables deserves a careful read. - **Non-sensitive variables** are readable from the Vercel dashboard in plaintext. They are meant to hold things like feature flags, public URLs, and non-secret configuration. In practice, teams regularly put API keys, database URLs, signing keys, and third-party tokens there—because the "sensitive" flag was not the default. - **Sensitive variables** are encrypted with read-prevention, meaning they cannot be retrieved in plaintext after being set—only referenced at build or runtime. These were **not** exposed. "Non-sensitive" never meant "low-value." It meant "readable." That's a subtle but costly distinction if the people reading are attackers. ## If You Operate a Vercel Project, Do This Today Vercel's guidance is specific. If you haven't worked through it yet, treat the following as a checklist: ### 1. Turn On MFA for Every Member of Every Team Authenticator apps or passkeys—not SMS. This is table stakes, but the breach is a reminder that an enterprise Workspace compromise will try to pivot into every connected SaaS. ### 2. Rotate Every Non-Sensitive Environment Variable API keys, database credentials, signing keys, third-party tokens, webhook secrets, OAuth client secrets—if it lives in a non-sensitive env var on an affected project, assume plaintext exposure and rotate. Do this **before** deleting or archiving projects, not after. ### 3. Flip the Sensitive Flag Going Forward Vercel has now shipped `sensitive: on` as the default for new environment variables, along with improved team-wide variable management. For existing variables, go back and turn the flag on for anything that shouldn't be readable from the dashboard. ### 4. Audit Activity Logs and Recent Deployments Look for unfamiliar IP addresses, team invites you didn't authorize, and deployments from branches you don't recognize. Vercel has also enhanced activity logging as part of this response—use it. ### 5. Rotate Deployment Protection Tokens If your project uses Deployment Protection, set it to at least "Standard" and rotate any tokens. A protection token that leaked is a bypass key to preview environments. ### 6. Audit Your Google Workspace OAuth App Inventory Whether or not you use Vercel, go into your Workspace admin console and look at the full list of third-party OAuth apps authorized by users in your org. For each, ask: **does this vendor need the scopes we granted, and what happens if they get breached?** Revoke aggressively. Most "Allow All" approvals are approvals by default, not by design. ## The Broader Lesson: AI Tools Are Now Supply-Chain Dependencies Every time your team authorizes a new AI product against a shared workspace, you are adding a supply-chain edge. That edge has all the same failure modes as any other vendor relationship—credential theft, insider risk, social engineering, malware—but it usually comes with broader scopes and less scrutiny, because AI tools demand access to content to be useful at all. Context.ai didn't fail spectacularly. It failed in exactly the way any SaaS vendor fails: an upstream machine got infected, credentials moved, an OAuth app became an entry point. The surprise isn't that it happened—it's how many downstream perimeters expanded without anyone noticing. The work for platform and security teams in 2026 is mundane and unglamorous: inventory OAuth grants, narrow scopes, require MFA on everything, and assume that every AI vendor you trust today will eventually have a bad day. ## After the Rotation Comes the Silent Breakage There is a second-order risk from this incident that doesn't show up in any bulletin: when a team rotates dozens of secrets under pressure, something usually breaks silently. A worker picks up the old key and starts 500-ing. A webhook endpoint that was verified with the old signing secret starts rejecting legitimate traffic. A build succeeds because the env var loads, but the application can no longer reach its database. The failure isn't the rotation. It's the delta between "rotated" and "verified in production." --- *At AutoSmoke, we build automated smoke tests that run against your production deployments—including the moments right after you've rotated secrets under incident pressure. If the Vercel breach triggered a mass rotation on your team, the next thing worth confirming is that your app still actually works. [Learn how automated smoke testing fits into your incident response workflow](/docs/getting-started).* --- # Vercel's Dubai Datacenter Failure: When a Single Region Takes Down Global Builds Published: 2026-03-03 Author: AutoSmoke Team URL: https://autosmoke.dev/blog/vercel-dubai-datacenter-outage-march-2026 Summary: On March 2, 2026, a datacenter failure in Vercel's Dubai (dxb1) region didn't just break deployments in the Middle East—it caused build failures worldwide for any project using Middleware. Here's what happened and what it means for your deployment strategy. If your Vercel deployments started failing on March 2, 2026, and you spent time convinced your code was broken—you weren't alone. A datacenter failure in Vercel's Dubai region (dxb1) cascaded far beyond the Middle East, catching thousands of developers off guard. Here's what happened, why the blast radius was so wide, and what you can do to protect yourself next time. ![Aerial view of the Dubai coastline with the sail-shaped Burj Al Arab hotel rising above a turquoise sea](https://images.unsplash.com/photo-1518684079-3c830dcef090?w=1600&q=80) ## What Happened Starting around **5:00 AM UTC on March 2**, Vercel's Dubai region (dxb1) began experiencing operational failures. Function invocations and deployments targeting dxb1 started failing with internal errors. If the outage had been limited to Dubai, most teams outside the region would never have noticed. But it wasn't. **The critical detail:** Vercel deploys Middleware Functions globally for production deployments. That means if your Next.js project uses middleware—for authentication, redirects, header manipulation, A/B testing, or any other purpose—your build needed to deploy to *every* region, including dxb1. When dxb1 couldn't accept deployments, those builds failed entirely. This turned a regional infrastructure problem into a global build outage. ## The Timeline - **~05:00 UTC** — Failures begin in the dxb1 region. Deployments and function invocations start returning internal errors. - **15:29 UTC** — Vercel officially identifies the issue. Traffic from dxb1 is rerouted to bom1 (Mumbai), the nearest available Edge region. - **Following hours** — Vercel begins excluding dxb1 from Middleware deployment targets so that builds using Middleware can complete. Eventually, all builds are configured to skip dxb1. - **18:23 UTC** — Vercel confirms continued monitoring and work toward restoring capacity in dxb1. - **Later that day** — The incident is resolved and dxb1 capacity is restored. For roughly **10+ hours**, developers who had no connection to the Dubai region were unable to deploy production builds if their projects included Middleware. ## Why This Matters This incident highlights an architectural reality of global edge deployments that's easy to overlook: **a single unhealthy region can block deployments that target all regions**. Most teams think of region selection as a performance optimization—pick the regions closest to your users. But when your deployment pipeline requires success across all regions (as Middleware does), every region becomes a dependency. Your deployment is only as reliable as the least reliable region in the set. This is the same class of problem we've seen repeatedly in major outages: a system's actual failure domain is wider than what teams assume. AWS's US-East-1 DNS outage in October 2025 proved this for cloud infrastructure. Vercel's dxb1 incident proves it for edge deployment platforms. ## What You Can Do ### 1. Know Your Deployment Targets If you're using Vercel, understand which regions your project deploys to and whether any features (like Middleware) force global deployment. The `regions` configuration in `vercel.json` gives you some control, but Middleware's global requirement can override it. ### 2. Have a Rollback Plan When builds fail, you need to know how to keep serving your last successful deployment. Vercel keeps previous deployments accessible—make sure your team knows how to promote a previous deployment if a new one can't complete. ### 3. Monitor the Platform, Not Just Your Code When a build fails, developers instinctively check their code. That's the right first step, but if you can't find the issue locally, check your platform's status page before spending hours debugging phantom problems. Subscribing to [Vercel's status page](https://www.vercel-status.com) notifications can save significant time. ### 4. Consider Multi-Platform Deployment For critical applications, having the ability to deploy to an alternative platform (even as a degraded fallback) can mean the difference between an outage and a minor inconvenience. This doesn't mean running two platforms full-time—it means having a tested escape hatch. ### 5. Test Your Middleware Necessity Not every project that uses Middleware actually needs it in production. If your Middleware handles something that could be done at the application level (like simple redirects), consider whether the global deployment requirement is worth the added failure surface. ## The Broader Lesson ![A view of Earth from orbit at night, with the glow of coastal cities tracing the continents](https://images.unsplash.com/photo-1451187580459-43490279c0fa?w=1600&q=80) Edge computing promises lower latency and better performance by running code closer to users. That's a real benefit. But it also means your deployment depends on infrastructure in regions you may never think about. The developers who were most frustrated on March 2 weren't teams deploying to Dubai. They were teams in North America and Europe who had no idea their builds depended on a datacenter thousands of miles away. The infrastructure was invisible—until it failed. Every abstraction that hides complexity also hides risk. The teams that recover fastest from incidents like this are the ones that understand their deployment topology *before* something goes wrong. --- *At AutoSmoke, we build automated smoke tests that run against your production deployments. When a platform outage breaks your builds, knowing whether your last successful deployment is still healthy is critical. [Learn how automated smoke testing fits into your incident response workflow](/docs/getting-started).* --- # When the Lights Go Out: Major Software Outages of 2024–2026 Published: 2026-02-23 Author: AutoSmoke Team URL: https://autosmoke.dev/blog/major-software-outages-2024-2025 Summary: From CrowdStrike's 8.5 million blue screens to AWS losing DNS for 15 hours—a breakdown of the biggest software outages in recent memory and what they reveal about modern system fragility. The modern software stack is a marvel of interconnected systems. It is also, as recent history keeps proving, profoundly fragile. Over the past two years, some of the most trusted names in tech—Microsoft, AWS, Google, CrowdStrike, Snowflake—have experienced outages that disrupted millions of businesses and users worldwide. These aren't just embarrassing postmortems. They're a window into how complex systems fail, and what every engineering team can learn from them. ![A row of dark server racks lit with green and orange status lights and a tangle of patch cables](https://images.unsplash.com/photo-1558494949-ef010cbdcc31?w=1600&q=80) ## CrowdStrike (July 2024): The Update That Broke the World No outage in recent memory had the sheer visual impact of the CrowdStrike incident. On July 19, 2024, approximately **8.5 million Windows machines** around the world simultaneously displayed the Blue Screen of Death. ![A digital signage pillar at LaGuardia Airport showing the Windows Blue Screen of Death during the CrowdStrike outage](/blog-images/crowdstrike-bsod-lga.jpg) _Even airport signage wasn't spared: a BSOD at LaGuardia during the July 19, 2024 outage. — [Wikimedia Commons, CC BY-SA 4.0](https://commons.wikimedia.org/wiki/File:CrowdStrike_BSOD_at_LGA.jpg)_ The cause was deceptively simple: a faulty configuration file shipped as part of a routine update to CrowdStrike's Falcon sensor. The file contained a logic error that caused the sensor—which runs at the kernel level—to trigger a crash on boot. Because the update was deployed automatically to CrowdStrike's entire user base at once, the impact was instantaneous and global. Airlines grounded flights. Hospitals reverted to paper records. Banks and broadcasters went dark. The estimated economic damage ran into the **billions of dollars**. What made recovery so painful was the manual nature of the fix. Affected machines couldn't boot, so IT teams had to physically access each one—often compounded by BitLocker encryption requiring recovery keys just to reach the command prompt. For organizations with thousands of endpoints across multiple locations, this took days. **The core failure:** A content update bypassed the staged rollout process that software code changes would normally go through. No canary deployment. No gradual rollout. One bad file, deployed everywhere, simultaneously. ## AWS US-East-1 (October 2025): A DNS Error, $650M in Losses On October 20, 2025, AWS experienced one of its most damaging outages in years. A **DNS error in the US-East-1 data center** (Northern Virginia) prevented applications from resolving DynamoDB's endpoint. Since DynamoDB underpins a vast number of AWS-hosted services, the failure cascaded rapidly. The outage lasted roughly **15 hours** and affected over **4 million users** and more than **1,000 companies**—including Snapchat, Reddit, and numerous payment and financial trading platforms. Economic losses were estimated between **$500 million and $650 million** for US companies alone. The concentration of workloads in us-east-1 has long been a known risk. The region hosts a disproportionate share of internet infrastructure, which means any instability there doesn't stay in us-east-1—it ripples outward through dependent services worldwide. **The core failure:** Single-region dependency, combined with a DNS layer that became a single point of failure for service discovery. ![Close-up of a network patch panel, numbered RJ45 ports with blue and grey ethernet cables plugged in](https://images.unsplash.com/photo-1544197150-b99a580bb7a8?w=1600&q=80) ## Google Cloud (June 2025): 54 Services, 7+ Hours On June 12, 2025, Google Cloud suffered a **global outage lasting over seven hours** that disrupted 54 services simultaneously—including API Gateway, App Engine, Cloud Run, and the Vertex Gemini API. The root cause was an **invalid automated quota update** pushed to the API management system, which caused external API requests to be rejected globally. The automation that was supposed to manage resource limits became the mechanism of failure. The blast radius extended far beyond Google's own products. Cloudflare, Spotify, Snapchat, and Discord all experienced cascading failures as their Google Cloud dependencies went dark. Within the same month, Cloudflare itself experienced a two-and-a-half-hour outage linked to a third-party cloud provider—a reminder that the dependencies of dependencies matter too. **The core failure:** An automated system making configuration changes at global scope, with no circuit breaker to limit blast radius. ## Microsoft Azure / M365 (Multiple, 2025–2026) Microsoft had an unusually turbulent stretch across 2025 and into 2026. **January 2025** brought a 50-hour Azure East US2 outage caused by networking configuration issues—notable both for its duration and for being the first major outage of the year. **October 2025** saw another Azure disruption affecting Office 365, Teams, Outlook, and Xbox Live, preceded by a September incident in the same quarter. Then, in **January 2026**, Microsoft 365 suffered a **nine-hour outage** affecting Outlook, Exchange Online, SharePoint, OneDrive, and Teams across North America. The cause: elevated service load during a maintenance window for a subset of North American infrastructure, compounded by a load balancing configuration change that worsened traffic distribution rather than improving it. Also in early 2026, an **inadvertent configuration change to Azure Front Door (AFD)**—a global networking layer—caused failures across all Azure regions simultaneously, affecting Entra, Defender, Purview, and downstream customers including Alaska Airlines. Recovery required rolling back to a "last known good" configuration. **The recurring pattern:** Configuration changes—applied too broadly, without sufficient validation or rollback safeguards—as the initiating event for cascading failures. ## Snowflake (December 2025): A Schema Change Knocks Out 10 Regions On December 16, 2025, Snowflake pushed a **backwards-incompatible database schema change** that caused a 13-hour outage spanning 10 of its 23 global regions, across AWS, Azure, and GCP simultaneously. Customers saw `SQL execution internal error` messages and were unable to query data or ingest files. For data-dependent businesses—analytics pipelines, dashboards, reporting workflows—the disruption was severe. The multi-cloud nature of the outage underscored something important: the assumption that distributing workloads across cloud providers protects against outages only holds if the failure doesn't originate in a shared layer. A schema change in Snowflake's own infrastructure can simultaneously affect all three of AWS, Azure, and GCP environments. **The core failure:** A breaking schema change deployed without compatibility validation or phased rollout across regions. ## Intercom (January 2026): 71 Minutes of Total Darkness Intercom's US region experienced a **complete service blackout on January 9, 2026**—71 minutes during which Inbox, Messenger, and all APIs were entirely unavailable. The cause traced back to a logic bug in the **Vitess database routing layer**. During a routine table-move operation, rollback logic incorrectly applied an empty routing configuration (VSchema), effectively disconnecting the application from its data shards. The application had no route to its own data. Seventy-one minutes is short compared to the other incidents in this list, but for a customer communications platform where businesses rely on Intercom to handle live support conversations, the impact on trust was real. **The core failure:** Rollback logic that could apply a destructive (empty) state, with no validation that the resulting configuration was valid before applying it. --- ## What These Outages Have in Common ![A dark laptop screen showing a live analytics dashboard with line charts and traffic metrics](https://images.unsplash.com/photo-1551288049-bebda4e38f71?w=1600&q=80) Across all of these incidents—different companies, different stacks, different failure modes—certain patterns keep appearing: **1. Configuration changes are the new deployment risk.** In most of these cases, no new code was shipped. A configuration update, a schema change, a routing map, a quota setting—these are the initiating events. Yet configuration changes often receive far less testing scrutiny than code changes. **2. Automation amplifies blast radius.** Automated updates, automated quota management, automated rollbacks—these systems are valuable precisely because they operate at scale and speed. But when they fail, they fail at scale and speed too. The CrowdStrike and Google Cloud incidents are the clearest examples: automation that was meant to help became the mechanism of global failure. **3. Staged rollouts are non-negotiable.** CrowdStrike's content update skipped the canary/staged deployment process. Snowflake's schema change hit 10 regions simultaneously. Had either been deployed incrementally—to one region, one cohort of machines, one data center first—the failure would have been detected before it became catastrophic. **4. Dependencies create hidden coupling.** When Google Cloud goes down, so does Cloudflare's infrastructure that depends on it. When AWS DNS fails, every service using DynamoDB fails with it. The real blast radius of any outage is measured not just by the failing system, but by everything downstream of it. **5. Recovery is harder than prevention.** The CrowdStrike incident required manual, physical intervention on millions of machines. The Azure Front Door incident required a full configuration rollback across all global regions. These recovery processes take orders of magnitude longer than the root cause change that triggered them. --- ## What This Means for Your Team If you're running a product on top of any of these platforms—and most teams are—these outages aren't just cautionary tales about other companies. They're a reminder that your users' experience depends on layers of infrastructure you don't control. The practical response isn't to eliminate cloud dependencies (you can't). It's to build for the assumption that any dependency can fail: - **Know your critical paths.** Which user journeys break when your cloud provider, database, or third-party service goes down? If you don't have a list, start there. - **Test your fallbacks, not just your happy paths.** Circuit breakers, graceful degradation, and fallback UIs only work if they've been tested. A fallback that's never been exercised is probably broken. - **Monitor what users experience, not just what servers report.** A server can report healthy while the user-facing flow is completely broken. Synthetic monitoring of actual user journeys catches what infrastructure metrics miss. - **Practice recovery.** The teams that recovered fastest from CrowdStrike were the ones that had tested their business continuity plans before they needed them. Disaster recovery is a skill that degrades without practice. The outages of the past two years haven't been caused by exotic zero-days or unprecedented failure modes. They've been caused by configuration changes, schema updates, and deployment automation—the same categories of change that every engineering team ships every week. The difference between an incident and a catastrophe is usually not the change itself. It's the blast radius, the detection time, and the recovery plan. --- _AutoSmoke helps teams catch critical failures before users do—with AI-powered smoke tests that run after every deploy and on a continuous schedule against your production environment. [Get started free](/new)._ --- # The Critical Role of Testing and Monitoring in the Age of AI Development Published: 2026-02-20 Author: AutoSmoke Team URL: https://autosmoke.dev/blog/testing-monitoring-ai-development Summary: As AI accelerates software development, the importance of robust testing and monitoring has never been greater. Here's why quality assurance is more critical than ever. The software development landscape is undergoing a fundamental transformation. AI-powered coding assistants can now generate entire features in minutes, startups ship products faster than ever, and the pace of iteration has reached unprecedented levels. But with great speed comes great responsibility—and a growing blind spot that many teams are overlooking: **quality assurance**. ![A MacBook displaying a dense wall of source code in a dark editor theme](https://images.unsplash.com/photo-1555066931-4365d14bab8c?w=1600&q=80) ## The AI Development Paradox Here's the paradox of AI-assisted development: the same tools that help us build faster also make it easier to introduce bugs faster. When an AI can generate hundreds of lines of code in seconds, it can also generate hundreds of potential issues just as quickly. Consider this scenario: A developer uses an AI assistant to implement a new checkout flow. The code looks correct, passes a quick manual test, and ships to production. Two days later, support tickets start rolling in—edge cases the AI didn't consider, browser compatibility issues, race conditions under load. This isn't a criticism of AI tools. They're incredibly powerful and are genuinely making developers more productive. But they're also shifting where bugs come from. Instead of typos and syntax errors, we're seeing more subtle issues: architectural problems, integration failures, and edge cases that weren't considered. ## Why Traditional Testing Falls Short The traditional approach to testing—writing unit tests, running them in CI, and calling it a day—isn't equipped for this new reality. ### The Coverage Illusion High code coverage doesn't mean high quality coverage. AI-generated code might have 90% line coverage while missing the critical paths that real users actually take. Unit tests verify that individual functions work, but they don't verify that your application works _as a product_. ### The Maintenance Burden End-to-end tests have always been the answer to testing real user flows, but they come with a notorious maintenance burden. Every UI change breaks a dozen tests. Selectors become stale. Tests become flaky. Eventually, teams start ignoring failures or abandoning E2E testing altogether. This creates a dangerous gap: the code is tested at the unit level, but the actual user experience goes unverified. ### Speed vs. Confidence In an AI-accelerated development cycle, spending hours maintaining brittle Selenium tests isn't feasible. Teams need testing that keeps up with their development pace. But cutting corners on testing leads to exactly the problems we described earlier—shipping bugs faster. ## The Case for Continuous Monitoring ![A laptop on a glass desk showing a colorful web analytics dashboard with line and bar charts](https://images.unsplash.com/photo-1460925895917-afdab827c52f?w=1600&q=80) The solution isn't to slow down development. It's to fundamentally rethink how we approach quality assurance. ### Shift from "Testing" to "Monitoring" Traditional testing is a gate: code passes tests, then ships. But in a world of continuous deployment and rapid iteration, testing needs to be continuous too. Think of your production application as a living system that needs constant health checks. You wouldn't deploy a server without monitoring its CPU and memory. Why would you deploy a web application without monitoring its actual functionality? ### Test What Users Do, Not What Developers Think The most valuable tests are the ones that verify real user journeys: - Can users sign up? - Can users complete a purchase? - Can users access their data? These aren't edge cases—they're the core of your business. If any of these break, nothing else matters. ### Embrace Self-Healing The brittleness of traditional E2E tests comes from their rigid selectors. Change a CSS class, and the test breaks. Modern AI-powered testing can adapt to UI changes automatically, the same way a human would. When a button moves from the header to the sidebar, a human tester would find it and click it. AI-powered tests can do the same—adapting to changes without requiring manual updates. ## Building a Quality Culture in the AI Era ![A small team working around a wooden table, each with a laptop open, pair-programming](https://images.unsplash.com/photo-1522071820081-009f0129c71c?w=1600&q=80) For teams embracing AI-assisted development, here's a practical framework for maintaining quality: ### 1. Automate Critical Path Testing Identify the 5-10 user journeys that absolutely must work at all times. Cover them with self-healing smoke tests that run after every deploy (and on a schedule between deploys). ### 2. Monitor, Don't Just Test Run your critical path tests on a schedule—hourly, or even more frequently. Treat test failures like you'd treat a server outage: something that demands immediate attention. ### 3. Make Testing Fast and Painless If testing is slow or painful, it won't happen. Choose tools that let you write tests quickly (plain English beats code), run them quickly (minutes, not hours), and maintain them easily (self-healing over manual updates). ### 4. Close the Feedback Loop When a test fails in monitoring, you should know within minutes—not days. Set up notifications that reach the right people immediately. ### 5. Test in Production(-like) Environments The closer your test environment is to production, the more valuable your tests are. Test against real data, real integrations, and real network conditions. ## The Future of Quality Assurance As AI continues to transform software development, quality assurance will become both more challenging and more critical. The teams that thrive will be those that embrace AI not just for building software, but for testing it too. The goal isn't to slow down the AI-assisted development process. It's to ensure that speed doesn't come at the cost of quality. With the right tools and practices, teams can have both: the productivity gains of AI-assisted development and the confidence that comes from robust, continuous testing. The age of AI development is here. The question is whether your testing strategy is ready for it. --- _AutoSmoke helps teams maintain confidence in their applications with AI-powered, self-healing smoke tests that run after every deploy. [Get started free](/new) and see how easy modern testing can be._ --- ## Contact - Email: contact@autosmoke.dev - Website: https://autosmoke.dev