This guide covers the most common issues teams run into when setting up and running AutoSmoke tests, along with step-by-step fixes.

Test Timing Out

Symptom: Test fails with "Timeout waiting for page to load" or "Step timed out."

Causes and fixes:

Slow staging environment — Increase the step timeout in your test settings. The default is 30 seconds per step; set it higher for pages that load large datasets.
Waiting for an element that never appears — Check that the element text or description in your test step matches what's actually on the page. AutoSmoke uses AI vision, but a completely wrong description won't match.
Network issues in CI — If timeouts only happen in CI, check that your CI runner has network access to the staging URL. Firewalled environments may need allowlisting.

Authentication Failures

Symptom: Test fails at the login step or lands on an unexpected page after login.

Fixes:

Verify that the test credentials are still valid and the account isn't locked
If using SSO or OAuth, consider using a direct email/password login for test accounts
For MFA-protected flows, either disable MFA on the test account in staging or use the self-hosted runner where you control the environment
See Handling Authentication for detailed setup instructions

Symptom: "Could not find element matching: ..." error.

AutoSmoke uses computer vision and contextual understanding to find elements. If it can't find one:

Check your description — Use the visible label text, not internal IDs. For example, write "Click the Sign In button" rather than "Click #btn-signin."
Check for overlays — Cookie banners, modals, or onboarding tooltips can obscure elements. Dismiss them in an earlier test step.
Dynamic content — If content loads asynchronously, add a "Wait for" step before interacting with the element.
See Understanding Selectors for how AutoSmoke locates elements.

Common causes:

Different viewport size — CI runners default to a smaller viewport. Set an explicit viewport in your test configuration to match what you expect.
Missing environment variables — Double-check that AUTOSMOKE_API_KEY and STAGING_URL are set in your CI secrets.
DNS resolution — If your staging URL is on a private network, make sure the CI runner can resolve it.
See CI/CD Integration for setup details.

Symptom: Test passes sometimes and fails other times with no code changes.

Strategies to reduce flakiness:

Add explicit "Wait for" steps before assertions on dynamically loaded content
Avoid testing against production if data changes frequently — use staging with seed data instead
Check for race conditions in your app (e.g., a dashboard that briefly shows empty state before data loads)
Use the test run history in the dashboard to identify patterns in failures

Symptom: Screenshots in the report don't match what you expect.

Redirect — The URL you provided may redirect. Check the actual URL in the screenshot metadata.
Auth redirect — You may have been redirected to a login page. Add an authentication step at the beginning of your test.
A/B testing or feature flags — Different users may see different content. Use a deterministic test account.

If you see "Rate limit exceeded" errors:

If none of the above resolves your issue: