Solutions to common issues when running AutoSmoke tests, including timeouts, authentication problems, element detection, and CI failures.

This guide covers the most common issues teams run into when setting up and running AutoSmoke tests, along with step-by-step fixes.

Test Timing Out#

Symptom: Test fails with "Timeout waiting for page to load" or "Step timed out."

Causes and fixes:

Slow staging environment — Increase the step timeout in your test settings. The default is 30 seconds per step; set it higher for pages that load large datasets.
Waiting for an element that never appears — Check that the element text or description in your test step matches what's actually on the page. AutoSmoke uses AI vision, but a completely wrong description won't match.
Network issues in CI — If timeouts only happen in CI, check that your CI runner has network access to the staging URL. Firewalled environments may need allowlisting.

Authentication Failures#

Symptom: Test fails at the login step or lands on an unexpected page after login.

Fixes:

Verify that the test credentials are still valid and the account isn't locked
If using SSO or OAuth, consider using a direct email/password login for test accounts
For MFA-protected flows, either disable MFA on the test account in staging or use the self-hosted runner where you control the environment
See Handling Authentication for detailed setup instructions

Symptom: "Could not find element matching: ..." error.

AutoSmoke uses computer vision and contextual understanding to find elements. If it can't find one:

Check your description — Use the visible label text, not internal IDs. For example, write "Click the Sign In button" rather than "Click #btn-signin."
Check for overlays — Cookie banners, modals, or onboarding tooltips can obscure elements. Dismiss them in an earlier test step.
Dynamic content — If content loads asynchronously, add a "Wait for" step before interacting with the element.
See Understanding Selectors for how AutoSmoke locates elements.

Common causes:

Different viewport size — CI runners default to a smaller viewport. Set an explicit viewport in your test configuration to match what you expect.
Missing environment variables — Double-check that AUTOSMOKE_API_KEY and STAGING_URL are set in your CI secrets.
DNS resolution — If your staging URL is on a private network, make sure the CI runner can resolve it.
See CI/CD Integration for setup details.

Symptom: Test passes sometimes and fails other times with no code changes.

Strategies to reduce flakiness:

Add explicit "Wait for" steps before assertions on dynamically loaded content
Avoid testing against production if data changes frequently — use staging with seed data instead
Check for race conditions in your app (e.g., a dashboard that briefly shows empty state before data loads)
Use the test run history in the dashboard to identify patterns in failures

Symptom: Screenshots in the report don't match what you expect.

Redirect — The URL you provided may redirect. Check the actual URL in the screenshot metadata.
Auth redirect — You may have been redirected to a login page. Add an authentication step at the beginning of your test.
A/B testing or feature flags — Different users may see different content. Use a deterministic test account.

If you see "Rate limit exceeded" errors:

If none of the above resolves your issue: