A lot of automated tests begin in an unusually pleasant universe.
The user is logged in.
The session is valid.
The database contains exactly the right data.
The feature flags are known.
The browser has no weird state.
The network works.
Nothing unexpected is covering the screen.
And the deployment that happened five minutes ago is assumed to be fine because CI passed.
Then the test runs.
Green.
Fantastic.
Unfortunately, real users don't live in that universe.
They leave tabs open overnight.
Their access tokens expire halfway through a workflow.
They get assigned to experiments.
They lose network connectivity.
A modal opens while another component still has focus.
A deployment changes one service but not another.
And sometimes the application quietly logs them back in while they're doing something important.
If your test automation strategy assumes those things don't happen, you might have excellent coverage of a product that doesn't actually exist.
The clean-session assumption is particularly dangerous
Here's a very normal test:
Login Open Dashboard Create Project Invite User Verify Invitation Looks good.
Now try this:
Login Open Dashboard Wait until access token expires Start creating project API returns 401 Application silently refreshes token Request retries Continue workflow Did the project get created once?
Twice?
Not at all?
Did the UI recover?
Did the application dump the user back onto the login page?
Did it preserve what they had typed?
Did the test framework even notice that authentication changed underneath it?
This is one of those situations where a test can pass hundreds of times simply because you're always starting with a fresh session.
There's a useful benchmark plan for session expiry recovery, silent re-login, and token-refresh evidence that treats this as something browser testing platforms should be evaluated against.
I agree.
Authentication isn't just a setup step.
It's application state.
Users don't politely expire between tests
Most automation frameworks encourage isolation.
That's generally a good thing.
Fresh context.
Fresh cookies.
Fresh test data.
Predictable environment.
But isolation can accidentally remove the exact states that make real applications difficult.
A real user might:
09:00 Log in 09:15 Open a form 09:22 Switch tabs 10:04 Come back 10:07 Submit Your automated test might do the entire thing in 14 seconds.
Those are technically the same workflow.
They're not the same conditions.
If your product has session expiration, refresh tokens, idle timeouts, or role changes, time itself becomes part of the test.
Feature flags create a similar illusion
Feature flags are great.
They let you roll things out gradually.
Test with internal users.
Run experiments.
Disable broken features quickly.
But they also create a new dimension of application state.
The same URL can now represent two different products.
checkout_v2 = false vs. checkout_v2 = true And if your test suite handles this by copying flag setup into 80 scenarios, you eventually get something like:
Set checkout_v2 = true Set new_nav = false Set pricing_test = variant_b Set recommendations = true Repeated everywhere.
Then somebody changes the feature configuration and half the suite becomes archaeological evidence.
This guide on testing feature flag overrides without hardcoding the same flag values into every scenario tackles that exact problem.
The larger lesson is useful even if you're not using feature flags heavily.
Test setup is part of your architecture.
If every test independently recreates the world it needs, maintenance eventually becomes expensive.
UI state is more than "element visible"
Here's another situation I think browser automation often oversimplifies.
Modals.
Testing a modal usually looks something like:
Click "Delete" Verify confirmation modal is visible Click "Confirm" Okay.
But accessibility changes what "working" actually means.
If a modal is open, what happens to the rest of the page?
Can keyboard focus escape?
Can you tab into controls behind the modal?
Can screen-reader users interact with background content that visually appears locked?
The relatively new inert attribute gives browsers a cleaner way to make parts of a page non-interactive.
But testing it properly requires more than:
modal.isVisible() === true This walkthrough on testing the inert attribute, modal background locking, and tabbable leakage is a good example of how much behavior can hide behind something that looks visually correct.
A screenshot can show you a perfect modal.
The keyboard experience can still be completely broken.
This is why I don't love "end-to-end coverage" as a metric
Teams will say:
We have 80% of our critical user journeys automated.
That sounds great.
But what does it mean?
If you test:
Login → Dashboard → Checkout → Confirmation do you also test:
Login ↓ Session expires ↓ Checkout begins ↓ Token refreshes ↓ Feature flag changes rendering ↓ Modal opens ↓ Keyboard navigation continues ↓ Purchase completes Probably not.
And I'm not saying you should automate every imaginable combination.
That would be ridiculous.
The point is that "journey coverage" can give you false confidence if your journeys are modeled as clean sequences instead of changing states.
AI-generated tests can amplify this problem
AI makes it easier to generate a lot of automation.
Type:
Test checkout.
And an agent can potentially create the flow for you.
That's useful.
But the first generated version will often represent the obvious path.
Open checkout Enter card Click Pay Verify success The interesting part is what happens later.
Can you edit the steps?
Can you add recovery behavior?
Can another person understand why something failed?
Can you attach notes explaining why a strange step exists?
Can the test survive being handed to a different team six months from now?
Those questions matter much more once AI starts generating a significant percentage of the suite.
This rubric on editable steps, recovery notes, and team handoffs in AI testing platforms focuses on exactly that.
I think "editable" is an underrated feature in AI systems.
Autonomy sounds cooler.
Editability is what saves you when reality gets complicated.
Fully autonomous QA sounds great until you ask what happens at 3 AM
There's a lot of interest now in autonomous QA agents.
And I understand why.
The idea is compelling:
Give agent application ↓ Agent explores ↓ Agent creates tests ↓ Agent runs continuously ↓ Agent finds failures That's a very different model from manually maintaining hundreds or thousands of scripts.
There's a demonstration of that approach in Endtest Bot: Your Autonomous QA Agent.
I think agentic testing will become a meaningful part of QA.
But the important question isn't whether the agent can click around your product.
It's:
What happens when the application behaves unexpectedly?
An autonomous system becomes much more valuable when it can recognize:
Session expired Token refreshed Expected state restored Continue scenario rather than simply reporting:
Step failed That difference is huge.
Recovery may be more important than self-healing
"Self-healing" became one of the favorite phrases in test automation.
Usually it means:
Locator broke ↓ Tool found another locator ↓ Test continued Useful.
But that's a pretty narrow definition of healing.
Real applications need recovery from things like:
- expired sessions
- intermittent APIs
- reauthentication
- WebSocket reconnects
- temporary loading states
- feature changes
- interrupted transactions
- background refreshes
Changing #submit-button to [aria-label="Submit"] is the easy version.
Recovering application state without changing the meaning of the test is much harder.
That's where I think the next generation of testing tools will either become genuinely useful or just generate impressive demos.
And then we deploy
There's another weird boundary in a lot of testing strategies.
Everything before deployment gets enormous attention.
Everything after deployment gets:
We'll check Sentry.
But a successful deployment pipeline doesn't mean the product actually works.
You can still have:
Bad environment variable Broken CDN asset Missing migration Expired production credential Misconfigured feature flag Third-party API failure DNS issue Regional outage CI may have been completely green.
Which raises a practical question:
What exactly should we run after deployment?
There isn't one answer.
Sometimes a fast API check is enough.
Sometimes you need a real browser.
Sometimes an important customer journey should be monitored continuously.
This decision guide for API smoke checks, full browser flows, and synthetic monitoring gives a useful framework for deciding where each belongs.
I'd think about them as different layers.
API Smoke Check "Is the service basically alive?" Browser Flow "Can a user actually complete this?" Synthetic Monitoring "Does it keep working after we leave?" Those are related questions.
They're not interchangeable.
Your production test strategy probably deserves more attention than your CI strategy
Suppose your checkout service deploys successfully.
CI is green.
Your API smoke check passes.
But the production cookie-consent configuration changed.
Now the checkout button is covered for European users.
The backend is healthy.
The API is healthy.
The deployment is healthy.
Checkout is broken.
This is why I increasingly think the traditional line between:
Testing and:
Monitoring is becoming artificial.
If an automated browser checks checkout before deployment, we call it a test.
If the same browser checks checkout every five minutes after deployment, we call it synthetic monitoring.
The user doesn't care what we call it.
They care whether checkout works.
Try auditing your suite for assumptions instead of tests
Here's an exercise that may be more useful than counting test cases.
Take ten important automated tests.
For each one, write down its assumptions.
Something like:
User has valid session Feature X enabled Feature Y disabled Network stable No third-party overlay API responds within 5 seconds User role unchanged No background token refresh Desktop viewport English locale Now ask:
Which of these assumptions are actually guaranteed?
You might find that your tests aren't really validating a workflow.
They're validating a workflow under one carefully curated version of reality.
That's still useful.
But it's important to know what you're getting.
I wouldn't try to test every combination
There's an obvious objection.
If we start considering:
sessions flags roles locales networks devices browser state accessibility state deployment state the number of combinations explodes.
Absolutely.
You shouldn't test all of them.
The solution isn't combinatorial madness.
It's risk-based selection.
Ask:
Which state changes have broken before? Which ones affect money? Which ones affect authentication? Which ones affect permissions? Which ones are difficult to recover from? Which ones differ between staging and production? Which ones would be embarrassing if a customer found them first? Test those.
The uncomfortable question
So here's the question I'd ask your team:
If our application behaved exactly like a normal production application instead of a freshly reset test environment, how much of our automation would still work?
Expired sessions.
Changing flags.
Half-finished interactions.
Browser history.
Modal focus.
Background refreshes.
Post-deployment configuration.
Real user state.
If the answer is "not much," that's worth knowing.
Because maybe your biggest automation gap isn't another missing test case.
Maybe your suite is simply making assumptions that production never agreed to.
