AI Can Write Tests. But Are We Testing the Right Things?

AI is changing how we approach software testing. But generating more tests doesn't necessarily mean delivering better software.

AI Is Making Test Automation Faster

Over the past year, I've been exploring how AI can improve our QA processes, especially when working with Playwright and automated regression testing.

And one thing is clear: AI can be incredibly helpful.

It can generate test scenarios, write automation scripts, suggest edge cases, and help investigate failing tests.

Tasks that previously took hours can sometimes be completed much faster.

But I've started asking myself an important question:

Are we actually improving software quality, or are we just generating more tests?

More Tests Don't Always Mean Better Quality

Imagine asking AI to generate tests for a new feature.

It produces 20 automated tests. They all pass. CI is green.

Looks great, right?

But what if the requirements were incomplete?

What if an important business rule was never discussed?

Or what if the AI generated tests based on how the implementation works, rather than how it should work?

You can end up with a perfectly green test suite that validates the wrong behavior.

And that's where I believe QA becomes even more important.

QA Needs to Be Involved Earlier

Traditionally, QA is often involved when a feature is ready for testing.

But with AI accelerating development, waiting until the end becomes even more risky.

If developers can generate working implementations faster, unclear requirements can become working code just as quickly.

That's why I believe QA should be involved before development begins.

We need to ask questions like:

• What is the expected behavior?
• What could go wrong?
• Which edge cases are we missing?
• What are the risks?
• How do we know the feature is actually correct?

Once we understand these things, AI becomes much more powerful.

Instead of asking AI to guess what needs testing, we can give it clear expectations and let it help us turn them into reliable automated tests.

AI Should Support QA, Not Replace Judgment

I'm also interested in concepts such as LLM-as-a-Judge and self-healing tests.

Both have exciting potential, but they introduce new questions.

Should an AI automatically fix a failing test?

How do we know it isn't hiding a real regression?

And who verifies that an AI-generated test actually checks the correct behavior?

I believe the answer is a combination of AI assistance, deterministic checks, and human judgment.

AI can help us work faster, but we still need to understand what we're validating.

Final Thoughts

I don't think the future of QA is about writing more automated tests.

It's about finding better ways to prevent defects, identify risks earlier, and make sure we're building the right things.

AI can help us test faster. QA must help us test smarter.

I'm curious how other teams are approaching this.

Are you using AI mainly to generate more tests, or are you also changing how QA is involved in the development process?