We recently wrote about why Vargus should be deterministic enough, not perfectly deterministic. The next lesson is more specific: if an AI testing agent keeps hallucinating failures, the fix may not be a better repair step.
The better fix is often grounding.
That sounds abstract, but the product lesson is concrete. A testing agent first has to discover what kind of project it is looking at, what routes or commands exist, where the test surface lives, and which paths are unsupported. If that discovery stays separate from generation, the agent can still write plausible tests against imaginary behavior.
Then the system spends the rest of the run cleaning up avoidable mistakes.
Repair Comes Too Late
Post-generation repair is tempting because it feels like a safety net. Let the agent generate tests, run them, inspect the failures, and ask another step to fix what broke.
That loop is useful, but it should not be the first line of defense.
If the generated test targets a route that does not exist, or assumes an application path the project never exposes, the problem is not only bad syntax. The model generated without enough project truth in front of it. A repair loop can patch around that one failure, but it does not address why the false positive appeared.
Grounding changes the shape of the task. Discovery output becomes input to generation. The agent is not merely told to test the app. It is constrained by the project facts it already found.
Discovery Should Become A Contract
The useful move is to treat discovery as a contract, not a preamble.
If discovery finds routes, commands, entry points, supported frameworks, unsupported project types, or known setup requirements, those facts should flow into the generation step as hard context. The generator should write tests against that discovered surface. It should also avoid surfaces that discovery did not confirm.
This is not just prompt engineering. It is product design.
An AI testing agent has several jobs inside it: inspect, classify, generate, execute, repair, and summarize. If those jobs pass around loose prose, each step has room to drift. If they pass around structured facts, the next step can be narrower and more accountable.
The system gets easier to debug too. When a generated test fails, the team can ask a sharper question: did discovery miss the surface, did generation ignore the surface, or did execution fail for an environmental reason?
False Positives Are Expensive
False positives are especially costly in an AI testing product because they look like progress.
A generated test file can be syntactically convincing while testing behavior the project does not actually support. A failure report can look serious while only proving that the agent invented the wrong path. The user then has to spend attention separating real product risk from agent noise.
That is the opposite of the job.
The agent should reduce review burden. If it creates plausible failures from ungrounded assumptions, it has moved work from the model to the human. That is worse than failing cleanly.
Grounding is one way to keep the burden in the system. The route map, command map, project type, and setup facts are not decorative metadata. They are the rails that keep generated tests attached to the real codebase.
Refusal Is Part Of Grounding
This also connects to unsupported cases.
The mature behavior is not to generate something for every repository. A CLI, API, library, and web application need different discovery paths and different test harnesses. Some are easier to support than others. Some should be explicit no-go cases until the system has the right tools.
That refusal is not a product weakness. It is part of trust.
A grounded system can say: this project type is supported, these surfaces were discovered, these tests were generated against those surfaces, and these limitations remain. An ungrounded system can only produce output and hope the user sorts out the rest.
The Instrument Changes
Once grounding is in place, the measurement problem changes.
It is still useful to track pass rates, semantic similarity, model-role combinations, and execution cost. But the most important signal becomes more diagnostic: is the agent testing the actual surface it discovered?
That is a different question from "did the generated tests pass?" A test can pass while missing the real product risk. A test can fail because the environment is wrong. A test can fail because the agent invented a route. Those cases need different fixes.
Grounding gives the team a way to separate them.
Better Agents Carry Their Own Evidence
The broader lesson is that useful AI agents should carry evidence from one step into the next.
Do not ask a model to rediscover the world every time it acts. Let it inspect the project once, turn that inspection into structured facts, and make later steps operate against those facts. The goal is not to remove judgment from the model. The goal is to remove avoidable guessing.
For Vargus, that means test generation should start from what the system has actually found. The agent should know the project type, confirmed surfaces, unsupported paths, and execution context before it writes a test.
That is the difference between an impressive demo and a tool people can trust with a codebase.
Ground the agent before it tests.
