Perfectly repeatable AI test generation sounds like the goal. It is not.
The more useful target is knowing when variation matters. If two runs produce different test files but cover the same behavior, that may be fine. If two runs disagree about what kind of project they are testing, that is a product problem.
That distinction is where the engineering work starts.
That has been the useful lesson from recent Vargus work. A codebase can ask an agent to inspect a project, discover likely test surfaces, write tests, run them, and report what happened. The first version can look impressive quickly. Then the real work starts: failures caused by missing environment setup, hallucinated interfaces, authentication paths, unsupported project types, brittle assumptions, and model output that changes between runs.
At that point, the product question is not "how do we remove all variation?" It is "which variation is acceptable, and which variation means the system does not understand the job?"
Perfect Repeatability Is The Wrong First Goal
Perfect determinism sounds attractive because it gives product teams something stable to reason about. But AI test generation is not a compiler. The output is a proposal for how to exercise a codebase, and there can be more than one sensible proposal.
That does not mean instability is harmless. Some variation is useful. Some is noise. Some is evidence that the system is guessing.
If two generated suites test the same behavior through different assertions, the system may be working. If one run identifies a CLI and another run treats the same repository like a web application, the system is not deterministic enough for the product decision in front of it.
The goal is bounded variation, not fake certainty.
Measure The Variation
For Vargus, the work has shifted toward measurement before control because measurement is how the team decides which instability matters.
One axis is ordinary execution quality. Do the generated tests run? Do they pass? Did the agent choose a sensible test target? Did it miss obvious project setup? Those are concrete signals, and they expose different problems from a human reading the output.
Another axis is semantic similarity. If the same project is run through multiple generations, the exact files and assertions may differ. That does not automatically mean the system failed. Two test suites can be textually different while testing the same behavior. Two outputs can look similar while covering different risks.
So the better measurement is not just text comparison. It is whether the outputs are meaningfully testing the same surface. That is why a semantic comparison layer matters. It gives the team a way to ask whether repeated generations are converging on the same understanding of the codebase, even when the syntax differs.
The score is not the product. The score is an instrument. It tells the team whether the system is converging on the same understanding of the codebase, even when the syntax differs.
Unsupported Cases Should Be Explicit
Another lesson is less glamorous: the system needs to know when not to generate.
An AI testing agent can create a lot of noise if it treats every repository as equally supported. A CLI, a web API, a library, and a full web application do not have the same discovery path. They do not fail the same way. They do not need the same harness.
If the agent has weak support for a project type, the mature behavior is not to bluff. It should identify the unsupported path, explain the limitation, and avoid producing low-confidence output that looks useful but wastes review time.
That is product quality. Refusing badly is better than generating confidently.
Unsupported cases are also part of determinism. A system that consistently says "not supported yet" is easier to trust than a system that sometimes invents a fragile answer.
Model Roles Fail Differently
The other reason to measure first is that model choice is not a single decision.
A test-generation system has multiple jobs inside it: understand the project, decide what matters, write tests, repair failures, summarize results. One model may be better at discovery. Another may be better at dense generation. A third may be too verbose for the same task even if it is strong in other settings.
This is the same pattern we wrote about in Prompts Are Tightly Coupled To Models. Model behavior is part of the product surface. Changing the model changes the system.
The next useful experiment is not "which model is best?" It is more specific: which model should do discovery, which model should generate, and how do we know the combination improved the end result?
That matters because a system can be deterministic in one role and unstable in another. A model might reliably summarize failures but wander during project discovery. Another might identify the right surface but write brittle tests. Treating the whole pipeline as one model choice hides those differences.
Deterministic Enough For The Decision
Deterministic enough is not a universal threshold. It depends on the decision the product is asking the user to make.
If the user is deciding whether a generated suite is worth reviewing, semantic agreement may be enough. If the user is deciding whether to trust a coverage claim, the bar is higher. If the system cannot identify the project type consistently, it should not pretend the generated tests are reliable.
This is the shape of real AI engineering. The product improves when the team adds instruments, support boundaries, and model-role experiments, not just stronger instructions.
The useful goal is not perfect repeatability. The useful goal is a system where variation is visible, measured, and bounded tightly enough for the decision at hand.
