One Vargus experiment made the model-routing problem hard to ignore.
We ran two hosted open models against the same small CLI codebase. GLM-5.2 found 21 areas to test, generated 21 suites containing 348 tests, and finished with a 96.6% execution success rate. The run cost $8.25.
Gemma 26B cost $0.47. It was roughly 17 times cheaper, but it found seven areas, generated 83 tests, and finished at 81.8%. Seventeen tests timed out.
The cheaper model did not simply do the same work for less money. It did less work. That distinction matters when a product reports cost, coverage, or success as if every run were comparable.
The Cheap Run Was Still Useful
It would be easy to read that result as a model leaderboard. We do not think it is one.
This was one project, not a benchmark. Gemma still produced real tests against the actual CLI instead of inventing a different application. For a model that cost less than fifty cents, that was useful. The weakness was concentrated in the parts of the run that required breadth and judgment: finding enough interesting areas, turning them into a full plan, and carrying that plan through execution.
That fits what we had already learned while making Vargus deterministic enough. Different stages fail differently. It also reinforces why we ground the agent before it tests. If discovery is weak, later stages operate against a smaller understanding of the project, however competently they execute it.
The practical question became whether the cheaper model needed to own the whole run.
A Hybrid Run Changed The Question
The next experiment used Claude for discovery, GLM for generation, and Gemma for execution. Vargus already had separate stages, so the work was mainly in making model and provider configuration travel with each stage instead of sitting above the pipeline as one global choice.
The hybrid run completed successfully. Its results were close enough to the heavier runs to justify continuing, while using expensive model capacity more selectively. We had not finished a full metrics analysis, so we could not claim that this was the optimal combination. It did show that hybrid configuration was a real product path rather than a theoretical cost-saving idea.
Discovery benefited most clearly from the stronger model. Generation could use a capable hosted open model once it received grounded project facts. Execution was narrower again. Reporting and cost reconciliation did not need model judgment everywhere.
That is a more useful shape than asking which single model should power Vargus.
A Model Name Is Not A Runtime
The experiments also exposed a complication: selecting a model is not enough.
The same Gemma model produced similar results through two providers, but one route was about 2.4 times faster. Provider defaults for reasoning and temperature were not always the same, and some parameters were translated differently or ignored. A run could therefore change after switching provider even when the model name stayed constant.
We saw a related effect with a local model that behaved better through its native local runtime than through a hosted compatibility layer. Tool calling, context handling, and protocol translation were part of the model's effective capability.
Vargus needs to record that full execution context. Model, provider, parameters, protocol, and stage all belong in the run history. Otherwise a comparison can look controlled while several important variables changed underneath it.
Routing Has To Be A Product Feature
Step-level configuration is only useful if a user can understand it without editing files between stages.
The current work is making backends discoverable during setup, validating endpoints and models before a run, carrying configuration through the pipeline, and showing where cost figures came from. A hybrid run may mix subscription quota, direct provider charges, local compute, and estimates. One total can hide more than it explains.
The interface should therefore answer a few concrete questions: which model handled each stage, why the system used it, what that stage produced, what it cost, and whether the number was billed or estimated. It should also be possible to repeat the run with the same configuration.
Giving users a choice matters. Some teams will want to use their own provider, run a local model, or optimize for a different balance of cost, speed, and coverage.
More important is that Vargus selects the right defaults. A user should not need to repeat our model and provider experiments before they can test their own code. We should turn that evaluation work into a strong default route for each stage, keep it current as models change, and leave the controls available when a team has a reason to override it.
We are still testing combinations across more project types, and the best policy will change as models and providers change. The durable part is not today's winning model. It is the separation of the work.
Once discovery, generation, execution, repair, and reporting can be measured independently, Vargus can choose better defaults, spend more where judgment changes the outcome, and spend less where the task is already narrow. Users keep meaningful choice without carrying the evaluation burden themselves. That gives us a system we can improve one decision at a time.
