Valyu’s 0.6B Multi-Agent Router: Results, Limits and When to Train One

A small router can outperform a larger prompted model when it has learned which retrieval agents actually return useful evidence. In Valyu's multi-agent routing study, a post-trained Qwen3-0.6B achieved a reported mean NDCG@10 of 0.771, versus 0.594 for Amazon Nova Lite and 0.552 for Claude Haiku 4.5. Mean selection latency was 120.1 ms. Those are results for a particular routing pipeline, not evidence that a 0.6B model is generally better than Haiku. [1]

The practical question is not whether to replace every large model with a small one. It is whether your retrieval system has enough repeated routing failures, reliable evaluation data and request volume to justify training a specialist. This review explains the result, an easily missed limitation in the metric, and the evidence to collect before paying for an implementation.

Research reviewed on 13 September 2026. This is a source-based assessment, not a Wavect reproduction of the experiment. The paper appears in the AgentSearch workshop programme at SIGIR 2026, not as a claim of main-conference acceptance. [2]

What did the Valyu SLM router actually beat?

The authors compare a router trained through supervised fine-tuning (SFT) and reinforcement learning (RL) with two prompted baselines. All choose from a fixed pool of 11 domain agents. The small model also produces keywords and date constraints for the downstream search call. [1]

The absolute mean-score gains are 0.177 over Nova Lite and 0.219 over Haiku. NDCG is not a percentage of correct answers. The similar medians and different means suggest that difficult or failed queries deserve more attention than the headline average. Inspect the distribution, not just its best-looking row. [1]

On a deliberately selected agent-query mismatch subset, the reported scores are 0.918, 0.539 and 0.490 respectively. That is useful evidence about a failure category, but it is not the overall benchmark. The paper illustrates a protein-structure query that prompted baselines send to genomics, while the trained router selects academic retrieval. The issue is where the relevant material resides, not merely which topic the words resemble. [1]

Retrieval-agent routing is not LLM model routing

This decision sits inside a retrieval-augmented generation (RAG) pipeline: selecting evidence is a separate step from generating the final answer.

A model router chooses which model should answer or perform a step. A retrieval-agent router chooses which search specialist or data source should supply evidence. A reranker orders documents after retrieval. These are different decisions, even when one product contains all three.

Valyu's system predicts an agent-selection bitmask, keywords and a temporal interval. It can select more than one agent. It is therefore doing more than topic classification, but less than coordinating an arbitrary team of autonomous workers. Improvements can come from source choice, parameter generation or their interaction. [1]

For broader architecture, use our multi-model agent stack guide. For vendor gateways, see the LLM gateway and router comparison. This article addresses the narrower decision: training a selector for retrieval outcomes across a known set of sources.

How SFT and retrieval-feedback training work together

SFT establishes the output contract. The paper describes 56,000 queries for the SFT dataset, partitioned 70% for training, 15% for validation and 15% for testing. The targets include source selection, keywords and dates. Training uses LoRA with rank 16 and alpha 32, bf16 precision, two epochs and a maximum sequence length of 512. Treat these as the authors' configuration, not universal deployment settings. [1]

RL evaluates what the chosen route retrieves. Starting from that checkpoint, the authors use REINFORCE++ with 11,000 production-log queries. The reward combines routing alignment with retrieved-content relevance and source credibility. It is not a pure NDCG reward: intent, keyword and date quality remain part of the objective. A KL penalty discourages excessive drift from the SFT checkpoint. [1]

Do not add the two dataset counts and assume 67,000 unique examples. The counts describe separate stages, not a documented non-overlapping corpus. For your own system, keep query families and time periods separate between training and final evaluation.

The feedback happens during training. The proposed runtime router predicts where to send a new query using what it learned. It does not first inspect every agent's live results for that request. Detecting a newly empty index, changed source or degraded connector still needs runtime health checks and fallback logic.

The important general lesson is outcome-informed supervision, not that SFT can never learn from outcomes. An SFT dataset could itself contain labels derived from retrieval evaluations. This paper compares its progressive training pipeline with intent-prompted models; it does not eliminate every alternative way to teach retrieval quality.

The public Qwen3-0.6B model card [3] describes the base model, not Valyu's trained router. Loading those base weights does not reproduce the scores. The reviewed paper does not provide a trained-router download link, so confirm checkpoint, data and training-code availability before planning a drop-in installation.

The NDCG@10 detail that changes how to read the result

NDCG rewards placing more relevant results near the top. Here, Claude Sonnet 4.6 judges pooled, shuffled results without seeing which router produced them. Results receive grades from 0 to 3, then return to their original search order. The authors do not add a reranking stage. [1]

The crucial detail is the denominator. Section 4.5.1 calculates the ideal ranking by sorting each system's own returned relevance scores, rather than using one shared ideal result set across systems. It assigns zero when every result is irrelevant and does not pad short lists to ten. [1]

This limits what the score alone can establish. As a mathematical illustration, ten results all graded 1 and ten results all graded 3 each obtain NDCG@10 = 1 when normalized against their own sorted list. Their usefulness is plainly different. A short, well-ordered list can also score highly without demonstrating comprehensive coverage.

This does not erase the reported advantage. It means 0.771 is not 77.1% answer accuracy, recall or completeness, and it should not be treated as a conventional common-pool retrieval-quality comparison without qualification. Before adopting the design, measure shared-pool NDCG, relevant-result coverage, all-zero-result rate and final answer correctness alongside the paper's metric.

A practical evaluation keeps a human-reviewed set of relevant documents or passages for each query. Every router is scored against the same target set, source snapshot and access permissions. Retain the original ranking scores too, so an improvement cannot hide behind a changed denominator.

What the latency numbers do and do not promise

The 120.1 ms mean and 179.4 ms P99 describe the authors' measured selector. They do not include the complete cost of retrieval, reranking and generating a grounded answer. Nor does an observed P99 become a production SLA. Queueing, warm-up, batching, token limits and traffic shape can change the result. [1]

There is also a hardware detail to clarify before reproducing the setup. The paper describes vLLM serving on an NVIDIA L4 with 48 GB RAM, while NVIDIA lists 24 GB of GPU memory for one L4. The 48 GB could refer to host RAM; the wording does not establish that. Do not turn it into a 48 GB VRAM requirement or a purchasing recommendation. [1] [4]

Benchmark router-only and end-to-end latency separately, on the infrastructure you will operate. Include cold starts, representative concurrency, timeouts and fallback requests. A faster selector is valuable, but a slower search backend can still dominate the user experience.

Four limits to resolve before commissioning a trained router

• Training is not matched across baselines. A domain-trained SLM is compared with prompted larger models, not identically trained models of different sizes. The study supports this specialized system, not a universal rule that model size is irrelevant.
• The incremental RL gain is not isolated by the reported retrieval table. The SFT-only results cover source matching, keywords, dates and latency, but the main comparison does not report an SFT-only NDCG@10. Request that ablation before attributing the entire improvement to RL.
• The source pool is fixed. Eleven known agents are different from a changing marketplace. Newly registered sources, index updates and new user languages need evaluation; capability-conditioned routing for new agents is described as future work.
• Judges and test composition matter. Blind shuffled judging is useful, but it is still model-based assessment. Ask for final evaluation and mismatch-subset sizes, duplicate controls, confidence intervals and independent human relevance checks. The SFT split alone does not answer these questions.

These boundaries are consistent with the paper's stated need for learned-selection baselines, other agent configurations and complementary human judgments. They are reasons to run a controlled comparison, not reasons to dismiss the research. [1]

Should you use rules, a learned selector or SFT plus RL?

Start with the cheapest method that can meet an explicit acceptance test. LTRR [5] learns to rank retrievers by downstream utility, while RAGRoute [6] studies lightweight source selection for federated RAG. They are relevant comparison ideas, not interchangeable implementations of Valyu's router.

Measure economics as monthly benefit = eligible requests × net variable saving per request − additional fixed monthly costs. Count source calls, model inference and retries on both paths. Fixed costs include training amortization, labels, evaluation and operations. A small weight file is not a business case. Our cost-per-task guide covers the broader budgeting question.

A production design that keeps routing separate from permission

The following is Wavect's proposed implementation pattern, not a claim that the paper implements these controls.

1. Filter the eligible source catalogue first. Resolve tenant, identity, document permissions, permitted regions and source availability outside the model. The router chooses only from what that caller may access.
2. Validate the complete routing output. Accept only known route IDs, permitted multi-source combinations and schema-valid keywords and dates. Bound fan-out, request size and execution time. Invalid output is not permission to query everything.
3. Execute through permission-aware adapters. Preserve caller identity at retrieval time. A topically relevant source is not automatically an authorized source.
4. Use a bounded fallback. Empty results, timeouts, unknown domains and uncertain selections can trigger a separately evaluated route that obeys the same permissions and budget. Avoid recursive retries.
5. Version and observe the whole decision. Record router version, eligible-agent catalogue version, source/index version, selected routes, normalized parameters, latency and outcome. Evaluate drift and roll back the policy when coverage deteriorates.

Store only the information required for evaluation, with retention and access controls. Queries and returned snippets can contain customer data. Cheap local inference does not make training logs safe by default. For the surrounding system, use the RAG production-readiness checklist and MCP authorization architecture.

A 30-day pilot for retrieval-agent routing

Week 1: establish the actual failure. Collect permission-safe queries, source versions and outcomes. Label which failures come from selection versus chunking, indexing, stale documents or generation. Training a router will not repair a broken corpus.

Week 2: compare inexpensive baselines. Evaluate rules, the current prompted router and an outcome-labelled small selector on the same held-out queries. Keep a distinct set of ambiguous, empty-result and multi-source cases. Test each production language rather than assuming multilingual base-model capability transfers to the trained router.

Week 3: test the extra value of RL. Add a retrieval-feedback policy only when the earlier baseline leaves material value on the table. Compare SFT-only with SFT+RL, hold the judge protocol constant and obtain human review on a blinded sample. Run a shadow deployment that does not change user-visible answers.

Week 4: canary or stop. Pre-agree quality, p95/p99 latency, cost-per-accepted-answer and authorization criteria. Roll out to a limited query segment only if it passes; keep a rollback path. A pilot that shows ordinary hybrid search is sufficient is a useful result.

Need to decide whether this belongs in your product? Wavect's AI consulting can help scope the evaluation, integration and rollout. The Twinsoft AI case study provides related delivery context, not evidence that Wavect has reproduced Valyu's router. Use the custom software versus off-the-shelf guide to compare owning the routing layer with buying it.

Discuss a retrieval-routing assessment with your current sources, recurring failure examples, traffic and latency budget. The first deliverable should be an evidence-backed build decision, not a promise to beat a public benchmark.

Sources and methodology

1. Valyu paper: methods, benchmark tables and evaluation protocol
2. AgentSearch at SIGIR 2026: workshop programme
3. Qwen3-0.6B: public base-model card
4. NVIDIA L4: official GPU memory specification
5. LTRR: Learning to Rank Retrievers for LLMs
6. RAGRoute: lightweight source selection for federated RAG

Header: illustration supplied with the brief, based on Figure 1 by Kondapalli et al. The original paper is available under CC BY 4.0. The header image is reproduced unchanged, not presented as a Wavect implementation. [1]