--- title: "Valyu SLM Router: What the 0.6B Retrieval Results Mean" canonical: https://wavect.io/blog/valyu-slm-multi-agent-router/ language: en description: "Review Valyu’s 0.6B retrieval router: SFT and RL, NDCG@10 caveats, latency, source selection and a practical build-or-buy evaluation for your team." image: "https://wavect.io/img/blog/headers/header_valyu-slm-multi-agent-router.png" --- [**Back**](/blog/overview/) [![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/) [Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt 14 min read · 13 Sep 2026 Last reviewed September 13, 2026 [**Next**](/blog/rag-production-readiness-checklist-eu/) # Valyu’s 0.6B Multi-Agent Router: Results, Limits and When to Train One TL;DR Valyu trained Qwen3-0.6B to select retrieval agents and generate keywords and date constraints using SFT followed by REINFORCE++. The paper reports mean NDCG@10 of 0.771 against 0.594 for prompted Nova Lite and 0.552 for prompted Claude Haiku 4.5, with 120.1 ms mean selection latency. This is not a general model ranking or end-to-end answer benchmark. Its NDCG normalizes each system’s own returned results, and the main retrieval table does not isolate the incremental RL gain with an SFT-only score. Evaluate shared-pool relevance, coverage, final answers, permissions and total operating cost before training a specialist router. **A small router can outperform a larger prompted model when it has learned which retrieval agents actually return useful evidence.** In Valyu's multi-agent routing study, a post-trained Qwen3-0.6B achieved a reported mean NDCG@10 of 0.771, versus 0.594 for Amazon Nova Lite and 0.552 for Claude Haiku 4.5. Mean selection latency was 120.1 ms. Those are results for a particular routing pipeline, not evidence that a 0.6B model is generally better than Haiku. [[1]](#source-paper) The practical question is not whether to replace every large model with a small one. It is whether your retrieval system has enough repeated routing failures, reliable evaluation data and request volume to justify training a specialist. This review explains the result, an easily missed limitation in the metric, and the evidence to collect before paying for an implementation. Research reviewed on 13 September 2026. This is a source-based assessment, not a Wavect reproduction of the experiment. The paper appears in the AgentSearch workshop programme at SIGIR 2026, not as a claim of main-conference acceptance. [[2]](#source-workshop) ## What did the Valyu SLM router actually beat? The authors compare a router trained through supervised fine-tuning (SFT) and reinforcement learning (RL) with two prompted baselines. All choose from a fixed pool of 11 domain agents. The small model also produces keywords and date constraints for the downstream search call. [[1]](#source-paper) | Router | Mean NDCG@10 | Median NDCG@10 | Mean latency | P99 latency | | --- | --- | --- | --- | --- | | Qwen3-0.6B, SFT + REINFORCE++ | 0.771 | 0.960 | 120.1 ms | 179.4 ms | | Amazon Nova Lite, prompted | 0.594 | 0.926 | 683.6 ms | 4,659.2 ms | | Claude Haiku 4.5, prompted | 0.552 | 0.929 | 2,457 ms | 6,985 ms | The absolute mean-score gains are **0.177 over Nova Lite and 0.219 over Haiku**. NDCG is not a percentage of correct answers. The similar medians and different means suggest that difficult or failed queries deserve more attention than the headline average. Inspect the distribution, not just its best-looking row. [[1]](#source-paper) On a deliberately selected agent-query mismatch subset, the reported scores are 0.918, 0.539 and 0.490 respectively. That is useful evidence about a failure category, but it is not the overall benchmark. The paper illustrates a protein-structure query that prompted baselines send to genomics, while the trained router selects academic retrieval. The issue is where the relevant material resides, not merely which topic the words resemble. [[1]](#source-paper) ## Retrieval-agent routing is not LLM model routing This decision sits inside a [retrieval-augmented generation (RAG) pipeline](/glossary/rag/): selecting evidence is a separate step from generating the final answer. A model router chooses which model should answer or perform a step. A retrieval-agent router chooses which search specialist or data source should supply evidence. A reranker orders documents after retrieval. These are different decisions, even when one product contains all three. Valyu's system predicts an agent-selection bitmask, keywords and a temporal interval. It can select more than one agent. It is therefore doing more than topic classification, but less than coordinating an arbitrary team of autonomous workers. Improvements can come from source choice, parameter generation or their interaction. [[1]](#source-paper) For broader architecture, use our [multi-model agent stack guide](/blog/multi-model-ai-coding-agent-stack-2026/). For vendor gateways, see the [LLM gateway and router comparison](/blog/llm-gateway-router-comparison-2026/). This article addresses the narrower decision: training a selector for retrieval outcomes across a known set of sources. ## How SFT and retrieval-feedback training work together **SFT establishes the output contract.** The paper describes 56,000 queries for the SFT dataset, partitioned 70% for training, 15% for validation and 15% for testing. The targets include source selection, keywords and dates. Training uses LoRA with rank 16 and alpha 32, bf16 precision, two epochs and a maximum sequence length of 512. Treat these as the authors' configuration, not universal deployment settings. [[1]](#source-paper) **RL evaluates what the chosen route retrieves.** Starting from that checkpoint, the authors use REINFORCE++ with 11,000 production-log queries. The reward combines routing alignment with retrieved-content relevance and source credibility. It is not a pure NDCG reward: intent, keyword and date quality remain part of the objective. A KL penalty discourages excessive drift from the SFT checkpoint. [[1]](#source-paper) Do not add the two dataset counts and assume 67,000 unique examples. The counts describe separate stages, not a documented non-overlapping corpus. For your own system, keep query families and time periods separate between training and final evaluation. **The feedback happens during training.** The proposed runtime router predicts where to send a new query using what it learned. It does not first inspect every agent's live results for that request. Detecting a newly empty index, changed source or degraded connector still needs runtime health checks and fallback logic. The important general lesson is outcome-informed supervision, not that SFT can never learn from outcomes. An SFT dataset could itself contain labels derived from retrieval evaluations. This paper compares its progressive training pipeline with intent-prompted models; it does not eliminate every alternative way to teach retrieval quality. The public [Qwen3-0.6B model card [3]](#source-qwen) describes the base model, not Valyu's trained router. Loading those base weights does not reproduce the scores. The reviewed paper does not provide a trained-router download link, so confirm checkpoint, data and training-code availability before planning a drop-in installation. ## The NDCG@10 detail that changes how to read the result NDCG rewards placing more relevant results near the top. Here, Claude Sonnet 4.6 judges pooled, shuffled results without seeing which router produced them. Results receive grades from 0 to 3, then return to their original search order. The authors do not add a reranking stage. [[1]](#source-paper) The crucial detail is the denominator. Section 4.5.1 calculates the ideal ranking by sorting **each system's own returned relevance scores**, rather than using one shared ideal result set across systems. It assigns zero when every result is irrelevant and does not pad short lists to ten. [[1]](#source-paper) This limits what the score alone can establish. As a mathematical illustration, ten results all graded 1 and ten results all graded 3 each obtain NDCG@10 = 1 when normalized against their own sorted list. Their usefulness is plainly different. A short, well-ordered list can also score highly without demonstrating comprehensive coverage. This does not erase the reported advantage. It means **0.771 is not 77.1% answer accuracy, recall or completeness**, and it should not be treated as a conventional common-pool retrieval-quality comparison without qualification. Before adopting the design, measure shared-pool NDCG, relevant-result coverage, all-zero-result rate and final answer correctness alongside the paper's metric. A practical evaluation keeps a human-reviewed set of relevant documents or passages for each query. Every router is scored against the same target set, source snapshot and access permissions. Retain the original ranking scores too, so an improvement cannot hide behind a changed denominator. ## What the latency numbers do and do not promise The 120.1 ms mean and 179.4 ms P99 describe the authors' measured selector. They do not include the complete cost of retrieval, reranking and generating a grounded answer. Nor does an observed P99 become a production SLA. Queueing, warm-up, batching, token limits and traffic shape can change the result. [[1]](#source-paper) There is also a hardware detail to clarify before reproducing the setup. The paper describes vLLM serving on an NVIDIA L4 with 48 GB RAM, while NVIDIA lists 24 GB of GPU memory for one L4. The 48 GB could refer to host RAM; the wording does not establish that. Do not turn it into a 48 GB VRAM requirement or a purchasing recommendation. [[1]](#source-paper) [[4]](#source-l4) Benchmark router-only and end-to-end latency separately, on the infrastructure you will operate. Include cold starts, representative concurrency, timeouts and fallback requests. A faster selector is valuable, but a slower search backend can still dominate the user experience. ## Four limits to resolve before commissioning a trained router - **Training is not matched across baselines.** A domain-trained SLM is compared with prompted larger models, not identically trained models of different sizes. The study supports this specialized system, not a universal rule that model size is irrelevant. - **The incremental RL gain is not isolated by the reported retrieval table.** The SFT-only results cover source matching, keywords, dates and latency, but the main comparison does not report an SFT-only NDCG@10. Request that ablation before attributing the entire improvement to RL. - **The source pool is fixed.** Eleven known agents are different from a changing marketplace. Newly registered sources, index updates and new user languages need evaluation; capability-conditioned routing for new agents is described as future work. - **Judges and test composition matter.** Blind shuffled judging is useful, but it is still model-based assessment. Ask for final evaluation and mismatch-subset sizes, duplicate controls, confidence intervals and independent human relevance checks. The SFT split alone does not answer these questions. These boundaries are consistent with the paper's stated need for learned-selection baselines, other agent configurations and complementary human judgments. They are reasons to run a controlled comparison, not reasons to dismiss the research. [[1]](#source-paper) ## Should you use rules, a learned selector or SFT plus RL? Start with the cheapest method that can meet an explicit acceptance test. [LTRR [5]](#source-ltrr) learns to rank retrievers by downstream utility, while [RAGRoute [6]](#source-ragroute) studies lightweight source selection for federated RAG. They are relevant comparison ideas, not interchangeable implementations of Valyu's router. | Your situation | First experiment | Why | | --- | --- | --- | | A few sources with clear eligibility rules | Deterministic routing plus hybrid retrieval | No training lifecycle unless the baseline fails. | | Repeated source-choice errors and trustworthy outcomes | Outcome-labelled classifier or SFT router | Tests whether learned source suitability adds value before RL. | | Routing and generated parameters interact in ways static labels miss | SFT + retrieval-feedback RL | Can optimize the consequences of the chosen route and inputs. | | Many new sources, little evaluation data or infrequent requests | Capability-based selection with a restricted fallback | A fixed trained route catalogue may be costly to maintain. | Measure economics as `monthly benefit = eligible requests × net variable saving per request − additional fixed monthly costs`. Count source calls, model inference and retries on both paths. Fixed costs include training amortization, labels, evaluation and operations. A small weight file is not a business case. Our [cost-per-task guide](/blog/cost-per-token-vs-cost-per-task/) covers the broader budgeting question. ## A production design that keeps routing separate from permission The following is Wavect's proposed implementation pattern, not a claim that the paper implements these controls. 1. **Filter the eligible source catalogue first.** Resolve tenant, identity, document permissions, permitted regions and source availability outside the model. The router chooses only from what that caller may access. 2. **Validate the complete routing output.** Accept only known route IDs, permitted multi-source combinations and schema-valid keywords and dates. Bound fan-out, request size and execution time. Invalid output is not permission to query everything. 3. **Execute through permission-aware adapters.** Preserve caller identity at retrieval time. A topically relevant source is not automatically an authorized source. 4. **Use a bounded fallback.** Empty results, timeouts, unknown domains and uncertain selections can trigger a separately evaluated route that obeys the same permissions and budget. Avoid recursive retries. 5. **Version and observe the whole decision.** Record router version, eligible-agent catalogue version, source/index version, selected routes, normalized parameters, latency and outcome. Evaluate drift and roll back the policy when coverage deteriorates. Store only the information required for evaluation, with retention and access controls. Queries and returned snippets can contain customer data. Cheap local inference does not make training logs safe by default. For the surrounding system, use the [RAG production-readiness checklist](/blog/rag-production-readiness-checklist-eu/) and [MCP authorization architecture](/blog/enterprise-mcp-authorization-architecture/). ## A 30-day pilot for retrieval-agent routing **Week 1: establish the actual failure.** Collect permission-safe queries, source versions and outcomes. Label which failures come from selection versus chunking, indexing, stale documents or generation. Training a router will not repair a broken corpus. **Week 2: compare inexpensive baselines.** Evaluate rules, the current prompted router and an outcome-labelled small selector on the same held-out queries. Keep a distinct set of ambiguous, empty-result and multi-source cases. Test each production language rather than assuming multilingual base-model capability transfers to the trained router. **Week 3: test the extra value of RL.** Add a retrieval-feedback policy only when the earlier baseline leaves material value on the table. Compare SFT-only with SFT+RL, hold the judge protocol constant and obtain human review on a blinded sample. Run a shadow deployment that does not change user-visible answers. **Week 4: canary or stop.** Pre-agree quality, p95/p99 latency, cost-per-accepted-answer and authorization criteria. Roll out to a limited query segment only if it passes; keep a rollback path. A pilot that shows ordinary hybrid search is sufficient is a useful result. Need to decide whether this belongs in your product? Wavect's [AI consulting](/services/artificial-intelligence/) can help scope the evaluation, integration and rollout. The [Twinsoft AI case study](/case-studies/twinsoft-ai/) provides related delivery context, not evidence that Wavect has reproduced Valyu's router. Use the [custom software versus off-the-shelf guide](/software-development-guide/custom-software-vs-off-the-shelf/) to compare owning the routing layer with buying it. [Discuss a retrieval-routing assessment](/contact/) with your current sources, recurring failure examples, traffic and latency budget. The first deliverable should be an evidence-backed build decision, not a promise to beat a public benchmark. ## Frequently asked questions about Valyu’s SLM router ### Did a 0.6B model really beat Claude Haiku 4.5? In this study’s particular retrieval-routing setup, the trained Qwen3-0.6B reports a higher mean NDCG@10 and lower mean selection latency than prompted Claude Haiku 4.5. That does not establish higher general intelligence, answer accuracy or performance on other tasks. ### What is a retrieval-agent router? It selects the search agents or data sources that should provide evidence for a query. Valyu’s router also generates keywords and date constraints. It is different from selecting a language model or reranking documents after retrieval. ### Does the router inspect every source before answering each request? The retrieval feedback is used during training. At runtime the router predicts a route for a new query. Live source failures still need health checks, evaluation and a bounded fallback. ### Does NDCG@10 of 0.771 mean 77.1% correct answers? No. It is a ranking score, not answer accuracy. The paper constructs each ideal ranking from that system’s own returned relevance scores, which limits conclusions about absolute relevance and coverage. Use shared-pool relevance and answer-level evaluation too. ### Can I download Qwen3-0.6B and reproduce the result? The public base model is not the trained router. The reviewed paper does not provide a trained-router download link. Confirm checkpoint, training data, code and evaluation access before promising a reproduction. ### Should my team add reinforcement learning immediately? Start with a measured baseline and outcome-labelled source selection. Compare SFT-only with SFT plus RL on the same holdout before paying for the extra training and operations. Keep source authorization outside the learned policy. ## Sources and methodology 1. [Valyu paper: methods, benchmark tables and evaluation protocol](https://arxiv.org/html/2608.00030v1) 2. [AgentSearch at SIGIR 2026: workshop programme](https://agent-search.github.io/agentsearch-sigir26/) 3. [Qwen3-0.6B: public base-model card](https://huggingface.co/Qwen/Qwen3-0.6B) 4. [NVIDIA L4: official GPU memory specification](https://www.nvidia.com/en-us/data-center/l4/) 5. [LTRR: Learning to Rank Retrievers for LLMs](https://arxiv.org/abs/2506.13743) 6. [RAGRoute: lightweight source selection for federated RAG](https://arxiv.org/abs/2502.19280) Header: illustration supplied with the brief, based on Figure 1 by Kondapalli et al. The original paper is available under CC BY 4.0. The header image is reproduced unchanged, not presented as a Wavect implementation. [[1]](#source-paper) Agent engineering ## Continue through this cluster Coding agents, MCP, context systems, evaluation and the controls required for dependable automation. [Start with the cornerstone**Graph Engineering for AI Agents: When Does a Knowledge Graph Pay Off?**](/blog/graph-engineering-ai-agents/) - [Spotify shunt Review: Setup, Savings and Limits](/blog/spotify-shunt-claude-code-token-routing/) - [OpenBot Review: Self-Hosted AI Coworkers, Costs & Controls](/blog/openbot-self-hosted-ai-coworkers-review/) - [Ramp Inspect Architecture 2026: Background Coding Agents at Scale](/blog/ramp-inspect-background-coding-agent-infrastructure-2026/) - [Model Hardware Standard: Enterprise Guide to Physical AI](/blog/model-hardware-standard-enterprise-guide/) - [Fonio AI Review 2026: Pricing, API, GDPR & Build vs Buy](/blog/fonio-ai-review-build-vs-buy-2026/) [**Back**](/blog/overview/) [![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/) [Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt 14 min read · 13 Sep 2026 Last reviewed September 13, 2026 [**Next**](/blog/rag-production-readiness-checklist-eu/) ## Structured Data ```json { "@context": "https://schema.org", "@graph": [ { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ], "employee": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "founder": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "legalRepresentative": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "name": "Wavect GmbH", "subjectOf": { "@id": "https://wavect.io/verified-claims.json#dataset", "@type": "Dataset", "creator": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "description": "A machine-readable registry of quantitative and qualitative claims published by Wavect, with review dates, localized page appearances and public third-party citations where available.", "inLanguage": "en", "isAccessibleForFree": true, "license": "https://creativecommons.org/licenses/by/4.0/", "name": "Wavect verified publication claims", "url": "https://wavect.io/verified-claims.json" }, "url": "https://wavect.io/" }, { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "sameAs": [ "https://www.wikidata.org/wiki/Q139796365", "https://www.linkedin.com/in/wsdt", "https://github.com/wsdt" ], "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "sameAs": [ "https://www.wikidata.org/wiki/Q139796367", "https://www.linkedin.com/in/jocr77/", "https://github.com/jo-chris" ], "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/#website", "@type": "WebSite", "inLanguage": [ "en", "de", "es", "zh" ], "name": "Wavect", "potentialAction": { "@type": "SearchAction", "query-input": "required name=search_term_string", "target": { "@type": "EntryPoint", "urlTemplate": "https://wavect.io/search/?q={search_term_string}" } }, "publisher": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "url": "https://wavect.io/" }, { "@id": "https://wavect.io/blog/valyu-slm-multi-agent-router/#webpage", "@type": "WebPage", "dateModified": "2026-09-13", "inLanguage": "en", "isPartOf": { "@id": "https://wavect.io/#website", "@type": "WebSite" }, "lastReviewed": "2026-09-13", "url": "https://wavect.io/blog/valyu-slm-multi-agent-router/" } ] } ``` ```json { "@context": "https://schema.org", "@type": "BlogPosting", "abstract": "Valyu trained Qwen3-0.6B to select retrieval agents and generate keywords and date constraints using SFT followed by REINFORCE++. The paper reports mean NDCG@10 of 0.771 against 0.594 for prompted Nova Lite and 0.552 for prompted Claude Haiku 4.5, with 120.1 ms mean selection latency. This is not a general model ranking or end-to-end answer benchmark. Its NDCG normalizes each system’s own returned results, and the main retrieval table does not isolate the incremental RL gain with an SFT-only score. Evaluate shared-pool relevance, coverage, final answers, permissions and total operating cost before training a specialist router.", "articleBody": " Blog overview/AI and agents/Agent engineering Valyu’s 0.6B Multi-Agent Router: Results, Limits and When to Train One TL;DR Valyu trained Qwen3-0.6B to select retrieval agents and generate keywords and date constraints using SFT followed by REINFORCE++. The paper reports mean NDCG@10 of 0.771 against 0.594 for prompted Nova Lite and 0.552 for prompted Claude Haiku 4.5, with 120.1 ms mean selection latency. This is not a general model ranking or end-to-end answer benchmark. Its NDCG normalizes each system’s own returned results, and the main retrieval table does not isolate the incremental RL gain with an SFT-only score. Evaluate shared-pool relevance, coverage, final answers, permissions and total operating cost before training a specialist router. A small router can outperform a larger prompted model when it has learned which retrieval agents actually return useful evidence. In Valyu's multi-agent routing study, a post-trained Qwen3-0.6B achieved a reported mean NDCG@10 of 0.771, versus 0.594 for Amazon Nova Lite and 0.552 for Claude Haiku 4.5. Mean selection latency was 120.1 ms. Those are results for a particular routing pipeline, not evidence that a 0.6B model is generally better than Haiku. [1] The practical question is not whether to replace every large model with a small one. It is whether your retrieval system has enough repeated routing failures, reliable evaluation data and request volume to justify training a specialist. This review explains the result, an easily missed limitation in the metric, and the evidence to collect before paying for an implementation. Research reviewed on 13 September 2026. This is a source-based assessment, not a Wavect reproduction of the experiment. The paper appears in the AgentSearch workshop programme at SIGIR 2026, not as a claim of main-conference acceptance. [2] What did the Valyu SLM router actually beat? The authors compare a router trained through supervised fine-tuning (SFT) and reinforcement learning (RL) with two prompted baselines. All choose from a fixed pool of 11 domain agents. The small model also produces keywords and date constraints for the downstream search call. [1] Authors' reported results, combining Tables 2 and 6. Latency is selection latency, not full retrieval and answer generation. RouterMean NDCG@10Median NDCG@10Mean latencyP99 latency Qwen3-0.6B, SFT + REINFORCE++0.7710.960120.1 ms179.4 ms Amazon Nova Lite, prompted0.5940.926683.6 ms4,659.2 ms Claude Haiku 4.5, prompted0.5520.9292,457 ms6,985 ms The absolute mean-score gains are 0.177 over Nova Lite and 0.219 over Haiku. NDCG is not a percentage of correct answers. The similar medians and different means suggest that difficult or failed queries deserve more attention than the headline average. Inspect the distribution, not just its best-looking row. [1] On a deliberately selected agent-query mismatch subset, the reported scores are 0.918, 0.539 and 0.490 respectively. That is useful evidence about a failure category, but it is not the overall benchmark. The paper illustrates a protein-structure query that prompted baselines send to genomics, while the trained router selects academic retrieval. The issue is where the relevant material resides, not merely which topic the words resemble. [1] Retrieval-agent routing is not LLM model routing This decision sits inside a retrieval-augmented generation (RAG) pipeline: selecting evidence is a separate step from generating the final answer. A model router chooses which model should answer or perform a step. A retrieval-agent router chooses which search specialist or data source should supply evidence. A reranker orders documents after retrieval. These are different decisions, even when one product contains all three. Valyu's system predicts an agent-selection bitmask, keywords and a temporal interval. It can select more than one agent. It is therefore doing more than topic classification, but less than coordinating an arbitrary team of autonomous workers. Improvements can come from source choice, parameter generation or their interaction. [1] For broader architecture, use our multi-model agent stack guide. For vendor gateways, see the LLM gateway and router comparison. This article addresses the narrower decision: training a selector for retrieval outcomes across a known set of sources. How SFT and retrieval-feedback training work together SFT establishes the output contract. The paper describes 56,000 queries for the SFT dataset, partitioned 70% for training, 15% for validation and 15% for testing. The targets include source selection, keywords and dates. Training uses LoRA with rank 16 and alpha 32, bf16 precision, two epochs and a maximum sequence length of 512. Treat these as the authors' configuration, not universal deployment settings. [1] RL evaluates what the chosen route retrieves. Starting from that checkpoint, the authors use REINFORCE++ with 11,000 production-log queries. The reward combines routing alignment with retrieved-content", "articleSection": "Engineering", "author": { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "name": "Kevin Riedl", "sameAs": [ "https://www.wikidata.org/wiki/Q139796365", "https://www.linkedin.com/in/wsdt", "https://github.com/wsdt" ], "url": "https://wavect.io/team/kevin-riedl/" }, "citation": [ { "@type": "WebPage", "name": "Valyu paper: methods, benchmark tables and evaluation protocol", "url": "https://arxiv.org/html/2608.00030v1" }, { "@type": "WebPage", "name": "AgentSearch at SIGIR 2026: workshop programme", "url": "https://agent-search.github.io/agentsearch-sigir26/" }, { "@type": "WebPage", "name": "Qwen3-0.6B: public base-model card", "url": "https://huggingface.co/Qwen/Qwen3-0.6B" }, { "@type": "WebPage", "name": "NVIDIA L4: official GPU memory specification", "url": "https://www.nvidia.com/en-us/data-center/l4/" }, { "@type": "WebPage", "name": "LTRR: Learning to Rank Retrievers for LLMs", "url": "https://arxiv.org/abs/2506.13743" }, { "@type": "WebPage", "name": "RAGRoute: lightweight source selection for federated RAG", "url": "https://arxiv.org/abs/2502.19280" } ], "dateModified": "2026-09-13", "datePublished": "2026-09-13", "description": "Valyu trained Qwen3-0.6B to select retrieval agents and generate keywords and date constraints using SFT followed by REINFORCE++. The paper reports mean NDCG@10 of 0.771 against 0.594 for prompted Nova Lite and 0.552 for prompted Claude Haiku 4.5, with 120.1 ms mean selection latency. This is not a general model ranking or end-to-end answer benchmark. Its NDCG normalizes each system’s own returned results, and the main retrieval table does not isolate the incremental RL gain with an SFT-only score. Evaluate shared-pool relevance, coverage, final answers, permissions and total operating cost before training a specialist router.", "headline": "Valyu’s 0.6B Multi-Agent Router: Results, Limits and When to Train One", "image": "https://wavect.io/img/blog/headers/header_valyu-slm-multi-agent-router.png", "inLanguage": "en", "keywords": "Retrieval routing, Small language models", "mainEntityOfPage": { "@id": "https://wavect.io/blog/valyu-slm-multi-agent-router/", "@type": "WebPage" }, "publisher": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "url": "https://wavect.io/blog/valyu-slm-multi-agent-router/", "wordCount": 2785 } ``` ```json { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "item": "https://wavect.io/", "name": "Home", "position": 1 }, { "@type": "ListItem", "item": "https://wavect.io/blog/overview/", "name": "Blog overview", "position": 2 }, { "@type": "ListItem", "item": "https://wavect.io/blog/topics/ai-agents/", "name": "AI and agents", "position": 3 }, { "@type": "ListItem", "item": "https://wavect.io/blog/clusters/agent-engineering/", "name": "Agent engineering", "position": 4 }, { "@type": "ListItem", "item": "https://wavect.io/blog/valyu-slm-multi-agent-router/", "name": "Valyu SLM Router: What the 0.6B Retrieval Results Mean", "position": 5 } ] } ``` ```json { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "In this study’s particular retrieval-routing setup, the trained Qwen3-0.6B reports a higher mean NDCG@10 and lower mean selection latency than prompted Claude Haiku 4.5. That does not establish higher general intelligence, answer accuracy or performance on other tasks." }, "name": "Did a 0.6B model really beat Claude Haiku 4.5?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "It selects the search agents or data sources that should provide evidence for a query. Valyu’s router also generates keywords and date constraints. It is different from selecting a language model or reranking documents after retrieval." }, "name": "What is a retrieval-agent router?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "The retrieval feedback is used during training. At runtime the router predicts a route for a new query. Live source failures still need health checks, evaluation and a bounded fallback." }, "name": "Does the router inspect every source before answering each request?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "No. It is a ranking score, not answer accuracy. The paper constructs each ideal ranking from that system’s own returned relevance scores, which limits conclusions about absolute relevance and coverage. Use shared-pool relevance and answer-level evaluation too." }, "name": "Does NDCG@10 of 0.771 mean 77.1% correct answers?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "The public base model is not the trained router. The reviewed paper does not provide a trained-router download link. Confirm checkpoint, training data, code and evaluation access before promising a reproduction." }, "name": "Can I download Qwen3-0.6B and reproduce the result?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "Start with a measured baseline and outcome-labelled source selection. Compare SFT-only with SFT plus RL on the same holdout before paying for the extra training and operations. Keep source authorization outside the learned policy." }, "name": "Should my team add reinforcement learning immediately?" } ] } ```