Series · Knowledge layer deep dives
SLMs for enterprise AI agents: routing, fine-tuning and limits
How to move an enterprise agent's routine model calls to small language models without losing accuracy: the published conversion method, routing and escalation, LoRA fine-tuning, the gates a model passes before it serves, hardware sizing with worked numbers, and where small models still fail.
Inovagio team · Published · Updated · 18 min read

On this page
Small language models (SLMs) for AI agents are compact models, usually under 10 billion parameters, that take over an enterprise agent's narrow, repetitive calls while a large language model (LLM) handles planning and the cases the small one can't confirm. The switch keeps data on your hardware, and cost per correct task shows whether it paid off. Done carelessly, though, it hides errors, and a cheap wrong answer that nobody checks is the most expensive kind.
What follows is the conversion method NVIDIA researchers published, plus the parts a production system adds: routing, evaluation gates and hardware sizing, worked through with real numbers. We also say where the evidence runs out. The business case, in plainer words, is in small language models for business.
Why AI agent calls suit small language models
#An agent's model calls aren't one workload. A single task can include a classification, three extractions into a fixed schema, a lookup, a comparison and a free-text note for a reviewer. Most of those have one right answer and a format you can check.
That's the argument of the paper at the center of the current debate, and its thesis fits in one line: small models are "sufficiently powerful, inherently more suitable, and necessarily more economical" for much of what agents do. Its authors define a small model as one that runs on a common consumer device fast enough to serve one user's agent requests, and they'd call most models under 10 billion parameters small as of 2025 (Belcak et al., "Small Language Models are the Future of Agentic AI", NVIDIA Research, arXiv 2506.02153, June 2025, revised September 2026). They cite work showing a 7-billion-parameter model is 10 to 30 times cheaper to serve than one of 70 to 175 billion, counting latency and energy as well as compute per answer.
The paper is a position paper, and it says so. Its appendix estimates how many calls a small model could take in three open agent frameworks (about 60%, 40% and 70%), but the authors got them by reading code, and nobody measured a run. Use them as a reason to count your own calls. They don't forecast your queue.
Gartner's view points the same way at the market level: it predicts that by 2027 organizations will use small, task-specific models at least three times more than general-purpose large models, by usage volume (Gartner, April 9, 2025). It also predicts that inference cost per agentic workflow will rise more than fivefold through 2028, and recommends tiering and routing so each task reaches a model whose cost fits it (Gartner, August 17, 2026).
The conversion method, step by step
#The NVIDIA paper lays out an LLM-to-SLM conversion in six steps. Here they are, each with what we'd add for regulated queues, where the training data is also evidence and the people who approve the work are named.
- Log the calls. Record every call that isn't a direct conversation with a person: the prompt, the response and the contents of each tool call, with latency as an option. The paper asks for encrypted pipelines with role-based access and anonymization before storage. We'd add the outcome: whether a person approved the result or the checks passed, because that label is what makes a log usable as training data. In a regulated business, the log is also evidence, so bind each entry to its tenant and keep it in the audit record.
- Curate and scrub. Remove personal and health data and anything else sensitive. The paper's rule of thumb is 10,000 to 100,000 examples to fine-tune a small model. Our addition: train only on results a person approved or the checks confirmed. A log of model outputs is not ground truth.
- Cluster the tasks. Group prompts and actions with unsupervised clustering to find the recurring task types. You'll usually find a long tail of rare calls and a short head of very common ones; the head is where conversion pays.
- Choose candidate models. The paper's criteria are instruction following, reasoning, context window, benchmarks relevant to the task, license and deployment footprint. Add provenance, for reasons covered below.
- Fine-tune. The paper names parameter-efficient fine-tuning such as LoRA or QLoRA, full fine-tuning if you have the budget, and distillation from a larger model.
- Iterate. Retrain the small models and the router as new approved data arrives.
What LoRA and QLoRA buy you
#LoRA (low-rank adaptation) freezes the base model and learns a low-rank update for each chosen weight matrix, stored as two thin matrices whose product is added to the frozen weights. In the original paper, against full fine-tuning of a 175-billion-parameter model, it cut the trainable parameters by a factor of 10,000 and the graphics processor (GPU) memory needed by a factor of three (Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models", arXiv, June 2021). QLoRA (quantized LoRA) goes further, training adapters on a base quantized to 4 bits with NormalFloat (NF4), a data type designed for normally distributed weights, and quantizing the quantization constants themselves (double quantization). That made it possible to fine-tune a 65-billion-parameter model on a single 48 GB GPU (Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs", arXiv, May 2023).
The practical effect for agents matters more than the training savings. A LoRA adapter is small, so one base model in memory can serve many task-specific adapters, chosen per request. Ten "task models" don't need ten copies of the weights. They need one. The catch is that an adapter belongs to one exact base: when the base model changes version, every adapter on it is retrained and re-scored, not copied across.
Distillation changes where the training targets come from. A larger teacher model either writes answers the small model is fine-tuned on (sequence-level distillation) or supplies its full output probabilities for the student to match, which needs access to the teacher's logits and a shared tokenizer (Hinton, Vinyals and Dean, "Distilling the Knowledge in a Neural Network", arXiv, March 2015; Kim and Rush, "Sequence-Level Knowledge Distillation", arXiv, June 2016). It's useful when you lack approved examples, with two cautions. The student inherits the teacher's mistakes, so the training set still needs a check that isn't the teacher. And read the teacher's license and terms of use first, because some forbid using a model's outputs to train another one.
Once a supervised model works, preference tuning is the next lever. Methods such as direct preference optimization (DPO) train directly on pairs of a preferred and a rejected answer, which is what a reviewer's correction gives you (Rafailov et al., "Direct Preference Optimization", arXiv, May 2023). Where the answer can be checked by code, reinforcement learning with verifiable rewards (RLVR) uses the same checks that gate a route as the reward.
Routing each call to the right model
#Routing is where most of the savings and most of the risk sit. There are three common designs, and production systems tend to combine them.
Static routing by task. Each task cluster from step 3 has a default model. A document classification goes to the small classifier; a reviewer's note goes to a larger model. This is predictable and easy to audit, and in a queue with well-defined steps it covers most traffic.
Learned routers. A router model predicts, per request, whether a weaker model will answer as well as a stronger one. RouteLLM's routers, trained on human preference data, more than halved cost in some cases without lowering response quality (Ong et al., "RouteLLM: Learning to Route LLMs with Preference Data", arXiv, June 2024).
Cascades. Try the cheap model first and move up only when the answer fails a check. In FrugalGPT the check is a small learned scorer of answer quality, and its cascade matched the best single model's performance with up to 98% cost reduction on its benchmarks (Chen, Zaharia and Zou, "FrugalGPT", Stanford, arXiv, May 2023).
We take a side on one design choice: escalate on external checks, not on the model's own confidence. A small model that's confidently wrong is the failure that costs most, and self-reported confidence is the signal least likely to catch it. External checks are concrete: the output validates against the step's JSON schema (the declared shape of a structured answer), the extracted totals reconcile with the purchase order, the cited clause exists in the document. Each one passes or it doesn't.
Here's the logic as a sketch (not product code). Each step kind has its own ladder, set by policy, so a hard step can start higher and a frontier rung appears only where policy allows one:
LADDERS = {
"classify_document": ["small-task-model", "larger-open-model"],
"extract_header": ["small-task-model", "larger-open-model"],
"split_freight": ["larger-open-model"],
}
def run_step(step, record):
ladder = LADDERS[step.kind] # an unrouted step kind raises; it never defaults to the cheapest model
failures = []
for model in ladder:
out = call(model, step, schema=step.schema) # decoding constrained to the step's schema
failures = check(out, step.schema, record) # schema, reconciliation, approved rules
log_attempt(step, model, failures) # every rung is recorded, including the ones that failed
if not failures:
return out
return to_person(step, reasons=failures) # never a silent fallbackTwo details make this safe. First, constrain the output: grammar- or schema-constrained decoding makes a malformed answer impossible rather than merely unlikely (a reply cut off at the token limit is the one exception, so check for it), and validation failures then mean wrong content, not broken JSON. Constraint fixes the shape, not the content. One study found strict output formats lowered scores on reasoning tasks (Tam et al., "Let Me Speak Freely?", arXiv, August 2024). Where a step has to reason before it answers, put the reasoning field ahead of the answer fields in the schema and test the step both ways. Second, the last rung is a person with the reason attached. A cascade that ends in "accept whatever the biggest model said" has quietly removed your control.
A worked example with one invoice and five calls
#Take the accounts payable queue at Tollbridge Fasteners (Tollbridge Fasteners · Example company). One vendor invoice produces these steps.
| Step | Route | Why |
|---|---|---|
| Classify the document: invoice, credit note or statement | Small task model | One label from a fixed list |
| Extract the header fields into a JSON schema | Small task model, schema-constrained | Fixed format, validated on output |
| Match the lines to the purchase order | An approved rule, no model | Arithmetic against an approved tolerance |
| Write the exception note for the reviewer | Larger open model | Free text that a person reads |
| Split a freight charge billed across two orders | Larger open model, then a person if the checks fail | Several steps with constraints to track |
Notice the third row. The cheapest model for line matching is no model at all. Deterministic checks against approved rules belong outside the model, where they're exact and can be replayed; our guide to how a rule is approved shows what such a rule contains. The fifth row is the kind of task the research says small models still struggle with, so it starts higher on the ladder.
Evaluation before a model serves
#A fine-tuned small model is a new model, and it earns its place on your work, not on a public leaderboard. The NVIDIA paper names reliance on generalist benchmarks as a barrier to adopting small models, because those benchmarks understate what a small model does on agent work. The reverse holds too: a model that scores well on a broad benchmark can still fail your vendor's invoice layout.
Five gates, in the order we run them:
- Task score on held-out approved results, scored by your own checks. Hold out examples by time as well as at random, so the test includes work the model never saw the shape of, and remove near-duplicates of test items from the training set first, or the score measures memory.
- Regression against the version in service. Same held-out set, both models. A new model that wins on average but loses on a category your auditors care about isn't an upgrade.
- Safety and red-team. Prompt injection in documents, requests outside the role, attempts to output data the step shouldn't see.
- Calibration. Where scores or confidences feed a route, check they mean what they say.
- A recorded deployment approval, by a named person, with the evidence attached.
Run evaluation in a deterministic serving mode, so a replay of the same input gives the same output. Otherwise a regression can hide in sampling noise, and nobody can reproduce what the auditor is asking about. If the step samples when it serves, score it at those settings as well, over several runs with the spread reported.
Then watch it in service. The measures that matter are first-pass success, the share of steps escalated, the minutes people spend on escalations, and cost per correct task, which counts all of it. The guide to measuring cost per correct task has the formula and a worksheet.
Where small models still fail
#The best recent evidence is AgentFloor, a deterministic benchmark of 30 tasks on a six-tier ladder from instruction following through tool use and multi-step coordination to long-horizon planning under persistent constraints (Karmakar and Chatterjee, "AgentFloor", arXiv 2605.00334, May 1, 2026).
Small and mid-size open models handled much of the short, structured tool use that dominates real pipelines, and the strongest open model matched the frontier model in aggregate. The tier results are worth reading closely. Equivalence was established statistically only for a single tool call. On two-tool chains, branching on an intermediate result and multi-source synthesis, the best open model came within about nine points of the frontier model without the test confirming a match. On the top tier, long-horizon planning under persistent constraints, the frontier model beat the strongest open model, which solved none of those tasks. But the frontier model itself solved only about one in ten, and the best open model on that tier, Mistral's 8-billion-parameter Ministral 3, solved about one in six, a difference the sample was too small to call. Every model failed most of the top tier. The authors conclude that scale alone doesn't explain the boundary; some failures responded to targeted fixes, with effects that varied by model.
Four other failure modes show up in practice:
- Out-of-distribution input. A narrowly tuned model meets a new document layout and answers fluently and wrong. Time-split held-out sets and schema checks catch most of this.
- Upstream schema drift. A source system renames a field and the tuned model keeps emitting the old one. Validation fails loudly, which is what you want.
- Long context. A model's advertised window is not the length it uses well (Hsieh et al., "RULER: What's the Real Context Size of Your Long-Context Language Models?", NVIDIA, arXiv, April 2024). Qwen3-8B, for one, is trained for 32,768 tokens and reaches 131,072 only with YaRN rope scaling, which its card advises against when inputs stay short. Test at the lengths your documents reach. Memory for that context also grows with every token, as the next section shows.
- Maintenance. Every tuned model has to be retrained and re-approved as the work changes, and every adapter again when its base model changes version. Budget for step 6 from the start.
Sizing the hardware for weights and the key-value cache
#Memory for a served model has two parts. The weights are fixed. The key-value (KV) cache, which stores attention state for every token in every active session, grows with use.
weights (bytes) ≈ parameters × bits per weight ÷ 8
KV cache (bytes) = 2 × layers × KV heads × head dimension × bytes per value × tokensThe 2 counts keys and values. The cache formula assumes every layer attends to the whole context; sliding-window layers stop growing at their window, and models with multi-head latent attention, such as DeepSeek's, cache a compressed latent instead, so check the architecture before you use it.
Take an 8-billion-parameter model with 36 layers and 8 key-value heads of 128 dimensions (grouped-query attention, with 32 query heads sharing those 8). Two published open models have those dimensions (Qwen3-8B model card; Ministral-8B model card). Qwen3-8B uses full attention on every layer. In Ministral-8B, three of every four layers use a sliding window of 32,768 tokens, so the arithmetic below holds for both up to that length and overstates Ministral's cache beyond it. Using Qwen3-8B's 8.2 billion parameters (Ministral-8B has 8.0 billion):
- Weights at 16 bits: about 16.4 GB. Quantized formats store a scale with every small block of weights, so the real cost per weight is above the nominal width: GGUF's Q8_0 spends 8.5 bits per weight and Q4_K spends 4.5, about 8.7 GB and 4.6 GB here. Files that keep the embedding and output layers at higher precision come out larger still.
- KV cache at 16 bits: 2 × 36 × 8 × 128 × 2 bytes = 147,456 bytes per token, about 147 KB. One session holding 32,768 tokens needs about 4.8 GB.
The cache is the part people under-budget. Eight concurrent sessions at 8,192 tokens each hold the same 65,536 tokens as two long ones, about 9.7 GB, more than the 8-bit weights. Quantizing the cache to 8 bits roughly halves it and 4 bits roughly quarters it, but the 4-bit cache costs accuracy on some models, so score it on your tasks before you turn it on. Sharing a common prefix (the same instructions and rules at the top of every request) keeps one copy of it in the cache instead of one per session, which is one reason to keep that prefix byte-identical across calls.
The same two model cards teach a second lesson. Both models have the dimensions above, but one is released under Apache 2.0, which permits commercial use, and the other under a research license that requires a separate agreement for commercial use. Licenses also move between releases: the same lab's next 8-billion-parameter model is Apache 2.0 (Ministral-3-8B model card). Read the license at step 4, before anyone fine-tunes, and again at every new base version.
Provenance is now an examiner question
#In October 2026, Reuters reported that US bank regulators were asking banks about limits on their AI agents, how they would shut them down, and whether they use technology developed in China (Reuters, October 13, 2026, as published by Khaleej Times). Several strong small-model families come from Chinese labs, Qwen among them, as do the DeepSeek distilled models the NVIDIA paper cites. Others come from NVIDIA's Nemotron line and from Mistral in France. Provenance is often layered, too: DeepSeek's distilled models were fine-tuned on reasoning data from DeepSeek-R1 but start from other labs' base models, Qwen's among them. The model card lists each base and its license.
That doesn't settle which to use. It means the choice needs an owner and a written reason. Pin the exact revision of the weights you pulled by its hash, prefer a format that can't run code when it loads (safetensors or GGUF rather than pickled checkpoints), and keep the hash and license with the trained artifact. Record the base family and version on every task, and make the allowed families a policy your model risk team can change.
How Inovagio runs small models
#Origin is Inovagio's model layer, built so small models you own take a Virtual Employee's routine steps and frontier models run only when you choose. Outside models are off by default; only an administrator you name can turn one on, for the tasks they choose. Nothing is billed per word to a lab. Which model may serve which task is set by policy, and every call is recorded with its model and cost, so you see what each task cost on the route it took, against what the work costs you today.
The Inference Server is our own engine, with its own model format, quantizers and GPU kernels. It serves open-weight models quantized by our own quantizers, from 4-bit to 8-bit weights. For the work in this guide it provides continuous batching, a paged KV cache with a prefix cache that survives across turns, KV cache in 16, 8 or 4 bits, multi-LoRA adapters selected per request, schema and grammar constraints with invalid requests refused rather than silently degraded, and a batch-invariant deterministic mode for audit replay. Speculative decoding with prompt-lookup, draft-head and multi-token-prediction drafters leaves the model's output distribution unchanged. Each request is bound to its authenticated tenant down to the cache keys, and model packages are signed.
Forge is the training side: curation with decontamination and a provenance chain for every dataset, then LoRA fine-tuning, DPO and other preference trainers, RLVR or distillation, then the gates above (task score on held-out approved results, regression against the version in service, plus the safety, red-team and calibration checks) and a recorded deployment approval. Each model ships with a model card, a software bill of materials and a signed evidence pack. The trained weights are yours, under the base model's license. Metrics and traces for every model go to Panoptes, and the architecture page shows where these layers sit.
Because the business rules live in the approved rulebook rather than inside any model, swapping a small model is a change to one route, re-scored on your approved results before it goes live. The model is a replaceable part makes that argument in full, and the context engineering guide covers what each step should be given to read, which matters even more when the model is small.
Further reading
#- Belcak et al., NVIDIA Research, Small Language Models are the Future of Agentic AI, arXiv, June 2, 2025, revised September 22, 2026.
- Karmakar and Chatterjee, AgentFloor: How Far Up the Tool Use Ladder Can Small Open-Weight Models Go?, arXiv, May 1, 2026.
- Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, arXiv, June 17, 2021.
- Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, arXiv, May 23, 2023.
- Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network, arXiv, March 9, 2015.
- Kim and Rush, Sequence-Level Knowledge Distillation, arXiv, June 25, 2016.
- Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, arXiv, May 29, 2023.
- Ong et al., RouteLLM: Learning to Route LLMs with Preference Data, arXiv, June 26, 2024.
- Chen, Zaharia and Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, arXiv, May 9, 2023.
- Tam et al., Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models, arXiv, August 5, 2024.
- Hsieh et al., NVIDIA, RULER: What's the Real Context Size of Your Long-Context Language Models?, arXiv, April 9, 2024.
- Gartner, Gartner Predicts by 2027, Organizations Will Use Small, Task-Specific AI Models Three Times More Than General-Purpose Large Language Models, press release, April 9, 2025.
- Gartner, Gartner Predicts AI Inference Costs Per Agentic Workflow Will Increase More Than Fivefold Through 2028, press release, August 17, 2026.
Questions
What people ask.
What is the difference between an SLM and an LLM for agents?
Size and fit. A small language model usually has fewer than 10 billion parameters and runs on one server; a large language model has tens of billions or more. For agents, small models suit narrow steps with a checkable answer, such as classification and schema extraction, while large models suit long, multi-step planning and open-ended text.
How many examples do I need to fine-tune a small model for one task?
The NVIDIA conversion method gives a rule of thumb of 10,000 to 100,000 examples. Use only results a person approved or your checks confirmed, remove personal and sensitive data first, and hold some out by time so the evaluation includes work the model has not seen.
Should an agent escalate to a larger model based on the model's confidence?
We recommend escalating on external checks instead: schema validation, reconciliation against records and your approved business rules. Self-reported confidence misses the confidently wrong answer, which is the costly one. If the larger model also fails the checks, the step goes to a person with the reason attached.
How much GPU memory does an 8-billion-parameter model need?
About 16 GB for the weights at 16 bits, or about 9 GB at 8 bits and 5 GB at 4 bits once the format's scale factors are counted, plus the key-value cache. For a model with 36 layers and 8 key-value heads of 128 dimensions, the cache is about 147 KB per token at 16 bits, so one 32,768-token session adds about 4.8 GB.
Where do small models still fail?
On long-horizon planning that has to track constraints across many steps, where frontier models keep a lead and neither is highly reliable. Narrowly tuned small models also fail on unfamiliar input and on renamed fields upstream, which is why every output should be validated and every model re-evaluated before it serves.
The products, industries and examples it covers.
- ModelsOriginRun open models on your own hardware, or a frontier model, through one API, never locked to one vendor.
- Model servingInference ServerRun open models, small to large, on hardware you control.
- Model trainingForgeTrain task-specific models on your data, with the evidence built in.
- Observability and analyticsPanoptesSee every log, trace, cost and decision across the platform.
- OverviewDevelopersBuild agents, workflows and apps, already governed.
- OverviewArchitectureHow the platform fits together, from your data to finished work.
- OverviewControl and costStop it any time, set limits, and see what every task costs.
Read next.
Article · Small language modelsSmall language models: why smaller AI does most business workMost steps an AI agent takes at work are narrow, like reading a field or sorting a document. A small language model can do them on your own server. Here's what the research says about how much work fits one, and where small models still fall short. · updated November 18, 2026 · 10 min read
Guide · QualityHow to measure AI cost per correct taskA per-task price shows what the AI charged. To know what the work cost, you need the formula, the baseline to take first, and a worksheet your finance team can fill in with its own numbers. · updated November 18, 2026 · 8 min read
Guide · Context engineeringContext engineering for enterprise AI agentsContext engineering decides what an AI agent sees at each step of a task. The research on why long contexts fail, the techniques that keep context small, how agent memory fits in, and the controls that stop context from becoming an attack path. · updated November 18, 2026 · 18 min read
Developer edition
Build on Inovagio.
The MNEMOS data engine, the command line and the desktop app, with guides and API references.







