Skip to content
InovagioInovagio

Series · Knowledge layer deep dives

SLMs for enterprise AI agents: routing, fine-tuning and limits

How to move an enterprise agent's routine model calls to small language models without losing accuracy: the published conversion method, routing and escalation, LoRA fine-tuning, the gates a model passes before it serves, hardware sizing with worked numbers, and where small models still fail.

· Published · Updated · 18 min read

On this page

Small language models (SLMs) for AI agents are compact models, usually under 10 billion parameters, that take over an enterprise agent's narrow, repetitive calls while a large language model (LLM) handles planning and the cases the small one can't confirm. The switch keeps data on your hardware, and cost per correct task shows whether it paid off. Done carelessly, though, it hides errors, and a cheap wrong answer that nobody checks is the most expensive kind.

What follows is the conversion method NVIDIA researchers published, plus the parts a production system adds: routing, evaluation gates and hardware sizing, worked through with real numbers. We also say where the evidence runs out. The business case, in plainer words, is in small language models for business.

Why AI agent calls suit small language models

#

An agent's model calls aren't one workload. A single task can include a classification, three extractions into a fixed schema, a lookup, a comparison and a free-text note for a reviewer. Most of those have one right answer and a format you can check.

That's the argument of the paper at the center of the current debate, and its thesis fits in one line: small models are "sufficiently powerful, inherently more suitable, and necessarily more economical" for much of what agents do. Its authors define a small model as one that runs on a common consumer device fast enough to serve one user's agent requests, and they'd call most models under 10 billion parameters small as of 2025 (Belcak et al., "Small Language Models are the Future of Agentic AI", NVIDIA Research, arXiv 2506.02153, June 2025, revised September 2026). They cite work showing a 7-billion-parameter model is 10 to 30 times cheaper to serve than one of 70 to 175 billion, counting latency and energy as well as compute per answer.

A quotation from NVIDIA researchers: small language models are sufficiently powerful, inherently more suitable, and necessarily more economical for agent work
A position paper: the authors argue the case and invite critique. Source: Belcak et al., NVIDIA Research, arXiv 2506.02153.

The paper is a position paper, and it says so. Its appendix estimates how many calls a small model could take in three open agent frameworks (about 60%, 40% and 70%), but the authors got them by reading code, and nobody measured a run. Use them as a reason to count your own calls. They don't forecast your queue.

Gartner's view points the same way at the market level: it predicts that by 2027 organizations will use small, task-specific models at least three times more than general-purpose large models, by usage volume (Gartner, April 9, 2025). It also predicts that inference cost per agentic workflow will rise more than fivefold through 2028, and recommends tiering and routing so each task reaches a model whose cost fits it (Gartner, August 17, 2026).

The conversion method, step by step

#

The NVIDIA paper lays out an LLM-to-SLM conversion in six steps. Here they are, each with what we'd add for regulated queues, where the training data is also evidence and the people who approve the work are named.

  1. Log the calls. Record every call that isn't a direct conversation with a person: the prompt, the response and the contents of each tool call, with latency as an option. The paper asks for encrypted pipelines with role-based access and anonymization before storage. We'd add the outcome: whether a person approved the result or the checks passed, because that label is what makes a log usable as training data. In a regulated business, the log is also evidence, so bind each entry to its tenant and keep it in the audit record.
  2. Curate and scrub. Remove personal and health data and anything else sensitive. The paper's rule of thumb is 10,000 to 100,000 examples to fine-tune a small model. Our addition: train only on results a person approved or the checks confirmed. A log of model outputs is not ground truth.
  3. Cluster the tasks. Group prompts and actions with unsupervised clustering to find the recurring task types. You'll usually find a long tail of rare calls and a short head of very common ones; the head is where conversion pays.
  4. Choose candidate models. The paper's criteria are instruction following, reasoning, context window, benchmarks relevant to the task, license and deployment footprint. Add provenance, for reasons covered below.
  5. Fine-tune. The paper names parameter-efficient fine-tuning such as LoRA or QLoRA, full fine-tuning if you have the budget, and distillation from a larger model.
  6. Iterate. Retrain the small models and the router as new approved data arrives.
The conversion cycle that moves AI agent calls to small language models: log and scrub the calls, cluster them into tasks, choose and fine-tune a small model, and retrain as approved work arrives
The six steps of the method, grouped into four. The last one never ends.

What LoRA and QLoRA buy you

#

LoRA (low-rank adaptation) freezes the base model and learns a low-rank update for each chosen weight matrix, stored as two thin matrices whose product is added to the frozen weights. In the original paper, against full fine-tuning of a 175-billion-parameter model, it cut the trainable parameters by a factor of 10,000 and the graphics processor (GPU) memory needed by a factor of three (Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models", arXiv, June 2021). QLoRA (quantized LoRA) goes further, training adapters on a base quantized to 4 bits with NormalFloat (NF4), a data type designed for normally distributed weights, and quantizing the quantization constants themselves (double quantization). That made it possible to fine-tune a 65-billion-parameter model on a single 48 GB GPU (Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs", arXiv, May 2023).

The practical effect for agents matters more than the training savings. A LoRA adapter is small, so one base model in memory can serve many task-specific adapters, chosen per request. Ten "task models" don't need ten copies of the weights. They need one. The catch is that an adapter belongs to one exact base: when the base model changes version, every adapter on it is retrained and re-scored, not copied across.

Distillation changes where the training targets come from. A larger teacher model either writes answers the small model is fine-tuned on (sequence-level distillation) or supplies its full output probabilities for the student to match, which needs access to the teacher's logits and a shared tokenizer (Hinton, Vinyals and Dean, "Distilling the Knowledge in a Neural Network", arXiv, March 2015; Kim and Rush, "Sequence-Level Knowledge Distillation", arXiv, June 2016). It's useful when you lack approved examples, with two cautions. The student inherits the teacher's mistakes, so the training set still needs a check that isn't the teacher. And read the teacher's license and terms of use first, because some forbid using a model's outputs to train another one.

Once a supervised model works, preference tuning is the next lever. Methods such as direct preference optimization (DPO) train directly on pairs of a preferred and a rejected answer, which is what a reviewer's correction gives you (Rafailov et al., "Direct Preference Optimization", arXiv, May 2023). Where the answer can be checked by code, reinforcement learning with verifiable rewards (RLVR) uses the same checks that gate a route as the reward.

A table of four ways to tune a small model and what each learns from: LoRA or QLoRA from approved examples as an adapter on a frozen base, distillation from a larger teacher, DPO from preferred and rejected pairs, and RLVR from the checks that gate a route
One base model in memory can serve many task-specific adapters, chosen per request.

Routing each call to the right model

#

Routing is where most of the savings and most of the risk sit. There are three common designs, and production systems tend to combine them.

Static routing by task. Each task cluster from step 3 has a default model. A document classification goes to the small classifier; a reviewer's note goes to a larger model. This is predictable and easy to audit, and in a queue with well-defined steps it covers most traffic.

Learned routers. A router model predicts, per request, whether a weaker model will answer as well as a stronger one. RouteLLM's routers, trained on human preference data, more than halved cost in some cases without lowering response quality (Ong et al., "RouteLLM: Learning to Route LLMs with Preference Data", arXiv, June 2024).

Cascades. Try the cheap model first and move up only when the answer fails a check. In FrugalGPT the check is a small learned scorer of answer quality, and its cascade matched the best single model's performance with up to 98% cost reduction on its benchmarks (Chen, Zaharia and Zou, "FrugalGPT", Stanford, arXiv, May 2023).

We take a side on one design choice: escalate on external checks, not on the model's own confidence. A small model that's confidently wrong is the failure that costs most, and self-reported confidence is the signal least likely to catch it. External checks are concrete: the output validates against the step's JSON schema (the declared shape of a structured answer), the extracted totals reconcile with the purchase order, the cited clause exists in the document. Each one passes or it doesn't.

Here's the logic as a sketch (not product code). Each step kind has its own ladder, set by policy, so a hard step can start higher and a frontier rung appears only where policy allows one:

python
LADDERS = {
    "classify_document": ["small-task-model", "larger-open-model"],
    "extract_header":    ["small-task-model", "larger-open-model"],
    "split_freight":     ["larger-open-model"],
}

def run_step(step, record):
    ladder = LADDERS[step.kind]       # an unrouted step kind raises; it never defaults to the cheapest model
    failures = []
    for model in ladder:
        out = call(model, step, schema=step.schema)    # decoding constrained to the step's schema
        failures = check(out, step.schema, record)     # schema, reconciliation, approved rules
        log_attempt(step, model, failures)             # every rung is recorded, including the ones that failed
        if not failures:
            return out
    return to_person(step, reasons=failures)           # never a silent fallback
The escalation ladder for one agent step: a small task model first, a larger open model when its output fails a check, a frontier model only where your policy allows one, and a person when the checks still fail
Each rung costs more. Most steps stop on the first one.

Two details make this safe. First, constrain the output: grammar- or schema-constrained decoding makes a malformed answer impossible rather than merely unlikely (a reply cut off at the token limit is the one exception, so check for it), and validation failures then mean wrong content, not broken JSON. Constraint fixes the shape, not the content. One study found strict output formats lowered scores on reasoning tasks (Tam et al., "Let Me Speak Freely?", arXiv, August 2024). Where a step has to reason before it answers, put the reasoning field ahead of the answer fields in the schema and test the step both ways. Second, the last rung is a person with the reason attached. A cascade that ends in "accept whatever the biggest model said" has quietly removed your control.

A worked example with one invoice and five calls

#

Take the accounts payable queue at Tollbridge Fasteners (Tollbridge Fasteners · Example company). One vendor invoice produces these steps.

StepRouteWhy
Classify the document: invoice, credit note or statementSmall task modelOne label from a fixed list
Extract the header fields into a JSON schemaSmall task model, schema-constrainedFixed format, validated on output
Match the lines to the purchase orderAn approved rule, no modelArithmetic against an approved tolerance
Write the exception note for the reviewerLarger open modelFree text that a person reads
Split a freight charge billed across two ordersLarger open model, then a person if the checks failSeveral steps with constraints to track

Notice the third row. The cheapest model for line matching is no model at all. Deterministic checks against approved rules belong outside the model, where they're exact and can be replayed; our guide to how a rule is approved shows what such a rule contains. The fifth row is the kind of task the research says small models still struggle with, so it starts higher on the ladder.

Evaluation before a model serves

#

A fine-tuned small model is a new model, and it earns its place on your work, not on a public leaderboard. The NVIDIA paper names reliance on generalist benchmarks as a barrier to adopting small models, because those benchmarks understate what a small model does on agent work. The reverse holds too: a model that scores well on a broad benchmark can still fail your vendor's invoice layout.

A model evaluation report for one task model, with five gates: task score on held-out approved results, regression against the version in service, safety and red-team, calibration, and a recorded deployment approval
A model that fails any gate doesn't serve.

Five gates, in the order we run them:

  • Task score on held-out approved results, scored by your own checks. Hold out examples by time as well as at random, so the test includes work the model never saw the shape of, and remove near-duplicates of test items from the training set first, or the score measures memory.
  • Regression against the version in service. Same held-out set, both models. A new model that wins on average but loses on a category your auditors care about isn't an upgrade.
  • Safety and red-team. Prompt injection in documents, requests outside the role, attempts to output data the step shouldn't see.
  • Calibration. Where scores or confidences feed a route, check they mean what they say.
  • A recorded deployment approval, by a named person, with the evidence attached.

Run evaluation in a deterministic serving mode, so a replay of the same input gives the same output. Otherwise a regression can hide in sampling noise, and nobody can reproduce what the auditor is asking about. If the step samples when it serves, score it at those settings as well, over several runs with the spread reported.

Then watch it in service. The measures that matter are first-pass success, the share of steps escalated, the minutes people spend on escalations, and cost per correct task, which counts all of it. The guide to measuring cost per correct task has the formula and a worksheet.

Where small models still fail

#

The best recent evidence is AgentFloor, a deterministic benchmark of 30 tasks on a six-tier ladder from instruction following through tool use and multi-step coordination to long-horizon planning under persistent constraints (Karmakar and Chatterjee, "AgentFloor", arXiv 2605.00334, May 1, 2026).

AgentFloor in numbers: 16 open-weight models from 0.27 to 32 billion parameters against a frontier model, across 16,542 scored runs on 30 tasks
Source: Karmakar and Chatterjee, AgentFloor, arXiv 2605.00334, May 1, 2026.

Small and mid-size open models handled much of the short, structured tool use that dominates real pipelines, and the strongest open model matched the frontier model in aggregate. The tier results are worth reading closely. Equivalence was established statistically only for a single tool call. On two-tool chains, branching on an intermediate result and multi-source synthesis, the best open model came within about nine points of the frontier model without the test confirming a match. On the top tier, long-horizon planning under persistent constraints, the frontier model beat the strongest open model, which solved none of those tasks. But the frontier model itself solved only about one in ten, and the best open model on that tier, Mistral's 8-billion-parameter Ministral 3, solved about one in six, a difference the sample was too small to call. Every model failed most of the top tier. The authors conclude that scale alone doesn't explain the boundary; some failures responded to targeted fixes, with effects that varied by model.

Four other failure modes show up in practice:

  • Out-of-distribution input. A narrowly tuned model meets a new document layout and answers fluently and wrong. Time-split held-out sets and schema checks catch most of this.
  • Upstream schema drift. A source system renames a field and the tuned model keeps emitting the old one. Validation fails loudly, which is what you want.
  • Long context. A model's advertised window is not the length it uses well (Hsieh et al., "RULER: What's the Real Context Size of Your Long-Context Language Models?", NVIDIA, arXiv, April 2024). Qwen3-8B, for one, is trained for 32,768 tokens and reaches 131,072 only with YaRN rope scaling, which its card advises against when inputs stay short. Test at the lengths your documents reach. Memory for that context also grows with every token, as the next section shows.
  • Maintenance. Every tuned model has to be retrained and re-approved as the work changes, and every adapter again when its base model changes version. Budget for step 6 from the start.

Sizing the hardware for weights and the key-value cache

#

Memory for a served model has two parts. The weights are fixed. The key-value (KV) cache, which stores attention state for every token in every active session, grows with use.

text
weights (bytes)  ≈ parameters × bits per weight ÷ 8
KV cache (bytes) = 2 × layers × KV heads × head dimension × bytes per value × tokens

The 2 counts keys and values. The cache formula assumes every layer attends to the whole context; sliding-window layers stop growing at their window, and models with multi-head latent attention, such as DeepSeek's, cache a compressed latent instead, so check the architecture before you use it.

Take an 8-billion-parameter model with 36 layers and 8 key-value heads of 128 dimensions (grouped-query attention, with 32 query heads sharing those 8). Two published open models have those dimensions (Qwen3-8B model card; Ministral-8B model card). Qwen3-8B uses full attention on every layer. In Ministral-8B, three of every four layers use a sliding window of 32,768 tokens, so the arithmetic below holds for both up to that length and overstates Ministral's cache beyond it. Using Qwen3-8B's 8.2 billion parameters (Ministral-8B has 8.0 billion):

  • Weights at 16 bits: about 16.4 GB. Quantized formats store a scale with every small block of weights, so the real cost per weight is above the nominal width: GGUF's Q8_0 spends 8.5 bits per weight and Q4_K spends 4.5, about 8.7 GB and 4.6 GB here. Files that keep the embedding and output layers at higher precision come out larger still.
  • KV cache at 16 bits: 2 × 36 × 8 × 128 × 2 bytes = 147,456 bytes per token, about 147 KB. One session holding 32,768 tokens needs about 4.8 GB.
Memory for one 8-billion-parameter model: weights at 16 bits and at 4 bits, and the key-value cache for one 32,768-token session at 16 bits
Worked from the published configuration: 36 layers, 8 key-value heads of 128 dimensions, 8.2 billion parameters.

The cache is the part people under-budget. Eight concurrent sessions at 8,192 tokens each hold the same 65,536 tokens as two long ones, about 9.7 GB, more than the 8-bit weights. Quantizing the cache to 8 bits roughly halves it and 4 bits roughly quarters it, but the 4-bit cache costs accuracy on some models, so score it on your tasks before you turn it on. Sharing a common prefix (the same instructions and rules at the top of every request) keeps one copy of it in the cache instead of one per session, which is one reason to keep that prefix byte-identical across calls.

The same two model cards teach a second lesson. Both models have the dimensions above, but one is released under Apache 2.0, which permits commercial use, and the other under a research license that requires a separate agreement for commercial use. Licenses also move between releases: the same lab's next 8-billion-parameter model is Apache 2.0 (Ministral-3-8B model card). Read the license at step 4, before anyone fine-tunes, and again at every new base version.

Provenance is now an examiner question

#

In October 2026, Reuters reported that US bank regulators were asking banks about limits on their AI agents, how they would shut them down, and whether they use technology developed in China (Reuters, October 13, 2026, as published by Khaleej Times). Several strong small-model families come from Chinese labs, Qwen among them, as do the DeepSeek distilled models the NVIDIA paper cites. Others come from NVIDIA's Nemotron line and from Mistral in France. Provenance is often layered, too: DeepSeek's distilled models were fine-tuned on reasoning data from DeepSeek-R1 but start from other labs' base models, Qwen's among them. The model card lists each base and its license.

That doesn't settle which to use. It means the choice needs an owner and a written reason. Pin the exact revision of the weights you pulled by its hash, prefer a format that can't run code when it loads (safetensors or GGUF rather than pickled checkpoints), and keep the hash and license with the trained artifact. Record the base family and version on every task, and make the allowed families a policy your model risk team can change.

How Inovagio runs small models

#

Origin is Inovagio's model layer, built so small models you own take a Virtual Employee's routine steps and frontier models run only when you choose. Outside models are off by default; only an administrator you name can turn one on, for the tasks they choose. Nothing is billed per word to a lab. Which model may serve which task is set by policy, and every call is recorded with its model and cost, so you see what each task cost on the route it took, against what the work costs you today.

The Inference Server is our own engine, with its own model format, quantizers and GPU kernels. It serves open-weight models quantized by our own quantizers, from 4-bit to 8-bit weights. For the work in this guide it provides continuous batching, a paged KV cache with a prefix cache that survives across turns, KV cache in 16, 8 or 4 bits, multi-LoRA adapters selected per request, schema and grammar constraints with invalid requests refused rather than silently degraded, and a batch-invariant deterministic mode for audit replay. Speculative decoding with prompt-lookup, draft-head and multi-token-prediction drafters leaves the model's output distribution unchanged. Each request is bound to its authenticated tenant down to the cache keys, and model packages are signed.

Forge is the training side: curation with decontamination and a provenance chain for every dataset, then LoRA fine-tuning, DPO and other preference trainers, RLVR or distillation, then the gates above (task score on held-out approved results, regression against the version in service, plus the safety, red-team and calibration checks) and a recorded deployment approval. Each model ships with a model card, a software bill of materials and a signed evidence pack. The trained weights are yours, under the base model's license. Metrics and traces for every model go to Panoptes, and the architecture page shows where these layers sit.

Forge's path for a small model: curate the data, train it, pass the gates for task score, regression, safety, red-team and calibration, and record a named approval before it ships with its evidence
Metrics and traces for every model go to Panoptes.

Because the business rules live in the approved rulebook rather than inside any model, swapping a small model is a change to one route, re-scored on your approved results before it goes live. The model is a replaceable part makes that argument in full, and the context engineering guide covers what each step should be given to read, which matters even more when the model is small.

Further reading

#

Share

Questions

What people ask.

What is the difference between an SLM and an LLM for agents?

Size and fit. A small language model usually has fewer than 10 billion parameters and runs on one server; a large language model has tens of billions or more. For agents, small models suit narrow steps with a checkable answer, such as classification and schema extraction, while large models suit long, multi-step planning and open-ended text.

How many examples do I need to fine-tune a small model for one task?

The NVIDIA conversion method gives a rule of thumb of 10,000 to 100,000 examples. Use only results a person approved or your checks confirmed, remove personal and sensitive data first, and hold some out by time so the evaluation includes work the model has not seen.

Should an agent escalate to a larger model based on the model's confidence?

We recommend escalating on external checks instead: schema validation, reconciliation against records and your approved business rules. Self-reported confidence misses the confidently wrong answer, which is the costly one. If the larger model also fails the checks, the step goes to a person with the reason attached.

How much GPU memory does an 8-billion-parameter model need?

About 16 GB for the weights at 16 bits, or about 9 GB at 8 bits and 5 GB at 4 bits once the format's scale factors are counted, plus the key-value cache. For a model with 36 layers and 8 key-value heads of 128 dimensions, the cache is about 147 KB per token at 16 bits, so one 32,768-token session adds about 4.8 GB.

Where do small models still fail?

On long-horizon planning that has to track constraints across many steps, where frontier models keep a lead and neither is highly reliable. Narrowly tuned small models also fail on unfamiliar input and on renamed fields upstream, which is why every output should be validated and every model re-evaluated before it serves.

Developer edition

Build on Inovagio.

The MNEMOS data engine, the command line and the desktop app, with guides and API references.

This is for

We use these details to answer your request. See our privacy policy.

What happens next

  1. 1We reply to every request.
  2. 2We send the next steps for your team.
  3. 3A direct line to the engineers who build it.