Series · The knowledge layer, explained
Small language models: why smaller AI does most business work
Most steps an AI agent takes at work are narrow, like reading a field or sorting a document. A small language model can do them on your own server. Here's what the research says about how much work fits one, and where small models still fall short.
Inovagio team · Published · Updated · 10 min read

On this page
An invoice lands in the accounts payable inbox at Pellworth Supply, and the AI agent working the queue makes 11 separate calls to a model before it's done (Pellworth Supply · Example company). A small language model (SLM) is an AI model compact enough to run on one server, usually under 10 billion parameters, and 10 of those 11 calls are its kind of work.
Ten are easy. They're small questions with one right answer: invoice or statement, the purchase order number, whether the total matches the order, which vendor sent it. The last one is hard: the vendor has billed freight on a separate invoice again, and the agent has to decide whether the two together match what was ordered. Sending all 11 to the biggest model you can buy is like asking a senior partner to stamp the mail.
A lot of back-office work looks like those 10 calls, which is why small models deserve more attention than they get. They have real limits, though, and for banks the question of where a model came from is now on the exam. If you build these systems, the companion guide on small language models for AI agents goes deeper on routing and fine-tuning, with hardware sizing worked through.
What makes a language model small
#A model's parameters are the numbers it learns during training. Think of them as the dials set while it studied. A large language model (LLM), the kind behind the best-known chat products, has tens of billions of them or more; a small one has a few billion or fewer.
NVIDIA researchers proposed a practical definition in 2025: a small model is one that runs on a common consumer device fast enough to serve one person's agent requests. They avoided a hard parameter count so the definition wouldn't date, but said they'd be comfortable calling most models below 10 billion parameters small (Belcak et al., "Small Language Models are the Future of Agentic AI", NVIDIA Research, June 2025, revised September 2026).
Many small models are open-weight. The trained numbers are published, so you can run the model on your own hardware and tune it on your own examples. Open-weight isn't the same as open source: the training data usually stays private, and the license still sets what you may do with the model. That matters more than the size. A small model you can't tune or host is still a rental.
Most agent work is narrow
#An AI agent is software that uses a model to take steps toward a goal: read a document, call a system, write a result. Each step is usually a separate call to a model, and most calls are narrow. They pull a field, sort a document into a category, fill in a form in a fixed format, or check one value against another.
The NVIDIA paper argues that this is exactly where small models fit, and that agents should use a mix of models, keeping large ones for open-ended conversation and hard reasoning. In an appendix, the authors read three popular open-source agent frameworks and estimated how many of their model calls a small model could take over: about 60% in MetaGPT, a software-development agent; about 40% in Open Operator, a workflow-automation agent; and about 70% in Cradle, an agent that runs desktop software from screenshots.
Read those as estimates, because that's what they are. Nobody ran the frameworks both ways and counted. Still, the direction matches how back-office queues work, where the rules are written down and most steps have one right answer. Our view is that regulated queues lean further toward small-model work than general-purpose agents do; a test on your own past cases is how you'd find out for your queue.
What small language models cost to run
#The same paper reports a wide gap in running cost. It cites work showing that serving a 7-billion-parameter model is 10 to 30 times cheaper than serving one of 70 to 175 billion, counting response time and energy as well as the computing work per answer. Whether that gap holds on your queue is what cost per correct task shows.
Gartner expects the market to follow the money. It predicts that by 2027 organizations will use small, task-specific AI models at least three times more than general-purpose large models, measured by usage volume. Its reasons are practical: general models lose accuracy on tasks that need business knowledge, while smaller models answer faster and use less computing power (Gartner, April 9, 2025).
Rising bills push the same way. Gartner also predicts that AI inference costs per agentic workflow will increase more than fivefold through 2028. An agent's workflow uses far more tokens than a chatbot exchange, often on pricier reasoning models, and falling prices per token don't keep up (Gartner, August 17, 2026). Its advice is to route each task to a model whose cost fits it.
A cheap call is only cheap if it's right, though. A small model that gets a field wrong sends the task to a person, and that person's minutes cost more than the model ever saved. That's why we judge models by cost per correct task, which counts review time and rework as well as the model. The guide to measuring cost per correct task shows the formula.
SLM vs LLM: which does what
#The best evidence on where small models fall short is recent. In May 2026, two researchers published AgentFloor, a benchmark of 30 tasks arranged as a six-tier ladder, from following instructions and using tools up to long-horizon planning under constraints. They ran 16 open-weight models, from 0.27 billion to 32 billion parameters, against a frontier model across 16,542 scored runs (Karmakar and Chatterjee, "AgentFloor", May 1, 2026).
Small and mid-size open models handled much of the short, structured tool use that fills real pipelines, and the strongest open model matched the frontier model in aggregate. At the top of the ladder, where a task needs a long plan and has to keep track of constraints along the way, the frontier model kept its edge over the strongest open model. Neither side was reliable there: on the top tier, every model failed most tasks. The authors also found that model size alone didn't explain where the failures fell.
There are quieter limits too. A small model tuned narrowly can stumble on input it has never seen, such as a new invoice layout. When an upstream system renames a field, a tuned model may keep producing the old one. And a tuned model has to be retrained whenever the work changes, so budget for that as a standing job.
What these limits call for is routing: the small model by default, a larger model for planning and for the cases the small one can't confirm, and a person for anything the checks can't settle.
Why running your own model helps
#A small model fits on hardware you control. Prompts and records stay on your servers, each call has a cost you can see instead of a line on a vendor's invoice, and nobody outside your company decides when the model you depend on changes. With the best open models only months behind the closed leaders, open vs closed AI models is less a trade-off than it was.
That's how Origin, Inovagio's model layer, is built: small models you own take a Virtual Employee's routine steps, and frontier models run only when you choose. Outside models are off by default, and only an administrator you name can turn one on, for the tasks they choose. The Inference Server runs open models from a single workstation to multi-GPU servers behind one API, and records the cost of every call.
Examiners now ask where a model came from
#Bank supervisors have started asking about models. In October, Reuters reported that the Federal Reserve, the Office of the Comptroller of the Currency (OCC) and the Federal Deposit Insurance Corporation (FDIC) were asking banks how they track, audit and shut down their AI agents, and whether they use technology developed in China (Reuters, October 13, 2026, via Khaleej Times). Several capable open model families come from Chinese labs, so picking a base model is now a governance decision as much as a technical one.
Licenses differ too, even between models of the same size. Two popular 8-billion-parameter models are built to nearly the same dimensions, yet one is published under the Apache 2.0 license, which allows commercial use, and the other under a research license that requires a separate agreement for commercial use (Qwen3-8B model card; Ministral-8B model card).
The same lab's next 8-billion-parameter model went out under Apache 2.0, so the license can change from one release to the next (Ministral-3-8B model card). Read the license before the benchmark, and read it again at every version.
A distilled model can carry two lines of provenance, the lab that trained it and the lab whose base model it started from, so read both.
That's why, in Inovagio, each role runs on an open-model family you choose, and the family is written on every task record. A procurement or model risk team can see which family did each task and decide which families are allowed. Our piece on why the model is a replaceable part explains how a role moves to a different family without touching its rules or records.
How a small task model is made from your own work
#The small model that does a queue's routine steps works best when it has been tuned on that queue's approved results. Forge, Inovagio's training pipeline, does this inside your security perimeter.
Results your people approved become training data, with the provenance of each source kept. Forge fine-tunes an open-weight base for the task, then the model has to pass gates before it ships: a task score on approved work held back for the test and graded by your own checks, a regression check against the version in service, plus safety and calibration checks.
A deployment approval is recorded, and each model gets a model card and a signed evidence pack. The trained weights are yours, under the base model's license, which is one of the assets you should own when a vendor's engineers leave. Origin then serves the model on the queue's routine steps, and the cost of every call is recorded.
Your business rules don't move into the model. They stay in the approved rulebook, which any model reads the same way, so a better small model next year can be swapped in without a rebuild.
Where to start
#Pick one queue. Record every model call it makes for a few weeks. Sort the calls by kind, from simple lookups and form-filling to the few that need real judgment. The sorted list tells you how much of the queue a small model could take, and the cost per correct task before and after tells you whether it did. The guide for engineers walks through each step, from logging the calls to routing them.
Further reading
#- Belcak et al., NVIDIA Research, Small Language Models are the Future of Agentic AI, arXiv, June 2, 2025, revised September 22, 2026.
- Gartner, Gartner Predicts by 2027, Organizations Will Use Small, Task-Specific AI Models Three Times More Than General-Purpose Large Language Models, press release, April 9, 2025.
- Gartner, Gartner Predicts AI Inference Costs Per Agentic Workflow Will Increase More Than Fivefold Through 2028, press release, August 17, 2026.
- Karmakar and Chatterjee, AgentFloor: How Far Up the Tool Use Ladder Can Small Open-Weight Models Go?, arXiv, May 1, 2026.
- Reuters, Banking watchdogs ramp up AI scrutiny as stability fears mount, sources say, October 13, 2026, as published by Khaleej Times.
Questions
What people ask.
What is a small language model?
A small language model is an AI model compact enough to run on a single server or workstation, usually with fewer than 10 billion parameters. It is often tuned for one narrow job, such as pulling fields from a document or sorting requests, and it runs on far smaller hardware than a large model.
Are small language models as accurate as large ones?
On short, structured tasks with one right answer, a well-tuned small model often matches a large one. In the AgentFloor benchmark, the strongest open model matched a frontier model in aggregate. Large models still lead on long, multi-step planning, so the sensible design routes each step to the smallest model that gets it right.
What does a small language model cost to run?
NVIDIA researchers cite work showing a 7-billion-parameter model is 10 to 30 times cheaper to serve than one of 70 to 175 billion parameters. What matters for a business is cost per correct task, which also counts the review time and rework a wrong answer causes, so measure it on your own cases.
Can a small language model run on our own servers?
Yes. A small model fits on a single server or workstation, so prompts and records stay on hardware you control. With Inovagio, Origin runs open models on your hardware, and outside models stay off unless an administrator you name turns one on for chosen tasks.
Does it matter which company made a small model?
It does for regulated firms. Bank regulators have asked whether banks use technology developed in China, and licenses differ between models of the same size. Choose model families deliberately, record which family did each task, and have each model pass your own checks before it goes into service.
The products, industries and examples it covers.
- ModelsOriginRun open models on your own hardware, or a frontier model, through one API, never locked to one vendor.
- Model trainingForgeTrain task-specific models on your data, with the evidence built in.
- Model servingInference ServerRun open models, small to large, on hardware you control.
- IndustryFinance operationsInvoices matched to orders and receipts and coded by your rules, and payments to changed bank details held until a callback is logged. Your team releases every payment.
- OverviewArchitectureHow the platform fits together, from your data to finished work.
- OverviewControl and costStop it any time, set limits, and see what every task costs.
Read next.
Guide · Small language modelsSLMs for enterprise AI agents: routing, fine-tuning and limitsHow to move an enterprise agent's routine model calls to small language models without losing accuracy: the published conversion method, routing and escalation, LoRA fine-tuning, the gates a model passes before it serves, hardware sizing with worked numbers, and where small models still fail. · updated November 18, 2026 · 18 min read
Article · ModelsThe model is a replaceable partThe best AI model this quarter won't be the best next quarter. When you can switch AI models without touching your rules, approvals or audit record, every better model shows up on the scoreboard in your cost per correct task. · updated November 20, 2026 · 7 min read
Guide · QualityHow to measure AI cost per correct taskA per-task price shows what the AI charged. To know what the work cost, you need the formula, the baseline to take first, and a worksheet your finance team can fill in with its own numbers. · updated November 18, 2026 · 8 min read
See it on your work
See how one of your tasks runs, start to finish.
A 30-minute walkthrough of one of your workflows, the approval step and the audit record, with the people who build the platform.



