Inference ServerModel serving
Run open models, small to large, on hardware you control.
Runs open models from a laptop to a rack of GPUs, behind one API that records what every call cost.
We’ll spend 30 minutes with you on one of your workflows, and you don’t need to share any data.
- Synesis
- Virtual Employees
- Agent Harness
- Opsis
- Origin
- Phylax
- Operations
- Platform services
Serving, on Keelstone hardware
One server · 4 GPUs · chat, embeddings and reranking side by side
Throughput
2,840 tok/s
Across all models
Requests in flight
46
Batched
Queue depth
3
Oldest 0.4 s
Prefix cache hits
71%
Across agent turns
Models loaded
| Model | Memory | State |
|---|---|---|
| Mid-size model · 30B | 34 GB | Loaded |
| Small model · 8B | 9 GB | Loaded |
| Keelstone task model · 3B | 4 GB | Loaded |
| Embeddings | 2 GB | Loaded |
| Reranker | 1 GB | Keep warm |
Throughput and queue, last hour
— Tokens per second— Queue depth
- Synesis
- Virtual Employees
- Agent Harness
- Opsis
- Origin
- Phylax
- Operations
- Platform services
Inovagio API · one call
Task, model tier and input in; output, model version, tokens, timing, cost, safety and its record out
Request
{ "task": "supplier-package-check", "model_tier": "mid-size", "input": "Does package SP-3318 carry the current drawing revision?", "context": ["record:SP-3318"], "max_output_tokens": 600, "stream": true}// The tenant comes from the signed-in credential,// never from the request body.
Response
{ "output": "Yes. Drawing 7741-B is at revision D, matching the release record.", "model": "mid-30b", "model_version": "v3 · sha256:7a1c…", "tenant": "Keelstone Aerospace", "tokens": { "input": 1840, "output": 212 }, "timing_ms": { "first_token": 210, "total": 3480 }, "cost": { "gpu_seconds": 3.4 }, "safety": { "input": "pass", "output": "pass" }, "record": "LR-20931"}
First token
0.21 s
Prefix cache hit
Total
3.5 s
64 tok/s
Tokens
2,052
1,840 in · 212 out
Record
LR-20931
In the audit ledger
Inference Server › Serving: the models loaded, throughput and queue depth. Sample data from Inovagio’s example companies.
Where Inference Server works
Inference Server isn’t named in a step of the onboarding. It is part of the platform every step runs on.
What Inference Server does
The Inference Server is Inovagio’s own model server: a native engine that runs open-weight models from a laptop to multi-GPU servers, serves generation, embeddings and reranking side by side, and can route chosen tasks to a frontier model, all behind one API that records the cost of every call.
Our own inference engine
A native C++ engine with its own model format, quantizers and GPU kernels, with no interpreter on the hot path.
Open models or frontier models
Serve leading open-weight dense, mixture-of-experts and hybrid models, or route a task to a frontier provider through the same API with retries and circuit breakers.
Built for agent workloads
Continuous batching, paged KV cache and a prefix cache that survives across turns cut the work each agent step repeats.
Speculative decoding
Speculative decoding with prompt-lookup, draft-head and multi-token-prediction drafters raises tokens per second without changing the model’s output distribution.
Structured, reliable output
JSON schema, grammar and regex constraints, tool calling and stop sequences, with invalid requests refused rather than silently degraded.
Chat, embeddings and reranking together
One server hosts chat, embedding and reranking models on one memory plane, loading and unloading them as demand changes.
Cost and telemetry on every call
Each request carries its token counts, cost and timing, with metrics and traces sent to Panoptes.
Isolated and audited
Each request is bound to its authenticated tenant down to the cache keys, and every call is written to a hash-chained audit ledger.
Small task models
Compact models trained on your approved results take over the tasks you run most, built with Forge and served by the Inference Server.
Everything Inference Server does, by area.
Serving
- Continuous in-flight batching on one engine loop per replica
- Paged KV cache with copy-on-write prefix sharing
- Radix prefix cache and session cache persisted to MNEMOS
- Chunked prefill and memory- and SLO-aware admission control
- Tiered preemption and per-request priority
- Multi-model residency with load, eject, keep-alive and idle unload
- Multi-LoRA adapters selected per request
- Batch-invariant deterministic mode for audit replay
Models and formats
- GGUF files served directly
- Safetensors models converted to Inovagio’s own format
- Weight quantization from 4-bit to 8-bit
- KV cache in f16, 8-bit and 4-bit, with mixed precision per layer
- Mixture-of-experts, sliding-window and hybrid linear-attention architectures
- Vision and multimodal input
- Speech-to-text and text-to-speech models for voice interviews, on your servers
- Thinking-mode and reasoning-effort control per request
APIs
- Native Inovagio API for generation, embeddings, reranking and model control
- Streaming with server-sent events
- Reranking endpoint
- Tool and function calling with per-model parsers
- Model Context Protocol (MCP) tool execution
- Health and metrics endpoints
Scale and trust
- Metal, CUDA, Vulkan and CPU backends from one codebase
- Tensor, pipeline and expert parallelism across GPUs
- Horizontal scale-out with prefix-aware load balancing and graceful drain
- Authenticated tenant bound to every cache key, fail-closed
- Hash-chained audit ledger of every conversation
- Signed and encrypted model packages
What Inference Server replaces
Inference Server does the work of these, inside one platform with one identity, one policy and one record:
- Separate inference servers for chat, embeddings and reranking
- Hosted model APIs for workloads that must stay private
- Model routing and gateway layers
- Standalone embedding and reranking services
- Tokenizer libraries and services
The controls Inference Server brings
Every Inovagio product keeps the same five commitments, because regulated work needs all five at once.
Security
Prompts stay on your hardware unless your policy routes a task to a frontier model you chose; tenant isolation extends to every cache key.
Governance
Model access and routing follow your policy.
On the record
Each call is written to the audit ledger with its model and cost.
Monitoring
Tokens, latency and queue depth for every model, sent to Panoptes.
Cost per task
Batching and caching cut the work each agent step repeats, and the cost of every task is recorded.
Inference Server in your industry
Inference Server runs the same way in every industry Inovagio serves. See the work in yours.
- Financial servicesBanks, credit unions, lenders and insurersWorked example: Alder Ridge Community Bank · Example company
- Healthcare providersHealth systems, hospitals and physician groupsWorked example: Meridale Health · Example company
- Health plansMedicare Advantage, Medicaid and commercial plansWorked example: Ashgrove Health Plan · Example company
- LegalLaw firms and in-house legal teamsWorked example: Calloway Finch · Example company
- GovernmentState, county and city agenciesWorked example: City of Larkhaven · Example company
- DefenseDefense contractors and their suppliersWorked example: Keelstone Aerospace · Example company
- Finance operationsFinance teams in any industry
For developers: Inference Server at a glance
- Interfaces
- Native Inovagio API over REST with server-sent-event streaming, and gRPC
- Endpoints
- Generate, embed, rerank, models (load / eject / keep), health, metrics
- Backends
- Metal, CUDA, Vulkan and CPU
- Platforms
- macOS (Apple silicon), Linux and Windows servers
- Model formats
- GGUF (served directly), safetensors (converted)
- Quantization
- 4-bit to 8-bit weights; 16-, 8- and 4-bit KV cache
- Deployment
- One inovagio app on a laptop or workstation, a private server, or multi-GPU cloud nodes
- Standards
- OpenTelemetry, Model Context Protocol
Inference Server in the platform.
Where Inference Server fits
One platformAgent HarnessAgent harness
Inovagio CLIInovagio DesktopWeb ChatInovagio SDK
OpsisExperience layer
FolioMorphe
Virtual EmployeesVirtual Employee layer
NexusErgon
SynesisKnowledge layer
PerceptionMNEMOS
Origin (this layer)Model layer
Inference Server (this product)Forge
PhylaxTrust layer
AuthInovagio SecretsGuard
OperationsOperations
PanoptesArchon
Platform servicesPlatform services
ServerAPI ManagementConnect
Further reading
Guides and posts on Inference Server
Guide · KnowledgeKnowledge layer: questions and answersStraight answers to the questions buyers and architects ask about the knowledge layer for AI agents, from ontologies, knowledge graphs and semantic layers to context engineering, GraphRAG, semantic caching, small models and who owns it all. · updated November 18, 2026 · 12 min read
Guide · Small language modelsSLMs for enterprise AI agents: routing, fine-tuning and limitsHow to move an enterprise agent's routine model calls to small language models without losing accuracy: the published conversion method, routing and escalation, LoRA fine-tuning, the gates a model passes before it serves, hardware sizing with worked numbers, and where small models still fail. · updated November 18, 2026 · 18 min read
Article · Small language modelsSmall language models: why smaller AI does most business workMost steps an AI agent takes at work are narrow, like reading a field or sorting a document. A small language model can do them on your own server. Here's what the research says about how much work fits one, and where small models still fall short. · updated November 18, 2026 · 10 min read
Article · ModelsThe model is a replaceable partThe best AI model this quarter won't be the best next quarter. When you can switch AI models without touching your rules, approvals or audit record, every better model shows up on the scoreboard in your cost per correct task. · updated November 20, 2026 · 7 min read
See Inference Server on one of your workflows.
Every project starts with one workflow your people know well, measured against what it costs you today.