Skip to content
InovagioInovagio

Inference ServerModel serving

Run open models, small to large, on hardware you control.

Runs open models from a laptop to a rack of GPUs, behind one API that records what every call cost.

We’ll spend 30 minutes with you on one of your workflows, and you don’t need to share any data.

Inference Server › Serving: the models loaded, throughput and queue depth. Sample data from Inovagio’s example companies.

Where Inference Server works

Inference Server isn’t named in a step of the onboarding. It is part of the platform every step runs on.

How the ten steps work

What Inference Server does

The Inference Server is Inovagio’s own model server: a native engine that runs open-weight models from a laptop to multi-GPU servers, serves generation, embeddings and reranking side by side, and can route chosen tasks to a frontier model, all behind one API that records the cost of every call.

  • Our own inference engine

    A native C++ engine with its own model format, quantizers and GPU kernels, with no interpreter on the hot path.

  • Open models or frontier models

    Serve leading open-weight dense, mixture-of-experts and hybrid models, or route a task to a frontier provider through the same API with retries and circuit breakers.

  • Built for agent workloads

    Continuous batching, paged KV cache and a prefix cache that survives across turns cut the work each agent step repeats.

  • Speculative decoding

    Speculative decoding with prompt-lookup, draft-head and multi-token-prediction drafters raises tokens per second without changing the model’s output distribution.

  • Structured, reliable output

    JSON schema, grammar and regex constraints, tool calling and stop sequences, with invalid requests refused rather than silently degraded.

  • Chat, embeddings and reranking together

    One server hosts chat, embedding and reranking models on one memory plane, loading and unloading them as demand changes.

  • Cost and telemetry on every call

    Each request carries its token counts, cost and timing, with metrics and traces sent to Panoptes.

  • Isolated and audited

    Each request is bound to its authenticated tenant down to the cache keys, and every call is written to a hash-chained audit ledger.

  • Small task models

    Compact models trained on your approved results take over the tasks you run most, built with Forge and served by the Inference Server.

Everything Inference Server does, by area.

Serving

  • Continuous in-flight batching on one engine loop per replica
  • Paged KV cache with copy-on-write prefix sharing
  • Radix prefix cache and session cache persisted to MNEMOS
  • Chunked prefill and memory- and SLO-aware admission control
  • Tiered preemption and per-request priority
  • Multi-model residency with load, eject, keep-alive and idle unload
  • Multi-LoRA adapters selected per request
  • Batch-invariant deterministic mode for audit replay

Models and formats

  • GGUF files served directly
  • Safetensors models converted to Inovagio’s own format
  • Weight quantization from 4-bit to 8-bit
  • KV cache in f16, 8-bit and 4-bit, with mixed precision per layer
  • Mixture-of-experts, sliding-window and hybrid linear-attention architectures
  • Vision and multimodal input
  • Speech-to-text and text-to-speech models for voice interviews, on your servers
  • Thinking-mode and reasoning-effort control per request

APIs

  • Native Inovagio API for generation, embeddings, reranking and model control
  • Streaming with server-sent events
  • Reranking endpoint
  • Tool and function calling with per-model parsers
  • Model Context Protocol (MCP) tool execution
  • Health and metrics endpoints

Scale and trust

  • Metal, CUDA, Vulkan and CPU backends from one codebase
  • Tensor, pipeline and expert parallelism across GPUs
  • Horizontal scale-out with prefix-aware load balancing and graceful drain
  • Authenticated tenant bound to every cache key, fail-closed
  • Hash-chained audit ledger of every conversation
  • Signed and encrypted model packages

What Inference Server replaces

Inference Server does the work of these, inside one platform with one identity, one policy and one record:

  • Separate inference servers for chat, embeddings and reranking
  • Hosted model APIs for workloads that must stay private
  • Model routing and gateway layers
  • Standalone embedding and reranking services
  • Tokenizer libraries and services

The controls Inference Server brings

Every Inovagio product keeps the same five commitments, because regulated work needs all five at once.

  • Security

    Prompts stay on your hardware unless your policy routes a task to a frontier model you chose; tenant isolation extends to every cache key.

  • Governance

    Model access and routing follow your policy.

  • On the record

    Each call is written to the audit ledger with its model and cost.

  • Monitoring

    Tokens, latency and queue depth for every model, sent to Panoptes.

  • Cost per task

    Batching and caching cut the work each agent step repeats, and the cost of every task is recorded.

Inference Server in your industry

Inference Server runs the same way in every industry Inovagio serves. See the work in yours.

For developers: Inference Server at a glance

Interfaces
Native Inovagio API over REST with server-sent-event streaming, and gRPC
Endpoints
Generate, embed, rerank, models (load / eject / keep), health, metrics
Backends
Metal, CUDA, Vulkan and CPU
Platforms
macOS (Apple silicon), Linux and Windows servers
Model formats
GGUF (served directly), safetensors (converted)
Quantization
4-bit to 8-bit weights; 16-, 8- and 4-bit KV cache
Deployment
One inovagio app on a laptop or workstation, a private server, or multi-GPU cloud nodes
Standards
OpenTelemetry, Model Context Protocol

Request the developer editionSee the architecture

Inference Server in the platform.

Where Inference Server fits

One platform
  1. Agent HarnessAgent harness

    Inovagio CLIInovagio DesktopWeb ChatInovagio SDK

  2. OpsisExperience layer

    FolioMorphe

  3. Virtual EmployeesVirtual Employee layer

    NexusErgon

  4. SynesisKnowledge layer

    PerceptionMNEMOS

  5. Origin (this layer)Model layer

    Inference Server (this product)Forge

  6. PhylaxTrust layer

    AuthInovagio SecretsGuard

  7. OperationsOperations

    PanoptesArchon

  8. Platform servicesPlatform services

    ServerAPI ManagementConnect

Where Inference Server fits

Further reading

Guides and posts on Inference Server

See Inference Server on one of your workflows.

Every project starts with one workflow your people know well, measured against what it costs you today.