Series · Knowledge layer deep dives
Data quality checks for AI agents in automated processes
The checks a record must pass before an AI agent may act on it, and the monitoring that catches what slips through. Shapes, business rules, freshness limits, confidence and quarantine, drift, and how a bad record is traced and fixed at the source, with the audit record each step leaves.
Inovagio team · Published · Updated · 13 min read

On this page
Data quality checks for AI agents are the tests a record must pass before an agent is allowed to act on it, plus the monitoring that catches what gets past them. There are five kinds: shape, business rules, freshness and confidence on each record, and completeness on each feed, with a quarantine for any record that fails a blocking check. They run on every record, every time, because an agent acts at machine speed on whatever it is given.
Preparing data once isn't enough. Our guide to preparing data for a knowledge layer gets one queue's sources clean and under contract. From then on, new records arrive every hour, sources change their formats without telling anyone, and someone saves a newer version of a policy next to the old one. Keeping that data fit while agents work on it takes checks on every record, a place for failures to wait and someone watching the feeds.
The feed matters as much as the record
#The most instructive case we know has nothing to do with AI. In 2016 Metro Bank switched on automated transaction monitoring. A data-feed error meant some transactions never reached the monitoring system: those on the day an account was opened, and later ones until the account record updated. The UK Financial Conduct Authority found that between June 2016 and December 2020 Metro lacked the systems and controls to adequately monitor over 60 million transactions for money laundering risk. Junior staff raised concerns in 2017 and 2018, the bank identified the problem in April 2019 and fixed it that July, and a check that all relevant transactions actually reached the system only arrived in December 2020 (FCA, November 12, 2024).
The monitoring rules were not the problem. Records that never arrive can't fail a check. That's the first lesson for agents: check the completeness of the feed, not only the quality of the records in it. Count what the source says it sent and what arrived. Stop when the two disagree.
Where data quality checks sit in an AI agent's work
#There are three places to check, and each catches something the others can't.
- Before the agent acts, on the records it is about to read. Most of what follows is about this gate.
- After the agent produces its work, against the business rules the work must satisfy. Our post on business rules and AI guardrails covers that check and how it differs from filtering what a model says.
- Continuously, on the feeds themselves and on stored data, to catch drift that no single record shows, such as a source that quietly stops sending one kind of record.
Data engineers have a name for the first pattern: write, audit, publish. Data lands in a staging area and is checked there. Only then does it become visible to the people and agents downstream. Synesis, our knowledge layer, applies it to knowledge: anything new lands as a candidate in a change set, the change set is validated before it's approved, and by default Virtual Employees read only what has been published. For the records an agent reads, the same idea becomes a gate with four layers.
At Pellmoor Savings Bank (Example company), a community bank, an AI agent prepares commercial loan renewal files and the credit team decides every loan. Before the agent reads a borrower's financial statement, the statement has to pass all four layers below.
Check the shape
#Shape checks ask whether a record is well formed. Are the fields there, with the right types and the required values? For tables and messages the check is a schema, ideally the one in the source's data contract. The Open Data Contract Standard (ODCS) puts quality rules beside the schema and gives each a severity. The standard only records it; your gate decides what it means, typically blocking on severity: error and letting a warning through with a flag (ODCS releases).
For the knowledge graph the standard is SHACL, the W3C Shapes Constraint Language (W3C, July 20, 2017). A shape for an extracted financial statement at Pellmoor looks like this:
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix pmo: <https://example.com/pellmoor/ontology/> .
@prefix pms: <https://example.com/pellmoor/shapes/> .
pms:FinancialStatementShape a sh:NodeShape ;
sh:targetClass pmo:FinancialStatement ;
sh:property [ sh:path pmo:borrower ; sh:class pmo:Borrower ; sh:minCount 1 ; sh:maxCount 1 ] ;
sh:property [ sh:path pmo:fiscalYearEnd ; sh:datatype xsd:date ; sh:minCount 1 ; sh:maxCount 1 ] ;
sh:property [ sh:path pmo:sourceDocument ; sh:minCount 1 ;
sh:message "Every statement must point to the page it came from"@en ] ;
sh:property [ sh:path pmo:extractionConfidence ; sh:datatype xsd:decimal ;
sh:minInclusive 0.90 ; sh:severity sh:Warning ;
sh:message "Low OCR confidence: send to a person before use"@en ] .Two details are worth copying. Every statement must point to its source document, so an agent can never cite a figure it can't trace. And the confidence rule is a warning, not a violation. A low-confidence figure isn't wrong; it needs a person to look. One trap: a SHACL report says the data doesn't conform when it holds any result at all, warnings included, so the gate has to read each result's severity rather than the overall verdict. SHACL 1.2 is a Working Draft and adds features worth watching, but the 2017 Recommendation already covers everything here. Synesis validates against SHACL shapes over the inferred graph, so a statement isn't reported missing when the ontology already entails it. A constraint it doesn't support fails loudly instead of passing, and a version can't publish while any shape fails.
Check the business rules
#Shape says a record is well formed. Business rules say it is usable for this piece of work. At Pellmoor, a covenant test needs statements for the fiscal year the credit policy names, and the policy says a statement older than a set number of months can't be used. Those are business rules, not data engineering choices, so they're written down and approved by whoever your organization makes responsible, with a version, like any other rule. How a rule is approved walks through that process, and the rule's version lands in the record of every task that applied it.
Keep the two layers apart. When a shape check fails, the data is broken and its owner fixes it. When a business rule fails, the data may be fine and the case just isn't ready yet, which is a message for the analyst, not the data team.
Check freshness
#Every source in a contract has a freshness limit: how old its data may be before an agent stops trusting it. The ODCS service-level properties include latency and frequency for exactly this. Pellmoor's borrower table refreshes nightly, so a renewal agent that finds it more than a day old holds its work.
Some data doesn't go stale on a timer. It needs someone to confirm it is still true. Gartner's January 2026 release on zero-trust data governance recommends active metadata that alerts when data is stale or needs recertification, and predicts that by 2028, 50% of organizations will implement a zero-trust posture for data governance (Gartner, January 21, 2026). An approved-appraiser list is a good example. Nothing in it changes on its own, but someone you name should confirm it each quarter, and an agent should treat an unconfirmed list as stale.
Check confidence, and quarantine what fails
#Anything extracted by a model, such as text read by optical character recognition (OCR) or a figure lifted from a scanned statement, carries a confidence. Perception keeps coordinates and confidence on the text it extracts, so the check can be specific: a debt service figure below the threshold goes to a person with the page and the highlighted region, and the rest of the statement goes on.
A threshold is only as good as the score behind it. A score of 0.9 should mean the value is right about nine times in ten, and raw model scores rarely behave that way; a model's own statement of how sure it is is worse. In Synesis no extracted value is accepted on its score until that kind of value has been calibrated against a random audit of items people checked. Until then, every one goes to review.
Not every failure deserves the same response. Decide per check, and write the decision into the contract:
| Response | When to use it | Pellmoor example |
|---|---|---|
| Block | The agent would act wrongly on this record | A statement with no borrower, or a borrower with no tax ID |
| Warn | The record is usable but a person should look | A figure read with low OCR confidence |
| Sample | The check is expensive or the risk is low | A person reads one extracted statement in fifty |
A blocked record goes to a dead-letter queue: a holding queue for records that failed, kept with everything needed to fix and replay them. Two rules make one work. Separate transient failures, such as a source that timed out, from permanent ones, such as a missing field: retry the first with backoff and send the second to a person. And make replay idempotent, so a record fixed and sent through twice doesn't produce two renewal files.
In our platform, Synesis ingestion is governed and retryable with dead-letter lanes, and MNEMOS provides durable queues with dead-letter handling and an idempotent task API. The dead-letter lane is only one exit. Content with no department scope is rejected, a fact that contradicts approved knowledge goes to review as a conflict card, and only an item the pipeline can't process lands in the dead-letter lane. Every item is identified by its source and a hash of its content, so a file sent twice changes nothing. A record that fails a check is never dropped silently or let through.
One statement through the gate
#A scanned 2025 financial statement for Harlan & Pike Holdings, a Pellmoor borrower, arrives from the shared drive. The shape check passes: it names one borrower, has a fiscal year end and points to its source file. The business rule passes too, because it is the fiscal year the current credit policy asks for. The freshness check looks at the borrower table it joins to, refreshed six hours earlier, and passes.
The confidence check doesn't. The debt service figure on page 4 was read at a confidence well under the threshold, so it goes to an analyst with the page and the highlighted region. The rest of the statement moves on, but the covenant test needs that figure, so it waits. The analyst sees that a fax has turned a 7 into a 1 and corrects it. Only then does the renewal agent test the covenant, and the task record shows the hold, the correction and who made it.
Drift and monitoring
#Some problems hide. They only show across many records: a field that slowly fills with a new code, a source whose volume halves overnight, a document collection where the share of scanned pages doubles. The NIST AI Risk Management Framework warns that data "can change over time, sometimes significantly and unexpectedly", and its MEASURE 2.4 subcategory asks that a system's behavior be monitored once it is deployed (NIST AI 100-1, January 26, 2023). GOVERN 6.2 adds contingency processes for failures in third-party data or AI systems deemed high-risk, and a feed you buy can be one.
Practitioners usually watch five signals per source: freshness, volume, schema changes, the distribution of values and lineage (which jobs feed which tables). Forrester's 2026 review of data quality tools put the shift bluntly.
Its point was that static validation is giving way to real-time monitoring (Forrester, March 29, 2026). For agents that means alerting on the feed before an agent reads from it, not after a week of wrong work.
Panoptes, our observability product, shows failures and performance by operation, keeps alert rules with their history and puts approvals and safety blocks beside the operational data. Phylax runs continuous control monitoring and anomaly detection on every Virtual Employee, so a sudden change in what one is reading or doing raises an alert. When the problem is serious, a Phylax kill rule can pull the same stop a person would, for one Virtual Employee, a department or everything at once; how to turn off AI agents covers when to use each level. Schema changes get caught closer to the source: a Synesis binding to another system's table checks a fingerprint of that table's schema before every use, so a dropped or retyped column fails loudly.
When a bad record gets through
#Some will. What matters is that each one is traced and fixed once, then stays fixed.
Start from the task. Every task leaves a record of what it read and of the checks and rule version that applied; read a task record shows how to follow it. That tells you which record was wrong and which version of it the agent saw. Fix it at the source, never in the agent's copy. The Consumer Financial Protection Bureau's 2022 order against Hyundai Capital America is the warning here: its monthly automated updates "reintroduced the data error after it had been disputed and corrected" (CFPB, July 26, 2022).
Then keep the lesson. If a check should have caught it, tighten that check or write a new one. The right person approves the change, as with any other rule. Replay the tasks that read the bad record, and confirm the check now holds. Our post on how a mistake fixed once stays fixed covers the same loop from the business side.
What the audit record should show
#For each task an auditor should be able to see these four things without asking an engineer:
- The records the agent read, with their versions
- Each check that ran, and its result
- Any record that was held, and who released it
- The rule version that decided the outcome
In our platform, MNEMOS keeps a data log alongside a write-once audit log, and Phylax keeps a hash-chained audit trail that can be replayed for an auditor. The trust page describes both.
Metrics worth a weekly look
#| Metric | What it tells you |
|---|---|
| Pass rate per check, per source | Which source is slipping, before an agent notices |
| Records arrived against records sent | Whether the feed is complete, the Metro Bank lesson |
| Records held per week, and time to release | Whether reviewers keep up with the queue |
| Share of fixes made at the source | Whether fixes will survive the next feed |
| Agent actions held by data checks | How often the gate saved a wrong action |
| Freshness lag against each contract | Which sources are drifting past their limits |
Count the held actions in money, too. Every record the gate stops is a task nobody has to unwind later, and counting only the tasks done hides that rework. Our guide to measuring cost per correct task shows how to count it.
Before agents start on a new queue, run the AI-ready data checklist; its run-time section is the short version of this guide.
Further reading
#- FCA press release on Metro Bank's financial crime failings, FCA, November 12, 2024: fca.org.uk
- Shapes Constraint Language (SHACL), W3C, July 20, 2017: w3.org
- Open Data Contract Standard, Bitol at LF AI & Data, v3.2.0, September 8, 2026: github.com
- Gartner Predicts by 2028, 50% of Organizations Will Adopt Zero-Trust Data Governance, Gartner, January 21, 2026: gartner.com
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 26, 2023: nist.gov
- The Forrester Wave: Data Quality Solutions, Q1 2026, Forrester, March 29, 2026: forrester.com
- CFPB order against Hyundai Capital America for credit reporting failures, CFPB, July 26, 2022: consumerfinance.gov
Questions
What people ask.
What data quality checks should run before an AI agent acts?
Four layers: shape checks that the record is well formed, business rules that it is usable for this work, freshness limits on how old it may be, and confidence thresholds on anything a model extracted. A record that fails a blocking check goes to a dead-letter queue for a person instead of reaching the agent; a warning lets it through with a flag.
What is a dead-letter queue?
A dead-letter queue is a holding queue for records that failed processing. Each entry keeps the record, the check it failed, the source and contract version, the retry count and the owner who fixes it, so the record can be corrected and replayed without being lost or processed twice.
Should a failed data check always block the agent?
No. Decide per check. Block when the agent would act wrongly on the record, warn when the record is usable but a person should look, and sample when the check is expensive or the risk is low. Write the decision into the source's data contract so it is consistent and reviewable.
How is data drift different from a bad record?
A bad record fails a check on its own. Drift shows only across many records, such as a field slowly filling with a new code or a source whose volume suddenly drops. It is caught by monitoring freshness, volume, schema and value distributions for each source over time.
What should happen when a bad record gets through to an agent?
Trace the task to the exact record and version the agent read, fix the record in its source system rather than in a copy, add or tighten the check that should have caught it with the right person's approval, and replay the tasks that read the bad record.
How do SHACL shapes help AI agents?
SHACL shapes describe what valid data in a knowledge graph looks like, such as required properties, value types and allowed ranges, and report each violation with a message. An agent that receives a violation can stop and say which rule the data broke instead of guessing.
The products, industries and examples it covers.
- Governance and auditPhylaxSign-in, policy, secrets and one audit trail from the screen to the disk, so you can show any examiner what happened.
- Observability and analyticsPanoptesSee every log, trace, cost and decision across the platform.
- Knowledge layerSynesisTurns your policies, procedures, records and your people’s know-how into a rulebook your experts approve, which you own and can take anywhere.
- Data engineMNEMOSRecords, documents, vectors, graph and agent memory in one engine, with a ledger behind every write.
- IndustryFinancial servicesCredit, account and claims files checked against your policy, and answers to exam requests drafted from records on file. Your compliance officer approves every response.
- IndustryInsurance servicing and reconciliationEach program’s bordereau matched to your policy system, the MGA’s cash matched to what is due, and every line checked against the delegated authority. Your team approves every write-off.
- IndustryHealth plansEnrollment and premium files reconciled line by line, directory entries verified on schedule, and appeal case files tracked on their clocks. A physician decides every medical appeal.
- OverviewTrust and governanceEvery action checked, every step recorded.
- OverviewDevelopersBuild agents, workflows and apps, already governed.
- OverviewHow it worksTen checked steps from your first decision to the day it handles real cases.
Read next.
Guide · Data qualityPreparing data for a knowledge layer: a step-by-step guideHow to get data ready for a knowledge layer and the AI agents that read it, in the order that works. Scope by the first queue, profile against named quality dimensions, name owners, resolve entities reversibly, clean documents, record lineage, map to the ontology and put a contract on every source. · updated November 18, 2026 · 17 min read
Guide · GovernanceHow an AI business rule is written and approvedEvery Virtual Employee works from rules approved one at a time by the people you make responsible. What a rule contains, who signs it, how a correction becomes a new version, and how any change is rolled back. · updated November 18, 2026 · 9 min read
Article · GovernanceBusiness rules or AI guardrails? Why regulated work needs bothGuardrails filter what an AI model takes in and says. Business rules check what the work itself must satisfy. In regulated work you need both, plus a model's judgment for the messy parts. What each one catches, a worked example, and what to ask a vendor. · updated November 18, 2026 · 12 min read
Developer edition
Build on Inovagio.
The MNEMOS data engine, the command line and the desktop app, with guides and API references.




