Series · Knowledge layer deep dives
Preparing data for a knowledge layer: a step-by-step guide
How to get data ready for a knowledge layer and the AI agents that read it, in the order that works. Scope by the first queue, profile against named quality dimensions, name owners, resolve entities reversibly, clean documents, record lineage, map to the ontology and put a contract on every source.
Inovagio team · Published · Updated · 17 min read

On this page
To prepare data for AI agents and the knowledge layer behind them, start from the records and documents your first AI work will read, make each one owned and traceable, then map it to the ontology your agents use. The work is scoped by one queue and a short list of questions, never a cleanup of everything you own.
Teams often start the other way. Someone commissions a data quality program for the whole estate, it produces a dashboard of red cells in its first quarter, and the AI project waits behind it for a year. The teams that get to working agents start from one queue, find the handful of sources it touches, and fix those at the source, with a named person accountable for each. The seven steps below follow that order. Each comes with the standard behind it, and one community bank runs through all of them as the worked example. The plain-English version is our post on data hygiene for AI, and data cleanup before AI puts it on a 90-day calendar.
Why AI-ready data is defined per use case
#Gartner's February 2025 release states the problem well. In a survey of 1,203 data management leaders, 63% of organizations either did not have or were unsure if they had the right data management practices for AI. Gartner predicted that through 2026, organizations would abandon 60% of AI projects unsupported by AI-ready data (Gartner, February 26, 2025). That window has nearly closed, so read the number as the forecast it was. The method in the same release has aged better: its first step is to align data with the use case, and it treats readiness as something proven per use case and kept up, not signed off once. Its test is blunt: "if the data has issues, then the data is not ready for AI."
That matters more for agents than for reports. A dashboard with a bad number misleads a reader, who may notice. An agent with a bad record acts on it. It sends the letter, or tests a covenant against last year's ratios. Gartner has since argued that AI-ready data has to expand to "agent-ready data", where leaders also check that agents use data appropriately (Gartner research note, August 27, 2026).
Forrester's Heidi Shey makes the same case from the security side. Agents amplify the weaknesses you already have: unclear classification, uncertain ownership, inconsistent permissions, incomplete lineage, questionable quality. She sees organizations putting consistent classification and better inventories first, before they move forward (Forrester, November 2, 2026). Each of those weaknesses has a step below.
The worked example
#Pellmoor Savings Bank (Example company) is a community bank with a commercial lending book. Its first queue is renewal file preparation: for every commercial loan maturing in the next 120 days, gather the borrower's and guarantors' financial statements, test the covenants under the credit policy in force, check the collateral appraisal and hand an analyst a complete file. The credit team decides every loan; the AI prepares the file. Our lending page shows the same work done by a Virtual Employee, and pick your first queue explains why a queue like this makes a good start.
The renewal analyst named four things every file must answer. They became the competency questions for the knowledge layer, and the ontology guide explains why every ontology should start from such a list.
| # | Competency question | Sources it touches |
|---|---|---|
| 1 | Which commercial loans mature in the next 120 days, and who are the borrowers and guarantors on each? | Core banking, loan origination, CRM |
| 2 | Which financial statements are on file for each of them, for which fiscal year, and which are missing? | Shared drive, loan origination |
| 3 | Which covenants apply under the credit policy version in force, and were they met at the last test? | Credit policy documents, loan origination |
| 4 | Is the collateral appraisal current under policy, and was it done by an approved appraiser? | Appraisal files, vendor master |
Every step below is scoped by those four. The rest can wait.
How to prepare data for AI agents, in order
#Each step needs the one before it. You can't resolve entities without stable identifiers, and you can't fix a source nobody owns. Mapping to an ontology before profiling only moves the mess into a graph.
1. Inventory the sources the questions touch
#For each question, list every system that holds part of the answer. The first two questions alone touched four systems at Pellmoor:
| System | Record or copy | Refresh | Owner |
|---|---|---|---|
| Core banking | Record for balances and maturity dates | Nightly | Deposit and loan operations |
| Loan origination | Record for borrowers, guarantors and covenants | On change | Head of commercial lending |
| CRM | Copy, edited by relationship managers | Hourly sync | Sales operations |
| Shared drive of borrower files | Copies of statements and appraisals | By hand | Nobody, at the start |
The record-or-copy column is the one people skip, and it decides where fixes go. In 2022 the Consumer Financial Protection Bureau ordered Hyundai Capital America to pay redress and a penalty for furnishing inaccurate credit information on more than 2.2 million consumer accounts. Its furnishing system often overrode staff corrections, sending monthly updates that "reintroduced the data error after it had been disputed and corrected" (CFPB, July 26, 2022). A fix made in a copy is undone by the next feed. Fix the record.
2. Profile each source against named dimensions
#Profiling means measuring a source before you trust it, using dimensions that two teams will read the same way. The six from the DAMA UK working group, adopted by the UK Government Data Quality Hub, are accuracy, completeness, uniqueness, consistency, timeliness and validity (UK Government Data Quality Hub, June 24, 2021). The formal model is ISO/IEC 25012:2008. It defines 15 characteristics seen from two points of view: inherent to the data (accuracy is one), dependent on the system holding it (availability is one), and several, such as confidentiality and traceability, that count as both. ISO last reviewed and confirmed the 2008 edition in 2025 (ISO), and ISO/IEC 25024 supplies the measures for each characteristic. The 2024 maintenance release of DAMA-DMBOK 2 adds currency as a recognized dimension (DAMA International).
Profile only the fields the questions need. For Pellmoor's borrower table that was one query:
-- Completeness, uniqueness and validity of the fields question 1 needs
SELECT
COUNT(*) AS borrowers,
COUNT(*) FILTER (WHERE tax_id IS NULL) AS missing_tax_id,
COUNT(tax_id) - COUNT(DISTINCT tax_id) AS rows_repeating_tax_id,
COUNT(*) FILTER (WHERE tax_id NOT SIMILAR TO '[0-9]{2}-[0-9]{7}') AS malformed_tax_id,
MAX(updated_at) AS last_refresh
FROM los.commercial_borrower
WHERE status = 'ACTIVE';It returned 2,316 active borrowers, 141 with no tax ID and 63 rows that repeated a tax ID already on another row. Those numbers became the cleanup list, and nothing else in the table was touched.
Keep the profile with the source, not in a one-off spreadsheet, and rerun it when the source changes. In Synesis, our knowledge layer, profiling is part of harvesting a database. It reads a sample under the caller's own permissions and stores only statistics, and a column's list of values has to pass the sensitivity classifier before it can be proposed as vocabulary, so customer data never turns into terms by accident.
3. Name an owner and an approver for each domain
#Every source needs an owner who answers for it. Every domain needs someone to approve what its terms mean, and you decide who; at Pellmoor, borrowers are one domain and credit policy another. This is the step that decides whether the rest lasts. Gartner's September 2026 survey of data and analytics leaders found cultural resistance, ahead of funding, was the main reason governance programs fail, and its advice was shared business and technology accountability (Gartner, September 21, 2026). Zhamak Dehghani's data mesh essay gives the working version: each domain owns its data as a product and states the integrity it offers as service-level objectives (martinfowler.com, May 20, 2019).
At Pellmoor the shared drive had no owner, which is why it had three copies of some statements. The credit administration manager took it on. Synesis lets you set who approves each stage and each class of rule. Many organizations have a department head approve their own terms and a governance body or compliance approve company-wide or regulated rules, and you can require two approvers for the most sensitive ones. Agents propose and never approve, and nobody approves their own change. How a rule is approved shows the same flow for rules.
4. Give every entity one identity
#Start with reference data. If the loan system says CRE-OO and the core says COMM RE OWNER OCC, map both to one approved term now, in the vocabulary, not in every query later. Keep each system's code on the concept as a SKOS notation, so the crosswalk is data someone owns rather than a CASE statement copied into every report. Our post on the five stages from vocabulary to knowledge graph explains why one approved term per concept comes first.
Then fix identifiers. The FAIR principles ask for "globally unique and persistent" identifiers (GO FAIR, from Wilkinson et al., Scientific Data, March 15, 2016). In practice that means an IRI (Internationalized Resource Identifier, the web-style name a knowledge graph gives each thing) built from a stable key: a loan number, or a party ID minted once and kept across systems. Never a row ID that changes on migration, never a name, and never a tax ID. The tax ID is restricted data, and an IRI gets copied into every link, log and cache. Pellmoor's IRIs follow the pattern in the ontology guide: instances under https://example.com/pellmoor/id/, ontology terms in a namespace of their own. For master data exchanged with other parties, ISO 8000-115:2024 sets the requirements for quality identifiers, including how an identifier resolves to the data it stands for (ISO).
Then resolve entities, the step teams most often underestimate, and make every merge reversible from day one. Resolving means finding the records that describe the same real-world thing. A widely cited survey splits it into blocking (narrowing the candidate pairs), matching (scoring each pair) and clustering (deciding which records form one entity) (Christophides et al., ACM Computing Surveys, December 2020). The probabilistic model most matching tools still offer is Fellegi and Sunter's 1969 theory of record linkage. It weighs each field's agreement by how informative it is and sets two thresholds; pairs that fall between them go to a person (Fellegi and Sunter, Journal of the American Statistical Association, December 1969).
"Harlan & Pike Holdings LLC" in Pellmoor's loan system and "HARLAN AND PIKE HOLDINGS" in the core share a tax ID. That's a match. Two guarantors, "Dale Oberlin" and "Dale Oberlin Jr.", share a street address and a phone number but have different dates of birth. A matcher that weights name and address would merge them, and the son's file would carry his father's personal guaranty.
A missed match costs a duplicate row. A false merge puts one person's obligations on another, and every agent that reads the record acts on it. Clustering makes it worse, because transitive links can chain four near-matches into one oversized entity.
So match on strong identifiers first, and block and score only what they leave. Treat facts that can't both be true, such as two different dates of birth, as a hard "never the same" that no score can outweigh. Send the middle band to the right person with the evidence that scored it. And record each merge as a link that can be undone, with no golden record overwriting either source.
Synesis does entity resolution this way. It scores pairs with Fellegi and Sunter's model and sets the two thresholds from error rates measured on an audit sample. Conflicting values for a property that can have only one, like the Oberlin dates of birth, are a cannot-link the clustering never crosses. An approved merge is applied as an owl:sameAs link and leaves the source records untouched, so an unmerge withdraws exactly what depended on it. A merge of two people is never automatic: a person approves it, and you can require a second approver.
5. Clean documents, and classify what's sensitive
#Most of a renewal file lives in documents: tax returns, financial statements, appraisals, the credit policy itself. Barnett and colleagues catalogued seven points where retrieval over documents fails, from missing content to incomplete answers, and found that adding the file name and chunk number to retrieved passages helped (Barnett et al., arXiv 2401.05856, January 11, 2024). Scans fail in their own way, at the optical character recognition (OCR) step that turns a page image into text. The OHRBench study built 8,561 document images across seven domains and found the OCR it tested was not adequate for building high-quality retrieval knowledge bases (Zhang et al., arXiv 2412.02592, ICCV 2025). Pellmoor's 2019 tax returns were scans of faxes.
| Check | Why it matters at Pellmoor |
|---|---|
| One current version, with its effective date | Credit policy v6 sat beside v7; an agent that found v6 would test last year's ratios |
| Near-duplicates removed before indexing | Three copies of one statement crowd better passages out of the results |
| OCR checked on a sample a person reads | Faxed returns misread a 7 as a 1 in debt service figures |
| Owner, date, classification and source on every passage | The agent can cite the page, and access can follow the classification |
| Retention set | A statement past its retention date shouldn't be answering questions |
Classify before you ingest, not after. The guarantors' tax returns carry Social Security numbers, and nothing the renewal agent does needs them. OWASP's 2025 list for language model applications names vector and embedding weaknesses (LLM08) and calls for permission-aware retrieval (OWASP, 2025).
Perception, our ingestion product, reads born-digital files directly with their text positions and runs vision-language OCR only on scanned pages. It keeps coordinates and confidence on extracted text and masks personal data before storage. Our ingestion pipeline treats a source nobody has classified as restricted until someone does, and an extraction that fails its check never becomes a candidate fact. Every item is identified by its source and a hash of its content, so a re-sent file changes nothing and a new version supersedes the old one. MNEMOS, our data engine, trims every result to the documents the person an agent acts for could open, and a Virtual Employee never sees more than its own role allows either.
6. Record lineage, then map to the ontology
#An agent's answer is only as defensible as your ability to show where each fact came from. W3C PROV-O models provenance as entities produced by activities that agents are responsible for (W3C, April 30, 2013). OpenLineage, an LF AI & Data project, defines the events a pipeline emits for each run of a job and the datasets it read and wrote (OpenLineage). The two answer different questions. PROV-O says where a fact in the graph came from; OpenLineage says which job produced the table it sits in. For high-risk AI systems, Article 10 of the EU AI Act, as amended in July 2026 by Regulation (EU) 2026/1744, requires governance of the data used to train, validate and test them, covering how it was collected, where it came from and, for personal data, why it was originally collected (EU AI Act, Article 10). It's written for training and test data rather than what an agent reads at run time, but it's the bar an examiner will hold that data to.
With lineage in place, map the sources. R2RML, a W3C Recommendation since 2012, maps relational tables to the ontology's classes and properties, and the mapping can be queried in place or materialized into a graph (W3C, September 27, 2012). RML-Core extends the same structure to CSV, JSON and XML. It is a Draft Community Group Report of the W3C Knowledge Graph Construction Community Group, dated March 16, 2026, not a W3C Recommendation (RML-Core). Pellmoor's borrower table maps like this, with the IRI built from the party ID:
@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix pmo: <https://example.com/pellmoor/ontology/> .
@prefix pmm: <https://example.com/pellmoor/mapping/> .
pmm:BorrowerMap a rr:TriplesMap ;
rr:logicalTable [ rr:tableName "los.commercial_borrower" ] ;
rr:subjectMap [
rr:template "https://example.com/pellmoor/id/party/{party_id}" ;
rr:class pmo:Borrower ] ;
rr:predicateObjectMap [
rr:predicate pmo:legalName ;
rr:objectMap [ rr:column "legal_name" ] ] .The tax ID stays in the source for matching and never becomes part of the graph's identifiers. Validate the output with shapes written in SHACL, the W3C Shapes Constraint Language, run over the inferred graph so a record isn't flagged for a fact the ontology already entails. The ontology guide has a longer mapping and shapes for its accounts payable example.
In Synesis you rarely write the first mapping by hand. Harvesting starts from the W3C Direct Mapping of each table, then proposes refinements with their evidence: a junction table becomes a many-to-many relationship, keys come from functional dependencies found in the sample, and undeclared foreign keys from inclusion dependencies. The output is R2RML, and every class and property binding is a candidate for the right person to approve. Everything harvested or extracted lands in a change set, never straight in the graph. A shape violation blocks publication and shows up on a review card, a contradiction with approved knowledge becomes a conflict card, and an item the pipeline can't process at all goes to a dead-letter lane with its retries. Nothing disappears or skips review. Each import writes into a named graph of its own, and every approved fact keeps a PROV-O chain back to its table and column, or its document, page and character offsets.
7. Put a contract on every source
#A data contract is a written agreement between a source's owner and everyone who uses it, including agents. It states the schema, the quality rules and the service levels, and it names the owner. The Open Data Contract Standard (ODCS), maintained by the Bitol project at LF AI & Data, is the open format. Version 3.2.0, released September 8, 2026, added a context block written for AI agents. It tells an agent how to use the data and what it must never do, and it can carry approved answers to common questions (ODCS releases).
Part of Pellmoor's contract for the borrower table, in ODCS v3.2 syntax:
apiVersion: v3.2.0
kind: DataContract
id: pellmoor-commercial-borrower
version: 1.3.0
status: active
domain: commercial-lending
schema:
- name: commercial_borrower
physicalName: los.commercial_borrower
properties:
- name: tax_id
logicalType: string
required: true
unique: true
classification: restricted
quality:
- metric: nullValues
mustBe: 0
dimension: completeness
severity: error
context:
instructions: >
One row per legal entity. Match borrowers on tax_id, never on legal_name.
constraints:
- constraint: "Never treat two parties as one on name and address alone."
slaProperties:
- property: latency
value: 1
unit: d
team:
name: commercial-lendingTwo lines earn their place. severity: error marks the check as one that blocks: the standard records the severity, and your gate acts on it by holding the data instead of letting it reach an agent; data quality checks for AI agents covers what happens to it next. The constraint writes the Oberlin lesson down where every future agent will read it.
The contract states what the owner commits to. Someone downstream still has to notice when the source stops matching it. In Synesis, a binding to a table in another system carries a fingerprint of the schema it was written against and checks it before every use. A dropped or retyped column fails loudly there, before it can surface as a wrong answer.
Gartner predicts that by 2030, 50% of organizations will use autonomous AI agents to interpret governance policies and technical standards into machine-verifiable data contracts (Gartner, March 11, 2026). You don't need to wait. A contract per source, written by its owner, is a short job for the few sources one queue touches.
What done looks like for one queue
#For Pellmoor it meant this. The four questions run as queries and return the right files. Every source they touch is profiled and owned, and has a contract. Borrowers and guarantors are resolved, with every merge explainable. The credit policy exists at one current version, scans carry a confidence on every figure, and the Social Security numbers in guarantors' returns are masked. Every number in a renewal file traces to its page. Synesis holds the approved knowledge, stored in MNEMOS beside the records and their history; the architecture page shows how they fit. None of that work is locked to us. Your rulebook exports free, in open standards (SKOS, OWL and SHACL), and the R2RML mappings and ODCS contracts are open formats too.
It took one queue's worth of data, not the bank's, and most of the effort went into owners and identifiers rather than into any single tool. The next queue reuses the borrower resolution and the policy versions, and it starts at step 1 with its own questions. Before you start, run through the AI-ready data checklist; once agents are working the queue, data quality checks for AI agents keeps the data clean.
Further reading
#- Lack of AI-Ready Data Puts AI Projects at Risk, Gartner, February 26, 2025: gartner.com
- AI Agents Aren't Waiting For Data Security, Forrester, November 2, 2026: forrester.com
- Meet the data quality dimensions, UK Government Data Quality Hub, June 24, 2021: gov.uk
- ISO/IEC 25012:2008, Data quality model, ISO, confirmed 2025: iso.org
- An Overview of End-to-End Entity Resolution for Big Data, Christophides et al., ACM Computing Surveys, December 2020: arxiv.org
- A Theory for Record Linkage, Fellegi and Sunter, Journal of the American Statistical Association, December 1969: jstor.org
- Seven Failure Points When Engineering a Retrieval Augmented Generation System, Barnett et al., January 11, 2024: arxiv.org
- OCR Hinders RAG, Zhang et al., ICCV 2025: arxiv.org
- The FAIR Guiding Principles for scientific data management and stewardship, Wilkinson et al., Scientific Data, March 15, 2016: gofair.foundation
- PROV-O: The PROV Ontology, W3C, April 30, 2013: w3.org
- EU AI Act, Article 10: Data and data governance, as amended by Regulation (EU) 2026/1744: artificialintelligenceact.eu
- R2RML: RDB to RDF Mapping Language, W3C, September 27, 2012: w3.org
- RML-Core, Draft Community Group Report, W3C Knowledge Graph Construction Community Group, March 16, 2026: w3id.org
- Open Data Contract Standard, v3.2.0, Bitol at LF AI & Data, September 8, 2026: github.com
- CFPB order against Hyundai Capital America for credit reporting failures, CFPB, July 26, 2022: consumerfinance.gov
Questions
What people ask.
What does it mean to prepare data for AI?
Preparing data for AI means making the records and documents an AI system will read fit for that specific use: owned by someone, measured against named quality dimensions, identified consistently across systems, traceable to their source and protected where they are sensitive. It is defined per use case, so the work starts with the questions the first piece of AI work must answer.
Do we have to clean all our data before using AI agents?
No. Clean the sources the first queue's questions touch, and fix them at the system of record so the next feed doesn't undo the work. Everything else waits until a later queue needs it. Cleaning the whole estate first is one of the most common reasons AI projects stall.
What are the data quality dimensions?
The six most often used are accuracy, completeness, uniqueness, consistency, timeliness and validity, from the DAMA UK working group and adopted by the UK Government Data Quality Hub. ISO/IEC 25012 defines a fuller model of 15 characteristics, seen as inherent to the data, dependent on the system holding it, or both.
What is entity resolution, and why does it matter for AI?
Entity resolution finds the records in different systems that describe the same customer, company or asset and links them. It matters for AI because an agent acts on whatever record it is given. A missed match leaves a duplicate; a false merge puts one person's obligations on another, so merges should be explainable and reversible.
What is a data contract?
A data contract is a written agreement between the owner of a data source and its users, covering the schema, quality rules, service levels and ownership. The Open Data Contract Standard is the open format, and its version 3.2.0 added a context section with instructions and constraints written for AI agents.
Where does R2RML fit in data preparation?
R2RML is the W3C standard for mapping relational tables to the classes and properties of an ontology. It comes after profiling, ownership and identifiers are settled, so the knowledge graph is built from clean, owned sources instead of copying their problems into a new store.
The products, industries and examples it covers.
- Data ingestionPerceptionTurn documents, scans, email and system data into cited knowledge.
- Knowledge layerSynesisTurns your policies, procedures, records and your people’s know-how into a rulebook your experts approve, which you own and can take anywhere.
- Data engineMNEMOSRecords, documents, vectors, graph and agent memory in one engine, with a ledger behind every write.
- IndustryLending and account operationsBorrowers’ statements and tax returns spread, covenants tested and annual review memos drafted against your credit policy. Your credit team decides.
- IndustryFinancial servicesCredit, account and claims files checked against your policy, and answers to exam requests drafted from records on file. Your compliance officer approves every response.
- OverviewArchitectureHow the platform fits together, from your data to finished work.
- OverviewDevelopersBuild agents, workflows and apps, already governed.
- OverviewHow it worksTen checked steps from your first decision to the day it handles real cases.
Read next.
Guide · Data qualityAI-ready data checklist for regulated teamsTwenty-eight checks to run against the data an AI process will read, before it goes live and at set intervals after. Grouped by stage, from sources and owners to documents, mapping and run-time checks, each with why it matters to an examiner and how to test it. · updated November 18, 2026 · 8 min read
Guide · OntologyOntology for AI agents: how to build one in five stagesHow to build an ontology AI agents can use, in five gated stages from a controlled vocabulary to a knowledge graph. The W3C standards for each stage, competency questions, Turtle and SPARQL for a worked example, entity resolution, versioning and "as of" queries. · updated November 18, 2026 · 21 min read
Guide · Data qualityData quality checks for AI agents in automated processesThe checks a record must pass before an AI agent may act on it, and the monitoring that catches what slips through. Shapes, business rules, freshness limits, confidence and quarantine, drift, and how a bad record is traced and fixed at the source, with the audit record each step leaves. · updated November 18, 2026 · 13 min read
Developer edition
Build on Inovagio.
The MNEMOS data engine, the command line and the desktop app, with guides and API references.






