Skip to content
InovagioInovagio

Series · Knowledge layer deep dives

Preparing data for a knowledge layer: a step-by-step guide

How to get data ready for a knowledge layer and the AI agents that read it, in the order that works. Scope by the first queue, profile against named quality dimensions, name owners, resolve entities reversibly, clean documents, record lineage, map to the ontology and put a contract on every source.

· Published · Updated · 17 min read

On this page

To prepare data for AI agents and the knowledge layer behind them, start from the records and documents your first AI work will read, make each one owned and traceable, then map it to the ontology your agents use. The work is scoped by one queue and a short list of questions, never a cleanup of everything you own.

Teams often start the other way. Someone commissions a data quality program for the whole estate, it produces a dashboard of red cells in its first quarter, and the AI project waits behind it for a year. The teams that get to working agents start from one queue, find the handful of sources it touches, and fix those at the source, with a named person accountable for each. The seven steps below follow that order. Each comes with the standard behind it, and one community bank runs through all of them as the worked example. The plain-English version is our post on data hygiene for AI, and data cleanup before AI puts it on a 90-day calendar.

Why AI-ready data is defined per use case

#

Gartner's February 2025 release states the problem well. In a survey of 1,203 data management leaders, 63% of organizations either did not have or were unsure if they had the right data management practices for AI. Gartner predicted that through 2026, organizations would abandon 60% of AI projects unsupported by AI-ready data (Gartner, February 26, 2025). That window has nearly closed, so read the number as the forecast it was. The method in the same release has aged better: its first step is to align data with the use case, and it treats readiness as something proven per use case and kept up, not signed off once. Its test is blunt: "if the data has issues, then the data is not ready for AI."

Gartner's line that data with issues is not ready for AI
Source: Gartner press release, February 26, 2025.

That matters more for agents than for reports. A dashboard with a bad number misleads a reader, who may notice. An agent with a bad record acts on it. It sends the letter, or tests a covenant against last year's ratios. Gartner has since argued that AI-ready data has to expand to "agent-ready data", where leaders also check that agents use data appropriately (Gartner research note, August 27, 2026).

Forrester's Heidi Shey makes the same case from the security side. Agents amplify the weaknesses you already have: unclear classification, uncertain ownership, inconsistent permissions, incomplete lineage, questionable quality. She sees organizations putting consistent classification and better inventories first, before they move forward (Forrester, November 2, 2026). Each of those weaknesses has a step below.

The worked example

#

Pellmoor Savings Bank (Example company) is a community bank with a commercial lending book. Its first queue is renewal file preparation: for every commercial loan maturing in the next 120 days, gather the borrower's and guarantors' financial statements, test the covenants under the credit policy in force, check the collateral appraisal and hand an analyst a complete file. The credit team decides every loan; the AI prepares the file. Our lending page shows the same work done by a Virtual Employee, and pick your first queue explains why a queue like this makes a good start.

The renewal analyst named four things every file must answer. They became the competency questions for the knowledge layer, and the ontology guide explains why every ontology should start from such a list.

#Competency questionSources it touches
1Which commercial loans mature in the next 120 days, and who are the borrowers and guarantors on each?Core banking, loan origination, CRM
2Which financial statements are on file for each of them, for which fiscal year, and which are missing?Shared drive, loan origination
3Which covenants apply under the credit policy version in force, and were they met at the last test?Credit policy documents, loan origination
4Is the collateral appraisal current under policy, and was it done by an approved appraiser?Appraisal files, vendor master

Every step below is scoped by those four. The rest can wait.

How to prepare data for AI agents, in order

#
The seven steps to prepare data for AI agents and a knowledge layer, climbing from an inventory of sources to a contract on each one
The order of work for one queue, from sources to contracts.

Each step needs the one before it. You can't resolve entities without stable identifiers, and you can't fix a source nobody owns. Mapping to an ontology before profiling only moves the mess into a graph.

1. Inventory the sources the questions touch

#

For each question, list every system that holds part of the answer. The first two questions alone touched four systems at Pellmoor:

SystemRecord or copyRefreshOwner
Core bankingRecord for balances and maturity datesNightlyDeposit and loan operations
Loan originationRecord for borrowers, guarantors and covenantsOn changeHead of commercial lending
CRMCopy, edited by relationship managersHourly syncSales operations
Shared drive of borrower filesCopies of statements and appraisalsBy handNobody, at the start

The record-or-copy column is the one people skip, and it decides where fixes go. In 2022 the Consumer Financial Protection Bureau ordered Hyundai Capital America to pay redress and a penalty for furnishing inaccurate credit information on more than 2.2 million consumer accounts. Its furnishing system often overrode staff corrections, sending monthly updates that "reintroduced the data error after it had been disputed and corrected" (CFPB, July 26, 2022). A fix made in a copy is undone by the next feed. Fix the record.

2. Profile each source against named dimensions

#

Profiling means measuring a source before you trust it, using dimensions that two teams will read the same way. The six from the DAMA UK working group, adopted by the UK Government Data Quality Hub, are accuracy, completeness, uniqueness, consistency, timeliness and validity (UK Government Data Quality Hub, June 24, 2021). The formal model is ISO/IEC 25012:2008. It defines 15 characteristics seen from two points of view: inherent to the data (accuracy is one), dependent on the system holding it (availability is one), and several, such as confidentiality and traceability, that count as both. ISO last reviewed and confirmed the 2008 edition in 2025 (ISO), and ISO/IEC 25024 supplies the measures for each characteristic. The 2024 maintenance release of DAMA-DMBOK 2 adds currency as a recognized dimension (DAMA International).

The six DAMA UK data quality dimensions, each written as the question a profile asks of one borrower table
Pellmoor Savings Bank (Example company): each dimension becomes a question about the borrower table.

Profile only the fields the questions need. For Pellmoor's borrower table that was one query:

sql
-- Completeness, uniqueness and validity of the fields question 1 needs
SELECT
  COUNT(*)                                              AS borrowers,
  COUNT(*) FILTER (WHERE tax_id IS NULL)                AS missing_tax_id,
  COUNT(tax_id) - COUNT(DISTINCT tax_id)                AS rows_repeating_tax_id,
  COUNT(*) FILTER (WHERE tax_id NOT SIMILAR TO '[0-9]{2}-[0-9]{7}') AS malformed_tax_id,
  MAX(updated_at)                                       AS last_refresh
FROM los.commercial_borrower
WHERE status = 'ACTIVE';

It returned 2,316 active borrowers, 141 with no tax ID and 63 rows that repeated a tax ID already on another row. Those numbers became the cleanup list, and nothing else in the table was touched.

A source profile card for one table, with numbered pins for its owner, refresh, quality results and what it is the record for
Pellmoor Savings Bank (Example company): one profile card per source, kept with the source.

Keep the profile with the source, not in a one-off spreadsheet, and rerun it when the source changes. In Synesis, our knowledge layer, profiling is part of harvesting a database. It reads a sample under the caller's own permissions and stores only statistics, and a column's list of values has to pass the sensitivity classifier before it can be proposed as vocabulary, so customer data never turns into terms by accident.

3. Name an owner and an approver for each domain

#

Every source needs an owner who answers for it. Every domain needs someone to approve what its terms mean, and you decide who; at Pellmoor, borrowers are one domain and credit policy another. This is the step that decides whether the rest lasts. Gartner's September 2026 survey of data and analytics leaders found cultural resistance, ahead of funding, was the main reason governance programs fail, and its advice was shared business and technology accountability (Gartner, September 21, 2026). Zhamak Dehghani's data mesh essay gives the working version: each domain owns its data as a product and states the integrity it offers as service-level objectives (martinfowler.com, May 20, 2019).

At Pellmoor the shared drive had no owner, which is why it had three copies of some statements. The credit administration manager took it on. Synesis lets you set who approves each stage and each class of rule. Many organizations have a department head approve their own terms and a governance body or compliance approve company-wide or regulated rules, and you can require two approvers for the most sensitive ones. Agents propose and never approve, and nobody approves their own change. How a rule is approved shows the same flow for rules.

4. Give every entity one identity

#

Start with reference data. If the loan system says CRE-OO and the core says COMM RE OWNER OCC, map both to one approved term now, in the vocabulary, not in every query later. Keep each system's code on the concept as a SKOS notation, so the crosswalk is data someone owns rather than a CASE statement copied into every report. Our post on the five stages from vocabulary to knowledge graph explains why one approved term per concept comes first.

Then fix identifiers. The FAIR principles ask for "globally unique and persistent" identifiers (GO FAIR, from Wilkinson et al., Scientific Data, March 15, 2016). In practice that means an IRI (Internationalized Resource Identifier, the web-style name a knowledge graph gives each thing) built from a stable key: a loan number, or a party ID minted once and kept across systems. Never a row ID that changes on migration, never a name, and never a tax ID. The tax ID is restricted data, and an IRI gets copied into every link, log and cache. Pellmoor's IRIs follow the pattern in the ontology guide: instances under https://example.com/pellmoor/id/, ontology terms in a namespace of their own. For master data exchanged with other parties, ISO 8000-115:2024 sets the requirements for quality identifiers, including how an identifier resolves to the data it stands for (ISO).

Then resolve entities, the step teams most often underestimate, and make every merge reversible from day one. Resolving means finding the records that describe the same real-world thing. A widely cited survey splits it into blocking (narrowing the candidate pairs), matching (scoring each pair) and clustering (deciding which records form one entity) (Christophides et al., ACM Computing Surveys, December 2020). The probabilistic model most matching tools still offer is Fellegi and Sunter's 1969 theory of record linkage. It weighs each field's agreement by how informative it is and sets two thresholds; pairs that fall between them go to a person (Fellegi and Sunter, Journal of the American Statistical Association, December 1969).

"Harlan & Pike Holdings LLC" in Pellmoor's loan system and "HARLAN AND PIKE HOLDINGS" in the core share a tax ID. That's a match. Two guarantors, "Dale Oberlin" and "Dale Oberlin Jr.", share a street address and a phone number but have different dates of birth. A matcher that weights name and address would merge them, and the son's file would carry his father's personal guaranty.

Two halves comparing a merge on name and address with a candidate link held for review with its evidence
A false merge is usually worse than a missed one, because everything downstream acts on the merged record.

A missed match costs a duplicate row. A false merge puts one person's obligations on another, and every agent that reads the record acts on it. Clustering makes it worse, because transitive links can chain four near-matches into one oversized entity.

So match on strong identifiers first, and block and score only what they leave. Treat facts that can't both be true, such as two different dates of birth, as a hard "never the same" that no score can outweigh. Send the middle band to the right person with the evidence that scored it. And record each merge as a link that can be undone, with no golden record overwriting either source.

Synesis does entity resolution this way. It scores pairs with Fellegi and Sunter's model and sets the two thresholds from error rates measured on an audit sample. Conflicting values for a property that can have only one, like the Oberlin dates of birth, are a cannot-link the clustering never crosses. An approved merge is applied as an owl:sameAs link and leaves the source records untouched, so an unmerge withdraws exactly what depended on it. A merge of two people is never automatic: a person approves it, and you can require a second approver.

5. Clean documents, and classify what's sensitive

#

Most of a renewal file lives in documents: tax returns, financial statements, appraisals, the credit policy itself. Barnett and colleagues catalogued seven points where retrieval over documents fails, from missing content to incomplete answers, and found that adding the file name and chunk number to retrieved passages helped (Barnett et al., arXiv 2401.05856, January 11, 2024). Scans fail in their own way, at the optical character recognition (OCR) step that turns a page image into text. The OHRBench study built 8,561 document images across seven domains and found the OCR it tested was not adequate for building high-quality retrieval knowledge bases (Zhang et al., arXiv 2412.02592, ICCV 2025). Pellmoor's 2019 tax returns were scans of faxes.

Two numbers about document quality for retrieval: seven failure points, and an OCR benchmark of 8,561 document images
Sources: Barnett et al., arXiv 2401.05856, January 2024; Zhang et al., arXiv 2412.02592, ICCV 2025.
CheckWhy it matters at Pellmoor
One current version, with its effective dateCredit policy v6 sat beside v7; an agent that found v6 would test last year's ratios
Near-duplicates removed before indexingThree copies of one statement crowd better passages out of the results
OCR checked on a sample a person readsFaxed returns misread a 7 as a 1 in debt service figures
Owner, date, classification and source on every passageThe agent can cite the page, and access can follow the classification
Retention setA statement past its retention date shouldn't be answering questions

Classify before you ingest, not after. The guarantors' tax returns carry Social Security numbers, and nothing the renewal agent does needs them. OWASP's 2025 list for language model applications names vector and embedding weaknesses (LLM08) and calls for permission-aware retrieval (OWASP, 2025).

Perception, our ingestion product, reads born-digital files directly with their text positions and runs vision-language OCR only on scanned pages. It keeps coordinates and confidence on extracted text and masks personal data before storage. Our ingestion pipeline treats a source nobody has classified as restricted until someone does, and an extraction that fails its check never becomes a candidate fact. Every item is identified by its source and a hash of its content, so a re-sent file changes nothing and a new version supersedes the old one. MNEMOS, our data engine, trims every result to the documents the person an agent acts for could open, and a Virtual Employee never sees more than its own role allows either.

6. Record lineage, then map to the ontology

#

An agent's answer is only as defensible as your ability to show where each fact came from. W3C PROV-O models provenance as entities produced by activities that agents are responsible for (W3C, April 30, 2013). OpenLineage, an LF AI & Data project, defines the events a pipeline emits for each run of a job and the datasets it read and wrote (OpenLineage). The two answer different questions. PROV-O says where a fact in the graph came from; OpenLineage says which job produced the table it sits in. For high-risk AI systems, Article 10 of the EU AI Act, as amended in July 2026 by Regulation (EU) 2026/1744, requires governance of the data used to train, validate and test them, covering how it was collected, where it came from and, for personal data, why it was originally collected (EU AI Act, Article 10). It's written for training and test data rather than what an agent reads at run time, but it's the bar an examiner will hold that data to.

With lineage in place, map the sources. R2RML, a W3C Recommendation since 2012, maps relational tables to the ontology's classes and properties, and the mapping can be queried in place or materialized into a graph (W3C, September 27, 2012). RML-Core extends the same structure to CSV, JSON and XML. It is a Draft Community Group Report of the W3C Knowledge Graph Construction Community Group, dated March 16, 2026, not a W3C Recommendation (RML-Core). Pellmoor's borrower table maps like this, with the IRI built from the party ID:

turtle
@prefix rr:  <http://www.w3.org/ns/r2rml#> .
@prefix pmo: <https://example.com/pellmoor/ontology/> .
@prefix pmm: <https://example.com/pellmoor/mapping/> .

pmm:BorrowerMap a rr:TriplesMap ;
  rr:logicalTable [ rr:tableName "los.commercial_borrower" ] ;
  rr:subjectMap [
    rr:template "https://example.com/pellmoor/id/party/{party_id}" ;
    rr:class pmo:Borrower ] ;
  rr:predicateObjectMap [
    rr:predicate pmo:legalName ;
    rr:objectMap [ rr:column "legal_name" ] ] .

The tax ID stays in the source for matching and never becomes part of the graph's identifiers. Validate the output with shapes written in SHACL, the W3C Shapes Constraint Language, run over the inferred graph so a record isn't flagged for a fact the ontology already entails. The ontology guide has a longer mapping and shapes for its accounts payable example.

In Synesis you rarely write the first mapping by hand. Harvesting starts from the W3C Direct Mapping of each table, then proposes refinements with their evidence: a junction table becomes a many-to-many relationship, keys come from functional dependencies found in the sample, and undeclared foreign keys from inclusion dependencies. The output is R2RML, and every class and property binding is a candidate for the right person to approve. Everything harvested or extracted lands in a change set, never straight in the graph. A shape violation blocks publication and shows up on a review card, a contradiction with approved knowledge becomes a conflict card, and an item the pipeline can't process at all goes to a dead-letter lane with its retries. Nothing disappears or skips review. Each import writes into a named graph of its own, and every approved fact keeps a PROV-O chain back to its table and column, or its document, page and character offsets.

A rail of four stops for anything harvested or extracted: proposed with its evidence, checked against the shapes, approved by the person you name, and published with a PROV-O chain back to its source
In Synesis, a mapping or an extracted fact reaches the graph only through a change set.

7. Put a contract on every source

#

A data contract is a written agreement between a source's owner and everyone who uses it, including agents. It states the schema, the quality rules and the service levels, and it names the owner. The Open Data Contract Standard (ODCS), maintained by the Bitol project at LF AI & Data, is the open format. Version 3.2.0, released September 8, 2026, added a context block written for AI agents. It tells an agent how to use the data and what it must never do, and it can carry approved answers to common questions (ODCS releases).

Part of Pellmoor's contract for the borrower table, in ODCS v3.2 syntax:

yaml
apiVersion: v3.2.0
kind: DataContract
id: pellmoor-commercial-borrower
version: 1.3.0
status: active
domain: commercial-lending
schema:
  - name: commercial_borrower
    physicalName: los.commercial_borrower
    properties:
      - name: tax_id
        logicalType: string
        required: true
        unique: true
        classification: restricted
        quality:
          - metric: nullValues
            mustBe: 0
            dimension: completeness
            severity: error
    context:
      instructions: >
        One row per legal entity. Match borrowers on tax_id, never on legal_name.
      constraints:
        - constraint: "Never treat two parties as one on name and address alone."
slaProperties:
  - property: latency
    value: 1
    unit: d
team:
  name: commercial-lending

Two lines earn their place. severity: error marks the check as one that blocks: the standard records the severity, and your gate acts on it by holding the data instead of letting it reach an agent; data quality checks for AI agents covers what happens to it next. The constraint writes the Oberlin lesson down where every future agent will read it.

The contract states what the owner commits to. Someone downstream still has to notice when the source stops matching it. In Synesis, a binding to a table in another system carries a fingerprint of the schema it was written against and checks it before every use. A dropped or retyped column fails loudly there, before it can surface as a wrong answer.

Gartner predicts that by 2030, 50% of organizations will use autonomous AI agents to interpret governance policies and technical standards into machine-verifiable data contracts (Gartner, March 11, 2026). You don't need to wait. A contract per source, written by its owner, is a short job for the few sources one queue touches.

What done looks like for one queue

#

For Pellmoor it meant this. The four questions run as queries and return the right files. Every source they touch is profiled and owned, and has a contract. Borrowers and guarantors are resolved, with every merge explainable. The credit policy exists at one current version, scans carry a confidence on every figure, and the Social Security numbers in guarantors' returns are masked. Every number in a renewal file traces to its page. Synesis holds the approved knowledge, stored in MNEMOS beside the records and their history; the architecture page shows how they fit. None of that work is locked to us. Your rulebook exports free, in open standards (SKOS, OWL and SHACL), and the R2RML mappings and ODCS contracts are open formats too.

It took one queue's worth of data, not the bank's, and most of the effort went into owners and identifiers rather than into any single tool. The next queue reuses the borrower resolution and the policy versions, and it starts at step 1 with its own questions. Before you start, run through the AI-ready data checklist; once agents are working the queue, data quality checks for AI agents keeps the data clean.

Further reading

#
  • Lack of AI-Ready Data Puts AI Projects at Risk, Gartner, February 26, 2025: gartner.com
  • AI Agents Aren't Waiting For Data Security, Forrester, November 2, 2026: forrester.com
  • Meet the data quality dimensions, UK Government Data Quality Hub, June 24, 2021: gov.uk
  • ISO/IEC 25012:2008, Data quality model, ISO, confirmed 2025: iso.org
  • An Overview of End-to-End Entity Resolution for Big Data, Christophides et al., ACM Computing Surveys, December 2020: arxiv.org
  • A Theory for Record Linkage, Fellegi and Sunter, Journal of the American Statistical Association, December 1969: jstor.org
  • Seven Failure Points When Engineering a Retrieval Augmented Generation System, Barnett et al., January 11, 2024: arxiv.org
  • OCR Hinders RAG, Zhang et al., ICCV 2025: arxiv.org
  • The FAIR Guiding Principles for scientific data management and stewardship, Wilkinson et al., Scientific Data, March 15, 2016: gofair.foundation
  • PROV-O: The PROV Ontology, W3C, April 30, 2013: w3.org
  • EU AI Act, Article 10: Data and data governance, as amended by Regulation (EU) 2026/1744: artificialintelligenceact.eu
  • R2RML: RDB to RDF Mapping Language, W3C, September 27, 2012: w3.org
  • RML-Core, Draft Community Group Report, W3C Knowledge Graph Construction Community Group, March 16, 2026: w3id.org
  • Open Data Contract Standard, v3.2.0, Bitol at LF AI & Data, September 8, 2026: github.com
  • CFPB order against Hyundai Capital America for credit reporting failures, CFPB, July 26, 2022: consumerfinance.gov

Share

Questions

What people ask.

What does it mean to prepare data for AI?

Preparing data for AI means making the records and documents an AI system will read fit for that specific use: owned by someone, measured against named quality dimensions, identified consistently across systems, traceable to their source and protected where they are sensitive. It is defined per use case, so the work starts with the questions the first piece of AI work must answer.

Do we have to clean all our data before using AI agents?

No. Clean the sources the first queue's questions touch, and fix them at the system of record so the next feed doesn't undo the work. Everything else waits until a later queue needs it. Cleaning the whole estate first is one of the most common reasons AI projects stall.

What are the data quality dimensions?

The six most often used are accuracy, completeness, uniqueness, consistency, timeliness and validity, from the DAMA UK working group and adopted by the UK Government Data Quality Hub. ISO/IEC 25012 defines a fuller model of 15 characteristics, seen as inherent to the data, dependent on the system holding it, or both.

What is entity resolution, and why does it matter for AI?

Entity resolution finds the records in different systems that describe the same customer, company or asset and links them. It matters for AI because an agent acts on whatever record it is given. A missed match leaves a duplicate; a false merge puts one person's obligations on another, so merges should be explainable and reversible.

What is a data contract?

A data contract is a written agreement between the owner of a data source and its users, covering the schema, quality rules, service levels and ownership. The Open Data Contract Standard is the open format, and its version 3.2.0 added a context section with instructions and constraints written for AI agents.

Where does R2RML fit in data preparation?

R2RML is the W3C standard for mapping relational tables to the classes and properties of an ontology. It comes after profiling, ownership and identifiers are settled, so the knowledge graph is built from clean, owned sources instead of copying their problems into a new store.

Developer edition

Build on Inovagio.

The MNEMOS data engine, the command line and the desktop app, with guides and API references.

This is for

We use these details to answer your request. See our privacy policy.

What happens next

  1. 1We reply to every request.
  2. 2We send the next steps for your team.
  3. 3A direct line to the engineers who build it.