> ## Documentation Index
> Fetch the complete documentation index at: https://docs.textql.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Module 0 · Concepts & The Method

> An ontology is the mapping between your technical assets (tables, catalogs, documents) and your business (metrics, workflows, terminology). In TextQL it’s just files — Markdown … (~15 min)

## What an ontology is here

An ontology is the mapping between your technical assets (tables, catalogs, documents) and your business (metrics, workflows, terminology). In TextQL it's **just files** — Markdown for human context and `.tql` for a typed, SQL-rendering semantic layer — kept in a **git-backed repository** with review, versioning, and access control. Ana reads it, renders inspectable warehouse SQL from it, and helps you extend it.

## The principles

* **Just files.** Business definitions, SQL templates, notes, and operating rules live in one versioned file tree. Portable, diff-able, and yours.
* **Locality of behavior.** You should understand a metric by reading its file and the few it imports — no black-box compiler. `.tql` renders **inspectable native warehouse SQL**.
* **Governed by default.** Semantic changes are proposed → diffed → reviewed → approved → audited, like software.
* **Ana does the heavy lifting.** She inspects a `.tql` file, renders SQL from parameters, and executes it — preferring a governed surface over ad-hoc SQL.
* **Discovery first.** Build from what's actually there — start by reading the schema and the documents, not from assumptions.

## Why this approach wins — field lessons

Most data leaders you'll build with have spent **months** on a clean semantic layer in Looker, Omni, Collibra, or Palantir — and they're rightly proud of it. The rigid-tool trap: those tools force big **design decisions upfront, before anyone has seen a single real question** from a real analyst. So the model captures what seemed important at modeling time and **drifts** from what actually turns out to matter. Our approach inverts that.

* **Their prior work isn't wasted — it's a corpus.** dbt models, LookML, Omni configs, Power BI datasets, wikis, spreadsheets — none needs to be reformatted or migrated. Drop it in; Ana reads it and starts from a warm place.
* **You're not writing the ontology — Ana is.** She makes most of the structural decisions and physically writes the content. You are the **captain and reviewer**, keeping her directionally correct. Ana is self-referential: she knows what context an agent needs.
* **You're not building blind — you need a North Star.** Establish **what the ontology is for** before anything goes in the repo. BI parity, analyst self-serve, and operational/embedded queries each want a **different shape** (governance, strictness, flexibility). An ontology without a North Star is expensive guesswork.

<Note>
  **The cardinal rule** —

  **Do not hand a skeptic a form to fill out.** Templates, question libraries, and "North Star archetypes" exist to **signal expertise** (you've solved this in their domain) and to **force the right conversation** — not as a schema to deploy. Naming domain-specific failure modes ("here's where naive implementations break") earns your opinions. Then **bias hard toward action**: the answer to "where do we start?" is *"let me show you."* Establish the North Star → feed existing context → start asking real questions → review & ratify what Ana proposes. The model improves every cycle.
</Note>

## The research behind it

This isn't just philosophy — it's measured. TextQL's paper *"Malleability Is All You Need: A Self-Maintaining Semantic Layer for Data Agents"* (Baumstark, Tomitsuka, Ma — VLDB 2026 / NOVAS) names the **"amnesia tax"**: a stateless agent that re-explores the warehouse on every question spends **92–94% of its tokens** rediscovering the same tables, joins, and filters. Cost scales with **task count, not novelty**; identical questions resolve to **divergent SQL** (non-reproducible); long contexts **rot** (latency and failure rates climb with depth); and naive memory is **non-portable** across engines.

* **Invert the lifecycle:** the agent **reads prior knowledge → explores only the frontier → answers → commits** new definitions back to a version-controlled, executable semantic layer. Discovery is amortized across tasks and agents, not re-incurred per request.
* **Measured results:** ≥**30% fewer tokens** (up to \~50% on complex multi-join queries), **\~41% fewer SQL probe-turns**, per-query cost **falls toward a "novelty floor"** as the layer accretes knowledge (most of a warehouse learned in \~35 queries), and switching backends (DuckDB→Postgres) required **no change** — definitions are dialect-independent.
* **Trust by construction:** every write-back is a **git commit** with an author + diff; definitions are **executable**, so they're validated mechanically (recompile + run) rather than re-read as prose; high-stakes paths route to human review (CODEOWNERS-style).

That's the "why" behind everything below: you're not asking the customer to build a perfect model upfront — you're standing up a **living** one that gets cheaper, more accurate, and more reproducible the more it's used.

## The repository shape

```text theme={null}
ontology/
  schema.tql            central table backings (rename once, everywhere follows)
  relations/            per-source modules: table source + reusable expressions + metric keys
  dimensions/           grouping expressions, each with the join it needs
  filters/              reusable filter predicates
  queries/              public query surfaces (expose metrics / dimensions / filters)
  notes/                business rationale + decisions, colocated with the code
```

## Where each input lands

<table>
  <tr><th>Input</th><th>Lands as</th></tr>
  <tr><td>Business documentation (SOPs, policies, glossaries)</td><td>`notes/*.md` + term definitions</td></tr>
  <tr><td>Metrics documentation</td><td>governed `.tql` query surfaces + notes</td></tr>
  <tr><td>Database + data dictionaries</td><td>schema backings + relations + table docs</td></tr>
  <tr><td>Process-flow diagrams</td><td>entities + relationships + workflow notes</td></tr>
  <tr><td>Call transcripts / unstructured text</td><td>terminology, intents, entities → glossary + entities</td></tr>
  <tr><td>Golden datasets</td><td>golden-query test fixtures (accuracy)</td></tr>
  <tr><td>Validation sets</td><td>eval cases that verify answers</td></tr>
  <tr><td>Governed / access-controlled documents</td><td>fail-closed access rules + governed notes</td></tr>
</table>

## How you know it's right

* Every query renders **inspectable SQL** — you can read exactly what ran.
* **Golden-query tests** pin a known-correct value and alert on drift.
* Flexibility ↔ accuracy is a per-metric dial: exact where it must be, flexible where exploration matters.

### The expert shortcut — seed from what you already know

Discovery (Modules 2–5) is for when the knowledge doesn't exist yet. If your team **already has** the join logic, ERDs, dbt models, or canonical SQL — don't make Ana rediscover it. Feed it in as a corpus and let her formalize it:

```text Prompt theme={null}
Here are assets that already encode how our data joins and what our metrics mean: [attach or paste: ERD, canonical SQL queries, dbt models, join documentation]. Seed the ontology from these directly - sources, join paths, and metric definitions - instead of rediscovering them by exploration. Show me everything you extracted as a reviewable change, and flag anything ambiguous or conflicting rather than guessing.
```

<Check>
  **You'll see:** Ana turn your existing SQL/ERD knowledge into governed ontology entries in one pass — discovery reserved for what your assets don't cover.
</Check>

<Note>
  **Two rules for a long, live session** —

  **1 · Checkpoint every couple of modules.** Long threads have a ceiling. After every module or two, ask Ana: *“Save a handoff document summarizing what we've built, what we decided, and what's next — so we can continue in a new thread.”* If a thread ever maxes out, you lose nothing — open a new thread, paste the handoff, keep going.<br /><br />
  **2 · Pin the scope in every prompt.** Name the **entity** you're analyzing and the **source-of-truth tables** in each prompt (“…for \[entity X], using the \[base audit] tables, not the summary table”). Without that, Ana may drift to a convenient summary table or query every source at once — and a plausible-looking answer from the wrong table is worse than no answer.
</Note>

### ✅ Checkpoint

* [ ] You can explain "just files" and why git-backed matters
* [ ] You know the six folders of the repo shape and what goes in each
* [ ] You know where a metrics doc vs. a call transcript lands in the model
* [ ] You can pitch a skeptic: prior work = corpus, Ana writes it, North Star first, never a form
* [ ] You can cite the research: the "amnesia tax" and the ≥30% token reduction from a malleable layer
