From 40 Million Tokens to a 450K-Token Graph: How Miriel Distills What It Learns

TL;DR — Every document that flows through learn() gets a second, quieter life. After it's chunked, embedded, and indexed, a background worker distills it into knowledge-graph triples: [Contract 12] [governed by] [Florida law], [Josh] [is married to] [Riddhi]. The graph isn't a replacement for anything — at query time it runs alongside vector search, SQL-style filtering and lookups, and Miriel's other retrieval methods — but it's the surface that answers a class of question the others structurally can't, and it does so in tens of tokens instead of thousands. On a whole-Gmail-account import (~50,000 emails, ~40M tokens of raw text), distillation produces a graph of roughly 30,000 triples — about 450K tokens, an ~89× compression — for a one-time extraction cost of about $10.50 on Miriel's default extraction model (Gemini 2.5 Flash-Lite). Retrieval that reads the graph instead of document chunks cuts per-query input tokens by 15–65×, the graph surfaces cross-source connections that similarity search cannot find, and relationships strengthen when used and decay when they aren't, with a 30-day half-life. This post walks through how the graph gets built, what it costs, what it saves, and what falls out of it that surprised us.

A note on prices: every dollar figure in this post uses the public per-token API rates of popular models (OpenAI, Google, Anthropic; July 2026) — they're there to make the token economics concrete and reproducible. They represent what raw LLM usage costs at list price, not what Miriel charges: Miriel's pricing is structured differently and is not a per-token pass-through.

The pipeline: what actually happens on learn()#

When you call learn() — via the API, the MCP tool, or a connected source like Gmail — the content flows through a stream pipeline: fetch → parse → index → entities. The first three stages normalize, chunk, embed, and store the content, feeding the document store, the vector indexes, and the structured metadata that Miriel's other retrieval surfaces (SQL-style filtering, exact lookups, time- and source-scoped queries) run against. The fourth stage is the distillation pass, and it's deliberately decoupled — entity enrichment never blocks a learning job from completing. If the entities worker is slow or down, your documents still land in storage; the graph just catches up later.

Fig. 1 — The write path distills each document once, asynchronously; the read path fans each question out across every retrieval surface in parallel and merges the results. The highlighted blue boxes are what this post is about — everything else was already there.

The entities worker does three things to each document:

1. Find the entities. A two-pass extraction: a regex pre-pass grabs capitalized runs as provisional entities, then a BERT NER model (dbmdz/bert-large-cased-finetuned-conll03-english) tags people, organizations, and locations. NER runs on a shared inference service — no LLM tokens spent. If BERT can't run, a small LLM fallback extracts entities instead.

2. Ask a small LLM how the entities relate. The entity list and the document go to a deliberately cheap model (the entities_model deployment setting — a nano/mini-class model, not a frontier one) with a prompt that is strict about what counts as a fact:

The predicate must accurately describe the relationship between term1 and term2 specifically. […] The triple must be reversible in meaning: "Alice [works at] Acme" = "Acme [employs] Alice". If you cannot express the reverse, it is not a relationship. […] "score" (0.0–1.0) rates how useful this relationship is to remember long-term. […] It is better to return an empty list than to return poorly-formed triples.

The model returns at most 10 scored triples per document. Anything below 0.7 usefulness is discarded before it ever touches the database. Most emails yield zero to three triples; that selectivity is the point — the graph is a distillate, not a transcript.

3. Upsert into the graph. Each surviving triple is canonicalized (predicate normalization, direction handling for symmetric/reverse relations), each node name is resolved against known aliases and existing nodes ("ElonMusk" and "Elon Musk" land on the same node), and the triple is upserted with provenance:

INSERT INTO entity_relationships
  (node1, edge, node2, score, user_id, project, ..., source_document_id,
   strength, last_accessed_at, created_at)
VALUES (...)
ON CONFLICT (user_id, project, lower(node1), edge, lower(node2))
DO UPDATE SET
  score    = GREATEST(entity_relationships.score, EXCLUDED.score),
  strength = LEAST(GREATEST(entity_relationships.strength,
                            EXCLUDED.strength) + 0.15, 1.0),
  last_accessed_at = now();

Note the ON CONFLICT branch: seeing the same fact twice doesn't create a duplicate edge — it reinforces the existing one. Repetition is signal. After each batch, a merge pass collapses duplicate nodes that slipped past insert-time resolution, including pairs the LLM itself linked with alias-style predicates like "is another name for."

Fig. 2 — The Knowledge Graph explorer after learning four short documents (a Discord message, two emails, and a contract summary): 7 entities, 6 relationships. Node color encodes current strength (green = strong, red = dormant); edge labels are the extracted predicates. The contract cluster — Miriel and Acme Corp party to a Master Services Agreement, governed by Florida, with Sarah Chen as counsel — came out of one paragraph.

The small case: one contract, one fact, forever#

Start with the simplest possible economic argument. A 100-page master services agreement is roughly 65,000 tokens. Somewhere on page 87 it says the agreement is governed by Florida law. After learn(), the graph contains:

[Master Services Agreement] --[governed by]--> [Florida law]

That's about 12 tokens, with a source_document_id pointing back at the PDF.

Every future query that needs that fact has options, priced here with real July-2026 rates for the model doing the answering:

The one-time distillation of that contract — a single pass through Miriel's default extraction model, Gemini 2.5 Flash-Lite ($0.10/MTok in, $0.40/MTok out) — costs about $0.0067. Two-thirds of a cent. It pays for itself before the first frontier-model query completes, and every query after that is a 65× reduction in input tokens against re-reading. The provenance pointer means the answer is still auditable: the triple tells you what, the source document link tells you where it came from if you need the exact clause.

Notice the middle rows, because this is the nuance that's easy to miss: the graph wins even when vector search works perfectly. Retrieval in Miriel is progressive — start small, pull more context only when the question demands it, and ultimately hand the model everything it needs. Chunk-based retrieval's floor is the size of a chunk: even if the vector store returns the one correct chunk on the first try, that's 1,000–4,000 tokens of prose wrapping a 12-token fact, and in practice you retrieve top-k of them, not one. When the graph can answer, the progressive ladder terminates at its first rung — a handful of triples — and far fewer chunks ever enter the context window. The graph doesn't replace the chunks; it makes most of them unnecessary for most questions.

This is the access-pattern argument in miniature: documents are written once and read many times. Facts inside them follow a power law — a handful get asked about constantly, most never. Distillation moves the cost of finding a fact from read time (paid every query, at frontier prices) to write time (paid once, at nano-model prices).

The large case: a whole Gmail account#

Small examples qualify the idea; large ones quantify it. Here's the arithmetic for a realistic full-account import — 50,000 emails averaging ~800 tokens each, about 40M tokens of raw corpus. Extraction reads every email once (75M input tokens including the prompt scaffold and entity lists) and writes triple JSON (~7.5M output tokens). Priced across popular extraction-tier models at current rates:

For comparison, reading the corpus once — no distillation, just pushing all 40M tokens through the model that answers your questions — costs $200 on Claude Opus 4.8 ($5/MTok in), $120 on Claude Sonnet 5, or $100 on GPT-5.4. Distilling the entire mailbox costs about 5% of a single frontier read. And the output is durable: roughly 30,000 triples, ~450K tokens — an ~89× compression of the corpus into its who-knows-whom, who-works-where, what-owns-what skeleton.

On the read side, compare against a retrieval answer that stuffs the top-20 email chunks (~16,000 tokens) into Claude Opus 4.8:

• Graph-backed context: ~1,000 tokens → $0.075 saved per query on input alone.
• Break-even against the $10.50 distillation: ~140 queries (~70 if extraction ran as a batch job).

For a mailbox you'll query daily for years, that's days. And the progressive-retrieval point from the contract example compounds at this scale: every question the graph answers outright is twenty chunks that never enter the context window, and every question it partially answers narrows which chunks the deeper retrieval passes still need to fetch.

What this does to a real workload#

Per-query comparisons flatter whichever row you point at, so let's model a realistic month instead. Real Miriel workloads aren't uniform: most queries are top-k retrievals, but some are exhaustive — Miriel detects questions like "what's the sum of all statement balances" or "how many contracts reference indemnification" and runs a map-reduce pass over every document matching the SQL-style criteria, not just the top k. Exhaustive queries are rare and enormous. Chunks vary in size but are often ~4K tokens. And the graph leg doesn't return one triple — a traversal returns a neighborhood, typically dozens of triples (~1,200 tokens), and it runs on every query, so we charge for it even when it contributes nothing.

Assumptions, stated so you can disagree with them:

The exhaustive rows deserve a word. First: at baseline, exhaustive queries are 8% of volume but nearly half the spend ($160 of $344/month) — they're where the token money actually goes. Second, some exhaustive questions stop being exhaustive at all once the graph exists: "which contracts are governed by Florida law" is a map-reduce over every contract without the graph, and a single indexed lookup over [* ] [governed by] [Florida] edges with it. Others can't be answered from triples but can be scoped by them — the graph names which entities (and via provenance, which documents) are involved, so the map stage reads 100 documents instead of 500.

The month, priced:

Against that: the one-time $10.50 distillation, plus about $0.32/month to distill ~1,500 new emails as they arrive. The write side is noise; the graph pays for itself in the first few days of a normal month. And the estimate is not fragile — cut every graph hit-rate assumption in half and the net savings is still ~18%. The single biggest lever isn't the hit rate on ordinary queries at all; it's converting or scoping the handful of exhaustive queries, because that's where the tokens live.

But the workload math still undersells the graph, because it prices it as if it were only a cost optimization. It isn't — it answers a class of question the other surfaces can't. Which brings us to insights.

Fig. 3 — The demo account after learning 73 documents spanning email, Slack, Discord, CRM notes, support tickets, and meeting minutes: 67 entities, 139 relationships, average strength 0.90. The structure that matters is the bridges — Acme Corp connects the legal cluster (Harper & Vance, the MSA, Florida) to the people cluster (Dana Whitfield, Marcus Lee, Josh) to the venture cluster (BetaWorks Robotics, Foothill Ventures, Granite Peak Capital), while banking, health-tech, and conference clusters hang off their own hub entities. No single document mentions more than a few of these.

Insights: what the graph knows that no document says#

Vector search retrieves what is similar to your question. But the facts you need are usually similar to the answer's dependencies, not to the question itself — and for any question that requires a join, no single document contains the answer at all.

The example that convinced us: planning a dinner party.

In Discord, months ago: "josh's wife is riddhi btw" In an email thread, weeks ago: "…Josh mentioned his wife is a vegetarian, so we skipped the steakhouse." The query, today: "What food should we serve at a party with guests Josh, Andy, Jason, and Madison, plus spouses?"

Ask ChatGPT and it has none of this context. Ask a naive RAG system and it fails more instructively: the Discord message about who Josh is married to has near-zero embedding similarity to a question about party menus. It won't be in the top-k. Even if "Josh" gets a lexical match, top-k is dominated by documents about food and parties — the relevant documents are relevant only because of a dependency chain the retriever can't see: guests → spouses → dietary constraints. Is Josh appearing in both documents enough for a similarity engine? No — similarity is not a join.

The graph makes the join trivial, because extraction already did the reasoning at write time, when each fact was cheap to see:

[Josh] --[is married to]--> [Riddhi]      (source: Discord)
[Riddhi] --[is vegetarian]--> ...         (source: email)

At query time, entity extraction on the question finds "Josh", and a depth-2 traversal from Josh surfaces Riddhi and the dietary fact in two hops — a few hundred tokens of context that no similarity metric would ever have retrieved. The answer ("have solid vegetarian mains") falls out.

The same mechanics generalize across an organization and across modalities. A sales-call transcript (audio → text) says "Acme's CTO mentioned they're migrating off Postgres." A support ticket filed by a different teammate references latency complaints from acme-corp. A CRM note says the Acme renewal lands in Q3. Three sources, three owners, three modalities — and entity resolution folds Acme, acme-corp, and "Acme Corp." into one node. A traversal from Acme now reads like an account brief nobody wrote: migration risk, open incidents, renewal timing. The non-obvious insight isn't in any document; it's in the bridge — and on the dashboard's force-directed view, those bridges are literally visible as the edges connecting clusters that have no business touching.

Fig. 4 — A two-hop traversal from 'Josh': 13 nodes, 20 relationships. One hop reaches his employer and colleagues; the second hop crosses into the legal and conference clusters through them. This is exactly the join the retriever performs at query time — and none of it would rank in a top-k similarity search for a question about Josh.

Traversal: retrieval that walks#

We're shipping graph traversal as a first-class retrieval surface. At query time, Miriel fans a question out across its retrieval methods in parallel — vector search over chunks, SQL-style filtering and exact lookups over document metadata, and now the graph — then merges and re-ranks what comes back. The graph leg works like this: NER runs on the query text, and each extracted entity seeds a breadth-first walk over the graph, currently two hops deep. Edges below the strength floor are skipped; everything traversed is deduplicated and handed to the model alongside the other surfaces' results, and also feeds a graph dimension in relevance re-ranking.

The traversal runs in a thread pool that starts before the vector search dispatches, so the graph walk overlaps the other retrieval I/O rather than adding latency in series. The output is compact — {node1, edge, node2, strength, source_document_id} rows — which is exactly why the token math above works: the model receives conclusions with citations, not raw material to re-derive them from.

There's a second-order effect worth noting: traversal touches edges, and touching an edge reinforces it (more below). The act of retrieving along a path makes that path stronger and cheaper to find next time. Access patterns don't just save money — they shape the graph.

We ran it, so you don't have to take our word#

Everything above is architecture and arithmetic; here's a live run. On the demo account from the screenshots (73 short documents — email, Slack, Discord, CRM notes, support tickets, meeting minutes), we asked the same question three ways through the query API, answering with GPT-5-nano:

"Which of our agreements are governed by Florida law, and who is our counsel of record at Acme Corp?"

Fig. 5 — The live run on the preview. The Response answers from both agreements; below it, the vector results with their scores, and the per-query Relationship Graph panel showing the 23-entity, 30-edge neighborhood the traversal contributed.

Three honest observations from the timing instrumentation:

• The graph walk is not where latency lives. The breadth-first traversal itself — the actual Postgres query — measured 5–19 ms. Vector search took 50–120 ms. The answering LLM took 7–10 seconds. The graph's latency contribution rounds to zero; what it can do is shrink the prompt, and prefill time scales with prompt size, so on the 40K-token contexts of the workload section the smaller graph context also buys back time-to-first-token, not just tokens. (One caveat from this preview: query-side NER ran on an LLM fallback here, costing ~2.4s cold and ~100ms warm; in production it runs on the shared inference tier at BERT speeds.)
• At 73 documents, vector search doesn't lose. All three modes answered correctly — a corpus this small fits comfortably in top-k, and the graph-augmented run actually spent ~1,000 more tokens than vector alone, exactly the neighborhood tax the workload model charges on every query. The economic separation needs corpus scale, not demo scale: top-k degrades as the corpus grows and chunks compete; a keyed lookup doesn't.
• Graph-only answering already works. The third row is the interesting one: the model produced the complete, correct answer — including the amendment — from 21 triples and no document text at all. "The triple is the answer" isn't a roadmap item; the mode exists today. What's ahead is making the router smart about when to trust it.

Forgetting on purpose: strength, decay, and reinforcement#

A graph that only grows becomes noise. Every relationship carries two numbers:

• score — the extraction-time judgment of how useful this fact is long-term (the 0.7 filter's survivor).
• strength — a live value that behaves like a synapse.

Strength decays exponentially with time since last access, with a 30-day half-life, computed lazily at read time:

decay_factor = 0.5 ** (days_since_last_access / 30.0)
effective_strength = max(strength * decay_factor, 0.01)

Every access — a retrieval hit, a traversal step, a re-extraction of the same fact — bumps strength by +0.15 (capped at 1.0) and resets the clock. A relationship nobody has touched in months fades toward the 0.01 floor and goes dormant: it stops appearing in retrieval, but it isn't deleted — one mention in a new document, or one traversal that reaches it from a hot neighbor, revives it.

The result is a memory with a working set. Facts you use stay bright green in the explorer; facts you don't slide through yellow to red and eventually step out of the retrieval path entirely — without anyone writing a retention policy. Clicking any edge in the explorer shows both numbers side by side — the stored strength and today's effective (decayed) strength — along with a link to the source document that produced it.

The unglamorous parts: canonicalization and identity#

Two triples about the same fact are only useful if they're the same row. Most of the engineering here is identity plumbing:

• Predicate canonicalization. A relation_predicates table (with per-user overrides) maps surface predicates onto canonical edges and declares direction: forward, reverse (so "is employed by" and "employs" store identically), or symmetric (so "is married to" stores its endpoints in deterministic order and Josh→Riddhi and Riddhi→Josh can't diverge).
• Insert-time entity resolution. Every node name passes through normalization, a per-user alias table, and a match-key lookup against existing nodes before it's written.
• Post-extraction merge. After each batch, a merge pass builds a rename plan from occurrence counts, recorded aliases, and alias-predicate edges the LLM emitted, then collapses duplicates — keeping the strongest edge, merging scores (max), strengths (max), and attributes, and recording the alias so the next insert resolves instantly.
• The prompt fights the problem at the source. The extraction prompt explicitly demands one canonical surface form per entity ("write 'Elon Musk', never 'ElonMusk' or 'Musk'; do not mix 'Meta', 'Meta Platforms' and 'Meta Platforms, Inc.'"). Cheap models follow this surprisingly well, and every triple that arrives pre-canonicalized is a merge the database never has to do.

Identity is also what makes the cross-user story work. When two users in an org learn documents that mention the same customer, entity resolution gives their graphs a shared vocabulary — and org-level traversal can walk from one user's sales transcript to another user's support ticket through a node both of them helped build.

Whose graph is it? Scoping and access#

A graph distilled from your entire mailbox is exactly as sensitive as the mailbox, so it inherits the same boundaries. Every relationship row is scoped to a user and a project: the triple's unique key literally begins with (user_id, project, …), and every read path — retrieval, traversal, the explorer, stats — filters on the requesting user before anything else. There is no global graph. Deleting a document doesn't strand its facts either: each edge carries its source_document_id, so provenance-based cleanup knows exactly which rows a removed source contributed. Project labels partition the graph further — the explorer's project filter isn't cosmetic; it's the same predicate the retrieval path applies.

Cross-user insight, when it happens, happens on shared material, under the same grant-based access controls that govern documents everywhere else in Miriel — content learned with specific grant_ids is visible to those grants and no one else, and the graph follows the documents, not the other way around. Entity resolution giving two users' graphs a shared vocabulary means their permitted views can be joined; it does not mean either can see edges distilled from the other's private mail. The dinner-party answer is only impressive if Josh's colleagues can't casually query who he's married to.

What's next#

Three directions follow from here. First, deeper traversal with smarter frontier selection — depth-2 BFS is a start, but strength-weighted walks should let us go deeper without dragging in noise. Second, richer cross-modal extraction: image entity extraction already runs through the same pipeline, and wiring video and audio-derived entities into the same identity machinery turns "search across your media" into "reason across your media." Third, graph-first answering by default: the live run above showed the mode already works — the remaining problem is the router knowing, per question, when the triples are sufficient and the chunks can stay home.

The through-line is the same economic bet as the rest of Miriel: pay once, cheaply, at write time — so that every read, forever, is fast, small, and grounded in something you can click through to verify.