Most Questions About Video Aren't AI Questions

There is a default architecture for asking questions about video, and it goes like this: retrieve some clips, caption them, stuff the captions into a context window, ask a model. It works. It is also the most expensive possible way to answer most of the questions people actually have.

Here is a question a warehouse asks every week: did more people go through that door than badges were swiped? That is tailgating, and it is a real security event with a real cost. It is also, structurally, a GROUP BY with a HAVING clause. There is nothing in it a language model is uniquely good at.

The interesting engineering problem is making that true — getting video into a shape where a question with a schema compiles to a join instead of a prompt.

The write path produces rows, not blobs#

Our perception pipeline doesn't emit "a description of what the camera saw." It emits typed rows into project-scoped, read-only views:

• api.detections — one row per detected thing in an analyzed frame, with its crop, its geometry, and its embedding references
• api.objects — the resolved entity behind those detections, one row per physical object rather than per sighting
• api.object_events — episode boundaries: entered, dwelled, moved, left
• api.frame_captions — the language layer, where it was worth generating

Every row is stamped with time, stream, and evidence identifiers. And critically, those views sit beside the project's own tables in the same query surface: badge swipes, WMS transactions, POS lines, GPS pings, alarm panel events. The vectors live in CyborgDB, encrypted; the time-series and graph stores are there too. One read-only SQL statement reaches all of them.

The tailgate query#

-- Tailgating: more people through a door than badges swiped.
SELECT door, entered_at,
       COUNT(DISTINCT object_id) AS people,   -- one id per person, across cameras
       COUNT(DISTINCT badge_id)  AS badges    -- the access-control system's table
FROM   api.object_events
LEFT   JOIN badge_swipes USING (door)         -- matched within ±10 seconds
WHERE  SEMANTIC_SCORE('a person') > 0.6       -- "person" isn't a column — it's vector
GROUP  BY door, entered_at                    -- similarity, scored on ciphertext
HAVING people > badges;

Two things are worth pausing on.

The one concept the schema doesn't contain is computed where it should be. There is no is_person boolean in api.object_events, and we don't want one — hard-coding a class list into the schema is how you end up re-ingesting a fleet when the question changes. SEMANTIC_SCORE('a person') is vector similarity, evaluated at query time, and it works equally well for 'a person in a red jacket' or 'a forklift carrying a pallet'. It is a column, usable in SELECT, in ORDER BY, or as a WHERE threshold.

The scoring happens on ciphertext. CyborgDB searches encrypted vectors without decrypting them — that's the same encrypted-in-use scheme behind our 100-billion-vector search benchmark, at ~1% overhead on index build and under 15% on search. The SQL engine never sees a plaintext embedding, and neither does the server.

And the answer to that query costs milliseconds of database time. In our production retrieval stack, graph traversal runs 5–19 ms and vector search 50–120 ms. LLM answer synthesis runs 7–10 seconds. When the question has a schema, we're paying the first bill, not the second.

The other stores, in the same statement#

Four relations pull the non-SQL stores into an ordinary query:

So the semantic question doesn't need a prompt either:

-- Semantic similarity as a column: CyborgDB scores ciphertext,
-- SQL does the rest; the LLM is saved for advanced analysis.
SELECT object_id, stream_id, captured_at,
       SEMANTIC_SCORE('person in a red jacket') AS score
FROM   api.detections
WHERE  stream_id = 'cam-02'
  AND  captured_at > now() - INTERVAL '12 hours'
ORDER  BY score DESC
LIMIT  10;

Every one of these relations returns normalized evidence columns — evidence id, store, kind, source id, object id, stream id. Provenance is part of the query surface rather than something an application layer reconstructs afterward. That is what makes an evidence package assemblable by query: the clip, the crop, the badge row, and the timestamp all arrive with identifiers that point back at where they came from.

Why the join is the whole point#

The questions that matter to operators are usually not answerable from any single sensor. They exist only in the join:

Take the vanished pallet, because it's the cheapest possible demonstration of the argument. A cycle count comes up one short in aisle 40, and the WMS says the pallet was put away Tuesday. Today that's hours of scrubbing footage, if anyone bothers for one pallet — which is to say it's an investigation nobody opens.

As a query it's a join across three tables that each look completely normal on their own: pallet-shaped episodes leaving aisle 40 since Tuesday, the RFID reader's scan rows, and the outbound dock's episodes in the hour after. An episode at 14:52 Wednesday with no matching scan and no pick order behind it is the answer, and it arrives with evidence identifiers pointing at the clip. No model. Milliseconds.

That's the shape of the thing. The exception doesn't exist in the video, and it doesn't exist in the WMS. It exists in the join, and the join is ordinary SQL.

So what is the model for?#

Language. Specifically, the questions with no schema to compile to:

• "Describe the second person."
• "Summarize the incident."
• "What usually happens here at 9 p.m.?"

That division of labor — SQL for structure, vectors for similarity, LLM for language — is the design decision this whole surface exists to enable. Its consequence is economic. If most questions never reach a model, the effective cost per answered question falls faster than model pricing does, because the denominator is shrinking at the same time as the numerator. Query costs ride the database curve, not the model curve.

It also has a reliability consequence we care about more than the cost one. A join either matches or it doesn't. When a security team asks whether someone entered without badging, the answer should be a row count with evidence identifiers attached, not a paragraph that is usually right.

Honest limits#

SEMANTIC_SCORE thresholds are not free of judgment — 0.6 is a tuned number per embedding space, and a threshold that works for 'a person' on dock cameras is not automatically right for 'a person' on a body-cam feed. Model-and-version contracts are pinned per vector space precisely so that a threshold means the same thing next year, but they don't eliminate the tuning.

Cross-modal enrichment of the knowledge graph — pulling video and audio entities into GRAPH_TRAVERSE's reach automatically — is further along in our document and email pipelines than in our video ones. The fusion joins above are architecturally supported and running; their economics at full video scale are something we intend to publish, not something we've published.