Schema Drift, Detected From a Header Fingerprint

The worst import failures are not the ones that throw. They are the ones where a partner quietly renames member_id to subscriber_id, your matcher shrugs, the column lands unmapped, and six weeks of rows arrive with a blank identity field that nobody notices until reconciliation.

You do not need a schema registry to catch that, and you certainly do not need a model. You need to remember what the header row looked like the last time a human confirmed a mapping for this source, and to notice when it changes. That is what POST /v1/layouts/drift does, and the whole mechanism fits in a paragraph of code.

The fingerprint

A layout is identified by a sha256 over the normalised header row, joined with a pipe:

const canonical = headers.map(normalize).join('|')
const digest    = await crypto.subtle.digest('SHA-256', encode(canonical))

normalize() is the cascade’s own canonical form — lowercase, NFKD decompose, strip combining accents, strip every non-alphanumeric run. It is the same function layer 2 of the matcher uses, which is the point: a layout is “the same layout” exactly when the matcher would treat the headers as the same strings.

Two consequences fall straight out of that definition:

• Cosmetic changes are invisible. Date of Birth, date_of_birth and DATE-OF-BIRTH normalise to the same string, so an export tool switching its capitalisation does not raise a false alarm.
• Order is significant. The join is positional, so a reordered header row is a different fingerprint. That is deliberate — a reorder is a real event in a positional format like CSV — but it is reported as a distinguishable one: the drift response carries reordered_only: true when the field set is unchanged and only its order moved.

Three answers, not two

The route takes a template_id and a headers array (capped at 200 headers per call) and returns one of three statuses. The third one is the reason this is useful rather than annoying.

POST /v1/layouts/drift
{ "template_id": "patient_demographics_v1",
  "headers": ["Vorname","Nachname","Geb_Datum","AHV","Mail"] }

→ { "template_id": "patient_demographics_v1",
    "status": "drift",
    "header_fingerprint": "9f2c…",
    "baseline_fingerprint": "41ab…",
    "baseline_last_used_at": "2026-07-30T09:12:44Z",
    "baseline_use_count": 37,
    "reordered_only": false,
    "diff": { "added": ["Mail"], "removed": ["email"], "common": [...] } }

One asymmetry in that payload is intentional and worth knowing about: added reports the raw incoming header exactly as it arrived — "Mail", capital M — because that is the string a human has to go and look for in the partner’s file. removed and common report the normalised form, because those are drawn from what we stored, and what we stored is normalised.

• stable — this exact layout has been confirmed before. Nothing to do.
• first_seen — there are no confirmed layouts for this (workspace, template) pair yet. This is not drift. A detector that shouts on the first file it ever sees gets muted in week one, and then it is not a detector.
• drift — prior layouts exist and this shape is not one of them. The response carries the field-level diff against the most recently used baseline, plus that baseline’s fingerprint, last-used timestamp and use count so you can judge how established the thing you are drifting from actually is.

Picking the most-recently-used layout as the baseline rather than the most-frequently-used one is a small decision with a clear rationale: when a partner migrates their export, the new shape becomes the truth, and you want tomorrow’s comparison to be against what you saw yesterday rather than against a shape that dominated the counter two quarters ago.

Where the layouts come from

Nothing here requires a separate registration step. Committing a mapping records the layout automatically: the full confirmed mapping (source column → target field, nulls included) is upserted against the fingerprint, with a use counter and a last-used timestamp. Two more routes expose the same store directly — POST /v1/layouts/lookup to read a previously confirmed mapping, and POST /v1/layouts/record to write one from an importer that manages its own commit flow.

The lookup is what makes “reuse last mapping” a one-click affair in an importer UI: an exact fingerprint hit returns the whole confirmed mapping and the cascade does not have to run at all. It is also the cheapest path through the product — but not a free one. Every map draws the small flat per-map fee, layout cache hits included; what a hit saves you is the metered AI layer and the latency, not the fee.

Layouts are workspace-scoped. There is no cross-tenant sharing of layouts or mappings — your partner’s header row is not evidence for someone else’s import, and it would be an odd thing to discover that it was.

Deterministic on purpose

Drift detection calls no model. It is a hash, a set difference and a table read, which buys three properties that matter more than sophistication here:

1. It is safe to run on everything. Read-only, read scope, no sample rows in the request — you send headers and a template id. There is no reason not to call it on every file that lands, including the ones you are confident about.
2. The answer is reproducible. The same header row returns the same fingerprint forever, so an alert you investigate three weeks later still refers to the same thing.
3. It fails in the boring direction. Without persistence configured there is no layout history, and every shape reads as first_seen — the detector goes quiet rather than fabricating a baseline.

What a header fingerprint cannot see

This technique is deliberately narrow, and it is worth being explicit about the drift it will not catch:

• Value drift. A column that keeps its name and changes its contents — dates switching from dd.mm.yyyy to ISO, codes switching from ICD-10 to something local — is invisible to a header hash. That is what field validators are for, and they run on the values at commit time.
• Semantic drift. If status quietly stops meaning what it meant, the header row is identical and the fingerprint is stable. No header-level mechanism can help you there.
• Headerless and positional feeds. A file with no header row has no fingerprint to take. Fixed-position feeds need a different check entirely.

What it does catch is the most common and most expensive class: a column added, a column removed, a column renamed. That is the class that silently degrades an import instead of breaking it — and one sha256 per file is a remarkably cheap way to stop being surprised by it.

The layout store and its three routes are documented in the layouts & drift reference, the request and response shapes are in the API docs, and the cost model — including why a cache hit still draws the flat fee — is written up in what a matching cascade costs.