What a Five-Layer Matching Cascade Costs You (And What It Doesn’t)

Ask a language model to map Geb_Datum onto date_of_birth and it will get it right. Ask it forty times a day, across two hundred columns, for every partner file your importer touches, and you have built a system whose cost, latency and failure mode all scale with the number of columns you were handed — including the thirty that were called email and needed no intelligence at all.

AdaptivMapr’s cascade exists to make the model the last resort rather than the first. Five layers, ordered cheapest-first, each with a published auto-accept threshold. A column that resolves at layer 1 is never described to a model at layer 5. That single property is the biggest cost lever in the product, and this post is the specific version of it: what each layer does, what it accepts at, and what it charges.

The five layers, in order

Layers 1 to 3 are pure compute — no network, no model, no key required. Layers 4 and 5 are configuration-gated and fail soft to OFF when their keys are absent, which means a deployment without an embedding key or an LLM key still maps files; it just stops earlier.

Layer 1 — Statistics

Every mapping a human confirms in your workspace becomes evidence. When the same source header shows up again, the cascade looks at what was confirmed for it before and auto-accepts on two production-tested rules:

{ minN: 100, minRatio: 0.95 }   // 100+ observations, 95% agreement
{ minN:  20, minRatio: 1.00 }   //  20+ observations, unanimous

Two rules rather than one because the shapes of certainty are different: a header seen a hundred times can carry a little disagreement, a header seen twenty times cannot. Neither rule is a tuning knob — loosening them is how a learning layer starts confidently repeating an early mistake.

Layer 2 — Heuristic

The header is normalised — lowercased, NFKD-decomposed, accents stripped, every non-alphanumeric run removed — and compared against the template field’s key, its human label, and every one of its hints. Hints ship in German, French, Italian, English and Spanish, so Geburtsdatum, date de naissance, fecha de nacimiento and DOB all land on the same field with no model involved.

This layer is also the one you can improve yourself. Adding a hint is the highest-leverage change in the whole engine: it converts a class of columns that used to reach layer 5 into a class that resolves for free, permanently, for every workspace.

Layer 3 — Fuzzy

A token-set ratio combined with Levenshtein distance over the normalised strings, auto-accepting at 0.80. This is what absorbs typos and token-order drift — patient_dob against dob_patient, e-mail adress against email_address — without a model and without a dictionary.

Layer 4 — Semantic

Cosine similarity between an embedding of the header and an embedding of the field’s label plus hints. It auto-accepts at 0.78 and surfaces a suggestion (rather than a decision) down to 0.65. Embeddings are cheap and cached, and the layer is silently disabled when no embedding key is configured — a missing key degrades the cascade, it never breaks it.

Layer 5 — The language model

Everything still unresolved goes up in one batched, collision-aware call. Not one call per header — one call per match(). The request is constrained to the allowed column set and is explicitly told which target fields other headers have already claimed, so the model is choosing inside a bounded space rather than inventing field names.

The call goes to phi-cloud, an OpenAI-compatible, PHI-aware gateway. On a full-data run it carries X-PHI: true and an X-Region header so the gateway forces a PHI-eligible, in-region model; jurisdiction and BAA coverage are enforced there, on the gateway that signs for them.

Contests are settled by quality, never by column order

There is a bug class that every first-generation matcher has, and we had it too. If you walk the headers left to right and let each one claim its best target field on the spot, the winner of a contested field is decided by position. Given ["e_mail_alt", "email"], the 0.85 heuristic hit on e_mail_alt claims email first, and the exact 1.00 match that arrives one column later is left unmapped. Reverse the two columns and you get a different answer for the same file.

A mapping engine whose output changes when you reorder the input columns is not a mapping engine. It is a race.

The cascade now proposes across layers 1 to 3 without claiming anything, arbitrates once, then runs the paid layers only for what is still unresolved. The sort is (layer rank, then confidence, then column index). Layer rank leads because the cascade’s layer order is its trust order: a statistics hit learned from real human confirmations outranks any heuristic hit, whatever the two confidence numbers say. Confidence only breaks ties inside a layer — which is exactly where the bug lived. Layer 5 arbitrates its own picks the same way, and a target field, once assigned, cannot be claimed twice in the same call.

Now the bill

AdaptivMapr is pay-as-you-go on a prepaid token wallet — a minimum top-up of about $10, shared across the phi-cloud suite. There is no free tier. That matters for reading the next paragraph honestly: “no model was called” is not the same as “this was free”.

• Every map draws a small flat fee — around $0.001, a few tokens. That includes a map that resolved entirely in layers 1 to 3, and it includes a re-map of a known template and input that hits the layout cache and uses no AI at all. We charge for the map, not for the drama.
• Only the metered work bills tokens. Layer-5 cleanup, any-to-any convert and structural reshape bill the phi-cloud tokens they actually consume at provider cost × 2 — or × 0.5 if you bring your own LLM key.
• PHI/enterprise routing multiplies the whole charge by 1.2. The flat fee included. It is locked until your workspace accepts the BAA in-app, and the surcharge follows the routing a run actually got, not the policy your workspace has set — charging a BAA premium for a run that bypassed the BAA gate is a claim nobody should be making.

So the economics of the cascade are not “layers 1 to 4 are free”. They are: the variable, unbounded, latency-carrying part of the cost — the model — is only spent on the columns that genuinely needed judgment, and everything else settles against a fixed, predictable per-map fee.

What it costs you that isn’t money

Three honest trade-offs, because a cascade is not free of them:

1. Thresholds are policy. A fuzzy auto-accept at 0.80 will occasionally accept something a human would have queried. That is why every medium- and high-risk template returns requires_hitl: true and hitl_status: "pending_review" — the engine is telling your commit workflow where to put a human, rather than pretending it does not need one.
2. The statistics layer is workspace-scoped. It learns from your confirmations, not from anyone else’s. That is the right privacy answer and it is also a slower start: a brand-new workspace spends more of its early maps in layers 2 and 3 than a mature one does.
3. Layers 4 and 5 can be switched off. That is a feature for a locked-down deployment, and it is a limitation everywhere else. A cascade with no metered layers still maps the columns it can name; it simply hands you a longer unresolved list instead of a guess.

The short version

The model is the last layer, not the first. It is called once per map rather than once per column, inside a bounded candidate set, with collisions already ruled out. Everything it never sees resolved against a published threshold you can look up, and the answer does not depend on the order your columns happened to arrive in.

The layer thresholds and the per-layer behaviour are written up in the cascade reference; what leaves your system in each mode is in schema-only vs full-data; and the exact fee arithmetic — flat fee, token markup, PHI surcharge — is on the pricing page.