Req Room

1The problem

eBay leadership cannot see consolidated headcount. Each team’s data lives in a different system — finance planning, team-owned spreadsheets, and an unknown long tail of other tools. There is no single view anyone trusts, which makes three downstream jobs painful:

1. Allocating limited headcount across teams under a fixed cap — both dollars and a req ceiling.
2. Forecasting how many people each function needs from its demand drivers.
3. The planning process itself — weeks of slides and spreadsheet reconciliation before any decision can be defended to a VP.

Consolidation is upstream of all three. You cannot allocate or forecast against numbers nobody believes.

The requirement behind the requirement

The exact source systems are not known. So the tool has to prove it can ingest from anywhere — not by shipping a connector for every SaaS product in existence (not credible, not demoable), but by making the onboarding of a new, unseen source fast, verifiable, and safe.

2Non-goals

Deferred to Phase 2 so Phase 1 stays shippable: the full leadership dashboard, the demand drivers and dual-constraint optimizer, the explainer / what-if / critic AI heads, policy-corpus retrieval, model routing, drift detection, and the written case study.

Out of scope permanently: agent frameworks, vector stores, orchestration libraries. Narrow prompts with schema-validated output plus real verification beat an agentic pipeline here — and demonstrating that judgment is part of the point.

3The invariant

Every AI output is either prose about numbers computed elsewhere, or a structured proposal that deterministic code validates and applies. There is no third category. Four corollaries, enforced mechanically rather than promised:

• Confidence and grounding are computed, never self-reported. The model is never asked how confident it is — the question is uncorrelated with correctness, and saying so out loud is part of the pitch.
• AI proposes at design time; deterministic code disposes at runtime. The mapper runs offline and its output is frozen. The runtime app makes zero LLM calls.
• Conflicts are surfaced, never silently resolved.
• No LLM judge in Phase 1. Mapping correctness is objectively checkable against a known key — a judge would add nothing and would violate this very invariant.

4Architecture

  any source (CSV / Sheets export / Anaplan export / unknown tool)
            │
   [AGENT: schema mapper]  ── profiles, hypothesizes, TESTS against
            │                  HARD constraints + HEURISTICS (kept distinct),
            │                  self-corrects, ESCALATES on heuristic-only evidence
            │               ── human approves ONCE
            ▼
   frozen mapping artifact (versioned, diffable, tested, deterministic)
            │
            ▼
   ┌──────────────────────────────────────────────────────────┐
   │  CANONICAL HEADCOUNT LEDGER  (bitemporal)                │
   │  person / req / approved-plan / org / cost-center        │
   │  + valid-time + transaction-time + source + freshness    │
   └──────────────────────────────────────────────────────────┘
            │
   [DETERMINISTIC: entity resolution + reconciliation]
            │  ── lineage on every number
            │  ── conflicts SURFACED, not resolved away
            ▼
   trusted ledger ──▶ thin surface (trace / contested queue / eval report)
                  ──▶ (Phase 2: leadership view, optimizer, AI heads)

   [EVAL HARNESS] ── labeled held-out set + BASELINES.
                     Built BEFORE the AI it grades.

5.1The canonical ledger — bitemporal

Person (with FTE fraction and employment type), Req (with backfill-vs-net-new), ApprovedPlan, and Org / CostCenter kept deliberately separate — because HRIS org boundaries and finance cost-center boundaries do not agree, and that disagreement is a fact to model, not a bug to paper over. Both hierarchies are effective-dated; otherwise a single reorg silently corrupts every point-in-time query ever run.

Two time axes, not one

Valid time is when the fact was true in the world — the person started on 1 October. Transaction time is when we learned it — the export we ingested on 20 October said so. Sources restate history: a correction lands and last month’s truth changes underneath you.

Without both axes you cannot separate “what did we believe on 1 October?” from “what was actually true on 1 October?” — and leadership asks those two questions in the same breath, usually without noticing they are different.

Deliberately excluded, and stated rather than forgotten: manager/reporting chain (a third hierarchy in matrix orgs), currency and cost basis, legal entity. Real, needed in production, out of Phase 1 — and the demo says so rather than pretending they don’t exist.

5.2Entity resolution — honest about where rules end

Two problems that deserve two different honest answers.

Org and cost-center crosswalks are genuinely rule-based: the set is finite and enumerable, and every match emits the rule that produced it.

Person and req matching is not, and will not be. Name variants, rehires, vendor IDs, and transfers guarantee it degrades into thresholded similarity matching. The spec says so — a “no fuzzy matching” claim would not survive first contact with a data engineer who has done this. So instead: explicit match keys and precedence, and an explicit ambiguity band. Above the upper threshold, auto-match. Below the lower, no match. Between them, escalate to a human — never silently guess in the band.

5.3Reconciliation — three states, N sources

Three corrections the council forced, each of which the original model got wrong:

• Partial disagreement is real. With five sources, a 3-vs-2 split is neither “agreed” nor cleanly “reconciled.” The state function takes the full multiset of claims, not a pair.
• Reconciled ≠ correct. “HRIS is authoritative” confidently misresolves whenever HRIS itself lags — a termination not yet processed. So reconciled output carries the rule and the dissenting claim, and rules are versioned so historical outputs stay reproducible under the rules that produced them.
• Agreed ≠ correct either. Sources sharing a common upstream error agree perfectly. This system cannot catch that, and the demo says so.

Freshness is rendered as data, not buried in a tooltip. “412 (Anaplan, as of 6 days ago)” is a materially different claim from “412 (as of this morning),” and quarter-end planning turns on the difference.

5.4The eval harness — built before the AI it grades

This is the interview-critical component, and it ships first. In the previous build it was scheduled last and never got finished — a mistake this ordering structurally prevents. Building the harness before the features it grades forces every AI component to be born measurable.

What gets measured, and against what

• A labeled held-out set. One entire source is held out and never tuned against — because the candidate authors both the fixtures and the answer key, and an interviewer will absolutely ask what passing your own test proves.
• Baselines, reported alongside. A number without a baseline is decoration. “The agent maps 94% of fields correctly, where a string-similarity baseline gets 61% and a human gets 97%” is a result. “94%” is not.
• The right confusion matrix: correct-map / wrong-map / correct-escalate / over-escalate. Over-escalation is a real cost — an agent that escalates everything is useless while scoring perfectly on any naive “never wrong” metric.
• Wilson confidence intervals on every rate. With ~50–150 field decisions, one miss swings recall by 10–20 points. Reporting three significant figures off n=60 is exactly the overclaiming this project exists to prevent.

5.5The mapper agent — with an honest feedback signal

The v1 claim that the agent’s validators constituted “ground truth” was partly circular, and the out-of-family critic caught it: the Q3 Ask (fte) trap column passes type-checking, referential integrity, and distribution sanity. A blind spot in the validators blinds the agent and the eval identically. The fix is to separate the checks by epistemic strength and never conflate them again.

Hard constraints — real ground truth. Failing one means the mapping is wrong.

• Schema and type validation
• Referential integrity — do these cost centers actually join to known cost centers?
• Control-total reconciliation — does the FTE sum match an independently known total? This is the only check that catches the trap column, and it works only because ApprovedPlan now exists to give “requested headcount” a correct home.

Heuristics — plausibility only. Passing one means nothing.

• Distribution sanity — does this look like a level? a geo? a date?
• Column-name semantics

The loop: profile (sample rows, distributions, cardinality, null rates) → hypothesize → test (hard constraints first, heuristics second, never conflated) → self-correct on hard failure → escalate where only heuristics apply → freeze the approved mapping as a versioned artifact that runtime replays deterministically.

The demoable artifact is the trace: three hypotheses, two killed by hard constraints, one escalated because only a heuristic supported it. That trace — including the escalation — beats a clean success, because it shows the agent knows the boundary of what it knows.

5.6The thin visible surface

Not the leadership view. Three screens, minimal chrome, existing purely so a non-builder can see something: the agent trace viewer (hypotheses, which checks killed them, what escalated and why), the contested queue (live disagreements, deltas, sources, owners, ages), and the eval report (confusion matrix, baselines, intervals, held-out results). Roughly two days — not a phase.

6Fixtures — synthetic, pathologically realistic

Fixture credibility is a first-order risk. If the planted disagreements don’t look like real corporate data, a planner in the room dismisses the whole tool — so these are drawn from real failure modes, not invented for convenience.

Five sources: an HRIS-shaped export, an ATS-shaped export, an Anaplan-shaped export, and two team spreadsheets — one of which is held out of tuning entirely and reserved for the eval set.

7Stack

Next.js 16 + TypeScript + Tailwind v4. Committed JSON fixtures, no database — which sidesteps a known deploy crash, while an explicit import contract and validation CLI keep ingestion a real boundary rather than a demo shortcut. AI runs offline via claude -p with zod-gated, cached outputs.

Reproducibility is enforced, not assumed: model ID, CLI version, prompt text, inputs and outputs are all captured as committed artifacts. “Offline” means nothing if the model silently changes underneath you. Cost and latency are measured and published per mapping — a cheap model for profiling and column classification, a frontier model for mapping hypotheses. You will be asked.

8Build order

Proof-first, and the ordering is the point.

1. Canonical ledger — bitemporal, including ApprovedPlan — plus entity resolution and the reconciliation engine. Deterministic, fully tested.
2. Eval harness, labeled held-out set, and baselines — so it grades the mapper from its very first commit.
3. The mapper agent — hard constraints versus heuristics, escalation, frozen artifacts.
4. The thin visible surface — trace viewer, contested queue, eval report.

Done when an unseen, never-tuned-against source can be dropped in, mapped, checked against hard constraints, escalated where only heuristics apply, frozen to a versioned artifact, replayed deterministically into the ledger, and reconciled against four other sources — while the harness reports a confusion matrix with baselines and Wilson intervals, and the trap column is caught by control-total reconciliation, on camera.

9Risks

10What this proves in the room

• Judgment about where AI belongs. Semantic, ambiguous work goes to the model. Anything a human acts on is deterministic.
• An agent with an honest feedback signal — and the ability to say precisely where that signal runs out and a person takes over. Most candidates cannot draw that line, and drawing it is the whole skill.
• Evaluation, not vibes. A real confusion matrix, baselines, confidence intervals, and a refusal to plot a curve the data can’t support.
• Trust engineering. Contested as a first-class state; bitemporal lineage; refusing to assert, as a feature.