9 · Data Dictionary, Pipeline, and Audit

9.1 · Data dictionary and quality states

Measurement systems live or die by their data contract. Table 8 is the system's data dictionary at the level of source families and quality states; the full registry is versioned in the engineering system. Two design properties of the dictionary deserve emphasis before the table. First, every field carries its observation time, its quality state, and its source and method version — a value without its provenance is not a measurement, it is a claim. Second, every field has an explicit non-observation semantics: a missing or failed field is recorded as a failure state, never as a silently fabricated zero, and the most recent valid value continues with an alert (Section 6.4).

Table 8. Data dictionary: source families, cadence classes, and the quality states that gate each observation. “Cadence class”: fast (minutes), daily, event.

The quality-state vocabulary is deliberately small and shared across families, because the downstream logic (circuit breakers, missing-value classification, audit) is state-driven: each state has a defined effect on how the observation enters (or does not enter) the computation. The mapping from raw state to effect is registered and versioned, so that “why did this value not update?” has a mechanical answer (Section 9.3).

9.2 · Collection–pricing–display pipeline

The pipeline that turns external reality into a published line has three stages, matching the three layers of Section 6.1 and the three latency calibers of Section 7.3:

• Collection (L1). Fast sources are polled on the order of 1–5 minutes; slow and event sources on daily and event cadences. Raw observations are written to an append-only store with their observation times and source versions. The collection stage never interprets; it records.
• Pricing (L2). The derived-indicator registry computes normalized factors, blocks, the settled anchor, and the RI envelope tracks. This stage is deterministic: the same L1 observations and versions produce the same L2 output. It is the stage where the missing-value taxonomy and circuit breakers act.
• Display (L3). The index family is computed and projected through the presentation-layer mapping (Section 6.8). This is the stage where the actual and nominal lines diverge and reconverge at each close.

Two pipeline disciplines deserve emphasis. Idempotency: every stage is designed to be recomputable from its inputs and versions, so the pipeline can in principle be replayed from any point (Section 9.3). No retroactive editing: the display history is written once; corrections propagate as new versions, never as edits to old ones (the no-backcasting discipline of Section 3.5, applied at the data level). A reader of the published series can therefore in principle tell what was believed when, which is the property designed to make the series auditable rather than merely available.

9.3 · Audit and replay protocols

The system's audit protocol is built from the invariants of Section 6.9 plus the replay property of the layers. It runs continuously and reports four checks:

1. Dual-track audit. The audit inequality $\mathrm{H}_i(t) \ge |\mathrm{RI}_i(t)|$ (Proposition 6.14) is checked per tick; a violation is a data-integrity alarm. Because both lines derive from the same envelope set by formulas differing only in the absolute value, the check is tight: it detects source desynchronization, misattribution, and envelope corruption.
2. Conservation audit. The flagship conservation (Proposition 6.9) is rechecked at every settlement; the bound approximation (Proposition 6.11) is rechecked whenever $D$ or $R$ is nonzero.
3. Replay audit. The RI renewal log is a deterministic ledger (Lemma 6.8, Proposition 6.13): any past instantaneous value is recomputed exactly from the log; the audit is designed so that the published live line can be checked against the log's arithmetic image.
4. Reconciliation audit. The self-collected observations are reconciled against third-party sources where both exist; a persistent unreconciled gap is flagged and investigated (Section 9.4).

The replay protocol is the most distinctive element: it means the live signal is auditable without being reproducible. An external party cannot reproduce the RI series from scratch (the parameters are reserved), but an auditor granted the renewal log can in principle verify that the published series matches what the log implies. Auditability and non-reproducibility are not in tension; they are separated by the log.

9.4 · Reconciliation and correction

Correction is a first-class process, not an afterthought. Three principles govern it. Correct to truth, forward. A confirmed error is corrected to the true value in the current computation and propagated forward; the historical record is annotated, not silently edited (Section 3.5). Never manufacture a fix. If the true value cannot be reconstructed, the field is marked unresolved and governance is escalated (Section 10.5); a recovery mechanism is never used to mask a fault. Log everything. Every correction carries its evidence, its decision, and its version; the correction log is itself auditable. These principles convert “the system makes errors” — which every measurement system does — into “the system's errors are bounded, visible, and correctable,” which is the property a fair measurement system must have.

Back to contents  |  ← 8 · Verification: Segment-by-Segment Acceptance and Evidence  |  10 · Operations and Governance →