Data provenance fails in familiar, expensive ways.
These are cases in which data left its source and someone later needed to prove what happened. Each case names the evidence Makoto needs to carry and the decision a receiver can make with it.
Can the receiver verify the story without trusting the producer’s system?
A producer creates source data, signs its origin, normalizes it, produces a public-safe derivative, and signs the handoff. The receiver starts with its own schemas and signer policy, reconstructs the graph, and hashes the bytes it actually received.
- 01SourceBind collection claims to exact source bytes.
- 02TransformPoint each step to its exact predecessor.
- 03HandoffSign roots, profiles, and the final artifact.
- 04VerifyApply receiver-owned policy and re-hash the data.
Six ways the history gets lost
- 01
Poisoned pipeline
A corrupted sensor file carries impossible readings, duplicate rows, and a malicious payload into downstream tables. Predecessor links expose the blast radius and the signed claims that admitted it.
Open case → - 02
Scientific reproducibility
A paper cites
Open case →experiment.csv, but the outlier removal, normalization, and original source have disappeared with the people who ran them. The lineage graph preserves a path back to the result. - 03
Dataset release
A release includes the data but not the evidence. Attach signed origin, transformation, and digest records so consumers can inspect exactly what shipped.
Open case → - 04
Configuration postmortem
Throughput falls after a weekend change made by a service account. A transformation claim records the configuration, code identity, executor, and affected artifact at the time of the run.
Open case → - 05
AI dataset gate
A plausible label has been poisoned. The model builder refuses data unless its source, processing steps, authorized attesters, required profiles, and final bytes verify.
Open case → - 06
Invisible Unicode
Signatures and hashes prove the hidden characters were not altered; they do not make the characters safe. A profile requires an explicit analysis and a normalization or rejection decision.
Open case →
The evidence patterns inside those cases
| Pattern | What it records | Why it matters |
|---|---|---|
| Origin statement | Source identity, collection method, acquisition time, governing claims, and the source artifact digest. | Every lineage graph has a defensible beginning. |
| Transformation statement | Input statement and artifact digests, output digest, transformation identity, code reference, parameters, and executor claims. | Each step can be checked without editing the step before it. |
| Stream window | A bounded time or offset range, partition identity, record count, and a digest or Merkle root for that window. | Streaming systems can attest to stable windows instead of pretending the stream is one static file. |
| Bundle and handoff | The statement graph, required profiles, roots, final artifact, and completeness claims selected for transfer. | The receiver knows what evidence was intentionally delivered and what it must resolve. |
| Signed envelope | The exact statement payload plus one or more DSSE signatures. | Content tampering becomes detectable, while authorization remains a receiver policy decision. |
Unknown is better than invented.
The retained SQuAD record is useful because it shows what a provenance tool must not do. The public dataset has a familiar source and a final artifact, but intermediate hashes were not published and no third party attested to bias, quality, or regulatory claims.
Those absences belong in the record. Makoto can distinguish signed evidence from reconstructed narrative; it cannot retroactively create proof that the source pipeline never emitted.
A useful example names the producer, recipient, bytes, transformation graph, trust policy, and failure that must close the gate.
Bring the handoff your data team struggles with. A good contribution can be a narrative, a schema profile, a failing fixture, a receiver policy, or a tested implementation.