Case 02 · scientific reproducibility

The chart survived. The method did not.

A paper cites an experiment file, but the source instrument export, outlier rule, normalization parameters, and exact plotting input disappeared when the postdoc left.

The code this page is about. Repeated from The output that catches it below.

$ makoto verify bundle handoffs/removed-predecessor/bundle \
  --policy receiver/policy.json \
  --schema-catalog receiver/catalog.json \
  --expected-manifest sha256:e24aae77ba1374162edff49aa0ab1d8dd7e38c333e134a2128ad1598fec99fb7 \
  --expected-head sha256:99aee76db46a720405da6d0015d427c791b0d4e02367b9e2bc36ce82ae812bce \
  --expected-artifact receiver/expected/removed-predecessor.artifact.json \
  --evaluation-time 2026-09-16T16:00:00Z
PASS        load-safely
PASS        parse-strictly
PASS        index-payloads
PASS        core-schemas
PASS        signatures
PASS        authorization-thresholds
PASS        metadata-profiles
PASS        authorization
PASS        graph-dependency-artifacts
FAIL        graph
PASS        roots-and-heads
FAIL        completeness-anchor
PASS        freshness-anchors
PASS        artifact-bytes
PASS        artifact-profiles
PASS        decision
DENY        decision
manifest sha256:e24aae77ba1374162edff49aa0ab1d8dd7e38c333e134a2128ad1598fec99fb7
E_PREDECESSOR_MISSING: predecessor 56b7be4394fe09c62ec7a3d5763cecc251e9696f267f35b2acc717b0d170a27a is absent
E_MANIFEST_SET: signed manifest sets do not equal the verified graph and bundle index
$ echo $?
1

Evidence boundary. Every terminal block on this page was captured by running the command shown against the checked-in reference fixtures with the Makoto core verifier at the pinned commit 276008e, which the gate below runs. Verification-summary tables are computed from the published reports with its property rules. The verification reports are byte-for-byte the files published under the end-to-end proof. Nothing below was typed by hand.

The problem

A postdoc submits a paper. The methods section cites one data file that supports the central claim. The reviewer asks four questions: which version of the raw data is this, were outliers removed and by what rule, who prepared the file, and can we get back to the original instrument readings?

Nobody can answer. The file passed through three hands over six months, the person who ran the cleaning step has graduated, and the parameters lived in a notebook nobody can find. The result is not a forged dataset. It is a lineage with a hole in it, and a hole looks exactly like a shortcut.

The runnable fixture uses the reference handoff, which has the same shape as a lab pipeline: a source export, two transformations, and a signed handoff. Two poisoned variants of the published evidence are tested: one drops the origin statement, one rewrites a step to claim it consumed the raw export directly.

The raw example

A complete evidence bundle answers the reviewer's questions from the statements themselves. Decoding each signed statement in the good bundle shows the chain: an origin claim over the raw export, a normalization step that consumed it, and a public-safe step that consumed the normalized file.

$ for f in handoffs/positive/bundle/attestations/*.dsse.json; do
  jq -r .payload "$f" | base64 -d | jq -c '{
    operation: (.predicate.operation.name // "origin"),
    input: .predicate.inputs[0]?.provenance.subjectName,
    output: .subject[0].name}'
done
{"operation":"Normalize customer records","input":"customers.raw.json","output":"customers.normalized.json"}
{"operation":"origin","input":null,"output":"customers.raw.json"}
{"operation":"Remove direct identifiers and bucket ages","input":"customers.normalized.json","output":"customers.public.json"}

The source export and normalized intermediate are published beside the final file, so every step's input and output bytes can be checked.

How it failed

Variant one, a missing step. The republished bundle simply leaves out the origin statement. Every remaining signature is still valid and the final bytes are unchanged, so a checksum or signature check passes. The bundle index now lists two statements instead of three.

Variant two, a rewritten step. The final statement is edited to say it consumed the raw export directly, skipping normalization, and is re-wrapped in a new bundle with the original signature copied across.

$ jq -r '.attestations[].path' handoffs/positive/bundle/bundle.json
attestations/1f28b72bcd4c1e9b7df71403ac6bb1670c2f2b09628ca6d76a2fa384db9a0848.dsse.json
attestations/56b7be4394fe09c62ec7a3d5763cecc251e9696f267f35b2acc717b0d170a27a.dsse.json
attestations/99aee76db46a720405da6d0015d427c791b0d4e02367b9e2bc36ce82ae812bce.dsse.json
$ jq -r '.attestations[].path' handoffs/removed-predecessor/bundle/bundle.json
attestations/1f28b72bcd4c1e9b7df71403ac6bb1670c2f2b09628ca6d76a2fa384db9a0848.dsse.json
attestations/99aee76db46a720405da6d0015d427c791b0d4e02367b9e2bc36ce82ae812bce.dsse.json
$ statement() { b=handoffs/$1/bundle; d=$(jq -r '.artifacts[0].statementDigest.sha256' $b/bundle.json); jq -r .payload $b/attestations/$d.dsse.json | base64 -d; }
$ statement positive | jq '.predicate.inputs[0].provenance'
{
  "statementDigest": {
    "sha256": "1f28b72bcd4c1e9b7df71403ac6bb1670c2f2b09628ca6d76a2fa384db9a0848"
  },
  "subjectName": "customers.normalized.json"
}
$ statement rewired-step | jq '.predicate.inputs[0].provenance'
{
  "statementDigest": {
    "sha256": "56b7be4394fe09c62ec7a3d5763cecc251e9696f267f35b2acc717b0d170a27a"
  },
  "subjectName": "customers.raw.json"
}
$ envelope() { b=handoffs/$1/bundle; d=$(jq -r '.artifacts[0].statementDigest.sha256' $b/bundle.json); cat $b/attestations/$d.dsse.json; }
$ diff <(envelope positive | jq .signatures) <(envelope rewired-step | jq .signatures) && echo 'signature bytes: copied unchanged from the original statement'
signature bytes: copied unchanged from the original statement

The rewired statement now points at 56b7be43… (the origin) instead of 1f28b72b… (normalization), while its signature bytes are unchanged.

Put Makoto in the path

Figures are rebuilt only from evidence that verifies end to end. A pull request that adds or changes a handoff runs the gate for each bundle the paper cites; the figure rebuild job depends on it.

The workflow

# Reproducibility gate for a paper's evidence bundles. A figure
# is rebuilt only from a handoff whose whole lineage verifies.
#
# Copy to .github/workflows/makoto-gate.yml in the repository that
# receives handoffs. Layout it expects:
#   handoffs/<name>/bundle/                  what the sender transferred
#   receiver/policy.json                     receiver-owned trust policy
#   receiver/catalog.json, receiver/resources/  pinned private schemas
#   receiver/expected/<name>.artifact.json   expected final artifact
name: Makoto gate before figure rebuild

on:
  workflow_dispatch:
  pull_request:
    paths:
      - "handoffs/**"

permissions:
  contents: read

env:
  # Pinned so these fixtures replay identically. Delete this line to
  # evaluate against the runner clock.
  EVALUATION_TIME: "2026-09-16T16:00:00Z"

jobs:
  gate:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        include:
          # The sender publishes manifest and head digests out of band.
          # The receiver pins them here, never inside the bundle.
          - handoff: removed-predecessor
            manifest: sha256:e24aae77ba1374162edff49aa0ab1d8dd7e38c333e134a2128ad1598fec99fb7
            head: sha256:99aee76db46a720405da6d0015d427c791b0d4e02367b9e2bc36ce82ae812bce
          - handoff: rewired-step
            manifest: sha256:c60982b592479600d4f8204c35b3215ebad80e7d7527f69bdea43e5eda4ecb7f
            head: sha256:5b9010046de6e13296dcc23db6bec9d1fe84326e81252402d9a549b45a2f20eb
    steps:
      - uses: actions/checkout@v4
      # Pin the verifier to an exact core commit, like any dependency.
      # Run it from a checkout: the CLI resolves its core schemas from
      # the repository tree. This is the core commit the hosted
      # schemas are pinned to.
      - uses: actions/checkout@v4
        with:
          repository: makoto-project/makoto
          ref: 276008eeec0f10d5322c4c627b8c103222fee841
          path: .makoto-core
          persist-credentials: false
      - uses: astral-sh/setup-uv@v6
      - name: Verify the received bundle
        id: verify
        shell: bash
        env:
          HANDOFF: ${{ matrix.handoff }}
          EXPECTED_MANIFEST: ${{ matrix.manifest }}
          EXPECTED_HEAD: ${{ matrix.head }}
        run: |
          verify() {
            uv run --project .makoto-core makoto verify bundle "handoffs/$HANDOFF/bundle" \
              --policy receiver/policy.json \
              --schema-catalog receiver/catalog.json \
              --expected-manifest "$EXPECTED_MANIFEST" \
              --expected-head "$EXPECTED_HEAD" \
              --expected-artifact "receiver/expected/$HANDOFF.artifact.json" \
              --evaluation-time "$EVALUATION_TIME" "$@"
          }
          status=0
          verify --json > makoto-report.json || status=$?
          verify || true
          exit "$status"
      - name: Keep the verification report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: makoto-report-${{ matrix.handoff }}
          path: makoto-report.json

  rebuild-figures:
    # Runs only when every handoff in the matrix was allowed.
    needs: gate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Rebuild figures from verified bytes
        run: echo "replace with your existing step"

Download makoto-gate.yml. Note the second matrix entry pins the manifest and head that the rewired bundle's own handoff declares. Even when the expectations match what was sent, the lineage still has to verify.

The gate keeps the JSON report as a build artifact, so a later job can summarize it into a signed verification summary without running verification again.

Which identity may sign which step

The receiver's trust policy binds each operation type to one authorized key. A statement cannot borrow another step's authority.

$ jq '.rules[] | {id, operationTypes, authorizedKeyIds}' receiver/policy.json
{
  "id": "urn:makoto:demo:rule:normalize",
  "operationTypes": [
    "urn:makoto:demo:operation:normalize"
  ],
  "authorizedKeyIds": [
    "sha256:47dea58ea00fae9417ee19d76755bfef690899021132effb04fe1f9e4f0c8059"
  ]
}
{
  "id": "urn:makoto:demo:rule:origin",
  "operationTypes": null,
  "authorizedKeyIds": [
    "sha256:fd110d301d2f077de1414b8f99f441b1403fab207b2052fbd2c065e4ee8e7dc2"
  ]
}
{
  "id": "urn:makoto:demo:rule:public-safe",
  "operationTypes": [
    "urn:makoto:demo:operation:public-safe"
  ],
  "authorizedKeyIds": [
    "sha256:8cef065b7af83669150b7d32704d3d3e75c3e9aee0d4f6740544c68e81b377aa"
  ]
}

The output that catches it

Missing step

Signatures, authorization, and final bytes all pass. The graph check fails because the normalization statement names a predecessor that is not in the bundle, and completeness-anchor fails because the signed manifest committed to three statements.

$ makoto verify bundle handoffs/removed-predecessor/bundle \
  --policy receiver/policy.json \
  --schema-catalog receiver/catalog.json \
  --expected-manifest sha256:e24aae77ba1374162edff49aa0ab1d8dd7e38c333e134a2128ad1598fec99fb7 \
  --expected-head sha256:99aee76db46a720405da6d0015d427c791b0d4e02367b9e2bc36ce82ae812bce \
  --expected-artifact receiver/expected/removed-predecessor.artifact.json \
  --evaluation-time 2026-09-16T16:00:00Z
PASS        load-safely
PASS        parse-strictly
PASS        index-payloads
PASS        core-schemas
PASS        signatures
PASS        authorization-thresholds
PASS        metadata-profiles
PASS        authorization
PASS        graph-dependency-artifacts
FAIL        graph
PASS        roots-and-heads
FAIL        completeness-anchor
PASS        freshness-anchors
PASS        artifact-bytes
PASS        artifact-profiles
PASS        decision
DENY        decision
manifest sha256:e24aae77ba1374162edff49aa0ab1d8dd7e38c333e134a2128ad1598fec99fb7
E_PREDECESSOR_MISSING: predecessor 56b7be4394fe09c62ec7a3d5763cecc251e9696f267f35b2acc717b0d170a27a is absent
E_MANIFEST_SET: signed manifest sets do not equal the verified graph and bundle index
$ echo $?
1

Output of jq '{decision, primaryError, errors, notPassed: [.checks[] | select(.status != "pass") | {id, status}]}' makoto-report.removed-predecessor.json against the report the workflow step wrote.

{
  "decision": "deny",
  "primaryError": "E_PREDECESSOR_MISSING",
  "errors": [
    {
      "causedByCheck": "graph",
      "code": "E_PREDECESSOR_MISSING",
      "context": {
        "statementDigest": {
          "sha256": "1f28b72bcd4c1e9b7df71403ac6bb1670c2f2b09628ca6d76a2fa384db9a0848"
        }
      },
      "message": "predecessor 56b7be4394fe09c62ec7a3d5763cecc251e9696f267f35b2acc717b0d170a27a is absent",
      "step": 9
    },
    {
      "causedByCheck": "completeness-anchor",
      "code": "E_MANIFEST_SET",
      "context": {
        "bundleId": "urn:uuid:55555555-5555-4555-8555-555555555555"
      },
      "message": "signed manifest sets do not equal the verified graph and bundle index",
      "step": 11
    }
  ],
  "notPassed": [
    {
      "id": "graph",
      "status": "fail"
    },
    {
      "id": "completeness-anchor",
      "status": "fail"
    }
  ]
}

What a verification summary records

A verification summary over this report is FAILED with E_REPORT_DENIED, because the decision is deny. Its evaluation sidecar still records each verified property's exact rule against the report's checks:

Property	What its rule reads in this report	Result
`MAKOTO_AUTHORIZED`	`signatures` pass, `authorization-thresholds` pass, `authorization` pass; handoff authorization pass	Established
`MAKOTO_GRAPH_COMPLETE`	`index-payloads` pass, `graph-dependency-artifacts` pass, `graph` fail, `roots-and-heads` pass, `completeness-anchor` fail	Not established: `E_PROPERTY_GRAPH_COMPLETE`
`MAKOTO_FRESHNESS_ANCHORED`	`freshness-anchors` pass; methods: `expected-artifacts` pass, `expected-heads` pass, `expected-manifest` pass, `max-age` not_checked, `nonce` not_checked	Established
`MAKOTO_SCHEMA_CONFORMANT`	`core-schemas` pass, `metadata-profiles` pass, `artifact-profiles` pass; the policy requires 2 digest-pinned profiles	Established
`MAKOTO_REPRODUCED`	Needs a second `allow` report from an independent Transform L2-or-higher platform; none is supplied	Not established: `E_PROPERTY_REPRODUCED`

Computed from the published report and the policy whose digest it records, with the property rules in core at the pinned commit. Track levels are not shown: they also need the statements themselves and any platform assessments.

Open the complete removed-predecessor report.

Rewritten step

The copied signature no longer matches the edited payload, so signatures fails first. Without a valid signature no policy rule is satisfied, and the graph checks that depend on it are skipped rather than reported as passing.

$ makoto verify bundle handoffs/rewired-step/bundle \
  --policy receiver/policy.json \
  --schema-catalog receiver/catalog.json \
  --expected-manifest sha256:c60982b592479600d4f8204c35b3215ebad80e7d7527f69bdea43e5eda4ecb7f \
  --expected-head sha256:5b9010046de6e13296dcc23db6bec9d1fe84326e81252402d9a549b45a2f20eb \
  --expected-artifact receiver/expected/rewired-step.artifact.json \
  --evaluation-time 2026-09-16T16:00:00Z
PASS        load-safely
PASS        parse-strictly
PASS        index-payloads
PASS        core-schemas
FAIL        signatures
FAIL        authorization-thresholds
PASS        metadata-profiles
FAIL        authorization
PASS        graph-dependency-artifacts
SKIPPED     graph
SKIPPED     roots-and-heads
FAIL        completeness-anchor
PASS        freshness-anchors
SKIPPED     artifact-bytes
SKIPPED     artifact-profiles
PASS        decision
DENY        decision
manifest sha256:c60982b592479600d4f8204c35b3215ebad80e7d7527f69bdea43e5eda4ecb7f
E_SIGNATURE_INVALID: signature verification failed
E_SIGNER_UNAUTHORIZED: no matching policy rule met its signature threshold
E_MANIFEST_SET: final artifact is not an exact terminal subject of its declared head
E_MANIFEST_SET: signed manifest sets do not equal the verified graph and bundle index
$ echo $?
1

Output of jq '{decision, primaryError, errors, notPassed: [.checks[] | select(.status != "pass") | {id, status}]}' makoto-report.rewired-step.json against the report the workflow step wrote.

{
  "decision": "deny",
  "primaryError": "E_SIGNATURE_INVALID",
  "errors": [
    {
      "causedByCheck": "signatures",
      "code": "E_SIGNATURE_INVALID",
      "context": {
        "keyid": "sha256:8cef065b7af83669150b7d32704d3d3e75c3e9aee0d4f6740544c68e81b377aa",
        "statementDigest": {
          "sha256": "5b9010046de6e13296dcc23db6bec9d1fe84326e81252402d9a549b45a2f20eb"
        }
      },
      "message": "signature verification failed",
      "step": 5
    },
    {
      "causedByCheck": "authorization-thresholds",
      "code": "E_SIGNER_UNAUTHORIZED",
      "context": {
        "statementDigest": {
          "sha256": "5b9010046de6e13296dcc23db6bec9d1fe84326e81252402d9a549b45a2f20eb"
        }
      },
      "message": "no matching policy rule met its signature threshold",
      "step": 6
    },
    {
      "causedByCheck": "completeness-anchor",
      "code": "E_MANIFEST_SET",
      "context": {
        "artifactDigest": {
          "sha256": "ecd821dbe6f4204c4aad64878643d7954cb8c94abbf8d0b28f9c2bfc272f781e"
        },
        "bundleId": "urn:makoto:demo:negative:rewired-step",
        "head": {
          "sha256": "5b9010046de6e13296dcc23db6bec9d1fe84326e81252402d9a549b45a2f20eb"
        },
        "subjectName": "customers.public.json"
      },
      "message": "final artifact is not an exact terminal subject of its declared head",
      "step": 11
    },
    {
      "causedByCheck": "completeness-anchor",
      "code": "E_MANIFEST_SET",
      "context": {
        "bundleId": "urn:makoto:demo:negative:rewired-step"
      },
      "message": "signed manifest sets do not equal the verified graph and bundle index",
      "step": 11
    }
  ],
  "notPassed": [
    {
      "id": "signatures",
      "status": "fail"
    },
    {
      "id": "authorization-thresholds",
      "status": "fail"
    },
    {
      "id": "authorization",
      "status": "fail"
    },
    {
      "id": "graph",
      "status": "skipped"
    },
    {
      "id": "roots-and-heads",
      "status": "skipped"
    },
    {
      "id": "completeness-anchor",
      "status": "fail"
    },
    {
      "id": "artifact-bytes",
      "status": "skipped"
    },
    {
      "id": "artifact-profiles",
      "status": "skipped"
    }
  ]
}

What a verification summary records

Property	What its rule reads in this report	Result
`MAKOTO_AUTHORIZED`	`signatures` fail, `authorization-thresholds` fail, `authorization` fail; handoff authorization pass	Not established: `E_PROPERTY_AUTHORIZED`
`MAKOTO_GRAPH_COMPLETE`	`index-payloads` pass, `graph-dependency-artifacts` pass, `graph` skipped, `roots-and-heads` skipped, `completeness-anchor` fail	Not established: `E_PROPERTY_GRAPH_COMPLETE`
`MAKOTO_FRESHNESS_ANCHORED`	`freshness-anchors` pass; methods: `expected-artifacts` pass, `expected-heads` pass, `expected-manifest` pass, `max-age` not_checked, `nonce` not_checked	Established
`MAKOTO_SCHEMA_CONFORMANT`	`core-schemas` pass, `metadata-profiles` pass, `artifact-profiles` skipped; the policy requires 2 digest-pinned profiles	Not established: `E_PROPERTY_SCHEMA_CONFORMANT`
`MAKOTO_REPRODUCED`	Needs a second `allow` report from an independent Transform L2-or-higher platform; none is supplied	Not established: `E_PROPERTY_REPRODUCED`

Open the complete rewired-step report.

Publish the evidence beside the result

Instrument export Start with exact collected bytes, acquisition time, instrument identity, and the source claim.
Outlier removal Append an operation statement with predecessor bindings, code identity, parameters, and exact output.
Normalization Append the next statement with the statistics and software evidence needed to interpret the claim.
Publication handoff Sign the exact graph and final plotting artifact distributed with the paper.

The honest limit

Makoto can preserve the inputs and signed claims needed to reproduce the workflow. It cannot reconstruct missing intermediate bytes after publication or prove that an instrument was calibrated merely because its identifier was signed.

Concrete publication bundle

Input	Instrument export, calibration reference, collection parameters, and a signed origin statement over the exact raw bytes.
Steps	Append outlier-removal, normalization, analysis, and plotting statements; bind code and parameter artifacts where available.
Handoff	Publish the final figure data with the signed graph, required profiles, and exact terminal artifacts.
Reproduction check	A new researcher verifies the received bytes and can identify every missing intermediate instead of guessing it.

The checked-in customer-data proof uses the same source → two transforms → signed handoff structure with downloadable intermediate artifacts.

What else this handles

Journal submissions that require data availability statements with verifiable provenance.
Clinical datasets where every processing step must be auditable by a regulator.
Collaborative research where datasets pass between institutions over months.
Machine-learning papers where preprocessing of the training data changes the result.

Key insight. Reproducibility is not about trusting people. It is about data that can answer questions about its own history, and refusing the answer when a step is missing or rewritten.

Make the next paper inspectable

Start the graph when data leaves the instrument, not when the PDF is submitted.

Run the source-to-handoff pattern Read the casebook entry