Guide · a sharded corpus kept in Git

Map a corpus index onto Makoto documents.

Many corpora live as a Git repository of small index files that point at large shards stored elsewhere. This guide follows one such corpus from its directory indexes down to a single shard, and shows which Makoto document carries each part, what a reader can then check, and what stays outside Makoto.

Every digest, signature and transcript in this guide is generated. scripts/corpus_example.py builds the corpus, signs its record with the Makoto reference CLI at the pinned core commit, runs each command shown, and saves what it printed. The keys are insecure example-only keys derived from fixed seeds. The archive, the letters and the society that transcribed some of them are fictional.

The command this guide builds toward. Repeated from Verify the record below.

$ makoto verify bundle text/letters/makoto \
  --policy lineage/policy.json \
  --schema-catalog lineage/catalog.json \
  --expected-artifact ../expected-letters.json
PASS        load-safely
PASS        parse-strictly
PASS        index-payloads
PASS        core-schemas
PASS        signatures
PASS        authorization-thresholds
PASS        metadata-profiles
PASS        authorization
PASS        graph-dependency-artifacts
PASS        graph
PASS        roots-and-heads
PASS        completeness-anchor
PASS        freshness-anchors
PASS        artifact-bytes
PASS        artifact-profiles
PASS        decision
ALLOW       decision
manifest sha256:b30f91d37734806e0b5e8d87f9a2a00c28202ea361c9737e2a9e70b7bc65157b

The corpus as it sits in Git

The repository holds index files, manifests and lineage records. The shards, which in a real corpus run to hundreds of megabytes each, live in a separate content-addressed store (a lookaside) and are fetched by digest. They never enter Git.

index.yaml                       lists the directories below it
text/
  index.yaml                     lists letters/
  letters/
    letters.yaml                 leaf manifest: languages, license, sources, shards
    ingest.yaml                  the settings the ingest ran with
    makoto/                      the lineage record for this leaf
lineage/
  policy.json                    which keys may sign which kind of step
  catalog.json                   schemas the policy names
  license.profile.json           the digest-pinned reference to one of them
  schemas/license-v1.schema.json
  check_lineage.py               the pull-request check
.github/workflows/lineage.yml    runs the check and a sign-off check

Every indexable directory has an index.yaml naming the directories below it, so a tool pointed at any directory can walk down to every leaf manifest beneath it.

# Directories beneath this one that hold corpora.
children:
  - text/

children:
  - letters/

The leaf manifest is the file contributors and readers care about. It declares the corpus's languages, its set-level license and evidence for it, the upstream source, and every shard by name, digest and size. Shard letters-00001 carries its own license, which overrides the set's.

# Leaf manifest, written by corpus-ingest. Edit ingest.yaml and re-run the
# ingest instead of editing this file: the lineage record pins its bytes.
name: harbour-letters
title: Harbour Letters
languages: [en]
license: CC0-1.0
license_evidence: https://archive.example/harbour-letters/rights
sources:
  - name: Harbour town archive, letters series
    uri: https://archive.example/harbour-letters/
    version: "2026-09-01"
    retrieved_at: "2026-09-02T09:00:00Z"
shards:
  - name: shards/letters-00000.ndjson
    sha256: 38b25153d8aee6004af8f13dcc459cf3be694fd35595303587637d9937dba952
    bytes: 898
    records: 3
  - name: shards/letters-00001.ndjson
    sha256: 6d2313de212a449feb04b7d9cd223305d4896fbd43371d43edd06048046b4dd3
    bytes: 743
    records: 2
    license: CC-BY-4.0
    attribution: "Transcribed by the Harbour Letters Society"
    license_evidence: https://archive.example/harbour-letters/society-terms

Inside a shard every row carries its own source, license and maintainer. These are the rows of shard letters-00001, one JSON object per line:

{"id":"letter-004","text":"Dear Mrs Pryce,\nThank you for the loan of the ledger. I have copied the tide\ntables for the society and will return it by Friday.\nE. Marsh\n","source":"https://archive.example/harbour-letters/letter-004.txt","license":"CC-BY-4.0","maintainer":"letters-maintainers","language":"en","attribution":"Transcribed by the Harbour Letters Society"}
{"id":"letter-005","text":"Dear Society members,\nThe copied ledgers are now shelved in the reading room.\nPlease sign the book when you borrow one.\nE. Marsh, secretary\n","source":"https://archive.example/harbour-letters/letter-005.txt","license":"CC-BY-4.0","maintainer":"letters-maintainers","language":"en","attribution":"Transcribed by the Harbour Letters Society"}

Which Makoto document carries which part

Corpus part	Carried by	Status
Directory index files	Nothing. Makoto has no catalog or directory concept, and forbids hashing a directory by walking it (section 12.5). The tree stays in Git.	Stays outside Makoto
Upstream source	An origin statement: `source.kind`, `uri`, `name`, `version`, `retrievedAt` (section 11.3).	Today
Raw files fetched from it	Entries in a dataset manifest (name, digest, size, media type) that the origin names as its subject.	Today
The ingest run	A transform statement. Its input is the raw manifest; `operation.tool` names the ingester; `operation.parametersDigest` pins the exact bytes of `ingest.yaml` (section 11.4).	Today
Shards	Entries in a second dataset manifest, a subject of the ingest statement.	Today
Leaf manifest	The ingest statement's other subject, handed off as the record's final artifact.	Today
The record in Git	A handoff bundle in `makoto/` beside the manifest, signed by an index maintainer's key (section 13.3).	Today
Set and shard licenses	A claim in the ingest statement's `extensions`, required by a profile. Carry licenses shows it.	Proposed extension
Languages, record counts	Would follow the same pattern as licenses: a named extension and a profile. Not included here.	Would need an extension
Row source, license, maintainer	Columns in the shard rows. The shard's digest commits to them; nothing in this record reads them. Carry licenses says when rows can be checked.	Stays outside this record

The documents, in the order they were signed

Each statement below is the exact payload its signature covers. Copy a block and hash it with shasum -a 256 to get the statement digest quoted beneath it. (The dataset manifests end with a newline the blocks omit; their signed bytes are in the record.) The signed envelopes are in makoto/attestations/.

The raw files, as a dataset manifest

A directory is not a finite byte sequence, so the ingest lists the files it fetched in a dataset manifest (section 12.5) and the origin names that manifest as its subject.

{"entries":[{"digest":{"sha256":"fb9186aeee2df0e682e23ef346d0d62f2d480ea8002d859144e153ddce1e42ba"},"mediaType":"text/plain","name":"letter-001.txt","size":145},{"digest":{"sha256":"d6592205ac62f1a4b8f1fbc3db50bcfdb0bc67aad07a8bc94f90ab950fe719b0"},"mediaType":"text/plain","name":"letter-002.txt","size":112},{"digest":{"sha256":"0473052deb29d1d987e8186945a3ba8fc0d124be0838f68fce49540a57eaea52"},"mediaType":"text/plain","name":"letter-003.txt","size":128},{"digest":{"sha256":"968b232149af2b66d3adf07414c03e3105c62f3ad43a9d291730709e113c82f5"},"mediaType":"text/plain","name":"letter-004.txt","size":139},{"digest":{"sha256":"d457e56492dbeeaede0c6ad15f839c44f57b28c2c36a242c5f419237917ca393"},"mediaType":"text/plain","name":"letter-005.txt","size":140}],"version":"0.2"}

Origin: where the bytes came from

{"_type":"https://in-toto.io/Statement/v1","predicate":{"event":{"id":"urn:uuid:3c0e8a52-7f1d-4b2a-9d6e-1a5b7c9e0f21","occurredAt":"2026-09-02T09:05:00Z"},"profiles":[{"closureDigest":{"sha256":"a16890114db598c65f7e35790e6f5b0193e0a3c8e452cd9727dd1d3442525bc2"},"critical":true,"digest":{"sha256":"5a0498a2911bbe3029c9dbcbcbbbb2c2ce683d280b796a5f09ad271d8e9b034f"},"id":"https://usemakoto.dev/schema/v0.2/dataset-manifest.schema.json","mediaType":"application/vnd.makoto.dataset-manifest.v0.2+json","resources":[],"subjectName":"letters.raw.json","target":"artifact"}],"schemaVersion":"0.2","source":{"kind":"urn:makoto:example:source:archive-series","mediaType":"text/plain","name":"Harbour town archive, letters series","retrievedAt":"2026-09-02T09:00:00Z","uri":"https://archive.example/harbour-letters/","version":"2026-09-01"}},"predicateType":"https://usemakoto.dev/predicate/v0.2/origin","subject":[{"digest":{"sha256":"0358912c7afe582ab9c93d2a1e19922b35ca4d72bcf5f86b02da74b911d2142b"},"name":"letters.raw.json"}]}

predicate.source: The archive series, its version, and when it was fetched. The uri is informative: a receiver cannot check from the statement that the URL still serves these bytes.
subject[0]: The raw-file manifest, by digest.
predicate.profiles[0]: The core dataset-manifest profile. Every dataset-manifest subject must carry it, and it makes the manifest's bytes required when the record is verified.

Statement digest cc67488.

The shards, as a second dataset manifest

Each entry carries a name, digest, size and media type. The shards here are NDJSON so their rows are readable on this page. Parquet shards are listed the same way, with application/vnd.apache.parquet; the verifier hashes and sizes them but cannot read inside them (section 12.4).

{"entries":[{"digest":{"sha256":"38b25153d8aee6004af8f13dcc459cf3be694fd35595303587637d9937dba952"},"mediaType":"application/x-ndjson","name":"shards/letters-00000.ndjson","size":898},{"digest":{"sha256":"6d2313de212a449feb04b7d9cd223305d4896fbd43371d43edd06048046b4dd3"},"mediaType":"application/x-ndjson","name":"shards/letters-00001.ndjson","size":743}],"version":"0.2"}

Ingest: what turned raw files into shards

{"_type":"https://in-toto.io/Statement/v1","predicate":{"event":{"id":"urn:uuid:8d4f2b61-0a3c-4e7d-b1f9-6c2e5a8d3b47","occurredAt":"2026-09-02T09:20:00Z"},"extensions":{"https://corpus.example/makoto/license/v1":{"default":{"evidence":"https://archive.example/harbour-letters/rights","license":"CC0-1.0"},"overrides":[{"attribution":"Transcribed by the Harbour Letters Society","entryName":"shards/letters-00001.ndjson","evidence":"https://archive.example/harbour-letters/society-terms","license":"CC-BY-4.0","subjectName":"letters.shards.json"}]}},"inputs":[{"digest":{"sha256":"0358912c7afe582ab9c93d2a1e19922b35ca4d72bcf5f86b02da74b911d2142b"},"name":"raw","provenance":{"statementDigest":{"sha256":"cc674882b6f8d3ba63a076a2f39e476bebffb52c144ca96cf735044f101645d8"},"subjectName":"letters.raw.json"}}],"operation":{"name":"Ingest the letters series into NDJSON shards","parametersDigest":{"sha256":"fdf257d83b86202ff5874f5c5aabb1d06942abd8e145700ab7bdf46454cad0db"},"tool":{"name":"corpus-ingest","uri":"https://tools.corpus.example/corpus-ingest","version":"1.4.0"},"type":"urn:makoto:example:operation:corpus-ingest"},"profiles":[{"closureDigest":{"sha256":"a16890114db598c65f7e35790e6f5b0193e0a3c8e452cd9727dd1d3442525bc2"},"critical":true,"digest":{"sha256":"5a0498a2911bbe3029c9dbcbcbbbb2c2ce683d280b796a5f09ad271d8e9b034f"},"id":"https://usemakoto.dev/schema/v0.2/dataset-manifest.schema.json","mediaType":"application/vnd.makoto.dataset-manifest.v0.2+json","resources":[],"subjectName":"letters.shards.json","target":"artifact"},{"closureDigest":{"sha256":"fbac8a68cb1a9ad0347e74c104ee885ed607bb7b837a4e1f3372163afc73c35d"},"critical":true,"digest":{"sha256":"ce30589e17abff371e461b6fd75723b055fdddef6e865bcc5fabcf327f07b9ab"},"id":"https://corpus.example/makoto/license/v1/predicate.schema.json","resources":[],"target":"predicate"}],"schemaVersion":"0.2"},"predicateType":"https://usemakoto.dev/predicate/v0.2/transform","subject":[{"digest":{"sha256":"599532ac3af859e59d2e9fad4c74cb3299fbb699357034db86c8881e95d6d49f"},"name":"letters.yaml"},{"digest":{"sha256":"d8df4b70de36f576e7d430cf970e462b28ab44f3eb1d1e44220def9058ab487b"},"name":"letters.shards.json"}]}

predicate.inputs[0]: The raw-file manifest, and the origin statement that described it (cc67488). This is the edge in the graph: it cannot be rewired without re-signing.
predicate.operation.parametersDigest: The SHA-256 of ingest.yaml exactly as committed (fdf257d). shasum -a 256 text/letters/ingest.yaml reproduces it.
subject: Two outputs: the leaf manifest's bytes (599532a) and the shard manifest.
predicate.extensions: The license claim. Carry licenses explains it.

Statement digest a4449e2.

Handoff: the record a reader receives

An index maintainer's key signs the handoff. It fixes the exact set of statements, their root and head, and the artifact handed off: the leaf manifest. The two dataset manifests travel in the bundle as supporting material, because their profile requires their bytes.

{"artifacts":[{"digest":{"sha256":"599532ac3af859e59d2e9fad4c74cb3299fbb699357034db86c8881e95d6d49f"},"head":{"sha256":"a4449e246f93d9a68a19481b2338590efe99e9b2f4e7614c2e80e8bd51bee031"},"name":"letters.yaml"}],"bundleId":"urn:uuid:e2b7c4d9-5f0a-4c18-a36e-9b1d7f2c8e05","heads":[{"sha256":"a4449e246f93d9a68a19481b2338590efe99e9b2f4e7614c2e80e8bd51bee031"}],"issuedAt":"2026-09-02T09:30:00Z","requiredProfiles":[],"roots":[{"sha256":"cc674882b6f8d3ba63a076a2f39e476bebffb52c144ca96cf735044f101645d8"}],"statements":[{"sha256":"a4449e246f93d9a68a19481b2338590efe99e9b2f4e7614c2e80e8bd51bee031"},{"sha256":"cc674882b6f8d3ba63a076a2f39e476bebffb52c144ca96cf735044f101645d8"}],"version":"0.2"}

Manifest digest b30f91d. The whole record is text/letters/makoto/: an unsigned index, the signed handoff, two signed statements, the artifact bytes, and the license profile's schema.

Verify the record

A reader runs the verifier from the repository root under a policy they hold themselves (section 14). This reader has reviewed the repository's lineage/policy.json and uses it; a reader who does not accept the maintainers' choice of keys writes their own. The expected-artifact binding sits beside the checkout, outside the repository, and names the leaf-manifest bytes they checked out:

{"digest":{"sha256":"599532ac3af859e59d2e9fad4c74cb3299fbb699357034db86c8881e95d6d49f"},"head":{"sha256":"a4449e246f93d9a68a19481b2338590efe99e9b2f4e7614c2e80e8bd51bee031"},"subjectName":"letters.yaml"}

$ makoto verify bundle text/letters/makoto \
  --policy lineage/policy.json \
  --schema-catalog lineage/catalog.json \
  --expected-artifact ../expected-letters.json
PASS        load-safely
PASS        parse-strictly
PASS        index-payloads
PASS        core-schemas
PASS        signatures
PASS        authorization-thresholds
PASS        metadata-profiles
PASS        authorization
PASS        graph-dependency-artifacts
PASS        graph
PASS        roots-and-heads
PASS        completeness-anchor
PASS        freshness-anchors
PASS        artifact-bytes
PASS        artifact-profiles
PASS        decision
ALLOW       decision
manifest sha256:b30f91d37734806e0b5e8d87f9a2a00c28202ea361c9737e2a9e70b7bc65157b

What ALLOW establishes: each statement is signed by a key the policy authorizes for that kind of step; the ingest's license claim validates against the profile the policy demands; the graph runs unbroken from the handoff back to the origin; and the leaf manifest's bytes are the ones the record names.

What it does not: that the archive's rights statement is correct, that the letters were transcribed faithfully, or that any shard you have not supplied is intact.

Fetch one shard and check it

A reader who wants only the society's letters reads the leaf manifest, fetches shard letters-00001 from the lookaside by digest, and checks the downloaded bytes against the signed shard manifest:

{
  "manifestStatementDigest": {
    "sha256": "a4449e246f93d9a68a19481b2338590efe99e9b2f4e7614c2e80e8bd51bee031"
  },
  "manifestSubjectName": "letters.shards.json",
  "entryName": "shards/letters-00001.ndjson",
  "digest": {
    "sha256": "6d2313de212a449feb04b7d9cd223305d4896fbd43371d43edd06048046b4dd3"
  },
  "path": "letters-00001.ndjson"
}

$ makoto verify bundle text/letters/makoto \
  --policy lineage/policy.json \
  --schema-catalog lineage/catalog.json \
  --expected-artifact ../expected-letters.json \
  --dataset-entry-binding ../downloads/letters-00001.json
PASS        load-safely
PASS        parse-strictly
PASS        index-payloads
PASS        core-schemas
PASS        signatures
PASS        authorization-thresholds
PASS        metadata-profiles
PASS        authorization
PASS        graph-dependency-artifacts
PASS        graph
PASS        roots-and-heads
PASS        completeness-anchor
PASS        freshness-anchors
PASS        artifact-bytes
PASS        artifact-profiles
PASS        decision
ALLOW       decision
manifest sha256:b30f91d37734806e0b5e8d87f9a2a00c28202ea361c9737e2a9e70b7bc65157b

The same run with --json reports the entry it checked:

[
  {
    "declaredSize": "743",
    "digest": {
      "sha256": "6d2313de212a449feb04b7d9cd223305d4896fbd43371d43edd06048046b4dd3"
    },
    "digestPrerequisiteChecks": [],
    "digestStatus": "pass",
    "entryName": "shards/letters-00001.ndjson",
    "manifestStatementDigest": {
      "sha256": "a4449e246f93d9a68a19481b2338590efe99e9b2f4e7614c2e80e8bd51bee031"
    },
    "manifestSubjectName": "letters.shards.json",
    "sizePrerequisiteChecks": [],
    "sizeStatus": "pass"
  }
]

Change one word in the downloaded shard and the same command refuses it:

$ makoto verify bundle text/letters/makoto \
  --policy lineage/policy.json \
  --schema-catalog lineage/catalog.json \
  --expected-artifact ../expected-letters.json \
  --dataset-entry-binding ../downloads/letters-00001.json
PASS        load-safely
PASS        parse-strictly
PASS        index-payloads
PASS        core-schemas
PASS        signatures
PASS        authorization-thresholds
PASS        metadata-profiles
PASS        authorization
PASS        graph-dependency-artifacts
PASS        graph
PASS        roots-and-heads
PASS        completeness-anchor
PASS        freshness-anchors
FAIL        artifact-bytes
PASS        artifact-profiles
PASS        decision
DENY        decision
manifest sha256:b30f91d37734806e0b5e8d87f9a2a00c28202ea361c9737e2a9e70b7bc65157b
E_ARTIFACT_DIGEST: dataset-entry bytes differ from the manifest member digest

A later step that uses only this shard names it in its own transform statement with inputs[].provenance.entryName (section 11.4), so its lineage chains to this record by digest without listing the other shard.

Limits worth knowing

Parallel work: Makoto records the results of sharded, parallel work. It does not schedule or describe the fan-out. One statement per worker means many statements and keys; one statement per ingest, as here, means trusting whoever aggregated the workers' output.
Scale: The specification's performance target is a 1,000-statement graph verified in at most 2 seconds at p95, excluding artifact hashing, on two named baseline machines (section 23.2). That is a target, not a measurement at the size of a large corpus.
Rows: The finest unit is the shard. Proving that a row belongs to a verified shard would need a new predicate, and current verifiers reject predicates they do not know (section 11.1).
Keys: Signatures use Ed25519 keys named in the reader's policy. Keyless signing, transparency logs and certificate authorities are outside the current specification (section 5).
Replacement: A re-ingest that replaces a corpus produces a new origin and a new head. The reader's policy decides whether an older record is still acceptable.
Implementation: The reference implementation is Python. From Go, drive the pinned CLI through os/exec and read its JSON report, as the Go page shows.

Next: carry set, shard and row licenses.

The license claim in the ingest statement, the profile that requires it, and what a reader sees when a step arrives without it.

Carry licenses Maintain it in Git Back to the casebook