Spark: attest the committed dataset, not the scheduler stage.
Spark can retry and reorder physical work. Stable Makoto evidence belongs after a DataFrame write, table snapshot, or declared streaming window has committed and exact inputs and outputs can be identified.
Implementation boundary. This is a lifecycle pattern, not a shipped Spark adapter or UDF.
The attachment point
| Batch write | After the writer commits, build a dataset manifest over the exact output files and append one transformation statement. |
|---|---|
| Iceberg or Delta | Use the committed snapshot identifier as platform context; bind exact files or an authenticated manifest as the artifact. |
| Structured Streaming | Finalize a bounded offset/time window in foreachBatch, then attest that window as one artifact or dataset manifest. |
| Listener events | Capture application, job, stage, retry, and code context as operation metadata. Do not treat the event itself as the data. |
Control flow
// Spark writes and commits the dataset first.
cleaned.write.mode("overwrite").parquet(outputPath)
# A post-write task resolves exact input and output artifacts.
uv run makoto attest transform --help
# It then supplies predecessor bindings, operation metadata,
# a signing key, and an output envelope to the checked-out CLI.The integration must fail closed if the output set is still changing, a predecessor cannot be resolved, or a partition manifest does not cover the exact handed-off files.
What the evidence can say
It can bind a committed Spark output to signed predecessor statements and platform context. It cannot prove that every physical stage ran as reported, that Unity Catalog already provides Makoto evidence, or that a signature establishes data quality.
Make the first adapter a conformance case.
Contribute one tested batch write or bounded streaming fixture with exact artifacts and an expected receiver report.