What the format checks, and when

Four independent layers verify different things. Knowing which one raised an error tells you where the problem is: a producer bug, a truncated upload, or a control document that no longer agrees with the objects it points at.

Terms used here are defined in the glossary, and each exception is covered in more detail in errors.

Generation does two unrelated jobs

One term is worth separating before the layers, because the same word covers a concurrency mechanism and an integrity mechanism.

As a compare-and-swap token, a generation answers “has this object changed since I read it?”. annotations/catalog.json is the one mutable object in the format, and it is replaced only when its version still matches what the writer observed. Each backend supplies its own token: Google Cloud Storage uses the object generation, S3 derives one from the ETag, and the local filesystem uses the file’s modification time. Without it, two jobs publishing annotation sets concurrently could erase each other.

As a read pin, a generation recorded inside a control document says “this reference means exactly this version of the object”. A reader following such a reference demands the pinned version rather than whatever currently occupies the path.

The two are independent. Compare-and-swap works on every backend. Read pinning only ever applied to GCS, because the field is an integer and only GCS versions are integers. Since 0.3.0 new writes record no generation at all, while readers still honour one written by an earlier release.

The four layers

Record validation, at encode time

validate() runs before a record is encoded. It rejects object dtypes, malformed axes, coordinates that do not match their axis length, references to axis IDs that do not exist, and metadata that exceeds the JSON depth or node limits in ResourcePolicy.

This protects a producer from writing something no reader can load. It runs before anything is written, so a failure leaves no objects behind.

Codec preflight, at decode time

Decoding parses the SafeTensors header and applies the same resource limits before allocating any tensor. A shard that declares a sixty-gigabyte tensor is refused at the header rather than at allocation.

This protects a consumer from a hostile or corrupt shard.

Publication checks, in publish()

Before writing any manifest, publish() compares the byte size of every shard a producer reported against the receipt. A shard that was truncated, half-uploaded, or rewritten between the producer finishing and the coordinator running never reaches a manifest. That check costs one HEAD request per shard.

Record counts are not checked by default. Counting them means re-opening and reading every shard, which on object storage is a full download of everything being published, twice, for data and metadata. Set PublicationOptions(verify_record_counts=True) if you want that exactness and are willing to pay for it. See the S3 guide for what it costs at scale.

Read-path checks, on every open

Opening a dataset walks a chain of references. Each hop verifies that what it fetched matches what the previous document promised, and raises CorruptDatasetError on any disagreement.

root.json          says: the snapshot is at <path>, N bytes
                   checks: byte size, and any pinned version the reference carries

snapshot.json      says: manifest page 0 covers ordinals 0-49, 4 shards, N bytes
                   checks: first ordinal, record count, shard count, byte totals

manifest.json      says: ordinal 0-49 lives in shard <path> at index <i>
                   checks: byte size, and that the index is in range

shard.arrayrecord  the encoded records

A reader never lists a directory or a bucket prefix. Every object it touches was named by a document it already verified, which is what makes an open cost a bounded number of requests regardless of dataset size. The reasoning behind that rule is in why signal-dataset.

What each layer does not catch

The publication and read-path checks compare sizes, counts, and ranges. They do not hash record payloads, so a same-size replacement of a shard’s contents passes both. A dataset published by an older release against a GCS root recorded a generation, and that catches this case because the version number changes even when the length does not. Datasets written by 0.3.0 and later record no generation, so on every backend a same-size substitution goes undetected.

Record-level integrity beyond that is a producer concern. Record.id and scene_id mean whatever the producer defines, and nothing verifies that a record’s samples correspond to its metadata.

If you need tamper-evidence rather than corruption-detection, hash your payloads yourself and put the digest in record metadata, where it travels with the record and is readable without fetching the samples.