Glossary

Terms this documentation uses in a specific sense. Where a term also has a general meaning in storage or signal processing, the definition here is the narrower one the format intends.

control document

Any of the small JSON documents that describe a dataset rather than hold its samples: root.json, snapshot.json, manifest pages, and the annotation catalog and set roots. They are pretty-printed deterministic JSON, so the same logical content always produces the same bytes. ControlPolicy bounds their size and structure before they are parsed.

dataset id

A producer-chosen name for the logical dataset, recorded in the snapshot. It labels the data and is not used to locate anything.

generation

A storage backend’s version token for a specific object, historically recorded alongside a reference so a reader could pin the exact version it resolved. Only Google Cloud Storage ever produced one. Since 0.3.0 new writes record no generation on any backend, while readers still honour one written by an earlier release. The rule is: stop writing it, keep reading it.

layout version

The version of the persisted format, currently 0.1, which moves independently of the package version. A reader accepts the versions in SUPPORTED_LAYOUT_VERSIONS. See the compatibility contract.

manifest

A page of the snapshot’s index. Each page names the exact data and metadata shard objects for a contiguous range of ordinals, with their sizes. Manifests are how a reader finds records without listing storage, and why an object nothing references is harmless rather than corrupting.

mount

A cloud bucket exposed as a local directory, for example through gcsfuse. A mount is declared by the caller, never detected by the library. See mounted buckets.

ordinal

A record’s integer position in the dataset, from 0 to len(dataset) - 1. Ordinals are the dataset’s stable addressing scheme: dataset[i] and dataset.record_metadata[i] refer to the same record.

profile-free

The property that the core format embeds no modality, training framework, or split policy. There is no radar or comms profile, no train and test split, and no trainer dependency. Metadata keys such as sample_rate_hz are recommended conventions rather than reserved names. See why signal-dataset.

publication

The act of making written shards visible as a dataset. Publication writes shards, manifest pages, and the snapshot, then creates root.json last. It is create-only, so it never overwrites, and re-running an identical publication resumes rather than failing.

root

Two related things, distinguished by context. The root is the URI of a dataset, such as captures.sds or gs://bucket/captures.sds, and is what you pass to open(). root.json is the control document at the top of that location, which pins exactly one snapshot. A root holds one snapshot for its lifetime.

shard

One ArrayRecord object holding the encoded records a single worker wrote. Every shard has a matching metadata shard carrying the same records’ identity and shape without their samples, which is what makes metadata reads cheap. Shard objects are content-addressed and immutable.

snapshot

One complete, immutable publication of a dataset, named by a snapshot_id and described by snapshot.json. A snapshot declares the global ordinal ranges its manifest pages cover. Readers see a whole snapshot or no dataset at all.

staging area

The seam that lets a shard container which can only open a local path serve an object it has no driver for. An S3 shard is materialized into an unnamed local file and read from there; local and GCS reads pass through untouched. See ShardStagingArea.

work id

The identifier of the process that wrote a shard, passed as work_id. It must be unique per writer within a publication. A retrying worker reuses its work_id with a higher attempt, so publication can select one winner per writer, and expected_work_ids can insist that every writer reported before anything is published.