Persisted format specification 0.1¶
This document is normative for layout 0.1. JSON documents are UTF-8 objects. Readers reject duplicate keys, non-finite numbers, and unsupported format or layout identifiers.
“Required” means the key must occur exactly once. Integer fields reject booleans. Metadata values
are JSON values; object keys are strings. Control documents are pretty-printed deterministic JSON
ending in LF. Embedded record/metadata descriptors are compact deterministic JSON. Keys are sorted,
UTF-8 is emitted directly, and separators are , and :. Hashes are lowercase SHA-256 hex.
Record¶
A record is one SafeTensors payload. Tensor __sds_descriptor__ is a one-dimensional uint8 JSON
descriptor with format = "signal-dataset.record" and version = "0.1". Every logical field maps
to exactly one other tensor. The descriptor contains record identity, optional scene identity, JSON
metadata, and ordered axes. Object arrays are forbidden.
The descriptor keys are format (required), version (required), id (nonempty string),
scene_id (string or null), metadata (object), and fields (nonempty object). Each field has
tensor (SafeTensors key), axes (array), and metadata (object). Tensor keys are assigned as
tensor_%06d over field names sorted by Unicode code point. Each axis has name, length, role,
id, coordinate, and metadata. A coordinate has values, start, step, reference, unit,
and metadata; values is mutually exclusive with regular start/step coordinates.
The metadata shard JSON has keys id, scene_id, metadata, and fields. Field metadata has
dtype (NumPy dtype string), shape (integer array), axes, and metadata. SafeTensors framing and
dtype names follow the SafeTensors format.
Dataset control graph¶
root.json identifies one snapshot. A snapshot declares record/shard counts, stored bytes, and
ordered manifest references. A manifest declares contiguous shard ordinal ranges. Each shard entry
references aligned data and metadata ArrayRecord objects. Optional record_indices has exactly
record_count physical indices and enables lightweight views.
References contain path, stored_bytes, and an optional object generation. New writes record no
generation on any backend; a reference written by an earlier release may carry one, and readers must
still honour it.
root.json additionally carries an optional min_reader_version: the oldest library release able
to read the dataset correctly. layout_version describes a document’s shape and is checked against
the set of versions a reader supports, not for equality; min_reader_version describes what a
reader must be prepared to do with a document whose shape has not changed. A reader older than the
value it finds must refuse the dataset and say so. Counts and byte sizes are nonnegative integers and ranges are contiguous.
Readers resolve references directly and never infer membership by listing storage. Publication
creates immutable shards, manifests, and the snapshot before create-only root.json.
Document |
Required keys |
|---|---|
dataset root |
|
snapshot |
|
manifest |
|
object reference |
|
shard entry |
|
Format identifiers are respectively signal-dataset.dataset, signal-dataset.snapshot, and
signal-dataset.manifest; layout_version is 0.1. Codec/container names are safetensors and
array_record. Snapshot paths are snapshots/<snapshot-id>/snapshot.json. Manifest page IDs are
<snapshot-id>-%08d and paths are manifests/<first-two-sha256-hex>/<page-id>.json. Shard paths are
objects/{data|metadata}/<first-two-shard-hash>/<shard-hash>.arrayrecord. ArrayRecord uses the
writer/reader option strings in StorageOptions; its container format is defined by the
ArrayRecord project.
Annotations¶
annotations/catalog.json points to a SHA-256-addressed revision. Revisions map portable set names
to roots under annotations/sets/<name>/<publication-id>/root.json. Roots bind a source snapshot and
declare positive, contiguous shard ranges covering every source ordinal. Annotation records contain
source ID/index, status, JSON values, tensor fields, provenance, and detail status.
Catalog updates use compare-and-swap. Worker pointers select one winner per logical shard ID; coordinators require contiguous dense coverage before publishing a root.
Catalog pointers contain exactly revision, whose reference has path and optional generation.
Revision paths are annotations/catalogs/<sha256-of-revision-bytes>.json. Revisions contain exactly
sets; each value has path, stored_bytes, and optional generation. Set roots contain exactly
name, publication_id, source_snapshot_id, record_count, metadata, and shards. Shard rows
contain first_ordinal, positive record_count, and object.
Annotation record status wire values are success, skipped, and failed. Worker pointer fields
are name, publication_id, source_snapshot_id, logical_shard_id, first_ordinal, record_ids,
uri, stored_bytes, generation, statuses, and terminal_statuses. Status arrays have the same
length as record_ids. Portable identifiers match [A-Za-z0-9][A-Za-z0-9._-]{0,127}.
Compatibility¶
Readers reject unknown required formats or versions. Changing tensor mapping, required meanings, ordinal semantics, or codec identifiers requires a new layout or record-format version.
Complete field contracts¶
Writers emit the required keys listed here. Layout 0.1 readers ignore unknown keys in record descriptors, dataset roots, snapshots, manifests, and their object/reference entries, permitting compatible optional additions. Annotation catalog pointers, revisions, set roots, and worker pointers are exact-key documents and reject unknown keys. Metadata objects are the general extension point.
ManifestReference requires first_ordinal, record_count, shard_count, stored_bytes, and
object. All are nonnegative integers except object, which is an object reference. References are
ordered and contiguous. Manifest shard entries use the same rule; record_indices, when present,
is an array of exactly record_count nonnegative physical indices.
Supported logical NumPy dtypes and SafeTensors names are:
NumPy kind |
SafeTensors dtype |
|---|---|
|
|
|
|
|
|
|
|
|
|
Field rank equals the number of axes; every axis length equals its tensor dimension. Axis names are
nonempty and unique within a field. Reused axis IDs have identical length and coordinate. Explicit
coordinate values length equals the axis length. Coordinate references target a record field name
or a declared axis ID.
Regular coordinates require both start and step; explicit values cannot also use them.
Layout 0.1 writes ArrayRecord shards with group_size:1. That belongs here because it changes the
bytes on disk: each record is its own compressed chunk, so reading one decompresses one, and an
interoperable reader has to know it.
Reader options are not part of the layout. StorageOptions.reader_options sets how a reader
buffers and parallelizes; it affects nothing another implementation could observe, and pinning it
here would make tuning throughput look like a compatibility change. The shipped default is
readahead_buffer_size:0,max_parallelism:0. See the performance notes.
The package depends on ArrayRecord >=0.8.3,<0.9; interoperable implementations must read the
corresponding ArrayRecord wire format rather than inventing another framing under the
array_record identifier.
Annotation carrier record¶
Each annotation row is encoded with the record codec above. The carrier record ID equals
source_record_id. Its record metadata has exactly one annotation object containing:
Key |
Wire value |
|---|---|
|
nonnegative integer source ordinal |
|
|
|
JSON object |
|
JSON object |
|
nonempty string or null |
|
boolean |
When annotation tensor fields exist, they are the carrier fields and empty_fields is false. When
none exist, the carrier contains one synthetic field named __empty__: an empty uint8 tensor with
one axis Axis(name="empty", length=0); empty_fields is true and the decoder removes that field.
Normative examples and fixtures¶
tests/test_compatibility_contract.py contains the normative layout-0.1 record and metadata byte
vectors (V01_RECORD, V01_METADATA) and generates a one-record root/snapshot/manifest graph whose
exact keys and format identifiers are asserted. Changes to those vectors require a format-version
decision, not an ordinary package refactor. tests/test_annotations.py asserts annotation catalog,
root, pointer, source-alignment, retry, and generation behavior. Release artifacts include these
tests in the source distribution so alternate implementations can run the same compatibility
journeys.