Why signal-dataset

Adopting a storage format is a long commitment, so this page argues the case rather than asserting it. If the argument does not apply to you, the non-goals at the end say so plainly.

The problem it solves

You have a large collection of multidimensional signal captures. Each one is an arbitrary-rank array, often complex64, with axes that mean something: channel, time, frequency, acquisition. Records vary in shape. Each carries metadata you need to filter and sample on, such as sample rate and centre frequency, and you would rather not read a gigabyte of samples to find out what a record is.

Several machines write the collection at once. Many machines then read it, in a shuffled order, for training. The collection lives on object storage, and it must never be seen half-written.

Why not an existing format

These are characterizations of design intent, not benchmark results. Every format listed is good at what it was built for, and several are used inside this one.

HDF5

Zarr

Parquet

WebDataset

TFRecord

signal-dataset

Arbitrary-rank complex tensors

yes

yes

awkward

opaque blobs

opaque blobs

yes

Named axes with coordinates

via convention

via attributes

no

no

no

yes

Random access by ordinal

yes

yes

row groups

no

no

yes

Read one record’s metadata without its samples

partly

partly

yes

no

no

yes

Concurrent multi-writer publication

no

with care

append-only

yes

yes

yes

Reader sees all or nothing

no

no

no

no

no

yes

Membership without listing the store

n/a

consolidated metadata

n/a

no

no

yes

The row that usually decides it is the last two.

Atomic publication. Most formats describe bytes on disk and leave “when does a reader see this?” to whoever operates the store. That is fine for a dataset written once by one process, and it is a recurring source of trouble when a job writes for six hours across forty machines. Here, publication writes every object first and root.json last. A reader either finds no dataset or a complete immutable snapshot. There is no window in which it sees half of one.

Membership without listing. A reader follows a generation-pinned manifest. It never lists a directory or a bucket prefix to discover what a dataset contains. That matters more than it sounds, and the next section is about why.

Why immutable, and why root-last

Immutability is not asceticism. It is what makes the other guarantees cheap.

Because no object is ever modified, a reader that has resolved an ordinal to an object can cache that mapping forever, and two readers of the same snapshot cannot disagree. Because publication writes the root last, and the root is created rather than replaced, the switch from “not published” to “published” is a single create of a single small object. That is the one operation object stores make atomic, so the guarantee rests on the narrowest possible foundation.

The failure this prevents is concrete. A training job starts, lists a dataset that is still being written, gets 80% of the records, and trains happily on a silently short dataset. Nothing errors. You find out weeks later, if at all.

Why never list a directory

Three reasons, in increasing order of how much trouble they cause.

  1. Listing costs. A prefix listing on object storage is paginated and priced per request. For a dataset of hundreds of thousands of objects, discovering membership by listing is slow and billable every time a reader opens it.

  2. Listing is eventually consistent in ways that surprise people. Object stores have improved here, but “the object is readable” and “the object appears in a listing” are still not the same guarantee everywhere.

  3. A listing cannot tell you what belongs. This is the one that matters. A prefix contains objects from the current publication, from an interrupted publication that nobody cleaned up, and from a retried worker whose first attempt also wrote objects. A listing shows all of them, identically. A manifest names exactly the objects the snapshot contains, so an orphan is harmless: it is simply never referenced.

That third point is why the rule is absolute rather than an optimization. Reading the manifest is not a faster way to list. It is the only way to get a correct answer.

Profile-free

The core embeds no modality, training framework, or split policy. There is no “radar profile” or “comms profile”, no notion of train and test splits, and no dependency on any trainer.

This is a deliberate trade. It means the format cannot validate that your radar cube has the axes a radar cube should have, because it does not know what a radar cube is. In exchange, the format does not need a new version when your field invents a new capture type, and two groups with different conventions can read each other’s data with the same reader.

Recommended metadata keys are exactly that, recommended. See conventions.

What this is not for

Reach for something else if:

  • Your data is tabular. Use Parquet. Columnar analytics is a different problem and Parquet is very good at it.

  • You need to mutate a published dataset. Nothing here can be edited in place. Publish a new snapshot instead. If your workload is “append a few records an hour, forever”, the immutability is working against you.

  • You have one array, not many records. A single enormous N-d array with chunked reads is what Zarr is for.

  • You need queries, retention, or authorization. The library provides none of these, and does not intend to. It reads and writes datasets; policy lives outside it.

  • You are on Windows. Atomic publication uses POSIX filesystem primitives. See install.

Next