signal-dataset

Immutable, indexed storage for multidimensional signal records on local filesystems, Google Cloud Storage, and Amazon S3. Numerical fields use SafeTensors, and ArrayRecord provides random access.

Use it when you have many arbitrary-rank signal captures, several machines writing them at once, and readers that must never see a half-written dataset.

Note

This is experimental 0.x software. The Python API may make documented breaking changes between minor releases. Persisted-format compatibility is versioned separately, and is covered by the compatibility contract.

Start here

  1. Why signal-dataset if you are deciding whether the format fits your problem.

  2. Install, then the quickstart to publish and read a dataset.

  3. Logical model for records, fields, axes, and coordinates.

  4. How-to guides for annotations, selection, training, and cloud storage.

The central read contract is small:

import signal_dataset as sds

dataset = sds.open("captures.sds")
record = dataset[1]                        # reads the sample tensors
descriptor = dataset.record_metadata[1]    # reads identity and shape only

Reading dataset[i] fetches sample tensors. Reading dataset.record_metadata[i] does not, which is what makes it cheap to plan a shuffle or a filter over a whole dataset before fetching anything.

Reference

  • API: public types, functions, and extension contracts.

  • Errors: what each exception means and what to do about it.

  • Format specification: the persisted layout, normatively.

  • Architecture: the four public domains and how they depend on each other.

  • Conventions: recommended, non-binding metadata keys.

  • Glossary: terms this documentation uses in a specific sense.