Quickstart

This page creates a dataset, publishes it, and reads it back. Every later page in the documentation opens the dataset you build here, so run this first.

Install the library, then run the example:

pip install signal-dataset

import numpy as np
import signal_dataset as sds

root = "captures.sds"
records = [
    sds.Record(
        id=f"observation-{index:04d}",
        fields={
            "samples": sds.Field(
                np.exp(1j * np.linspace(0, index + 1, 1024)).astype(np.complex64),
                axes=(
                    sds.Axis(
                        "time",
                        1024,
                        coordinate=sds.Coordinate(start=0, step=1e-6, unit="s"),
                    ),
                ),
            )
        },
        metadata={"sample_rate_hz": 1_000_000, "label": index % 2},
    )
    for index in range(3)
]

shard = sds.write_shard(records, root, work_id="local-000")
dataset = sds.publish(
    root,
    [shard],
    dataset_id="captures",
    snapshot_id="run-001",
)

assert len(dataset) == 3
assert dataset[1]["samples"].data.dtype == np.complex64
assert dataset.record_metadata[1]["samples"].shape == (1024,)

What you just made

captures.sds is a directory, not a file. It holds the record objects, the manifest that lists them, and the root.json that names one immutable snapshot. Nothing outside that directory changed.

sds.open("captures.sds") reopens the same snapshot later. Indexing dataset reads tensor bytes, while indexing dataset.record_metadata reads only identity and shape, without fetching the samples.

The work_id is the identifier of the process that wrote the shard, "local-000" here because there is one writer. The snapshot_id names this publication of the dataset. Both matter once several machines write at once, which distributed writing covers.

Running it a second time

Re-running the example exactly as written is safe. Every write is create-only and resumes on an identical object, so a run interrupted anywhere completes when repeated, and a run that already finished simply produces the same dataset again.

A root holds exactly one snapshot. root.json is written last and once, so publishing a different snapshot_id or dataset_id into captures.sds raises PublicationCollisionError rather than replacing what is there. To publish a different dataset, write its shards into a different root. A shard belongs to the root it was written into, so reuse of a shard across roots raises ValueError:

other = "captures-run-002.sds"
shard = sds.write_shard(records, other, work_id="local-000")
sds.publish(other, [shard], dataset_id="captures", snapshot_id="run-002")

To start over from scratch, remove the directory:

rm -rf captures.sds

One sharp edge to know about. Because the resume path assumes the repeated write is identical, and does not compare the content, re-publishing changed records under the same snapshot_id succeeds without error and readers continue to see the originally published data. Immutability is preserved, but you get no signal that the new data was ignored. Publish changed data under a new root.

Next