Amazon S3¶

Install the S3 extra and use the same API with an s3:// root:

import signal_dataset as sds

root = "s3://example-bucket/datasets/captures.sds"
shard = sds.write_shard(records, root, work_id="worker-000")
dataset = sds.publish(
    root,
    [shard],
    dataset_id="captures",
    snapshot_id="run-001",
)

record = sds.open(root)[42]

Authentication uses the standard AWS credential chain: environment variables, shared configuration, an instance or container role, or web identity. The library does not store credentials, choose a profile, or hardcode a bucket.

Mount the bucket for random-access reads¶

If you read records in a random order, meaning training, sampling, or shuffling, mount the bucket and open a local path instead of using an s3:// root.

mount-s3 example-bucket /mnt/s3

dataset = sds.open("/mnt/s3/datasets/captures.sds")

Mountpoint for Amazon S3 supports read, pread, and lseek, so the ArrayRecord reader indexes into a shard directly and Mountpoint fetches only the byte ranges it needs. As far as this library is concerned that is an ordinary local path, and it is the fastest way to read a dataset that lives in S3.

The s3:// scheme cannot do this. The ArrayRecord reader opens a path and knows one remote scheme, gs://, so a shard stored in S3 has to be fetched whole before any record can be read out of it. The library keeps one staged shard open and reuses it, so reading many records from the same shard costs one fetch. But a pattern that alternates between shards pays a whole shard each time it switches.

So: use s3:// to publish, and to read in shard order. Mount for anything else.

Publishing through a mount works, with one exception. Mountpoint implements no hard links, no POSIX file locks, and no directory fsync, so the mount must be declared – see mounted buckets – and the backend then publishes without them. Annotations are refused there, because updating the annotation catalog needs a compare-and-swap that no mount can provide; publish those against the s3:// root, which addresses the same dataset.

An s3:// root remains the better choice for publication where you have one: it uses S3’s own conditional writes, so creation stays all-or-nothing.

What the backend guarantees¶

Every object except the annotation catalog is written create-only, using If-None-Match: *. A second writer targeting the same object raises PublicationCollisionError rather than overwriting it.

annotations/catalog.json is the one mutable object in the format. It is updated with If-Match: <etag>, so two workers publishing annotation sets concurrently cannot erase each other; the loser retries.

S3 has no object generation, so references written against an S3 root carry no version, exactly as for a local root. Readers verify each control object’s size against the manifest that points at it.

Configuration¶

Endpoints, profiles, and retry policy live in S3Options. Pass a configured store directly:

from signal_dataset.storage import S3ObjectStore, S3Options

store = S3ObjectStore(options=S3Options(region_name="us-west-2"))
dataset = sds.open(root, object_store=store)

Or register it once so every call resolves it:

from signal_dataset.storage import default_registry
from signal_dataset.storage.backends.s3 import S3Options, s3_backend

default_registry().register(
    s3_backend(S3Options(endpoint_url="https://minio.example", addressing_style="path")),
    replace=True,
)

S3-compatible endpoints implement conditional writes unevenly. If one does not, the backend raises StorageError naming the problem rather than silently losing create-only safety.

Limits¶

A shard is published in a single request, which S3 caps at 5 GiB. Above that the backend raises StorageError; cap shard size with StorageOptions.max_encoded_shard_bytes.

Staged shards are written to unnamed temporary files, so they are reclaimed automatically. They are also invisible to ls and du while open, so point staging elsewhere, or bound it, with:

from signal_dataset.storage import ArrayRecordShardStore, EphemeralFileStaging

store = S3ObjectStore(options=S3Options(region_name="us-west-2"))
shards = ArrayRecordShardStore(
    staging=EphemeralFileStaging(
        objects=store,  # without this, staging builds its own default store
        directory="/mnt/nvme",
        max_object_bytes=2 << 30,
    )
)
dataset = sds.open(root, object_store=store, shard_store=shards)

Pass objects= when you build staging yourself. Omitting it makes staging resolve its own store from the URI, which silently ignores the endpoint, profile, and retry policy configured on the store you passed to open.

Publishing compares sizes and does not count records by default. publish() compares the byte size of each shard it wrote, which costs one HEAD per shard and catches the failure that actually happens: a truncated or half-written upload. It does not read the shards back.

PublicationOptions(verify_record_counts=True) additionally re-opens every shard and counts its records. That check is exact, and it costs a full read of everything being published. For a terabyte dataset that is a terabyte downloaded twice, once for data and once for metadata, plus cross-region egress if the writer is not in the bucket’s region. Publish from the bucket’s region if you turn it on.