Google Cloud Storage

Install the GCS extra and use the same API with a gs:// root:

import signal_dataset as sds

root = "gs://example-bucket/datasets/captures.sds"
shard = sds.write_shard(records, root, work_id="worker-000")
dataset = sds.publish(
    root,
    [shard],
    dataset_id="captures",
    snapshot_id="run-001",
)

record = sds.open(root)[42]

Authentication uses Google Application Default Credentials. Production jobs should use workload identity. The library does not store credentials, choose a project, or hardcode a bucket.

Writes use create-only generation preconditions (ifGenerationMatch: 0). Control documents record no object version: every object a reference can name is written once and never modified, so reading a path returns what a pinned version would have. Datasets published before this change still carry versions and are still read with them. Shard upload streams one caller-sized temporary file through the Google client; readers follow manifests without listing the prefix. Producers choose batch size, with optional writer guardrails available through StorageOptions.

Reading and writing through a gcsfuse mount

A mounted bucket is an ordinary path, and both reading and publishing work through one. See mounted buckets for what is different, what is refused, and why a dataset published through a mount is byte-identical to one published with a gs:// root.