Mounted buckets

A bucket mounted with gcsfuse or Mountpoint for Amazon S3 looks like a directory and is not. Reading through one has always worked. Publishing through one now works too, with one exception.

gcsfuse --implicit-dirs --metadata-cache-ttl-secs=0 example-bucket /mnt/gcs
export SIGNAL_DATASET_MOUNT_ROOTS=/mnt/gcs

shard = sds.write_shard(records, "/mnt/gcs/datasets/captures.sds", work_id="worker-000")
sds.publish("/mnt/gcs/datasets/captures.sds", [shard],
            dataset_id="captures", snapshot_id="run-001")

From any other machine, that dataset opens as sds.open("gs://example-bucket/datasets/captures.sds"). The two are the same bytes: a dataset published through a mount is byte-identical to one published natively, because references record no object version on either path.

Declaring a mount

SIGNAL_DATASET_MOUNT_ROOTS is a list of absolute path prefixes, separated the way PATH is. Anything beneath one is treated as a mounted bucket.

A mount is declared, never detected. Probing, meaning trying link(2) and degrading when it fails, would silently degrade on any filesystem that refused a link for an unrelated reason, which is the “appears to support it” behaviour worth avoiding. The environment variable is the right shape because “this machine has a gcsfuse mount at /mnt/gcs” is a property of the deployment: it belongs beside the gcsfuse invocation in your launcher or Dockerfile, and the training script stays identical whether it reads from a mount or from gs://.

It is also the only mechanism that reaches everywhere. DatasetView.publish and materialize take no object_store argument, so a capabilities= argument could never get to them.

For a single store, pass it directly:

from signal_dataset.storage import MOUNTED_BUCKET, LocalObjectStore

objects = LocalObjectStore(root="/mnt/gcs/datasets/captures.sds",
                           capabilities=MOUNTED_BUCKET)

If you forget, the failure says so: a missing link(2) or flock(2) is reported with the path, the filesystem type on Linux, and the variable to set.

What works where

	local POSIX	gs://	s3://	gcsfuse	Mountpoint
open, dataset[i], iterators	yes	yes	yes	yes	yes
read annotations	yes	yes	yes	yes	yes
write_shard(), publish	yes	yes	yes	yes	yes
publish_reference	yes	yes	yes	yes	yes
publish_annotations()	yes	yes	yes	no	no

Why annotations are refused

Every object in the format is written once and never modified, with one exception: annotations/catalog.json. Publishing a set writes a new content-addressed revision and then swings that one pointer to it, under a compare-and-swap, so that two publishers cannot erase each other’s sets.

A compare-and-swap needs either a lock or an atomic replace. gcsfuse implements neither POSIX file locks nor an atomic rename, and the local backend’s version token is the file’s modification time, which on a mount is derived from object metadata at coarse granularity, so two writes in the same tick are indistinguishable. Doing it anyway would silently drop one writer’s annotation set, which is the exact failure the revision-and-pointer design exists to prevent.

So it is refused, and refused before any shard is written: annotation shards are written first and the catalog last, so failing at the swap would leave objects nothing references, with no delete to clean them up.

The workaround is to publish the dataset through the mount and its annotations natively. It is sound precisely because the two addresses are the same dataset. Note that publish_annotations takes an open dataset rather than a URI, so the native address is opened first:

sds.publish("/mnt/gcs/datasets/captures.sds", shards, ...)   # through the mount

dataset = sds.open("gs://example-bucket/datasets/captures.sds")   # native
sds.publish_annotations(dataset, "detections", records)

What publishing through a mount gives up

On a POSIX filesystem the backend writes a temporary file and links it to its final name. The link is atomic, and it fails if the name is taken, so an object at its final name is always complete.

A mount has no link(2), so the file is created at its final name directly. The name therefore exists from the moment it is opened and the bytes arrive when it is closed. A writer killed mid-upload leaves a short or empty object at a real name.

That is detected rather than ignored. The retry compares what is stored against what it was about to write and refuses, naming the object. Recovery is manual. Delete it and run again, or republish under a new snapshot_id, because this library never deletes anything.

Root confinement loses exactly one check. (st_dev, st_ino) no longer pins the root directory’s identity, because a mount synthesizes inode numbers. In practice this costs nothing: gcsfuse keeps a stable root inode for a mount’s lifetime, so the check silently passed rather than protecting anything. The protections that stop a path escaping, namely openat, O_NOFOLLOW, and rejecting .., all still apply.

Mount options that matter

--implicit-dirs is effectively required for gcsfuse. This library never writes directory placeholder objects, so a natively published dataset has no directory entries and gcsfuse will not show objects/... without it.

--metadata-cache-ttl-secs=0 matters on a mount you publish through. Creating a file exclusively is a lookup followed by a create, and a cached negative entry can let the lookup miss an object that exists.

Mountpoint needs --allow-delete for nothing this library does, since it never deletes. Note that it implements no rename, which is why the publish path uses no temporary file at all.

One coordinator at a time

Mount publication assumes a single coordinator. Many workers may produce shards beforehand; one process publishes them.

This is not enforceable, because without a lock nothing here can exclude a second writer, so it is asserted rather than pretended. What is checked: every control document is verified to hold the bytes that were written, and two coordinators sharing a snapshot_id collide on the first manifest, while two with different ones collide on root.json, which is create-only.