Read performance

Tuning the input pipeline is the trainer’s job. This page is what SDS knows that a trainer would otherwise have to measure for itself: which settings exist, what they do, and what was actually observed.

Everything below was measured on one laptop against local files with a warm page cache, unless a source is named. That isolates CPU and reader structure honestly, and it means the numbers say nothing about network behaviour, which is where the largest effects are.

The setting that matters most is not one of ours

AWS, measuring the same data and the same GPU, found sequential access left every client GPU-bound at ~100% utilization across shard sizes from 4 MB to 256 MB, while random access ran at ~138 samples/sec. ArrayRecord’s own benchmark shows the same ratio from a different cause: 310,120 queries/sec for batched access against 4,933 for individual, both on a local filesystem.

Two independent measurements, roughly 60× apart, both saying the access order dominates everything else on this page.

Every comparable library converges on the same shape: shuffle the shard order, then shuffle within a bounded buffer. WebDataset, MosaicML, TFDS and DALI all do it, with buffer defaults of 1000, 1000 and 1024. Grain is the outlier in offering a true global index shuffle, and recommends it for random-access formats like ArrayRecord, which is sound only if the sampler batches.

dataset.record_metadata[i] exists to make this cheap: it reads one record’s identity and shape without its samples, so an order can be planned over metadata at a fraction of the byte cost and the tensors fetched in whatever order that produces. Pair it with dataset.read_instructions(...). See bring your own loader.

What the knobs do

StorageOptions field	default	effect
reader_options	readahead_buffer_size:0,max_parallelism:0	passed to ArrayRecord; see below
writer_options	group_size:1	one record per chunk. Do not change
records_per_read_batch	1024	indices per batched read, within one shard
manifest_cache_pages	16	manifest pages retained
max_open_shards	4	staged shards kept open, s3:// only
file_reader_buffer_size	None	ArrayRecord’s own default, 1 MiB

readahead_buffer_size:0 is correct and should stay. It looks like it is disabling something useful, and 0.8.3’s own docstring calls that string the random-access recipe, but Google changed their default to 0 in December 2025, with the rationale that read-ahead “occupies extra memory space with little benefit to both random access and batch access”. Grain hardcodes it too. It measured fastest here: 3,617 MB/s against 1,937 at 16 MiB.

max_parallelism:0 is the questionable half. Batch reads ignore it entirely, and on the sequential path it forces decode on the calling thread; it measured 1,807 MB/s against 3,370 unrestricted. Google leaves it at auto.

group_size:1 must not change. At group_size:65536 a single random read costs 378 ms against 797 µs, a factor of 460, because the whole group decompresses to yield one record. Grain logs an error if it is anything else, and TFDS writes 1.

max_open_shards, and why it was worth fixing

Only s3:// stages: local and gs:// hand the URI to the reader unchanged. A staging miss re-downloads a whole object, so under an order that alternates between more shards than are retained, every record costs a full shard. At a 250 MB shard and a 0.5 MB record that is roughly 500× read amplification.

The parameter existed all along and nothing could set it, so every shipped configuration retained exactly one shard. It is now StorageOptions.max_open_shards. Size it against your access order and the staging volume: retention × shard size is the working set.

Per-record CPU

Decoding a 525 KB record takes about 48 µs, of which roughly 21 µs is two avoidable copies. safetensors.load copies every tensor out of the payload, and Field.__post_init__ copies again to get an immutable buffer. A zero-copy np.frombuffer is 0.3 µs.

Validation runs on every decode and costs a flat ~35 µs. At 0.5 MB records that is 10.9 GB/s per core and cannot starve a GPU; at 2.5 KB records it is ~29k records/sec per core and starts to matter. Of six comparable formats surveyed, only the TFRecord family validates per record on every read, and where its cost has been measured it is a fraction of a percent of a decode.

Compression is not worth changing

On complex64 IQ, zstd:3 gives a ratio of 1.091 at 1,483 MB/s per core. The arithmetic runs the wrong way from intuition: decompression throughput is measured against the uncompressed stream while the network carries compressed bytes, so a better ratio means more CPU, not less.

cores = network_rate × compression_ratio / decompression_rate

At 25 Gbps that is 2.3 cores to avoid about 0.26 GB/s of network. Raising the level does not help: levels 1, 3 and 9 all give 1.091. FCBench’s median across 33 real float datasets is 1.16, and ALP measured 1.08 on neural-network weights, so this is the expected result rather than a quirk of the test data.

The change with real leverage is the storage dtype, not the codec. If your captures are 12–14 bit ADC output, storing complex int16 rather than complex64 halves the bytes outright and moves the ratio into the ~1.4× band measured on real 16-bit IQ, roughly 4× in total, at no read-side cost. The format already supports int16.

Mounted buckets

Google’s published gains for gcsfuse are ratios rather than throughputs: file cache “up to 2.3x faster training time”, parallel downloads “up to nine times faster model load”, buffered reads “2-5x”. For read-heavy training the settings that matter are metadata-cache:ttl-secs=-1 on read-only data, file-cache:max-size-mb=-1 on a local SSD, enable-buffered-read=true, and cache-file-for-range-read=true. That last one matters specifically because this library reads at non-zero offsets, and it is false by default.

For Mountpoint, note the shared --cache-xz retains objects only up to 1 MB, so it is useless at realistic shard sizes; only the local disk --cache applies.

See mounted buckets for what publication through a mount does and does not guarantee.

Findings this library does not act on

These are real, measured, and belong to the layer above. They are recorded so whoever owns the training pipeline can decide.

• Readers are rebuilt on every call. Opening one costs 2.1× a read with no network at all (263 µs against 124 µs); on GCS it additionally re-parses the shard’s chunk index, which at group_size:1 has one entry per record.

• Batching is the format’s largest lever and Grain cannot reach it. grep '__getitems__' over Grain returns zero matches, and its private _getitems pushdown is gated on a function that returns False unconditionally. Any batching has to sit below the single-index call. Note PyTorch’s DataLoader does call __getitems__, since 1.13.

• S3 fetches whole objects. AWS recommends concurrent 8 MiB ranges, and a single TCP flow is capped at 5 Gbps, so a whole-object GET cannot fill a 25 Gbps NIC.

• Annotation element reads cost one info() and one source-metadata record read each, to compare two strings.