Skip to content

Volumes

Armnet Volumes are durable per-user storage for data your jobs need across runs.

Use volumes for:

  • model weights and checkpoints;
  • log dumps and artifacts;
  • datasets;
  • other large files that should be available to runtime containers.

The cloud volume is the source of truth; the copy on the robot-cell node (and on your workstation) is a local cache. Volumes are user-scoped: your jobs only mount your own volume, and other users' volumes are not mounted into your containers.

Upload

armnet volume upload ./checkpoint openpi/checkpoints/my-checkpoint

Uploads skip files that already exist in the cloud volume with the same hash, unless --overwrite is provided. Large uploads (e.g. OpenPI checkpoints) show a progress bar with transferred/total bytes and ETA.

Download

armnet volume cp openpi/checkpoints/my-checkpoint ./checkpoint

List

armnet volume ls                       # whole volume
armnet volume ls openpi/checkpoints    # a sub-tree

Prints the cloud volume as a directory tree with per-file sizes, so you can see exactly what is stored.

Delete

armnet volume delete openpi/checkpoints/my-checkpoint

Mirroring (auto-sync on jobs)

You do not need to pre-stage data on the cell. When a job references a volume path (e.g. an OpenPI checkpoint passed as volume://openpi/checkpoints/my-run), the cell mirrors that path down from the cloud before the container starts:

  • missing files are downloaded;
  • files whose cloud copy changed (new object generation / hash) invalidate the local cache and are re-downloaded;
  • unchanged files are skipped without re-hashing (a manifest at the cache root records the last-synced generation, size, and mtime).

So the workflow "upload a checkpoint, run a job, later replace the checkpoint at the same path, run again" just works — the second run picks up the new bytes.

After the job finishes, any new or changed files the job wrote under a referenced volume path are mirrored back up to the cloud (best-effort; a write-back failure is logged but does not fail a successful job). To have job outputs persisted to your volume, write them under a volume:// path the job was given.

Progress for mirror-down / mirror-up is streamed into the job log.

How the cell reaches Cloud Storage

The cell never holds a static GCP credential, and it does not choose which bucket or prefix to sync — that is decided entirely server-side. When the orchestrator dispatches a job that references a volume:// path, it mints a short-lived, prefix-scoped GCS token for the job's server-stamped username (the username is set from the authenticated API key at job creation and cannot be influenced by the client) and attaches it to the JobDispatch message the cell receives over NATS. The cell uses those credentials verbatim.

If a job is dispatched by an older orchestrator that doesn't attach credentials, the cell falls back to fetching a token from POST /volume/credentials with its own ARMNET_API_KEY (a compatibility shim that only yields the right prefix when the cell key's user matches the job's user).

Multi-user note: the volume bucket currently grants the shared remoterobo-volume-user SA bucket-wide objectAdmin, so isolation between user prefixes is by convention (the server-chosen prefix), not enforced by IAM. To enforce per-user isolation, downscope the minted token with a Credential Access Boundary or use per-user GCS managed folders — see infra/volumes.tf.

Use in Runtime Code

Inside a runtime container, access the mounted volume through ctx.volume:

checkpoint_dir = ctx.volume.path("openpi/checkpoints/my-checkpoint")

The volume mount is separate from the cache mount:

  • REMOTEROBOT_VOLUME_HOME: durable user volume
  • REMOTEROBOT_CACHE_HOME: best-effort cache
  • HF_HOME: set under REMOTEROBOT_CACHE_HOME