feat(dstack-util): add streaming encrypt and decrypt commands

docs/stream-encryption.md

@@ -0,0 +1,77 @@
+# dstack Chunked Encryption Format
+
+`dstack-util encrypt` and `dstack-util decrypt` use a chunked format for
+bounded-memory encryption of arbitrary data. It uses the same app-scoped X25519
+key pair as encrypted environment variables, but it is a separate wire format.
+
+The header and frame metadata are defined with `binrw`, the fixed-layout binary
+codec already used by dstack. Fixed-width integers use little-endian encoding.
+The nonce construction below deliberately uses a big-endian chunk index so its
+byte representation follows counter order.
+
+## Header
+
+| Field | Size | Description |
+|---|---:|---|
+| Magic | 8 bytes | ASCII `dstkscrt` |
+| Version | 1 byte | Format version, currently `0` |
+| Ephemeral public key | 32 bytes | X25519 public key generated by the sender |
+| Nonce prefix | 8 bytes | Random prefix shared by all chunks |
+| Chunk size | 4 bytes | Maximum plaintext bytes in each chunk |
+
+The X25519 shared secret is used directly as the AES-256-GCM key, matching the
+encrypted environment variable protocol.
+
+## Frames
+
+Each frame contains:
+
+| Field | Size | Description |
+|---|---:|---|
+| Flags | 1 byte | Bit 0 marks the final chunk; all other bits must be zero |
+| Plaintext length | 4 bytes | Number of plaintext bytes in this chunk |
+| Ciphertext and tag | `plaintext length + 16` bytes | AES-256-GCM output |
+
+The 12-byte nonce is `nonce_prefix || chunk_index`, where `chunk_index` is a
+4-byte integer starting at zero. The authenticated additional data is:
+
+```text
+header || chunk_index || flags || plaintext_length
+```
+
+Every non-final frame must contain exactly `chunk_size` plaintext bytes. The
+final frame may be shorter or empty. A final frame is always emitted, including
+for empty input and for input whose length is an exact multiple of the chunk
+size. Missing final frames, trailing data, unknown flags, and authentication
+failures are rejected.
+
+## CLI
+
+Encrypt data after retrieving the app public key over verified TLS:
+
+```bash
+dstack-util encrypt \
+  --kms-url https://kms.example.com \
+  --app-id "$APP_ID" \
+  --kms-pubkey "$TRUSTED_KMS_SIGNER_PUBKEY" \
+  --input plaintext.bin \
+  --output ciphertext.bin
+```
+
+`--kms-pubkey` is the trusted compressed secp256k1 public key used to verify the
+KMS response's timestamped signature. For a KMS using a private CA, also pass
+`--root-ca ca.pem`. Decrypt inside the CVM:
+
+```bash
+dstack-util decrypt --input ciphertext.bin --output plaintext.bin
+```
+
+`decrypt` detects the magic string automatically. Inputs without the magic are
+handled as the legacy encrypted-environment format. Hex input remains available
+through `--hex`, but it is decoded in memory and should not be used for large
+files.
+
+Successfully authenticated chunks are written as they are processed. If a
+later chunk is corrupt or the final frame is missing, stdout or a file may
+therefore contain an authenticated but incomplete plaintext prefix. Callers
+must check the command's exit status and discard all output on failure.

dstack/dstack-util/Cargo.toml

@@ -55,6 +55,7 @@ cert-client.workspace = true
 x509-parser.workspace = true
 yaml-rust2.workspace = true
 bollard.workspace = true
+binrw.workspace = true
 sodiumbox.workspace = true
 libc.workspace = true
 luks2.workspace = true