diff --git a/src/http-gateways/path-gateway.md b/src/http-gateways/path-gateway.md index 84538837..8d11a3c4 100644 --- a/src/http-gateways/path-gateway.md +++ b/src/http-gateways/path-gateway.md @@ -4,7 +4,7 @@ description: > The comprehensive low-level HTTP Gateway enables the integration of IPFS resources into the HTTP stack through /ipfs and /ipns namespaces, supporting both deserialized and verifiable response types. -date: 2026-03-05 +date: 2026-08-24 maturity: reliable editors: - name: Marcin Rataj @@ -662,27 +662,125 @@ or non-HTTP URLs like `ipfs://cid`, where empty path component is implicit `/`. See [`Location` section](https://specs.ipfs.tech/http-gateways/subdomain-gateway/#location-response-header) of :cite[subdomain-gateway]. +### `Ipfs-Uri` (response header) + +Used for HTTP caching and indicating the IPFS address of the data. + +Indicates the original, requested content path before any path resolution and traversal is performed. + +The value is one valid URI: `ipfs://` for the `/ipfs/` namespace +(:cite[ipfs-uri]), or `ipns://` for `/ipns/` (:cite[ipns-uri]). + +The content path is the request path, normalized: percent-decode +each segment once, collapse duplicate slashes, then apply any `.` and `..` +segments. `%2F` decodes to a separator in the first step, so a request for +`/ipfs/{cid}/a%2F..%2Fb.txt` gives the content path `/ipfs/{cid}/b.txt`. + +One consequence: a UnixFS link whose name contains `/` cannot be +addressed by any content path, because path components never contain `/` +(:cite[unixfs]); such an entry is reachable only by its own CID. + +The URI authority SHOULD be the content root in the canonical form +:cite[ipfs-uri] and :cite[ipns-uri] define. A trailing dot on a DNS name +is stripped. Under `/ipns/`, a legacy base58 peer ID (`Qm...`, +`12D3Koo...`) becomes a `libp2p-key` CIDv1 in base36. A gateway omits +this header when it cannot produce such an authority: the root is +invalid, the IPNS key uses a codec it does not support, or the DNS name +does not convert to the `dnslink-name` form. In the DNSLink case, a name +with no dot, like `examplemissingtld`, never becomes an authority: on each +network it can point at different content. A dotted DNSLink name on a +private network, like `example.local`, is fine. + +A content root longer than 63 characters is emitted in full and HTTP +clients SHOULD accept it: the length guidance in :cite[ipfs-uri] and +:cite[ipns-uri] does not apply to this header. + +The URI path mirrors the rest of the content path: split it on `/`, +percent-encode each segment, and rejoin with `/`. A trailing slash is +kept, and an empty remainder means no path: +`/ipfs/{cid}` → `ipfs://{cid}`, `/ipfs/{cid}/` → `ipfs://{cid}/`, +`/ipfs/{cid}/dir/` → `ipfs://{cid}/dir/`. Interior empty segments MUST NOT +appear. + +Each segment MUST be percent-encoded over its UTF-8 bytes: every byte +outside the unreserved set `A-Z a-z 0-9 - . _ ~` (Section 2.3 of +:cite[rfc3986]) MUST be encoded as `%XX` with uppercase hexadecimal +digits, and unreserved bytes MUST NOT be encoded. This covers `%`, `?`, +`#`, spaces, control characters, and every non-ASCII byte, so the value is +ASCII-only and always a valid field value (Section 5.5 of :cite[rfc9110]). + +The one exception: a segment that is exactly `.` or `..` MUST be emitted +as `%2E` or `%2E%2E`, so it cannot be read as a dot segment. The +normalization above means such segments do not normally occur. + +The value MUST NOT contain a query or fragment. Request query parameters +such as `?format=car` never appear in it. + +An HTTP client that recovers the content path MUST apply the +URI-to-content-path mapping of :cite[ipfs-uri] and :cite[ipns-uri], then +percent-decode each segment once. + +Example: `Ipfs-Uri: ipfs://bafy..ul6/subdir/file.txt` + +Example: `Ipfs-Uri: ipns://dnslink.example.net/wiki/Bogot%C3%A1` + +This header SHOULD be returned with deserialized responses, and on +redirect and error responses once the content root has been parsed. +Implementations MAY omit it with trustless response types +(`application/vnd.ipld.raw` and `application/vnd.ipld.car`). They MAY also +omit it when the value would exceed 8192 bytes. + +This header supersedes the deprecated +[`X-Ipfs-Path`](#x-ipfs-path-response-header). HTTP clients SHOULD prefer +`Ipfs-Uri` when both headers are present. + +Gateways serving cross-origin traffic SHOULD include `Ipfs-Uri` in +`Access-Control-Expose-Headers`. + ### `X-Ipfs-Path` (response header) +:::warning + +This header is deprecated. HTTP header values use a more limited character set +than IPFS Paths so it is possible to include characters that may break some HTTP +Clients. + +The `Ipfs-Uri` response header should be used instead if it is present as it can +correctly encode all characters that might be found in an IPFS Path. + +::: + Used for HTTP caching and indicating the IPFS address of the data. Indicates the original, requested content path before any path resolution and traversal is performed. Example: `X-Ipfs-Path: /ipns/k2..ul6/subdir/file.txt` -This header SHOULD be returned with deserialized responses. +Gateways MAY keep returning this header for legacy clients, with the value +format unchanged. New implementations SHOULD return only +[`Ipfs-Uri`](#ipfs-uri-response-header). + +A gateway that returns this header MUST omit it when the content path +contains any byte other than HTAB (0x09), SP (0x20), or visible ASCII +(0x21-0x7E). + +Gateways that return this header SHOULD do so with deserialized responses. Implementations MAY omit it with trustless response types (`application/vnd.ipld.raw` and `application/vnd.ipld.car`). +Gateways that return this header for cross-origin traffic SHOULD also +include it in `Access-Control-Expose-Headers`. + ### `X-Ipfs-Roots` (response header) Used for HTTP caching. A way to indicate all CIDs required for resolving logical roots (path -segments) from `X-Ipfs-Path`. The main purpose of this header is allowing HTTP -caches to make smarter decisions about cache invalidation. +segments) from the requested content path. The main purpose of this +header is allowing HTTP caches to make smarter decisions about cache +invalidation. -Below, an example to illustrate how `X-Ipfs-Roots` is constructed from `X-Ipfs-Path` pointing at a DNSLink. +Below, an example to illustrate how `X-Ipfs-Roots` is constructed from a content path pointing at a DNSLink. The traversal of `/ipns/en.wikipedia-on-ipfs.org/wiki/Block_of_Wikipedia_in_Turkey` includes a HAMT-sharded UnixFS directory `/wiki/`. @@ -862,7 +960,7 @@ such as domains that cannot be resolved, or IPNS keys that cannot be resolved. should be produce acceptable cache hits. - Advanced caching strategies can be built using additional information in - `X-Ipfs-Path` and `X-Ipfs-Roots` headers. + `Ipfs-Uri` and `X-Ipfs-Roots` headers. - Implement support for requests sent with [`Cache-Control: only-if-cached`](#cache-control-request-header). diff --git a/src/ipips/ipip-0548.md b/src/ipips/ipip-0548.md new file mode 100644 index 00000000..64eb0139 --- /dev/null +++ b/src/ipips/ipip-0548.md @@ -0,0 +1,154 @@ +--- +title: "IPIP-0548: Sunset X-Ipfs-Path header" +date: 2026-08-22 +ipip: proposal +editors: + - name: Alex Potsides + github: achingbrain + url: https://achingbrain.net + affiliation: + name: Shipyard + url: https://ipshipyard.com + - name: Marcin Rataj + github: lidel + affiliation: + name: Shipyard + url: https://ipshipyard.com +relatedIssues: + - https://github.com/ipfs/specs/issues/547 +order: 548 +tags: ['ipips'] +--- + +## Summary + +Replace `X-Ipfs-Path` header with `Ipfs-Uri` version that can correctly encode +any special characters likely to be found in an IPFS Path. + +## Motivation + +HTTP header values can only include characters from a limited set. + +There is a gap in the existing gateway specification in that it does not say how +characters from outside this set are to be treated. + +Values already arrive broken: browser `fetch()` garbles raw UTF-8, and +Go's `net/http` replaces CR and LF with spaces. + +The spec is implemented and consumed widely so retrospectively adding encoding +rules would be disruptive, and we would have to agree on an encoding format. + +URIs already have a well-defined encoding format (percent encoding, defined in +[RFC 3986](https://www.rfc-editor.org/rfc/rfc3986.html#section-2.1)), so +introduce an `Ipfs-Uri` header to be used in preference to `X-Ipfs-Path` which +can handle any and all characters found in an IPFS path, and can be losslessly +converted back into an IPFS Path if the client desires it. + +## Detailed design + +The `Ipfs-Uri` header should be added which contains the IPFS/IPNS path as a +URI (e.g. `ipfs://...` or `ipns://...`) with any special characters +percent-encoded as per RFC 3986. + +The URI schemes are defined by :cite[ipfs-uri] and :cite[ipns-uri]; the +[`Ipfs-Uri` section](https://specs.ipfs.tech/http-gateways/path-gateway/#ipfs-uri-response-header) +of :cite[path-gateway] defines the exact serialization. + +This IPIP also updates :cite[unixfs]: names containing `/` join the +restricted names list, and the path escaping section defines the HTTP +gateway and URI behavior while leaving other contexts unspecified. + +`Ipfs-Uri` deprecates `X-Ipfs-Path`, and clients SHOULD prefer `Ipfs-Uri` +when both are present. The +[`X-Ipfs-Path` section](https://specs.ipfs.tech/http-gateways/path-gateway/#x-ipfs-path-response-header) +says when the legacy header MUST be omitted to avoid issues with unsafe +byte ranges. + +## Design rationale + +Retroactively adding encoding rules to `X-Ipfs-Path` would be too disruptive to +existing clients so adding a new header and deprecating the old one seems like +the least worst way forward. + +Path escaping was previously undefined across the stack: the UnixFS spec +explicitly declared it out of scope, and nothing said how gateways decode +request paths or how a content path becomes a header-safe string. This IPIP +locks that behavior down: request path components are percent-decoded once +(so `%2F` is a component separator), `Ipfs-Uri` is the canonical encoded +form, and names containing `/` are formally not path-addressable. + +### User benefit + +`Ipfs-Uri` correctly encodes otherwise illegal characters so users can determine +the original IPFS Path of a resource without data corruption. + +### Compatibility + +Since we are adding a new header this is a non-breaking change. + +Existing deployments can keep returning both headers; new implementations +return only `Ipfs-Uri`. + +### Security + +Percent-encoding keeps raw control bytes such as CR and LF out of `Ipfs-Uri` +values. + +## Test fixtures + +A UnixFS directory under +`bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae` +([`dir-with-tricky-filenames.car` in +ipip-0548-test-fixtures.zip](https://github.com/user-attachments/files/31286627/ipip-0548-test-fixtures.zip)), +and the header returned for each file in it: + +| UnixFS file name | Response header | +| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | +| `plain.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/plain.txt` | +| `with space.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/with%20space.txt` | +| `100% sure.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/100%25%20sure.txt` | +| `a#b?c.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/a%23b%3Fc.txt` | +| `łódź.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/%C5%82%C3%B3d%C5%BA.txt` | +| `emoji🚀.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/emoji%F0%9F%9A%80.txt` | +| `αρχείο.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/%CE%B1%CF%81%CF%87%CE%B5%CE%AF%CE%BF.txt` | +| `файл.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/%D1%84%D0%B0%D0%B9%D0%BB.txt` | +| `קובץ.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/%D7%A7%D7%95%D7%91%D7%A5.txt` | +| `ملف.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/%D9%85%D9%84%D9%81.txt` | +| `नमस्ते.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/%E0%A4%A8%E0%A4%AE%E0%A4%B8%E0%A5%8D%E0%A4%A4%E0%A5%87.txt` | +| `ไฟล์.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/%E0%B9%84%E0%B8%9F%E0%B8%A5%E0%B9%8C.txt` | +| `ファイル.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB.txt` | +| `你好.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/%E4%BD%A0%E5%A5%BD.txt` | +| `파일.txt` | `Ipfs-Uri: ipfs://bafybeiflhd5aimv4xavauidbetge3v3uadbqfibu5lfo4b26yyieqvnvae/%ED%8C%8C%EC%9D%BC.txt` | + +A second fixture, the preexisting +[`dir-with-percent-encoded-filename.car`](https://github.com/ipfs/gateway-conformance/raw/v0.13.1/fixtures/path_gateway_unixfs/dir-with-percent-encoded-filename.car) +directory under +`bafybeig675grnxcmshiuzdaz2xalm6ef4thxxds6o6ypakpghm5kghpc34`, holds a +name that already looks percent-encoded. The literal `%2C` is encoded +again (`%252C`), never decoded into a comma, and `+` and `=` do not pass +through raw: + +| UnixFS file name | Response header | +| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | +| `Portugal%2C+España=Peninsula Ibérica.txt` | `Ipfs-Uri: ipfs://bafybeig675grnxcmshiuzdaz2xalm6ef4thxxds6o6ypakpghm5kghpc34/Portugal%252C%2BEspa%C3%B1a%3DPeninsula%20Ib%C3%A9rica.txt` | + +A third fixture, +[`dir-with-slash-in-filename.car`](https://github.com/ipfs/gateway-conformance/raw/149f396ec8a229293bf7e96df3185d2ce4a93dcf/fixtures/path_gateway_unixfs/dir-with-slash-in-filename.car) +under `bafybeihuqitp4tzukehqfyaozl6zexd7szyzeywojuynfxopnn7dqjepv4`, holds +a real subdirectory `a` with `b.txt` inside, plus a sibling link literally +named `a/b.txt`. Such a link is legal in dag-pb but not addressable by any +content path: `%2F` decodes to a separator, so every spelling resolves to +the nested file: + +| Request path | Response header | +| ----------------------- | -------------------------------------------------------------------------------------- | +| `/ipfs/{cid}/a/b.txt` | `Ipfs-Uri: ipfs://bafybeihuqitp4tzukehqfyaozl6zexd7szyzeywojuynfxopnn7dqjepv4/a/b.txt` | +| `/ipfs/{cid}/a%2Fb.txt` | `Ipfs-Uri: ipfs://bafybeihuqitp4tzukehqfyaozl6zexd7szyzeywojuynfxopnn7dqjepv4/a/b.txt` | + +The [gateway-conformance](https://github.com/ipfs/gateway-conformance) +test suite uses these directories to test `Ipfs-Uri` and the legacy +`X-Ipfs-Path` behavior. + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). diff --git a/src/unixfs.md b/src/unixfs.md index 9e18be0c..8567fd5c 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -3,7 +3,7 @@ title: UnixFS description: > UnixFS is a Protocol Buffers-based format for describing files and directories as dag-pb DAGs and raw blocks in IPFS. -date: 2026-03-05 +date: 2026-08-22 maturity: draft editors: - name: Marcin Rataj @@ -680,13 +680,16 @@ inspired by POSIX paths. :::warning -Behavior is not defined. +Behavior is only defined for HTTP interfaces. -Until we agree on a specification for this, implementations SHOULD NOT depend on any escape -sequences and/or non-ASCII characters for mission-critical applications, or limit escaping to specific context. +HTTP Gateways percent-decode each request path component once before path +resolution, so `%2F` becomes a component separator, and the +[`Ipfs-Uri` response header](https://specs.ipfs.tech/http-gateways/path-gateway/#ipfs-uri-response-header) +defines the reverse: a canonical percent-encoded form of a content path. -- HTTP interfaces such as Gateways have limited support for [percent-encoding](https://developer.mozilla.org/en-US/docs/Glossary/Percent-encoding). -- The `\` may be used to trigger an escape sequence. However, it is currently broken and inconsistent across implementations. +Outside of that, the behavior is unspecified. +The `\` may be used to trigger an escape sequence, but it is currently broken +and inconsistent across implementations. ::: @@ -712,6 +715,9 @@ The following names SHOULD NOT be used in UnixFS directories: - Any string containing a `NULL` (`0x00`) byte, as this is often used to signify string terminations in some systems, such as C-compatible systems. Many unix file systems do not accept this character in path components. +- Any string containing a `/` (`0x2F`) codepoint, as path components cannot + contain `/` (see [Paths](#paths)): a directory entry with such a name cannot + be addressed by any UnixFS path and is reachable only by its own CID. # Appendix: Test Vectors