Skip to content

Commit c9e231c

Browse files
lidellidel
committed
ipip-412: refactor on top of ipip-402
moving spec details to trustless-gateway, rebasing on top of ipip-402
1 parent 742d641 commit c9e231c

3 files changed

Lines changed: 174 additions & 117 deletions

File tree

‎src/http-gateways/path-gateway.md‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -595,11 +595,7 @@ The following response types require an explicit opt-in, can only be requested w
595595
- Raw Block (`?format=raw`)
596596
- Opaque bytes, see [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw).
597597
- CAR (`?format=car`)
598-
- A CAR file or a stream that contains all blocks required to trustlessly verify the requested content path query, see [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) and :cite[trustless-gateway].
599-
- **Note:** by default, block order in CAR response is not deterministic,
600-
blocks can be returned in different order, depending on implementation
601-
choices (traversal, speed at which blocks arrive from the network, etc).
602-
An opt-in ordered CAR responses MAY be introduced in a future IPIP.
598+
- A CAR file or a stream that contains all blocks required to trustlessly verify the requested content path query, see [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) and Section 5 (CAR Responses) at :cite[trustless-gateway].
603599
- TAR (`?format=tar`)
604600
- Deserialized UnixFS files and directories as a TAR file or a stream, see :cite[ipip-0288].
605601
- IPNS Record

‎src/http-gateways/trustless-gateway.md‎

Lines changed: 150 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,10 @@ editors:
1313
- name: Henrique Dias
1414
github: hacdias
1515
url: https://hacdias.com/
16+
xref:
17+
- url
18+
- path-gateway
19+
- ipip-0412
1620
tags: ['httpGateways', 'lowLevelHttpGateways']
1721
order: 1
1822
---
@@ -25,11 +29,12 @@ The minimal implementation means:
2529

2630
- response type is always fully verifiable: client can decide between a raw block or a CAR stream
2731
- no UnixFS/IPLD deserialization
28-
- for CAR files:
29-
- the behavior is identical to :cite[path-gateway]
3032
- for raw blocks:
3133
- data is requested by CID, only supported path is `/ipfs/{cid}`
3234
- no path traversal or recursive resolution
35+
- for CAR files:
36+
- the pathing behavior is identical to :cite[path-gateway]
37+
3338

3439
# HTTP API
3540

@@ -63,13 +68,14 @@ Same as in :cite[path-gateway], but with limited number of supported response ty
6368

6469
### `Accept` (request header)
6570

66-
This HTTP header is required when running in a strict, trustless mode.
71+
A Client SHOULD sent this HTTP header to leverage content type negotiation
72+
based on section 12.5.1 of :cite[rfc9110].
6773

6874
Below response types MUST to be supported:
6975
- [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw) – requests a single, verifiable raw block to be returned
7076

7177
Below response types SHOULD to be supported:
72-
- [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) – disables IPLD/IPFS deserialization, requests a verifiable CAR stream to be returned, implementations MAY support optional parameters (:cite[ipip-0412])
78+
- [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) – disables IPLD/IPFS deserialization, requests a verifiable CAR stream to be returned, implementations MAY support optional CAR content type parameters (:cite[ipip-0412])
7379
- [application/vnd.ipfs.ipns-record](https://www.iana.org/assignments/media-types/application/vnd.ipfs.ipns-record) – requests a verifiable :cite[ipns-record] (multicodec `0x0300`).
7480

7581
Gateway SHOULD return HTTP 400 Bad Request when running in strict trustless
@@ -175,28 +181,29 @@ For example: `Content-Type: application/vnd.ipld.car; version=1`
175181

176182
MUST be returned and set to `attachment` to ensure requested bytes are not rendered by a web browser.
177183

178-
## Response Payload
179-
180-
### Block Response
184+
# Block Responses (application/vnd.ipld.raw)
181185

182186
An opaque bytes matching the requested block CID
183187
([application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw)).
184188

185189
The Body hash MUST match the Multihash from the requested CID.
186190

187-
### CAR Response
191+
# CAR Responses (application/vnd.ipld.car)
188192

189193
A CAR stream for the requested
190194
[application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car)
191-
content type, path and optional `dag-scope` and `entity-bytes` URL parameters.
195+
content type (with optional `order` and `dups` params), path and optional
196+
`dag-scope` and `entity-bytes` URL parameters.
197+
198+
Below MUST be implmented when a Gateway supports [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car).
192199

193-
#### CAR version
200+
## CAR version
194201

195202
Value returned in
196203
[`CarV1Header.version`](https://ipld.io/specs/transport/car/carv1/#header)
197204
field MUST match the `version` parameter returned in `Content-Type` header.
198205

199-
#### CAR roots
206+
## CAR roots
200207

201208
The behavior associated with the
202209
[`CarV1Header.roots`](https://ipld.io/specs/transport/car/carv1/#header) field
@@ -210,27 +217,148 @@ As of 2023-06-20, the behavior of the `roots` CAR field remains an [unresolved
210217

211218
:::
212219

213-
#### CAR determinism
220+
## CAR `order` (content type parameter)
221+
222+
The `order` parameter allows clients to specify the desired block order in the
223+
response. It supports the following values:
224+
225+
- `dfs`: [Depth-First Search](https://en.wikipedia.org/wiki/Depth-first_search)
226+
order, enables streaming responses with minimal memory usage.
227+
- `unk` or missing: Unknown order, which serves as the implicit default when the `order`
228+
parameter is missing. In this case, the client cannot make any assumptions
229+
about the block order: blocks may arrive in a random order or be a result of
230+
a custom DAG traversal algorithm.
231+
232+
A Gateway SHOULD always return explicit `order` in CAR's `Content-Type` response header.
233+
234+
A Gateway MAY skip `order` in CAR response if no order was explicitly requested
235+
by the client and the default order is unknown.
236+
237+
A Client MUST assume implicit `order=unk` when `order` is missing, unknown, or empty.
238+
239+
## CAR `dups` (content type parameter)
240+
241+
The `dups` parameter specifies whether duplicate blocks (the same block
242+
occurring multiple times in the requested DAG) will be present in the CAR
243+
response. Useful when a deterministic block order is used.
244+
245+
It accepts two values:
246+
- `y`: Duplicate blocks MUST be sent every time they occur during the DAG walk.
247+
- `n`: Duplicate blocks MUST be sent only once.
248+
249+
When set to `y`, light clients are able to discard blocks after
250+
reading them, removing the need for caching in-memory or on-disk.
251+
252+
Setting to `n` allows for more efficient data transfer of certain types of
253+
data, but introduces additional resource cost on the receiving end, as each
254+
block needs to be kept around in case its CID appears again.
255+
256+
A Client MUST not assume any implicit behavior when `dups` is missing.
257+
258+
If the `dups` parameter is not present in the `Content-Type` header, the
259+
behavior is unspecified, and the CAR response includes an arbitrary list of
260+
blocks. In this unknown state, the client MUST assume `n` as the default, but
261+
also MUST ignore duplicates if they are present.
262+
263+
A Gateway MUST return always return `dups` in `Content-Type` response header
264+
when the duplicate status is known at the time of response.
265+
266+
A Gateway MAY skip `dups` if it was not present in `Accept` header sent by the
267+
client or if it is not possible to tell the duplicate status.
268+
269+
:::warning
270+
271+
The specified parameter does not apply to virtual blocks identified by identity
272+
CIDs. CAR responses MUST never include these virtual blocks. The parameter in
273+
question is meant to control the behavior of non-virtual blocks in the
274+
response. Therefore, it does not have any effect on virtual blocks, and they
275+
should never be included in the CAR response, no matter if present, or what
276+
value is set.
277+
278+
:::
279+
280+
281+
## CAR parameters and determinism
214282

215-
The default CAR header and block order in a CAR response is not specified and is non-deterministic.
283+
The default header and block order in a CAR format is not specified by IPLD specifications.
216284

217285
Clients MUST NOT assume that CAR responses are deterministic (byte-for-byte identical) across different gateways.
218286

219287
Clients MUST NOT assume that CAR includes CIDs and their blocks in the same order across different gateways.
220288

289+
Clients MUST assume block order and duplicate status only if `Content-Type` returned with CAR responses includes optional `order` or `dups` parameters, as specified by :cite[ipip-0412].
290+
291+
A Gateway SHOULD support some aspects of determinism by implementing content type negotiation and signaling via `Accept` and `Content-Type` headers.
292+
221293
:::issue
222294

223-
In controlled environments, clients MAY choose to rely on undocumented CAR determinism,
224-
subject to the agreement of the following conditions between the client and the
225-
gateway:
295+
In controlled environments, clients MAY choose to rely on implicit and
296+
undocumented CAR determinism, subject to the agreement of the following
297+
conditions between the client and the gateway:
226298
- CAR version
227299
- content of [`CarV1Header.roots`](https://ipld.io/specs/transport/car/carv1/#header) field
228-
- order of blocks
229-
- status of duplicate blocks
300+
- order of blocks (`order` from :cite[ipip-0412])
301+
- status of duplicate blocks (`dups` from :cite[ipip-0412])
230302

231-
In the future, there may be an introduction of a convention to indicate aspects
232-
of determinism in CAR responses. Please refer to
233-
[IPIP-412](https://github.com/ipfs/specs/pull/412) for potential developments
234-
in this area.
303+
Mind this is undocumented behavior, and MUST NOT be used on public networks.
235304

236305
:::
306+
307+
### CAR format signaling in Request
308+
309+
Content type negotiation is based on section 12.5.1 of :cite[rfc9110].
310+
311+
Clients MAY indicate their preferred block order by sending an `Accept` header in
312+
the HTTP request. The `Accept` header format is as follows:
313+
314+
```
315+
Accept: application/vnd.ipld.car; version=1; order=dfs; dups=y
316+
```
317+
318+
In the future, when more orders or parameters exist, clients will be able to
319+
specify a list of preferences, for example:
320+
321+
```
322+
Accept: application/vnd.ipld.car;order=foo, application/vnd.ipld.car;order=dfs;dups=y;q=0.5
323+
```
324+
325+
The above example is a list of preferences, the client would really like to use
326+
the hypothetical `order=foo` however if this isn't available it would accept
327+
`order=dfs` with `dups=y` instead (lower priority indicated via `q` parameter,
328+
as noted in :cite[rfc9110]).
329+
330+
### CAR format signaling in Response
331+
332+
The Trustless Gateway MUST always respond with a `Content-Type` header that includes
333+
information about all supported and known parameters, even if the client did not
334+
specify them in the request.
335+
336+
The `Content-Type` header format is as follows:
337+
338+
```
339+
Content-Type: application/vnd.ipld.car;version=1;order=dfs;dups=n
340+
```
341+
342+
Gateway implementations SHOULD decide on the implicit default ordering or
343+
other parameters, and use it in responses when client did not explicitly
344+
specify any matching preference.
345+
346+
A Gateway MAY choose to implement only some of the parameters and return HTTP
347+
400 Bad Request or 406 Not Acceptable when client requested a response with
348+
unsupported content type variant.
349+
350+
A Client MUST verify `Content-Type` returned with CAR response before
351+
processing the payload, as the legacy gateway may not support optional content
352+
type parameters like `order` an `dups` and return plain
353+
`application/vnd.ipld.car`.
354+
355+
356+
# IPNS Record Responses (application/vnd.ipfs.ipns-record)
357+
358+
An opaque bytes matching the [Signed IPNS Record](https://specs.ipfs.tech/ipns/ipns-record/#ipns-record)
359+
for the requested [IPNS Name](https://specs.ipfs.tech/ipns/ipns-record/#ipns-name)
360+
returned as [application/vnd.ipfs.ipns-record](https://www.iana.org/assignments/media-types/application/vnd.ipfs.ipns-record).
361+
362+
A Client MUST confirm the record signature match `libp2p-key` from the requested IPNS Name.
363+
364+
A Client MUST [perform additional record verification according to the IPNS specification](https://specs.ipfs.tech/ipns/ipns-record/#record-verification).

‎src/ipips/ipip-0412.md‎

Lines changed: 23 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -61,96 +61,10 @@ The `order` parameter allows the client to indicate its preference for a
6161
specific block order in the CAR response, and the `dups` parameter specifies
6262
whether duplicate blocks are allowed in the response.
6363

64-
### Signaling in Request
64+
A Client SHOULD sent `Accept` HTTP header to leverage content type negotiation
65+
based on section 12.5.1 of :cite[rfc9110] to get the preferred response type.
6566

66-
Content type negotiation is based on section 12.5.1 of :cite[rfc9110].
67-
68-
Clients MAY indicate their preferred block order by sending an `Accept` header in
69-
the HTTP request. The `Accept` header format is as follows:
70-
71-
```
72-
Accept: application/vnd.ipld.car; version=1; order=dfs; dups=y
73-
```
74-
75-
In the future, when more orders or parameters exist, clients will be able to
76-
specify a list of preferences, for example:
77-
78-
```
79-
Accept: application/vnd.ipld.car;order=foo, application/vnd.ipld.car;order=dfs;dups=y;q=0.5
80-
```
81-
82-
The above example is a list of preferences, the client would really like to use
83-
the hypothetical `order=foo` however if this isn't available it would accept
84-
`order=dfs` with `dups=y` instead (lower priority indicated via `q` parameter,
85-
as noted in :cite[rfc9110]).
86-
87-
#### `order` CAR content type parameter
88-
89-
The `order` parameter allows clients to specify the desired block order in the
90-
response. It supports the following values:
91-
92-
- `dfs`: [Depth-First Search](https://en.wikipedia.org/wiki/Depth-first_search)
93-
order, enables streaming responses with minimal memory usage.
94-
- `unk`: Unknown order, which serves as the implicit default when the order
95-
parameter is missing. In this case, the client cannot make any assumptions
96-
about the block order: blocks may arrive in a random order or be a result of
97-
a custom DAG traversal algorithm.
98-
99-
#### `dups` CAR content type parameter
100-
101-
The `dups` parameter specifies whether duplicate blocks (the same block
102-
occuring multiple times in the requested DAG) will be present in the CAR
103-
response. Useful when a deterministic block order is used.
104-
105-
It accepts two values:
106-
- `y`: Duplicate blocks MUST be sent every time they occur during the DAG walk.
107-
- `n`: Duplicate blocks MUST be sent only once.
108-
109-
When set to `y`, light clients are able to discard blocks after
110-
reading them, removing the need for caching in-memory or on-disk.
111-
112-
Setting to `n` allows for more efficient data transfer of certain types of data,
113-
but introduces additional resource cost on the receiving end.
114-
115-
If the `dups` parameter is not present in the `Content-Type` header, the
116-
behavior is unspecified, and the CAR response includes an arbitrary list of
117-
blocks. In this case, the client should assume `n` as the default, but ignore
118-
duplicates if they are present.
119-
120-
:::warning
121-
122-
The specified parameter does not apply to virtual blocks identified by identity
123-
CIDs. CAR responses MUST never include these virtual blocks. The parameter in
124-
question is meant to control the behavior of non-virtual blocks in the
125-
response. Therefore, it does not have any effect on virtual blocks, and they
126-
should never be included in the CAR response, no matter if present, or what
127-
value is set.
128-
129-
:::
130-
131-
<!-- TODO: do we need a parameter for inclusion of identity CIDs?
132-
It seems to be only relevant in Filecoin due to legacy hiccup:
133-
https://github.com/ipfs/specs/pull/330#issuecomment-1274106892 -->
134-
135-
### Signaling in Response
136-
137-
The Trustless Gateway MUST always respond with a `Content-Type` header that includes
138-
information about all supported/known parameters, even if the client did not
139-
specify them in the request.
140-
141-
The `Content-Type` header format is as follows:
142-
143-
```
144-
Content-Type: application/vnd.ipld.car;version=1;order=dfs;dups=y
145-
```
146-
147-
148-
Gateway implementations are free to decide on the implicit default ordering or
149-
other parameters, and use it in responses when client did not explicitly
150-
specify any matching preference.
151-
152-
Implementations MAY choose to implement only some of the parameters and return
153-
HTTP 406 Not Acceptable when client requested a response with unsupported one.
67+
More details in Section 5. (CAR Responses) of :cite[trustless-gateway].
15468

15569
## Design rationale
15670

@@ -245,7 +159,7 @@ Several alternative approaches were considered before arriving at the proposed s
245159
saves a few bytes in each round-trip. Also, :cite[rfc6648] advises against
246160
use of `X-` and similar constructs in new protocols.
247161

248-
The proposed solution of negotiating the block order through headers si
162+
The proposed solution of negotiating the block order through headers is
249163
future-proof, allows for flexibility, interoperability, and customization while
250164
maintaining compatibility with existing implementations.
251165

@@ -255,11 +169,30 @@ Implementation compliance can be determined by testing the negotiation process
255169
between clients and Trustless Gateways using various combinations of `order` and
256170
`dups` parameters.
257171

172+
Relevant tests were added to
173+
[gateway-conformance](https://github.com/ipfs/gateway-conformance) test suite
174+
in [#87](https://github.com/ipfs/gateway-conformance/pull/87).
175+
176+
<!-- TODO after conformance release is tagged
177+
A detailed list of compliance checks for `order` and `dupes` can be found in
178+
[`v0.TODO.0/trustless_gateway_car_test.go`](https://github.com/ipfs/gateway-conformance/blob/v0.TODO.0/tests/trustless_gateway_car_test.go) or later.
179+
-->
180+
181+
Below are CIDs, CARs, and short summary of each fixture.
182+
258183
TODO:
259184
1. a CAR with blocks for a small file in DFS order
260185
2. a CAR with blocks for a small file with one block appearing twice
261186

262187

188+
189+
Tests for duplicates use a fixture where a directory contains two files that
190+
are the same. If `dups=n`, then there are no duplicates. If `dups=y`, then the
191+
blocks of the file are sent twice, by the order they show up in the DAG.
192+
193+
The same fixture is used for testing `order=dfs`.
194+
195+
263196
### Copyright
264197

265198
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

0 commit comments

Comments
 (0)