Skip to content

Commit 1d9ec9c

Browse files
guseggertguseggert
committed
feat: Delegated Routing HTTP API
1 parent 1ac9290 commit 1d9ec9c

2 files changed

Lines changed: 189 additions & 0 deletions

File tree

‎IPIP/0000-delegated-routing-http-api.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# IPIP 0000: Delegated Routing HTTP API
2+
3+
- Start Date: 2022-10-18
4+
- Related Issues:
5+
- (add links here)
6+
7+
## Summary
8+
9+
This IPIP specifies an HTTP API for delegated routing.
10+
11+
## Motivation
12+
13+
Idiomatic and first-class HTTP support for delegated routing is an important requirement for large content routing providers,
14+
and supporting large content providers is a key strategy for driving down IPFS latency.
15+
These providers must handle high volumes of traffic and support many users, so leveraging industry-standard tools and services
16+
such as HTTP load balancers, CDNs, reverse proxies, etc. is a requirement.
17+
To maximize compatibility with standard tools, IPFS needs an HTTP API specification that uses standard HTTP idioms and payload encoding.
18+
The [Reframe spec](https://github.com/ipfs/specs/blob/main/reframe/REFRAME_PROTOCOL.md) for delegated content routing was an experimental attempt at this,
19+
but it has resulted in a very unidiomatic HTTP API which is difficult to implement and is incompatible with many existing tools.
20+
The cost of a proper redesign, implementation, and maintenance of Reframe and its implementation is too high relative to the urgency of having a delegated routing HTTP API.
21+
22+
Note that this does not supplant nor deprecate Reframe. Ideally in the future, Reframe and its implementation would receive the resources needed to map the IDL to idiomatic HTTP,
23+
and this spec could then be rewritten in the IDL, maintaining backwards compatibility.
24+
25+
## Detailed design
26+
27+
See the [API design](../routing/DELEGATED_ROUTING_HTTP.md) included with this IPIP.
28+
29+
## Design rationale
30+
To understand the design rationale, it is important to consider the concrete Reframe limitations that we know about:
31+
32+
- Reframe methods are encoded inside messages
33+
- This prevents URL-based pattern matching on methods
34+
- Configuring different caching strategies for different methods
35+
- Configuring reverse proxies on a per-method basis
36+
- Routing methods to specific backends
37+
- Method-specific reverse proxy config such as timeouts
38+
- Developer UX is poor as a result, e.g. for CDN caching you must encode the entire request message and pass it as a query parameter
39+
- This was initially done by URL-escaping the raw bytes
40+
- Not possible to consume correctly using standard JavaScript
41+
- Shipped in Kubo 0.16
42+
- Packing a CID into a struct, encoding it with DAG-CBOR, multibase-encoding that, percent-encoding that, and then passing it in a URL, rather than merely passing the CID in the URL, is needlessly complex from a user's perspective
43+
- Added complexity of "Cacheable" methods supporting both POSTs and GETs
44+
- The required streaming support and message groups add a lot of implementation complexity but isn’t very useful
45+
- Ex for FindProviders, the response is buffered anyway for ETag calculation
46+
- There are no limits on response sizes nor ways to impose limits and paginate
47+
- This is useful for routers that have highly variable resolution time, to send results as soon as possible, but this is not a use case we are focusing on right now and we can add it later
48+
- The Identify method is not implemented because it is not currently useful
49+
- Client and server implementations are difficult to write correctly, because of the non-standard wire formats and conventions
50+
- The Go implementation is [complex](https://github.com/ipfs/go-delegated-routing/blob/main/gen/proto/proto_edelweiss.go) and [brittle](https://github.com/ipfs/go-delegated-routing/blame/main/client/provide.go#L51), and is currently maintained by IPFS Stewards who are already over-committed with other priorities
51+
- Only the HTTP transport has been designed and implemented, so it's unclear if the existing design will work for other transports, and what their use cases and requirements are
52+
53+
So this API proposal makes the following changes:
54+
55+
- The API is defined in HTTP directly
56+
- "Methods" and cache-relevant parameters are pushed into the URL path
57+
- Streaming support is removed, and optional pagination is added, which limits the response size and provides a scalable mechanism for iterating over arbitrarily-large collections
58+
- We might add streaming support w/ chunked-encoded responses in the future, but it's currently not an important feature for the use cases that an HTTP API will be used for
59+
- Bodies are encoded using standard JSON or CBOR, instead of using IPLD codecs
60+
- The "Identify" method and "message groups" are removed
61+
62+
### User benefit
63+
64+
The cost of building and operating content routing services will be much lower, as developers will be able to reuse existing industry-standard tooling.
65+
This will result in more content routing providers, each providing a better experience for users, driving down content routing latency across the IPFS netowrk
66+
and increasing data availability.
67+
68+
### Compatibility
69+
70+
#### Backwards Compatibility
71+
IPFS Stewards will implement this API in [go-delegated-routing](https://github.com/ipfs/go-delegated-routing), using breaking changes in a new minor version.
72+
Because the existing Reframe spec can't be safely used in JavaScript, the experimental support for Reframe in Kubo will be removed in the next release,
73+
and delegated routing will subsequently use this HTTP API. We may decide to re-add Reframe support in the future once these issues have been resolved.
74+
75+
#### Forwards Compatibility
76+
Standard HTTP mechanisms for forward compatibility are used--the API is versioned using a version number in the path. The `Accept` and `Content-Type` headers are used for content type negotiation. new methods will result in new paths, and parameters can be added using either new query parameters or new fields in the request/response body. Certain parts of bodies are labeled as "opaque bytes", which are passed through by the implementation, with no schema enforcement.
77+
78+
### Security
79+
80+
None
81+
82+
### Alternatives
83+
84+
This *is* an alternative.
85+
86+
### Copyright
87+
88+
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

‎routing/DELEGATED_ROUTING_HTTP.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# ![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) Delegated Routing HTTP API
2+
3+
**Author(s)**:
4+
- Gus Eggert
5+
6+
**Maintainer(s)**:
7+
8+
* * *
9+
10+
**Abstract**
11+
12+
"Delegated routing" is a mechanism for IPFS implementations to use for offloading content routing to another process/server. This spec describes an HTTP API for delegated routing.
13+
14+
# Organization of this document
15+
16+
- [Introduction](#introduction)
17+
- [Spec](#spec)
18+
- [Interaction Pattern](#interaction-pattern)
19+
- [Cachability](#cachability)
20+
- [Transports](#transports)
21+
- [Protocol Message Overview](#protocol-message-overview)
22+
- [Known Methods](#known-methods)
23+
- [Method Upgrade Paths](#method-upgrade-paths)
24+
- [Implementations](#implementations)
25+
26+
# API Specification
27+
By default, the Delegated Routing HTTP API uses the `application/json` content type. Clients and servers may optionally negotiate other content types such as `application/cbor`, `application/vnd.ipfs.rpc+dag-json`, etc. using the standard `Accept` and `Content-Type` headers.
28+
29+
- `GET /v1/providers/{CID}`
30+
- Reframe equivalent: FindProviders
31+
- Response
32+
33+
```json
34+
{
35+
"Providers": [
36+
{
37+
"PeerID": "...",
38+
"Multiaddrs": ["...", "..."]
39+
"Protocols": [
40+
{
41+
"Codec": 2320,
42+
"Payload": <opaque data>
43+
}
44+
]
45+
}
46+
]
47+
"NextPageToken": "<token>"
48+
}
49+
```
50+
51+
- Default limit: 100 providers
52+
- Optional query parameters
53+
- `transfer` only return providers who support the passed transfer protocols, expressed as a comma-separated list of multicodec IDs such as `2304,2320`,
54+
- `transport` only return providers whose published multiaddrs explicitly support the passed transport protocols, such as `/quic` or `/tls/ws`.
55+
- `GET /v1/providers/hash/{multihash}`
56+
- This is the same as `GET /v1/providers/{CID}`, but takes a hashed CID encoded as a multihash
57+
- `GET /v1/ipns/{ID}`
58+
- Reframe equivalent: GetIPNS
59+
- Response
60+
- record bytes
61+
- `POST /v1/ipns/{ID}`
62+
- Reframe equivalent: PutIPNS
63+
- Body
64+
- record bytes
65+
- No need for idempotency
66+
- `PUT /v1/providers/{CID}`
67+
- Reframe equivalent: Provide
68+
- Body
69+
70+
```json
71+
{
72+
"Keys": ["cid1", "cid2"],
73+
"Timestamp": 1234,
74+
"AdvisoryTTL": 1234,
75+
"Signature": "multibase bytes",
76+
"Provider": {
77+
"Peer": {
78+
"ID": "peerID",
79+
"Addrs": ["multiaddr1", "multiaddr2"]
80+
},
81+
"Protocols": [
82+
{
83+
"Codec": 1234,
84+
"Payload": <opaque data>
85+
}
86+
]
87+
}
88+
}
89+
```
90+
91+
- Idempotent
92+
- `GET /v1/ping`
93+
- This is absent from Reframe but is necessary for supporting e.g. the accelerated DHT client which can take many minutes to bootstrap
94+
- Returns 200 once the server is ready to accept requests
95+
- An alternate approach is w/ an orchestration dance in the server by not listening on the socket until the dependencies are ready, but this makes the “dance” easier to implement
96+
- Pagination
97+
- Responses with collections of results must have a default limit on the number of results that will be returned in a single response
98+
- Servers may optionally implement pagination by responding with an opaque page token which, when provided as a subsequent query parameter, will fetch the next page of results.
99+
- Clients may continue paginating until no `NextPageToken` is returned.
100+
- Clients making calls that return collections may limit the number of per-page results returned with the `limit` query parameter, i.e. `GET /v1/providers/{CID}?limit=10`
101+
- Additional filtering/sorting operations may be defined on a per-path basis, as needed

0 commit comments

Comments
 (0)