From 40623d2ae09dd6dc849d44cddee1af68c7215ca4 Mon Sep 17 00:00:00 2001 From: "Evgeny @ SimpleX Chat" <259188159+evgeny-simplex@users.noreply.github.com> Date: Sat, 11 Jul 2026 16:39:15 +0000 Subject: [PATCH 1/3] rfc: agent support for PRC pattern without duplex connection --- .../2026-07-11-service-rpc-implementation.md | 229 ++++++++++++++++++ rfcs/2026-07-11-service-rpc.md | 175 +++++++++++++ 2 files changed, 404 insertions(+) create mode 100644 plans/2026-07-11-service-rpc-implementation.md create mode 100644 rfcs/2026-07-11-service-rpc.md diff --git a/plans/2026-07-11-service-rpc-implementation.md b/plans/2026-07-11-service-rpc-implementation.md new file mode 100644 index 000000000..e7fa0fd75 --- /dev/null +++ b/plans/2026-07-11-service-rpc-implementation.md @@ -0,0 +1,229 @@ +# Service RPC implementation plan + +RFC: [../rfcs/2026-07-11-service-rpc.md](../rfcs/2026-07-11-service-rpc.md) + +Types and instances below must encode exactly the ABNF in the RFC. Constructor and event names are provisional. + +## Versions + +- `VersionSMPA` (agent protocol): new version for `AgentRequest`/`AgentResponse` envelopes and the service key bundle in link data. +- `VersionSMPC` (SMP client): new version for the hybrid public header. +- `VersionSMP` (SMP protocol): new version for the `SSND` command. + +## Address type - `Simplex.Messaging.Agent.Protocol` + +```haskell +data ContactConnType = CCTContact | CCTChannel | CCTGroup | CCTRelay | CCTService + +ctTypeChar CCTService = 'S' -- 's' in links: link encoding lowercases, parsing uppercases + +ctTypeP 'S' = pure CCTService +``` + +## Service key bundle - `Simplex.Messaging.Agent.Protocol` + +```haskell +data ServiceKeyBundle = ServiceKeyBundle + { keyId :: ByteString, + reqDhKey :: C.PublicKeyX25519, + reqKemKey :: KEMPublicKey + } + +instance Encoding ServiceKeyBundle where + smpEncode ServiceKeyBundle {keyId, reqDhKey, reqKemKey} = smpEncode (keyId, reqDhKey, reqKemKey) + smpP = do + (keyId, reqDhKey, reqKemKey) <- smpP + pure ServiceKeyBundle {keyId, reqDhKey, reqKemKey} +``` + +`UserContactData` gains a field, appended to the encoding (earlier versions skip it, its absence parses as `Nothing`): + +```haskell +data UserContactData = UserContactData + { direct :: Bool, + owners :: [OwnerAuth], + relays :: [ConnShortLink 'CMContact], + userData :: UserLinkData, + serviceKeys :: Maybe ServiceKeyBundle + } + +instance Encoding UserContactData where + smpEncode UserContactData {direct, owners, relays, userData, serviceKeys} = + B.concat [smpEncode direct, smpEncodeList owners, smpEncodeList relays, smpEncode userData, smpEncode serviceKeys] + smpP = do + direct <- smpP + owners <- smpListP + relays <- smpListP + userData <- smpP + serviceKeys <- fromMaybe Nothing <$> optional smpP -- absent in data from earlier versions + _ <- A.takeByteString -- ignoring tail for forward compatibility + pure UserContactData {direct, owners, relays, userData, serviceKeys} +``` + +The service address record stores current and previous private key bundles with rotation timestamps; expired bundles are deleted. + +## SSND command - `Simplex.Messaging.Protocol`, `Simplex.Messaging.Server` + +```haskell +-- Protocol.hs +SSND :: SndPublicAuthKey -> MsgFlags -> MsgBody -> Command Sender + +-- CommandTag, encoding "SSND" +SSND_ :: CommandTag Sender + +-- encodeProtocol +SSND k flags msg -> e (SSND_, ' ', k, ' ', flags, ' ', Tail msg) + +-- protocolP, gated by VersionSMP +SSND_ -> SSND <$> (smpP <* A.space) <*> (smpP <* A.space) <*> (unTail <$> smpP) +``` + +`checkCredentials`: as `SKEY` - authorization required, sender entity ID. Server verification: `vc SSender (SSND k _ _) = verifySecure k` - authorized by the key it sets. + +Server processing: `checkMode QMMessaging`, then `secureQueue_` (existing, idempotent), then deliver with deduplication. The queue store keeps the hash of the unacknowledged message delivered by `SSND`; a repeated `SSND` with an equal body hash responds `OK` without storing; the hash is dropped when the message is acknowledged. + +## Hybrid public header - `Simplex.Messaging.Protocol` + +```haskell +data PubHeader + = PubHeader + { phVersion :: VersionSMPC, + phE2ePubKey :: Maybe C.PublicKeyX25519 + } + | PubHeaderHybrid + { phVersion :: VersionSMPC, + phKeyId :: Maybe ByteString, -- present in requests, absent in replies + phDhKey :: C.PublicKeyX25519, + phKemCt :: KEMCiphertext + } + +instance Encoding PubHeader where + smpEncode = \case + PubHeader v k_ -> smpEncode (v, k_) -- Maybe encodes as '0' / '1' key, as today + PubHeaderHybrid v kId_ k ct -> smpEncode (v, '2', kId_, k, ct) + smpP = do + v <- smpP + A.anyChar >>= \case + '0' -> pure $ PubHeader v Nothing + '1' -> PubHeader v . Just <$> smpP + '2' -> PubHeaderHybrid v <$> smpP <*> smpP <*> smpP + _ -> fail "bad PubHeader" +``` + +Secret derivation and encryption (`Simplex.Messaging.Crypto`, used from `Simplex.Messaging.Agent.Client`): + +```haskell +hybridSecret :: C.DhSecretX25519 -> KEMSharedKey -> C.SbKey +hybridSecret dh (KEMSharedKey k) = C.unsafeSbKey $ C.hkdf "" (C.dhBytes' dh <> BA.convert k) "SimpleXSMPQueue" 32 +``` + +The body is encrypted with `C.sbEncrypt` (secret_box) instead of `C.cbEncrypt`, padded to the same lengths. Sending variant of `agentCbEncryptOnce`: generate ephemeral X25519 pair, encapsulate to the published KEM key, encode `PubHeaderHybrid`. Receiving: select private keys by `phKeyId` (requests) or use the reply queue keys from the request (replies); unknown key ID produces the error that makes the client re-fetch link data. + +## Agent envelopes - `Simplex.Messaging.Agent.Protocol` + +```haskell +data AgentMsgEnvelope + = ... -- existing constructors + | AgentRequest + { agentVersion :: VersionSMPA, + replyQueue :: RequestReplyQueue, + requestBody :: ByteString + } + | AgentResponse + { agentVersion :: VersionSMPA, + signature :: C.Signature 'C.Ed25519, -- root key signature of signedReply + signedReply :: ByteString -- encoded SignedReply, parsed after signature verification + } + +data RequestReplyQueue = RequestReplyQueue + { smpClientVersion :: VersionSMPC, + smpServer :: SMPServer, + senderId :: SMP.SenderId, + dhPublicKey :: C.PublicKeyX25519, -- fresh per request + kemPublicKey :: KEMPublicKey, -- fresh per request + sndAuthKey :: SndPublicAuthKey -- to secure the reply queue + } + +data SignedReply = SignedReply + { requestHash :: ByteString, -- SHA3-256 of the request message body + prevMsgHash :: ByteString, -- SHA3-256 of the previous reply envelope, empty in the first reply + final :: Bool, + replyBody :: ByteString + } +``` + +```haskell +instance Encoding AgentMsgEnvelope where + smpEncode = \case + ... -- existing constructors + AgentRequest {agentVersion, replyQueue, requestBody} -> + smpEncode (agentVersion, 'Q', replyQueue, Tail requestBody) + AgentResponse {agentVersion, signature, signedReply} -> + smpEncode (agentVersion, 'P', signature, Tail signedReply) + smpP = do + agentVersion <- smpP + smpP >>= \case + ... -- existing constructors + 'Q' -> do + (replyQueue, Tail requestBody) <- smpP + pure AgentRequest {agentVersion, replyQueue, requestBody} + 'P' -> do + (signature, Tail signedReply) <- smpP + pure AgentResponse {agentVersion, signature, signedReply} + +instance Encoding RequestReplyQueue where + smpEncode RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey} = + smpEncode (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey) + smpP = do + (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey) <- smpP + pure RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey} + +instance Encoding SignedReply where + smpEncode SignedReply {requestHash, prevMsgHash, final, replyBody} = + smpEncode (requestHash, prevMsgHash, final, Tail replyBody) + smpP = do + (requestHash, prevMsgHash, final, Tail replyBody) <- smpP + pure SignedReply {requestHash, prevMsgHash, final, replyBody} +``` + +Signing and verification follow the link data pattern (`Simplex.Messaging.Crypto.ShortLink`): + +```haskell +-- service +mkAgentResponse :: C.PrivateKeyEd25519 -> VersionSMPA -> SignedReply -> AgentMsgEnvelope +mkAgentResponse rootPrivKey agentVersion reply = + let signedReply = smpEncode reply + in AgentResponse {agentVersion, signature = C.sign' rootPrivKey signedReply, signedReply} + +-- client +verifyAgentResponse :: C.PublicKeyEd25519 -> AgentMsgEnvelope -> Either AgentErrorType SignedReply +verifyAgentResponse rootKey AgentResponse {signature, signedReply} + | C.verify' rootKey signature signedReply = parse smpP (AGENT A_MESSAGE) signedReply + | otherwise = Left ... -- verification error +``` + +## Agent processing - `Simplex.Messaging.Agent`, `Simplex.Messaging.Agent.Client` + +Client side: + +- `sendServiceRequest`: read link data (cached or `LGET`, proxied per config), create the reply queue (`NEW`, `QRMessaging`, subscribe), generate fresh X25519 + KEM keys and sender auth key, build and store the request message, send (proxied per config), return request ID. +- Retries re-send the stored byte-identical message. Deadline, cancellation and `final` delete the reply queue and the request record. +- Reply queue messages in `processSMPTransmissions`: decrypt with the stored hybrid secret (first reply establishes it from the hybrid header), verify with `verifyAgentResponse` and the hash chain, deliver as events; failures are acknowledged and dropped. +- New events: reply received (request ID, final flag, body), request failed (request ID, error). + +Service side: + +- Address queue messages with `AgentRequest`: compute request hash, look up the dedup store; on hit re-send stored replies (or the expiry error); on miss deliver a request event to the service application. +- Send replies: secure the reply queue and send the first reply with `SSND`, subsequent replies with `SEND`; store sent replies under the request hash until the window expires. + +Database: + +- Client: in-flight requests - request ID, stored request message, reply queue, hybrid secret, deadline, last reply hash. +- Service: dedup store - request hash, sent replies, expiry. + +## Phases + +1. SMP protocol: `SSND`, hybrid public header, version bumps, server dedup marker. +2. Agent: address type, key bundle, envelopes, client API and reply processing. +3. Agent: service-side dedup store and request events. +4. Adoption: `SSND` in the fast connection handshake; hybrid scheme for invitations and confirmations. diff --git a/rfcs/2026-07-11-service-rpc.md b/rfcs/2026-07-11-service-rpc.md new file mode 100644 index 000000000..b29e132c0 --- /dev/null +++ b/rfcs/2026-07-11-service-rpc.md @@ -0,0 +1,175 @@ +# One-off requests to service addresses + +Implementation plan: [../plans/2026-07-11-service-rpc-implementation.md](../plans/2026-07-11-service-rpc-implementation.md) + +## Problem + +Client applications need to interact with services, for example: badge issuance, directory requests, telemetry submissions, blockchain reads and writes, LLM calls. The only communication primitive available today is a duplex connection, so each of these interactions requires the full connection procedure - creating queues, key agreement, double ratchet initialization - and leaves persistent state on both sides: queues, ratchet state, connection records, message history. + +This is the wrong primitive for most service interactions: + +1. Cost. To send the first request to a not yet connected service, the client and the service exchange multiple commands across two servers. For "search the directory" all of it is overhead. The setup cost also creates an incentive to keep connections open, and a service with N users who used it once permanently holds N sets of queues and ratchet states. + +2. Privacy. A connection is a stable pairwise pseudonym. If a service were to use a duplex connection, it could link all requests made over it into a profile: search history in the directory, blockchain operations linked even when different on-chain keys are used, telemetry that becomes longitudinal tracking. The client also accumulates history that can be recovered from the device. Where continuity is needed, it can be provided in the application protocol (e.g., a token included in requests), without a transport-level identity. + +3. Encryption. Messages sent to contact addresses outside an established connection (invitations) have a single layer of X25519 encryption. This was acceptable while such messages contained only a connection link and profile (though it is planned to be improved); it is not acceptable for service requests. + +In-app service addresses should be stored as names resolving to links via the existing addressing layer (server host in the link authority, current link data retrieved with LGET), so that service links can be changed without redeploying the apps. Name resolution is already supported, and out of scope. + +## Security objectives + +1. Requests from the same client must not be linkable to each other by the service or by servers, and no state that outlives the exchange is created on either side. +2. Post-quantum resistant e2e encryption of requests and replies; decryptability of recorded requests must be bounded by key rotation, without changing the address. +3. Reply authenticity must be verifiable against the link; substitution, replay, dropping or reordering of replies by servers must be detectable. +4. A repeated or replayed request must not be executed twice. + +## Solution + +A service address is a contact address subtype. The client sends a request in a single message to the address queue and receives replies in a reply queue it creates for this request: + +1. Resolve the service name to a short link; retrieve and cache link data (`LGET`, via proxy when IP protection is needed). +2. Create a reply queue on a client-chosen server (`NEW`, messaging mode, sender can secure, subscribed at creation). +3. Send the request to the service address queue (unauthenticated `SEND`, via proxy when IP protection is needed). The request contains the reply queue address with fresh keys, and is encrypted to rotatable service keys published in link data, using hybrid X25519 + KEM encryption at the queue layer. +4. The service secures the reply queue and sends the first reply with one combined command. Replies are encrypted to the fresh keys from the request; each reply is signed with the root key over the reply and the request hash, and includes a flag whether more replies follow. +5. The client receives replies until the final one or a deadline, then deletes the reply queue (`DEL`). + +To the first request the client sends 3 commands (2 with cached link data). Streamed replies (LLM output) and delayed replies (credential issued after payment) are ordinary queue messages, distinguished only by the flag whether more replies follow. + +How this meets the objectives: + +1. Unlinkability: fresh keys and a fresh reply queue per request; the sender's IP address and session are protected by existing private routing; the reply queue is deleted after the exchange; nothing is shared between two requests. +2. Encryption: hybrid X25519 + sntrup761 secret at the queue layer for the request and all replies; keys are published in mutable link data, rotated with `LSET`, old keys deleted after a short window. The double ratchet is not used: it would create per-request state before the service decides to reply, and its properties serve long-lived sessions, not one-off exchanges. +3. Authenticity: every reply is signed with the root key committed in the immutable link data, over the reply and the request hash; the hash chain across replies detects dropped and reordered replies. +4. Replay protection: a retry or replay is the byte-identical request; the service agent stores sent replies for a limited window and re-sends them without executing the request again. + +## Design + +Syntax below uses [ABNF][1] with [case-sensitive strings extension][2]. `shortString`, `largeString` and `x509encoded` are defined in the [SMP protocol](../protocol/simplex-messaging.md); `smpServer` and `agentVersion` - in the [agent protocol](../protocol/agent-protocol.md). All hashes below are SHA3-256. + +### Service address + +A new contact address type - char `s` in short links, next to existing `a`/`c`/`g`/`r`: + +```abnf +contactType =/ %s"s" ; service address +``` + +Link data has the same structure as contact addresses: immutable fixed data (root key, connection request data with the queue address) committed by the link hash, and mutable user data signed by the root key. Fixed data and contact user data encodings ignore trailing bytes, so the fields below are added without a new link format. + +Agents dispatch on the address type and the envelope type: a normal address rejects requests, a service address rejects invitations and confirmations. + +### Service key bundle + +Mutable user data of a service address includes a key bundle, appended to the existing contact user data encoding: + +```abnf +userContactData = direct ownersList relaysList userLinkData [serviceKeys] +serviceKeys = %s"0" / (%s"1" serviceKeyBundle) ; skipped by earlier versions +serviceKeyBundle = keyId reqDhKey reqKemKey +keyId = shortString ; referenced by requests +reqDhKey = length x509encoded ; X25519, used instead of the fixed-data queue key +reqKemKey = largeString ; sntrup761 encapsulation key, 1158 bytes +``` + +The service periodically rotates the bundle with `LSET` and keeps previous private keys for a window covering client link data caching and queue message retention. A request encrypted to an unknown key ID fails with an error; the client then re-fetches link data and retries. Deleting expired keys is what bounds decryptability of recorded requests, so the window must be short and fixed. + +Link data is encrypted with a symmetric key derived from the link, which is not weakened by quantum computers, so publishing the bundle in link data does not undermine the scheme. + +Sizes: KEM public key is 1158 bytes, user data is padded to 13784 bytes - the bundle fits together with application data. + +### Queue-layer PQ encryption + +The single-shot queue encryption (today: crypto_box with a secret from the queue DH key and a sender ephemeral key, ephemeral key in the message public header) is extended to a hybrid scheme in a new SMP client version. The public header gains a third variant: + +```abnf +smpPubHeaderHybrid = smpClientVersion %s"2" optKeyId senderPublicDhKey kemCiphertext +optKeyId = %s"0" / (%s"1" keyId) ; present in requests, absent in replies +senderPublicDhKey = length x509encoded ; fresh X25519 key +kemCiphertext = largeString ; sntrup761 ciphertext, 1039 bytes +``` + +The secret is derived from both shared secrets, and the body is encrypted with NaCl secret_box, padded as today: + +``` +secret = HKDF(dh(recipient key, sender ephemeral key) || KEM shared secret) +``` + +The header is visible to the destination server, as invitation headers are today (with proxied sending it is not visible to the proxy). + +The same scheme applies to replies, using the keys from the request: the first reply includes the hybrid header, the hybrid secret is computed once per reply queue and used for subsequent replies with the empty header - the same one-secret-per-queue model as existing sender queues. Invitations and confirmations can adopt the same scheme independently of this design, removing the single-layer X25519 exposure of the profile. + +### Request + +A new agent envelope (alongside `agentConfirmation`, `agentMsgEnvelope`, `agentInvitation`, `agentRatchetKey`): + +```abnf +agentRequest = agentVersion %s"Q" replyQueue requestBody +replyQueue = smpClientVersion smpServer senderId dhPublicKey kemPublicKey sndAuthPublicKey +smpClientVersion = 2*2 OCTET +senderId = shortString ; sender ID of the reply queue +dhPublicKey = length x509encoded ; X25519, fresh per request +kemPublicKey = largeString ; sntrup761 encapsulation key, fresh per request +sndAuthPublicKey = length x509encoded ; key to secure the reply queue +requestBody = *OCTET ; remaining bytes, application-defined +``` + +The envelope type indicates to the receiving agent that no connection is created and replies are sent to the provided queue. The reply queue keys never appear outside the encrypted request. Requests must fit in one message; larger payloads are passed as an XFTP file description in the request body. + +A retry is the byte-identical stored request message, so all correlation uses one value: the request hash - the hash of the message body, the same bytes as sent by the client and as received by the service after the server encryption layer is removed. + +Request delivery failures follow existing SEND semantics: on QUOTA (the address queue is full) the client retries with backoff within the request deadline. + +### Replies + +The service secures the reply queue with the sender key from the request and sends the first reply in one combined SMP command (see below). Replies use a new agent envelope, following the signature-first structure of link data: + +```abnf +agentResponse = agentVersion %s"P" signature signedReply +signature = length 64*64 OCTET ; root key signature of signedReply +signedReply = requestHash prevMsgHash final replyBody +requestHash = shortString ; hash of the request message body +prevMsgHash = shortString ; hash of the previous agentResponse, empty in the first reply +final = %s"T" / %s"F" ; F - more replies follow +replyBody = *OCTET ; remaining bytes, application-defined +``` + +The client verifies each reply: the signature against the root key from link data, the request hash against the sent request, the hash chain against the previous reply. Forging a reply requires the root key even if the reply queue keys are compromised; a signed reply cannot be replayed for a different request; the hash chain detects replies dropped or reordered by the reply queue server. + +The client deletes the reply queue on the final flag, on the request deadline (application-defined per request), or on cancellation. Deleting the queue also cancels the stream: subsequent SEND commands from the service fail with AUTH. Router queue and message expiry limit the lifetime of abandoned queues. Stream length between client acknowledgements is bounded by queue capacity. + +Delayed replies (e.g., a credential issued after payment) are stored in the reply queue within message retention time and received when the client subscribes again; notification credentials can be added to the reply queue with existing commands. Messages in the reply queue that fail decryption or verification are acknowledged and dropped. + +### Combined SKEY+SEND command + +A new SMP command combining `SKEY` and `SEND` in one transmission: + +```abnf +secureSend = %s"SSND " senderAuthPublicKey SP msgFlags SP smpEncMessage +senderAuthPublicKey = length x509encoded +``` + +The command is authorized with the key it sets, as `SKEY`, and only accepted on messaging-mode queues. The server responds `OK` or `ERR`. It is idempotent in both parts: + +- key part: as `SKEY` today - repeated command with the same key succeeds, different key fails with AUTH +- send part: the server stores a hash of the message body until the message is acknowledged; within that window a repeated command with the same body is not delivered again and is reported as delivered + +A retry arriving after the message was acknowledged is delivered as a duplicate and suppressed by the receiving agent (by message hash), so the server-side marker covers the common case and the agent covers the rest. + +It is used by the service for the first reply, and by the joining party in the fast connection handshake (currently `SKEY` then `SEND` confirmation), removing one command and round trip from both flows. + +### Idempotency + +Handled uniformly by the agent: it stores the request hash and sent replies for a window declared in link data (with a default), and re-sends stored replies for a repeated request without invoking the service application. This gives exactly-once execution over at-least-once delivery for all services; services with idempotent semantics of their own lose nothing. + +The guarantee is bounded by the window: the client must not retry after the request deadline, and the deadline must not exceed the declared window. Stored replies may be evicted under storage pressure; a retry whose replies were evicted receives an error prompting a new request. Within the window a request is never executed twice. + +### Out of scope + +- Continuity: sessions across requests are an application concern (tokens in request and reply bodies). +- Service-initiated messages: there is no standing channel; use a connection where push is needed. +- Abuse protection beyond existing queue quotas: services can require application-level credentials (e.g., a badge) in request bodies; rate limiting options are a separate discussion. +- Scaling request reception: a single address queue bounds service throughput; distributing reception across multiple queues or relays (the existing `relays` field in contact link data) is a separate discussion. +- Name resolution: existing addressing layer. + +[1]: https://tools.ietf.org/html/rfc5234 +[2]: https://tools.ietf.org/html/rfc7405 From 97e7e9b7c5fc5ec0d217d713072da710ad7a6fb8 Mon Sep 17 00:00:00 2001 From: "Evgeny @ SimpleX Chat" <259188159+evgeny-simplex@users.noreply.github.com> Date: Sun, 12 Jul 2026 15:09:01 +0000 Subject: [PATCH 2/3] update rfc, plan --- .../2026-07-11-service-rpc-implementation.md | 270 ++++++++++++++---- rfcs/2026-07-11-service-rpc.md | 107 +++---- 2 files changed, 265 insertions(+), 112 deletions(-) diff --git a/plans/2026-07-11-service-rpc-implementation.md b/plans/2026-07-11-service-rpc-implementation.md index e7fa0fd75..68b9bff67 100644 --- a/plans/2026-07-11-service-rpc-implementation.md +++ b/plans/2026-07-11-service-rpc-implementation.md @@ -2,7 +2,7 @@ RFC: [../rfcs/2026-07-11-service-rpc.md](../rfcs/2026-07-11-service-rpc.md) -Types and instances below must encode exactly the ABNF in the RFC. Constructor and event names are provisional. +Types and instances below must encode exactly the ABNF in the RFC. Constructor, event, table and function names are provisional. ## Versions @@ -20,7 +20,9 @@ ctTypeChar CCTService = 'S' -- 's' in links: link encoding lowercases, parsing u ctTypeP 'S' = pure CCTService ``` -## Service key bundle - `Simplex.Messaging.Agent.Protocol` +The contact type is currently fixed to `CCTContact` when the agent reconstructs a contact link (`Agent.hs`, `cslContact`). It becomes a stored property of the address (column `link_contact_type` below), read when the link is reconstructed and when an address queue message is dispatched. + +## Service key bundle in link data - `Simplex.Messaging.Agent.Protocol` ```haskell data ServiceKeyBundle = ServiceKeyBundle @@ -36,7 +38,7 @@ instance Encoding ServiceKeyBundle where pure ServiceKeyBundle {keyId, reqDhKey, reqKemKey} ``` -`UserContactData` gains a field, appended to the encoding (earlier versions skip it, its absence parses as `Nothing`): +`UserContactData` gains a field, appended to the encoding, absent in data written by earlier versions. The agent sets it when it signs and updates link data; the application supplies only its own data. There is no retention value in link data - retention is service configuration (see Idempotency). ```haskell data UserContactData = UserContactData @@ -60,8 +62,6 @@ instance Encoding UserContactData where pure UserContactData {direct, owners, relays, userData, serviceKeys} ``` -The service address record stores current and previous private key bundles with rotation timestamps; expired bundles are deleted. - ## SSND command - `Simplex.Messaging.Protocol`, `Simplex.Messaging.Server` ```haskell @@ -74,13 +74,15 @@ SSND_ :: CommandTag Sender -- encodeProtocol SSND k flags msg -> e (SSND_, ' ', k, ' ', flags, ' ', Tail msg) --- protocolP, gated by VersionSMP +-- protocolP, in a new VersionSMP SSND_ -> SSND <$> (smpP <* A.space) <*> (smpP <* A.space) <*> (unTail <$> smpP) ``` `checkCredentials`: as `SKEY` - authorization required, sender entity ID. Server verification: `vc SSender (SSND k _ _) = verifySecure k` - authorized by the key it sets. -Server processing: `checkMode QMMessaging`, then `secureQueue_` (existing, idempotent), then deliver with deduplication. The queue store keeps the hash of the unacknowledged message delivered by `SSND`; a repeated `SSND` with an equal body hash responds `OK` without storing; the hash is dropped when the message is acknowledged. +Server processing: `checkMode QMMessaging`, then `secureQueue_` (existing, idempotent), then deliver with deduplication. All queue store backends (STM, journal, PostgreSQL) keep the hash of the message delivered by `SSND` until it is acknowledged; a repeated `SSND` with an equal message hash responds `OK` without delivering it again; the hash is removed when the message is acknowledged. Server stats gain an `SSND` counter. + +Proxying: `proxySMPCommand` is polymorphic in the sender command, so `SSND` forwards through `PFWD`/`RFWD` without changes. ## Hybrid public header - `Simplex.Messaging.Protocol` @@ -117,7 +119,9 @@ hybridSecret :: C.DhSecretX25519 -> KEMSharedKey -> C.SbKey hybridSecret dh (KEMSharedKey k) = C.unsafeSbKey $ C.hkdf "" (C.dhBytes' dh <> BA.convert k) "SimpleXSMPQueue" 32 ``` -The body is encrypted with `C.sbEncrypt` (secret_box) instead of `C.cbEncrypt`, padded to the same lengths. Sending variant of `agentCbEncryptOnce`: generate ephemeral X25519 pair, encapsulate to the published KEM key, encode `PubHeaderHybrid`. Receiving: select private keys by `phKeyId` (requests) or use the reply queue keys from the request (replies); unknown key ID produces the error that makes the client re-fetch link data. +The body is encrypted with `C.sbEncrypt` (secret_box) rather than `C.cbEncrypt`, padded to the same lengths. The sending side generates an ephemeral X25519 pair, encapsulates to the recipient KEM key, and writes `PubHeaderHybrid`. The receiving side selects private keys by `phKeyId` for a request, or uses the reply queue keys for a reply, and computes the same secret. A request with an unknown key ID is discarded and counted. + +Size constants: the hybrid header adds about 1.2KB (KEM ciphertext 1039 bytes plus the ephemeral key). Define the request and reply body length constants from the existing padded body lengths at implementation. ## Agent envelopes - `Simplex.Messaging.Agent.Protocol` @@ -127,103 +131,249 @@ data AgentMsgEnvelope | AgentRequest { agentVersion :: VersionSMPA, replyQueue :: RequestReplyQueue, - requestBody :: ByteString + requestPayload :: ByteString -- opaque, hashed } | AgentResponse { agentVersion :: VersionSMPA, - signature :: C.Signature 'C.Ed25519, -- root key signature of signedReply - signedReply :: ByteString -- encoded SignedReply, parsed after signature verification + signature :: C.Signature 'C.Ed25519, -- root key signature of signedResponse + signedResponse :: ByteString -- encoded SignedResponse, parsed after the signature is verified } data RequestReplyQueue = RequestReplyQueue { smpClientVersion :: VersionSMPC, smpServer :: SMPServer, - senderId :: SMP.SenderId, - dhPublicKey :: C.PublicKeyX25519, -- fresh per request - kemPublicKey :: KEMPublicKey, -- fresh per request - sndAuthKey :: SndPublicAuthKey -- to secure the reply queue + senderId :: SMP.SenderId, -- sender ID of the reply queue + dhPublicKey :: C.PublicKeyX25519, -- reply queue X25519 key (its e2e key) + kemPublicKey :: KEMPublicKey -- reply queue sntrup761 key } -data SignedReply = SignedReply - { requestHash :: ByteString, -- SHA3-256 of the request message body - prevMsgHash :: ByteString, -- SHA3-256 of the previous reply envelope, empty in the first reply - final :: Bool, - replyBody :: ByteString +data SignedResponse = SignedResponse + { requestHash :: ByteString, -- SHA3-256 of requestPayload + prevMsgHash :: ByteString, -- SHA3-256 of the previous AgentResponse, empty in the first + more :: Bool, -- True if more reply messages follow + responses :: NonEmpty ByteString -- opaque application responses } ``` +The request does not include a key to secure the reply queue: the service generates its own sender key and secures the queue with `SSND`. `requestPayload` is the only hashed part; the reply queue fields are not hashed. + ```haskell instance Encoding AgentMsgEnvelope where smpEncode = \case ... -- existing constructors - AgentRequest {agentVersion, replyQueue, requestBody} -> - smpEncode (agentVersion, 'Q', replyQueue, Tail requestBody) - AgentResponse {agentVersion, signature, signedReply} -> - smpEncode (agentVersion, 'P', signature, Tail signedReply) + AgentRequest {agentVersion, replyQueue, requestPayload} -> + smpEncode (agentVersion, 'Q', replyQueue, Tail requestPayload) + AgentResponse {agentVersion, signature, signedResponse} -> + smpEncode (agentVersion, 'P', signature, Tail signedResponse) smpP = do agentVersion <- smpP smpP >>= \case ... -- existing constructors 'Q' -> do - (replyQueue, Tail requestBody) <- smpP - pure AgentRequest {agentVersion, replyQueue, requestBody} + (replyQueue, Tail requestPayload) <- smpP + pure AgentRequest {agentVersion, replyQueue, requestPayload} 'P' -> do - (signature, Tail signedReply) <- smpP - pure AgentResponse {agentVersion, signature, signedReply} + (signature, Tail signedResponse) <- smpP + pure AgentResponse {agentVersion, signature, signedResponse} instance Encoding RequestReplyQueue where - smpEncode RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey} = - smpEncode (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey) + smpEncode RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey} = + smpEncode (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey) smpP = do - (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey) <- smpP - pure RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey, sndAuthKey} + (smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey) <- smpP + pure RequestReplyQueue {smpClientVersion, smpServer, senderId, dhPublicKey, kemPublicKey} -instance Encoding SignedReply where - smpEncode SignedReply {requestHash, prevMsgHash, final, replyBody} = - smpEncode (requestHash, prevMsgHash, final, Tail replyBody) +instance Encoding SignedResponse where + smpEncode SignedResponse {requestHash, prevMsgHash, more, responses} = + smpEncode (requestHash, prevMsgHash, more) <> smpEncodeList (map Large $ L.toList responses) smpP = do - (requestHash, prevMsgHash, final, Tail replyBody) <- smpP - pure SignedReply {requestHash, prevMsgHash, final, replyBody} + (requestHash, prevMsgHash, more) <- smpP + rs <- map unLarge <$> smpListP + responses <- maybe (fail "empty responses") pure $ L.nonEmpty rs + pure SignedResponse {requestHash, prevMsgHash, more, responses} ``` Signing and verification follow the link data pattern (`Simplex.Messaging.Crypto.ShortLink`): ```haskell --- service -mkAgentResponse :: C.PrivateKeyEd25519 -> VersionSMPA -> SignedReply -> AgentMsgEnvelope -mkAgentResponse rootPrivKey agentVersion reply = - let signedReply = smpEncode reply - in AgentResponse {agentVersion, signature = C.sign' rootPrivKey signedReply, signedReply} - --- client -verifyAgentResponse :: C.PublicKeyEd25519 -> AgentMsgEnvelope -> Either AgentErrorType SignedReply -verifyAgentResponse rootKey AgentResponse {signature, signedReply} - | C.verify' rootKey signature signedReply = parse smpP (AGENT A_MESSAGE) signedReply +-- service: signing key is linkPrivSigKey from the address ShortLinkCreds +mkAgentResponse :: C.PrivateKeyEd25519 -> VersionSMPA -> SignedResponse -> AgentMsgEnvelope +mkAgentResponse rootPrivKey agentVersion resp = + let signedResponse = smpEncode resp + in AgentResponse {agentVersion, signature = C.sign' rootPrivKey signedResponse, signedResponse} + +-- client: rootKey from the fixed link data +verifyAgentResponse :: C.PublicKeyEd25519 -> AgentMsgEnvelope -> Either AgentErrorType SignedResponse +verifyAgentResponse rootKey AgentResponse {signature, signedResponse} + | C.verify' rootKey signature signedResponse = parse smpP (AGENT A_MESSAGE) signedResponse | otherwise = Left ... -- verification error ``` +## Reply queue - no new connection type + +The reply queue on the client is a receive connection (`RcvConnection`), extended to carry the KEM key and the computed hybrid secret. There is no new connection type and no flag on the connection: a client service request row (below) references the reply queue connection, and its presence identifies the connection as an RPC reply queue for dispatch and cleanup. + +The reply queue reuses the existing receive queue fields. `e2ePrivKey` is its X25519 key, its public half is the queue address DH key sent in the request. Two nullable columns are added to `rcv_queues`: + +- `reply_kem_priv_key` - the reply queue KEM private key. +- `reply_secret` - the hybrid secret, empty until the first reply, then set from the first reply's header, as the per-queue DH secret is set by `setRcvQueueConfirmedE2E` on the first message today. Later replies are decrypted with it by the standard receive path. + +The service does not create a connection or queue for the reply queue. It sends replies directly to the reply queue address, as `sendInvitation` sends to a queue with no connection. + +## Database schema - `Agent/Store/SQLite/Migrations`, `Agent/Store/Postgres/Migrations` + +One migration (`M20260712_service_rpc`), SQLite shown, PostgreSQL mirrors it; column types follow existing schema conventions. + +```sql +-- address contact type (contact address queues), and reply queue keys (reply queues). +-- NULL link_contact_type means 'A' (contact) for existing rows. +ALTER TABLE rcv_queues ADD COLUMN link_contact_type TEXT; +ALTER TABLE rcv_queues ADD COLUMN reply_kem_priv_key BLOB; +ALTER TABLE rcv_queues ADD COLUMN reply_secret BLOB; + +-- service side: current and retired bundle private keys. +CREATE TABLE service_address_keys( + service_address_key_id INTEGER PRIMARY KEY AUTOINCREMENT, + conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, + key_id BLOB NOT NULL, + dh_priv_key BLOB NOT NULL, + kem_priv_key BLOB NOT NULL, + created_at TEXT NOT NULL, + retired_at TEXT -- set on rotation, row deleted when the key retention period ends +); +CREATE UNIQUE INDEX idx_service_address_keys ON service_address_keys(conn_id, key_id); + +-- client side: one pending request per reply queue connection. +CREATE TABLE snd_service_requests( + snd_service_request_id INTEGER PRIMARY KEY AUTOINCREMENT, + conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, -- reply queue connection + request_hash BLOB NOT NULL, + root_key BLOB NOT NULL, -- to verify replies + last_reply_hash BLOB, -- updated as reply messages arrive + deadline TEXT NOT NULL, + created_at TEXT NOT NULL +); + +-- service side: one record per distinct request hash on an address. +CREATE TABLE rcv_service_requests( + rcv_service_request_id INTEGER PRIMARY KEY AUTOINCREMENT, + conn_id BLOB NOT NULL REFERENCES connections ON DELETE CASCADE, -- address connection + request_hash BLOB NOT NULL, + ended INTEGER NOT NULL DEFAULT 0, -- a reply message with more = False was sent + expires_at TEXT NOT NULL, -- created + retention + created_at TEXT NOT NULL +); +CREATE UNIQUE INDEX idx_rcv_service_requests ON rcv_service_requests(conn_id, request_hash); + +-- service side: ordered signed reply messages for a request; the signed envelope +-- is independent of the reply queue, so it is stored once and encrypted per queue on send. +CREATE TABLE rcv_service_replies( + rcv_service_reply_id INTEGER PRIMARY KEY AUTOINCREMENT, + rcv_service_request_id INTEGER NOT NULL REFERENCES rcv_service_requests ON DELETE CASCADE, + reply_seq INTEGER NOT NULL, + more INTEGER NOT NULL, + signed_reply BLOB NOT NULL -- encoded AgentResponse +); +CREATE UNIQUE INDEX idx_rcv_service_replies ON rcv_service_replies(rcv_service_request_id, reply_seq); + +-- service side: reply queues subscribed under a request (the first, and any repeat). +CREATE TABLE rcv_service_reply_queues( + rcv_service_reply_queue_id INTEGER PRIMARY KEY AUTOINCREMENT, + rcv_service_request_id INTEGER NOT NULL REFERENCES rcv_service_requests ON DELETE CASCADE, + host TEXT NOT NULL, + port TEXT NOT NULL, + snd_id BLOB NOT NULL, + server_key_hash BLOB, + reply_secret BLOB NOT NULL, -- hybrid secret to this reply queue + secured INTEGER NOT NULL DEFAULT 0, -- reply queue secured with SSND + last_sent_seq INTEGER NOT NULL DEFAULT 0 +); +``` + +## Agent API - `Simplex.Messaging.Agent` + +Service side: + +```haskell +-- Creates a contact connection with CCTService link type, generates the root key +-- and the first key bundle, composes and signs link data. +createServiceAddress :: AgentClient -> UserId -> UserLinkData -> AE (ConnId, ConnShortLink 'CMContact) + +-- Re-signs and updates user data with LSET, keeping the agent-owned key bundle. +updateServiceAddressData :: AgentClient -> ConnId -> UserLinkData -> AE () + +-- Generates a new bundle, updates link data, retires the previous keys until the retention period ends. +-- Also runs on a schedule (config) while the address is subscribed. +rotateServiceAddressKeys :: AgentClient -> ConnId -> AE () + +-- Sends one reply message with a list of responses; more = False ends the exchange. +-- The first message to each reply queue secures it with SSND; later messages use SEND. +sendServiceReply :: AgentClient -> ServiceRequestRef -> Bool -> NonEmpty ByteString -> AE () +``` + +Address deletion: existing `deleteConnection`. + +Client side (name resolution to a link is an existing API; the link must have `CCTService` type and a key bundle in link data): + +```haskell +-- Retrieves link data, creates the reply queue, sends the request, and waits for the first +-- reply message up to the deadline. The callback receives later reply messages while the process runs. +sendServiceRequest :: + AgentClient -> UserId -> ConnShortLink 'CMContact -> UTCTime -> + ByteString -> (ServiceReply -> IO ()) -> AE ServiceReply + +-- Deletes the reply queue and the request row, and drops the callback. +cancelServiceRequest :: AgentClient -> ConnId -> AE () + +data ServiceReply = ServiceReply {responses :: NonEmpty ByteString, more :: Bool} +``` + +The first reply message returns from `sendServiceRequest`; later messages are delivered through the callback. Both the waiting call and the callback are held in an in-memory map in `AgentClient`, keyed by the reply queue connection, and filled by the receive path. They do not survive a restart. + +Service side events (`AEvent`, entity is the address connection): + +```haskell +SREQ :: ServiceRequestRef -> ByteString -> AEvent AEConn -- request received, payload for the bot +``` + +`ServiceRequestRef` identifies the request record; the bot passes it to `sendServiceReply`. + ## Agent processing - `Simplex.Messaging.Agent`, `Simplex.Messaging.Agent.Client` Client side: -- `sendServiceRequest`: read link data (cached or `LGET`, proxied per config), create the reply queue (`NEW`, `QRMessaging`, subscribe), generate fresh X25519 + KEM keys and sender auth key, build and store the request message, send (proxied per config), return request ID. -- Retries re-send the stored byte-identical message. Deadline, cancellation and `final` delete the reply queue and the request record. -- Reply queue messages in `processSMPTransmissions`: decrypt with the stored hybrid secret (first reply establishes it from the hybrid header), verify with `verifyAgentResponse` and the hash chain, deliver as events; failures are acknowledged and dropped. -- New events: reply received (request ID, final flag, body), request failed (request ID, error). +- `sendServiceRequest`: retrieve link data (`LGET`, proxied per config) for every request - no caching; create the reply queue (`NEW`, messaging mode, subscribed) with an added KEM key; write the `snd_service_requests` row; send the request (proxied per config); wait on the in-memory sink for the first reply until the deadline. +- Reply processing in `processSMPTransmissions`: a message on a receive queue that has a `snd_service_requests` row is a reply. Decrypt (the first message sets `reply_secret` from its header), verify with `verifyAgentResponse`, check the request hash and the previous-message hash, update `last_reply_hash`, deliver to the waiting call or the callback. A message that fails verification is acknowledged and discarded. On `more = False`, or on the deadline, or on `cancelServiceRequest`, delete the reply queue (`DEL`) and the row. +- `cleanupManager` gains one step: delete `snd_service_requests` past the deadline and mark their reply queue connections deleted; the existing deleted-connections step sends `DEL` and removes them. After a restart every row is stale, so this step removes all reply queues left behind. Service side: -- Address queue messages with `AgentRequest`: compute request hash, look up the dedup store; on hit re-send stored replies (or the expiry error); on miss deliver a request event to the service application. -- Send replies: secure the reply queue and send the first reply with `SSND`, subsequent replies with `SEND`; store sent replies under the request hash until the window expires. +- Address queue message dispatch reads `link_contact_type`: a service address accepts only `AgentRequest` and rejects invitations and confirmations; a non-service address rejects `AgentRequest`. +- On `AgentRequest`: select bundle private keys by `keyId`, decrypt, compute the request hash. Look up `rcv_service_requests` by (address conn, hash): + - new: insert the request, insert a `rcv_service_reply_queues` row for the reply queue in the request, deliver `SREQ` to the bot. + - existing: insert a `rcv_service_reply_queues` row for the reply queue in this request, and send it every `rcv_service_replies` message already stored, in order. +- `sendServiceReply`: append a `rcv_service_replies` message (sign with the address root key), then send it to every reply queue of the request - `SSND` for a queue not yet secured, `SEND` after - encrypting the stored envelope to each queue's `reply_secret`; set `ended` when `more = False`. +- `cleanupManager` gains one step: delete `rcv_service_requests` past `expires_at` (cascading to replies and reply queues) and `service_address_keys` past the key retention period. + +Configuration (`AgentConfig`): default request deadline, request retention period (1 to 24 hours), key rotation interval. + +Errors: API failures reuse `AgentErrorType` (for example `CMD PROHIBITED` for a link that is not a service address). A new constructor is added only if the chat library needs to distinguish service request failures - decided at implementation. + +## Correlation and chat + +A reply is connected to its request by the reply queue: one request has one reply queue, and every message in that queue is a reply to that request. The request hash is used only in the reply signature, to bind a reply to the request content. The application ID is content inside `requestPayload`, used only by the application to make two requests equal or different; the agent does not read it. + +Both ends are chat bots on the chat library. The chat library serializes a service command into `requestPayload` and deserializes the responses; the agent transports them and correlates by reply queue. The chat framework's `chatServiceCalls` correlation by `(AgentConnId, SharedMsgId)` is not used - the agent correlates by reply queue and the call returns the first reply directly. -Database: +## Tests -- Client: in-flight requests - request ID, stored request message, reply queue, hybrid secret, deadline, last reply hash. -- Service: dedup store - request hash, sent replies, expiry. +- Encoding roundtrips: envelopes, `ServiceKeyBundle` in `UserContactData`, `PubHeader` (all three variants), `SSND`, `SignedResponse` with one and several responses. +- Server: `SSND` on messaging and contact queues, repeated `SSND` before and after ACK, a different key, via proxy. +- Agent end-to-end: a request with one reply message; several responses in one message; responses in several messages delivered to the callback; a repeat request while pending coalesces onto the same operation; a repeat request after completion receives the stored messages without a second execution; key rotation with an old key still accepted while kept; a request whose key was deleted fails at the deadline; deadline; cancellation; a restart deletes reply queues; envelope rejection on both address types. ## Phases -1. SMP protocol: `SSND`, hybrid public header, version bumps, server dedup marker. -2. Agent: address type, key bundle, envelopes, client API and reply processing. -3. Agent: service-side dedup store and request events. -4. Adoption: `SSND` in the fast connection handshake; hybrid scheme for invitations and confirmations. +1. SMP protocol: `SSND`, hybrid public header, version bumps, server dedup hash, server tests. +2. Agent: address type, key bundle in link data, envelopes, reply queue columns, schema migration, `createServiceAddress`, `sendServiceRequest`, reply processing, cleanup. +3. Agent: service-side request store, `sendServiceReply`, coalescing and repeats, key rotation, end-to-end tests. +4. Adoption: `SSND` in the fast connection handshake; the hybrid scheme for invitations and confirmations. diff --git a/rfcs/2026-07-11-service-rpc.md b/rfcs/2026-07-11-service-rpc.md index b29e132c0..1eb7a42df 100644 --- a/rfcs/2026-07-11-service-rpc.md +++ b/rfcs/2026-07-11-service-rpc.md @@ -21,26 +21,26 @@ In-app service addresses should be stored as names resolving to links via the ex 1. Requests from the same client must not be linkable to each other by the service or by servers, and no state that outlives the exchange is created on either side. 2. Post-quantum resistant e2e encryption of requests and replies; decryptability of recorded requests must be bounded by key rotation, without changing the address. 3. Reply authenticity must be verifiable against the link; substitution, replay, dropping or reordering of replies by servers must be detectable. -4. A repeated or replayed request must not be executed twice. +4. A repeated request for the same operation must not be executed twice. ## Solution -A service address is a contact address subtype. The client sends a request in a single message to the address queue and receives replies in a reply queue it creates for this request: +A service address is a contact address subtype. Both the client and the service are chat bots using the chat library over the agent. The chat library serializes a request into opaque bytes and calls the agent; the agent transports it and returns replies; the chat library deserializes them. The correlation of a reply with its request is the reply queue: each request has its own reply queue, and every message in that queue is a reply to that request. -1. Resolve the service name to a short link; retrieve and cache link data (`LGET`, via proxy when IP protection is needed). -2. Create a reply queue on a client-chosen server (`NEW`, messaging mode, sender can secure, subscribed at creation). -3. Send the request to the service address queue (unauthenticated `SEND`, via proxy when IP protection is needed). The request contains the reply queue address with fresh keys, and is encrypted to rotatable service keys published in link data, using hybrid X25519 + KEM encryption at the queue layer. -4. The service secures the reply queue and sends the first reply with one combined command. Replies are encrypted to the fresh keys from the request; each reply is signed with the root key over the reply and the request hash, and includes a flag whether more replies follow. -5. The client receives replies until the final one or a deadline, then deletes the reply queue (`DEL`). +The exchange: -To the first request the client sends 3 commands (2 with cached link data). Streamed replies (LLM output) and delayed replies (credential issued after payment) are ordinary queue messages, distinguished only by the flag whether more replies follow. +1. Resolve the service name to a short link and retrieve link data (`LGET`, via proxy when IP protection is needed). Link data holds the root key for verifying replies and the service key bundle for encrypting the request. +2. Create a reply queue (`NEW`, messaging mode, subscribed at creation). The reply queue is a receive queue with an added KEM key; the service encrypts replies to the reply queue keys. +3. Send the request to the service address queue once (unauthenticated `SEND`, via proxy when IP protection is needed). The request includes the reply queue address and its KEM key, and the application payload. It is encrypted to the service key bundle with hybrid X25519 + KEM encryption at the queue layer. There is no transport retry; a reply is the success signal, and a hard error fails the call. +4. The service decrypts the request, delivers the payload to the service application, secures the reply queue, and sends replies. Each reply message includes the request hash, the previous message hash, a flag whether more messages follow, and a non-empty list of application responses; it is signed with the root key. +5. The client verifies each reply message and delivers its responses to the application. The first reply message returns from the call; later reply messages are delivered through a callback the application registered. The client deletes the reply queue on the final message, on the request deadline, or when the application cancels the request. How this meets the objectives: 1. Unlinkability: fresh keys and a fresh reply queue per request; the sender's IP address and session are protected by existing private routing; the reply queue is deleted after the exchange; nothing is shared between two requests. -2. Encryption: hybrid X25519 + sntrup761 secret at the queue layer for the request and all replies; keys are published in mutable link data, rotated with `LSET`, old keys deleted after a short window. The double ratchet is not used: it would create per-request state before the service decides to reply, and its properties serve long-lived sessions, not one-off exchanges. -3. Authenticity: every reply is signed with the root key committed in the immutable link data, over the reply and the request hash; the hash chain across replies detects dropped and reordered replies. -4. Replay protection: a retry or replay is the byte-identical request; the service agent stores sent replies for a limited window and re-sends them without executing the request again. +2. Encryption: hybrid X25519 + sntrup761 secret at the queue layer for the request and all replies. Request keys are published in mutable link data and rotated with `LSET`; the double ratchet is not used, because it would create per-request state before the service decides to reply and its properties serve long-lived sessions. +3. Authenticity: every reply message is signed with the root key committed in the immutable link data, over the request hash and the message; the previous-message hash in each message detects dropped and reordered messages. +4. Single execution: the service identifies a request by the hash of its payload and, within a fixed retention period, repeats the stored replies for a repeated request without running the operation again. ## Design @@ -54,7 +54,7 @@ A new contact address type - char `s` in short links, next to existing `a`/`c`/` contactType =/ %s"s" ; service address ``` -Link data has the same structure as contact addresses: immutable fixed data (root key, connection request data with the queue address) committed by the link hash, and mutable user data signed by the root key. Fixed data and contact user data encodings ignore trailing bytes, so the fields below are added without a new link format. +Link data has the same structure as contact addresses: immutable fixed data (root key, connection request data with the queue address) committed by the link hash, and mutable user data signed by the root key. Fixed data and contact user data encodings ignore trailing bytes, so the field below is added without a new link format. Agents dispatch on the address type and the envelope type: a normal address rejects requests, a service address rejects invitations and confirmations. @@ -64,80 +64,80 @@ Mutable user data of a service address includes a key bundle, appended to the ex ```abnf userContactData = direct ownersList relaysList userLinkData [serviceKeys] -serviceKeys = %s"0" / (%s"1" serviceKeyBundle) ; skipped by earlier versions +serviceKeys = %s"0" / (%s"1" serviceKeyBundle) ; absent in data from earlier versions serviceKeyBundle = keyId reqDhKey reqKemKey -keyId = shortString ; referenced by requests +keyId = shortString ; identifies the bundle, included in requests reqDhKey = length x509encoded ; X25519, used instead of the fixed-data queue key reqKemKey = largeString ; sntrup761 encapsulation key, 1158 bytes ``` -The service periodically rotates the bundle with `LSET` and keeps previous private keys for a window covering client link data caching and queue message retention. A request encrypted to an unknown key ID fails with an error; the client then re-fetches link data and retries. Deleting expired keys is what bounds decryptability of recorded requests, so the window must be short and fixed. +The service rotates the bundle with `LSET` and keeps the previous private keys long enough to decrypt requests already in the address queue - the queue message retention. The client retrieves link data for every request, so the published keys are current when the request is sent; a request becomes undecryptable only if it stays in the address queue longer than its key is kept, and such a request is not answered and fails at the client's deadline. Deleting old keys bounds the time recorded requests can be decrypted, so the retention is short and fixed. Nothing about retries or retention is published in link data. -Link data is encrypted with a symmetric key derived from the link, which is not weakened by quantum computers, so publishing the bundle in link data does not undermine the scheme. +Link data is encrypted with a symmetric key derived from the link, which is not weakened by quantum computers, so publishing the bundle in link data does not weaken the scheme. -Sizes: KEM public key is 1158 bytes, user data is padded to 13784 bytes - the bundle fits together with application data. +Sizes: the KEM public key is 1158 bytes and user data is padded to 13784 bytes, so the bundle fits together with application data. ### Queue-layer PQ encryption -The single-shot queue encryption (today: crypto_box with a secret from the queue DH key and a sender ephemeral key, ephemeral key in the message public header) is extended to a hybrid scheme in a new SMP client version. The public header gains a third variant: +The single-shot queue encryption (today: crypto_box with a secret from the queue DH key and a sender ephemeral key, the ephemeral key in the message public header) is extended to a hybrid scheme in a new SMP client version. The public header gains a third variant: ```abnf smpPubHeaderHybrid = smpClientVersion %s"2" optKeyId senderPublicDhKey kemCiphertext -optKeyId = %s"0" / (%s"1" keyId) ; present in requests, absent in replies -senderPublicDhKey = length x509encoded ; fresh X25519 key +optKeyId = %s"0" / (%s"1" keyId) ; present in requests to select the bundle, absent in replies +senderPublicDhKey = length x509encoded ; sender ephemeral X25519 key kemCiphertext = largeString ; sntrup761 ciphertext, 1039 bytes ``` -The secret is derived from both shared secrets, and the body is encrypted with NaCl secret_box, padded as today: +The secret combines both shared secrets, and the body is encrypted with NaCl secret_box, padded as today: ``` secret = HKDF(dh(recipient key, sender ephemeral key) || KEM shared secret) ``` -The header is visible to the destination server, as invitation headers are today (with proxied sending it is not visible to the proxy). +The header is readable by the destination server, as invitation headers are today; with proxied sending it is not readable by the proxy. -The same scheme applies to replies, using the keys from the request: the first reply includes the hybrid header, the hybrid secret is computed once per reply queue and used for subsequent replies with the empty header - the same one-secret-per-queue model as existing sender queues. Invitations and confirmations can adopt the same scheme independently of this design, removing the single-layer X25519 exposure of the profile. +For a request, the recipient keys are the service bundle keys selected by `keyId`. For a reply, the recipient keys are the reply queue keys included in the request. The first reply message includes the hybrid header; the client computes the secret once and stores it in the reply queue record, and later reply messages use it with the plain header. This is the one-secret-per-queue model of existing queues, and the secret is stored the same way the per-queue DH secret is stored on receiving the first message. Invitations and confirmations can adopt the same scheme independently of this design. ### Request A new agent envelope (alongside `agentConfirmation`, `agentMsgEnvelope`, `agentInvitation`, `agentRatchetKey`): ```abnf -agentRequest = agentVersion %s"Q" replyQueue requestBody -replyQueue = smpClientVersion smpServer senderId dhPublicKey kemPublicKey sndAuthPublicKey +agentRequest = agentVersion %s"Q" replyQueue requestPayload +replyQueue = smpClientVersion smpServer senderId dhPublicKey kemPublicKey smpClientVersion = 2*2 OCTET senderId = shortString ; sender ID of the reply queue -dhPublicKey = length x509encoded ; X25519, fresh per request -kemPublicKey = largeString ; sntrup761 encapsulation key, fresh per request -sndAuthPublicKey = length x509encoded ; key to secure the reply queue -requestBody = *OCTET ; remaining bytes, application-defined +dhPublicKey = length x509encoded ; reply queue X25519 key +kemPublicKey = largeString ; reply queue sntrup761 encapsulation key +requestPayload = *OCTET ; remaining bytes, opaque application payload ``` -The envelope type indicates to the receiving agent that no connection is created and replies are sent to the provided queue. The reply queue keys never appear outside the encrypted request. Requests must fit in one message; larger payloads are passed as an XFTP file description in the request body. +The envelope type indicates to the receiving agent that no connection is created and replies are sent to the reply queue. The request does not include a key to secure the reply queue: the service generates its own sender key and secures the queue with it, as a sender does in the fast handshake. The reply queue address and keys are not part of the hashed payload. -A retry is the byte-identical stored request message, so all correlation uses one value: the request hash - the hash of the message body, the same bytes as sent by the client and as received by the service after the server encryption layer is removed. +The request hash is the SHA3-256 of `requestPayload` - the same bytes the client sends and the service reads after the server encryption layer is removed. The application sets a request ID inside the payload; two requests are the same operation when their payloads, and therefore their hashes, are equal. A request must fit in one message; a larger payload is sent as an XFTP file description in the payload. -Request delivery failures follow existing SEND semantics: on QUOTA (the address queue is full) the client retries with backoff within the request deadline. +Request delivery follows existing SEND semantics: a hard error (AUTH, QUOTA) fails the call, and the application decides whether to send a new request. ### Replies -The service secures the reply queue with the sender key from the request and sends the first reply in one combined SMP command (see below). Replies use a new agent envelope, following the signature-first structure of link data: +The service secures the reply queue and sends the first reply message with one combined SMP command (see below). A reply message uses a new agent envelope, following the signature-first structure of link data: ```abnf -agentResponse = agentVersion %s"P" signature signedReply -signature = length 64*64 OCTET ; root key signature of signedReply -signedReply = requestHash prevMsgHash final replyBody -requestHash = shortString ; hash of the request message body -prevMsgHash = shortString ; hash of the previous agentResponse, empty in the first reply -final = %s"T" / %s"F" ; F - more replies follow -replyBody = *OCTET ; remaining bytes, application-defined +agentResponse = agentVersion %s"P" signature signedResponse +signature = length 64*64 OCTET ; root key signature of signedResponse +signedResponse = requestHash prevMsgHash more responses +requestHash = shortString ; hash of the request payload +prevMsgHash = shortString ; hash of the previous agentResponse message, empty in the first +more = %s"T" / %s"F" ; T - more reply messages follow +responses = length 1*responseItem ; non-empty list of application responses +responseItem = largeString ; opaque application response ``` -The client verifies each reply: the signature against the root key from link data, the request hash against the sent request, the hash chain against the previous reply. Forging a reply requires the root key even if the reply queue keys are compromised; a signed reply cannot be replayed for a different request; the hash chain detects replies dropped or reordered by the reply queue server. +A reply message includes a list of responses, so responses known together are sent in one message rather than several; responses that become known over time are sent in separate messages. The signature covers the whole message. -The client deletes the reply queue on the final flag, on the request deadline (application-defined per request), or on cancellation. Deleting the queue also cancels the stream: subsequent SEND commands from the service fail with AUTH. Router queue and message expiry limit the lifetime of abandoned queues. Stream length between client acknowledgements is bounded by queue capacity. +The client verifies each reply message: the signature against the root key from link data, the request hash against the sent request, the previous-message hash against the previous message. A message that fails verification is acknowledged and discarded. Forging a reply message requires the root key even if the reply queue keys are known; a signed message cannot be reused for a different request; the previous-message hash detects messages dropped or reordered by the reply queue server. -Delayed replies (e.g., a credential issued after payment) are stored in the reply queue within message retention time and received when the client subscribes again; notification credentials can be added to the reply queue with existing commands. Messages in the reply queue that fail decryption or verification are acknowledged and dropped. +The first reply message returns from the request call. Later reply messages are delivered through the callback the application registered with the request. The exchange ends on a message with `more` set to false. The client deletes the reply queue on that message, on the request deadline, or when the application cancels the request. Deleting the queue stops further replies: later SEND commands from the service fail with AUTH. Server queue and message expiry limit the lifetime of a reply queue left after a client restart. ### Combined SKEY+SEND command @@ -150,24 +150,27 @@ senderAuthPublicKey = length x509encoded The command is authorized with the key it sets, as `SKEY`, and only accepted on messaging-mode queues. The server responds `OK` or `ERR`. It is idempotent in both parts: -- key part: as `SKEY` today - repeated command with the same key succeeds, different key fails with AUTH -- send part: the server stores a hash of the message body until the message is acknowledged; within that window a repeated command with the same body is not delivered again and is reported as delivered +- key part: as `SKEY` today - a repeated command with the same key succeeds, a different key fails with AUTH. +- send part: the server keeps the hash of the message until it is acknowledged, and a repeated command with the same message within that period is reported as delivered without delivering it again. -A retry arriving after the message was acknowledged is delivered as a duplicate and suppressed by the receiving agent (by message hash), so the server-side marker covers the common case and the agent covers the rest. +A repeat arriving after the message was acknowledged is delivered as a duplicate and discarded by the receiving agent by message hash, so the server-side hash covers the common case and the agent covers the rest. -It is used by the service for the first reply, and by the joining party in the fast connection handshake (currently `SKEY` then `SEND` confirmation), removing one command and round trip from both flows. +The service uses this command for the first reply message. The joining party uses it in the fast connection handshake, where it replaces `SKEY` followed by the `SEND` confirmation, removing one command and one round trip. ### Idempotency -Handled uniformly by the agent: it stores the request hash and sent replies for a window declared in link data (with a default), and re-sends stored replies for a repeated request without invoking the service application. This gives exactly-once execution over at-least-once delivery for all services; services with idempotent semantics of their own lose nothing. +The service agent identifies a request by its hash and keeps, for a fixed retention period it chooses (in the 1 to 24 hour range, in service configuration, not in link data), a record of the ordered reply messages it has sent and the reply queues subscribed under that hash. A repeat request with the same hash does not reach the service application: -The guarantee is bounded by the window: the client must not retry after the request deadline, and the deadline must not exceed the declared window. Stored replies may be evicted under storage pressure; a retry whose replies were evicted receives an error prompting a new request. Within the window a request is never executed twice. +- while the first request is being answered, the repeat is added to the record and receives the reply messages already sent, and each later message is sent to every reply queue in the record. +- after the operation completed, the repeat receives the whole stored sequence of reply messages, in order. + +This gives single execution over at-least-once delivery. The retention period bounds it: after the record is deleted, a request with the same hash is a new operation and runs again. The application does not rely on the transport for recovery across a restart; it keeps its own state and, after a restart, sends whatever request fits what it knows, which is often a different request (for example, "is this payment still pending" rather than a repeat of "start this payment"). ### Out of scope -- Continuity: sessions across requests are an application concern (tokens in request and reply bodies). -- Service-initiated messages: there is no standing channel; use a connection where push is needed. -- Abuse protection beyond existing queue quotas: services can require application-level credentials (e.g., a badge) in request bodies; rate limiting options are a separate discussion. +- Recovery across restart: the transport keeps no exchange across a client restart. The application persists its own state and sends a new request when it needs to. +- Service-initiated messages: there is no standing channel; use a connection where the service must reach the client without a request. +- Abuse protection beyond existing queue quotas: services can require application-level credentials (e.g., a badge) in the request payload; rate limiting is a separate discussion. - Scaling request reception: a single address queue bounds service throughput; distributing reception across multiple queues or relays (the existing `relays` field in contact link data) is a separate discussion. - Name resolution: existing addressing layer. From 5633bbfb139e5c1e0733b8b3f7756c8ae4f50001 Mon Sep 17 00:00:00 2001 From: Evgeny Date: Sun, 12 Jul 2026 16:16:58 +0100 Subject: [PATCH 3/3] corrections Co-authored-by: Evgeny --- rfcs/2026-07-11-service-rpc.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rfcs/2026-07-11-service-rpc.md b/rfcs/2026-07-11-service-rpc.md index 1eb7a42df..074658e4f 100644 --- a/rfcs/2026-07-11-service-rpc.md +++ b/rfcs/2026-07-11-service-rpc.md @@ -12,13 +12,13 @@ This is the wrong primitive for most service interactions: 2. Privacy. A connection is a stable pairwise pseudonym. If a service were to use a duplex connection, it could link all requests made over it into a profile: search history in the directory, blockchain operations linked even when different on-chain keys are used, telemetry that becomes longitudinal tracking. The client also accumulates history that can be recovered from the device. Where continuity is needed, it can be provided in the application protocol (e.g., a token included in requests), without a transport-level identity. -3. Encryption. Messages sent to contact addresses outside an established connection (invitations) have a single layer of X25519 encryption. This was acceptable while such messages contained only a connection link and profile (though it is planned to be improved); it is not acceptable for service requests. +3. Encryption. Messages sent to contact addresses outside an established connection (invitations) have a single layer of X25519 encryption. This was acceptable while such messages contained only a connection link and profile (though it is planned to be improved); it is probably less acceptable for service requests. In-app service addresses should be stored as names resolving to links via the existing addressing layer (server host in the link authority, current link data retrieved with LGET), so that service links can be changed without redeploying the apps. Name resolution is already supported, and out of scope. ## Security objectives -1. Requests from the same client must not be linkable to each other by the service or by servers, and no state that outlives the exchange is created on either side. +1. Requests from the same client must not be linkable to each other by the service or by servers, and no long term state is created on either side in the transport layer. 2. Post-quantum resistant e2e encryption of requests and replies; decryptability of recorded requests must be bounded by key rotation, without changing the address. 3. Reply authenticity must be verifiable against the link; substitution, replay, dropping or reordering of replies by servers must be detectable. 4. A repeated request for the same operation must not be executed twice. @@ -171,7 +171,7 @@ This gives single execution over at-least-once delivery. The retention period bo - Recovery across restart: the transport keeps no exchange across a client restart. The application persists its own state and sends a new request when it needs to. - Service-initiated messages: there is no standing channel; use a connection where the service must reach the client without a request. - Abuse protection beyond existing queue quotas: services can require application-level credentials (e.g., a badge) in the request payload; rate limiting is a separate discussion. -- Scaling request reception: a single address queue bounds service throughput; distributing reception across multiple queues or relays (the existing `relays` field in contact link data) is a separate discussion. +- Scaling request reception: a single address queue bounds service throughput; distributing reception across multiple queues or relays (the existing `relays` field in contact link data) is a separate question, but it would fit well with name resolving to multiple addresses, both for redundancy, reliability and higher throughput. - Name resolution: existing addressing layer. [1]: https://tools.ietf.org/html/rfc5234