add v3 X25519 + ML-KEM-768 scheme

CHANGELOG.md

@@ -1,3 +1,9 @@
+# Unreleased
+
+* Add hybrid post-quantum `v3` encryption using X25519 + ML-KEM-768, HKDF-SHA256, and XChaCha20-Poly1305.
+* Add `ejson keygen -scheme v3` / `-pqc` and schema-aware keydir lookup using v3 key IDs.
+* Preserve legacy `v1` key generation and decryption compatibility.
+
 # 1.5.4
 
 * Bumps golang.org/x/crypto from 0.17.0 to 0.31.0

README.md

@@ -1,16 +1,13 @@
 # ejson
 
 `ejson` is a utility for managing a collection of secrets in source control. The
-secrets are encrypted using [public
-key](http://en.wikipedia.org/wiki/Public-key_cryptography), [elliptic
-curve](http://en.wikipedia.org/wiki/Elliptic_curve_cryptography) cryptography
-([NaCl](http://nacl.cr.yp.to/) [Box](http://nacl.cr.yp.to/box.html):
-[Curve25519](http://en.wikipedia.org/wiki/Curve25519) +
-[Salsa20](http://en.wikipedia.org/wiki/Salsa20) +
-[Poly1305-AES](http://en.wikipedia.org/wiki/Poly1305-AES)). Secrets are
-collected in a JSON file, in which all the string values are encrypted. Public
-keys are embedded in the file, and the decrypter looks up the corresponding
-private key from its local filesystem.
+secrets are encrypted using public-key cryptography. Legacy `v1` keys use
+[NaCl](http://nacl.cr.yp.to/) [Box](http://nacl.cr.yp.to/box.html)
+(Curve25519 + XSalsa20 + Poly1305). New hybrid `v3` keys use X25519 and
+ML-KEM-768 to derive a shared secret with HKDF-SHA256, then encrypt values with
+XChaCha20-Poly1305. Secrets are collected in a JSON file, in which all the
+string values are encrypted. Public keys are embedded in the file, and the
+decrypter looks up the corresponding private key from its local filesystem.
 
 ![demo](http://burkelibbey.s3.amazonaws.com/ejson-demo.gif)
 
@@ -80,6 +77,21 @@ $ cat /opt/ejson/keys/5339*
 888a4291bef9135729357b8c70e5a62b0bbe104a679d829cdbe56d46a4481aaf
 ```
 
+By default, `keygen` still generates legacy `v1` keys. To generate a hybrid
+post-quantum keypair, use `-scheme v3` or `-pqc`:
+
+```
+$ ejson keygen -scheme v3 -w
+v3:<base64 X25519 public key + ML-KEM-768 encapsulation key>
+$ ls /opt/ejson/keys
+<32-character key id>
+```
+
+Hybrid public keys are intentionally much longer than legacy keys because the
+full ML-KEM public key is embedded in the EJSON document. This preserves the
+existing workflow where anyone who can edit the document can add new secrets
+without access to the private keydir.
+
 ### 3: Create an `ejson` file
 
 The format is described in more detail [later on](#format). For now, create a
@@ -115,10 +127,13 @@ new secret will be encrypted.
 
 ### 5: Decrypt the file
 
-To decrypt the file, you must have a file present in the `keydir` whose name is
-the 64-byte hex-encoded public key exactly as embedded in the `ejson` document.
-The contents of that file must be the similarly-encoded private key. If you used
-`ejson keygen -w`, you've already got this covered.
+To decrypt the file, you must have the matching private key present in the
+`keydir`. For legacy `v1` documents, the keydir filename is the 64-character
+hex-encoded public key exactly as embedded in the `ejson` document, and the file
+contains the similarly-encoded private key. For hybrid `v3` documents, the
+keydir filename is a 32-character key ID derived from the public key, and the
+file contains an `ejson-key v3` private key file with both X25519 and ML-KEM key
+material. If you used `ejson keygen -w`, you've already got this covered.
 
 Unlike `ejson encrypt`, which overwrites the specified files, `ejson decrypt`
 only takes one file parameter, and prints the output to `stdout`:
@@ -138,9 +153,11 @@ The `ejson` document format is simple, but there are a few points to be aware
 of:
 
 1. It's just JSON.
-2. There *must* be a key at the top level named `_public_key`, whose value is a
-   32-byte hex-encoded (i.e. 64 ASCII byte) public key as generated by `ejson
-   keygen`.
+2. There *must* be a key at the top level named `_public_key`. For legacy `v1`
+   documents, its value is a 32-byte hex-encoded (i.e. 64 ASCII byte) public key
+   as generated by `ejson keygen`. For hybrid `v3` documents, its value begins
+   with `v3:` and contains a base64-encoded X25519 public key plus ML-KEM-768
+   encapsulation key.
 3. Any string literal that isn't an object key will be encrypted by default (ie.
    in `{"a": "b"}`, `"b"` will be encrypted, but `"a"` will not.
 4. Numbers, booleans, and nulls aren't encrypted.
@@ -149,6 +166,11 @@ of:
    encrypted, and is useful for implementing metadata schemes.
 6. Underscores do not propagate downward. For example, in `{"_a": {"b": "c"}}`,
    `"c"` will be encrypted.
+7. Encrypted values are schema-tagged. Legacy values use
+   `EJ[1:<ephemeral public key>:<nonce>:<ciphertext>]`. Hybrid post-quantum
+   values use
+   `EJ[3:<ephemeral X25519 public key>:<ML-KEM ciphertext>:<nonce>:<ciphertext>]`.
+   Existing `EJ[1:...]` values remain decryptable.
 
 ## See also
 

cmd/ejson/actions.go

@@ -3,6 +3,7 @@ package main
 import (
 	"fmt"
 	"os"
+	"strings"
 
 	"github.com/Shopify/ejson"
 )
@@ -43,15 +44,19 @@ func decryptAction(args []string, keydir, userSuppliedPrivateKey, outFile string
 	return err
 }
 
-func keygenAction(_ []string, keydir string, wFlag bool) error {
-	pub, priv, err := ejson.GenerateKeypair()
+func keygenAction(_ []string, keydir string, wFlag bool, scheme string) error {
+	pub, priv, keyID, err := ejson.GenerateKeypairForScheme(scheme)
 	if err != nil {
 		return err
 	}
 
 	if wFlag {
-		keyFile := fmt.Sprintf("%s/%s", keydir, pub)
-		err := writeFile(keyFile, append([]byte(priv), '\n'), 0o440)
+		keyFile := fmt.Sprintf("%s/%s", keydir, keyID)
+		contents := []byte(priv)
+		if !strings.HasSuffix(priv, "\n") {
+			contents = append(contents, '\n')
+		}
+		err := writeFile(keyFile, contents, 0o440)
 		if err != nil {
 			return err
 		}

cmd/ejson/main.go

@@ -78,9 +78,22 @@ func main() {
 					Name:  "write, w",
 					Usage: "rather than printing both keys, print the public and write the private into the keydir",
 				},
+				cli.StringFlag{
+					Name:  "scheme",
+					Value: "v1",
+					Usage: "key scheme to generate: v1 or v3",
+				},
+				cli.BoolFlag{
+					Name:  "pqc",
+					Usage: "generate a v3 hybrid X25519+ML-KEM-768 keypair",
+				},
 			},
 			Action: func(c *cli.Context) {
-				if err := keygenAction(c.Args(), c.GlobalString("keydir"), c.Bool("write")); err != nil {
+				scheme := c.String("scheme")
+				if c.Bool("pqc") {
+					scheme = "v3"
+				}
+				if err := keygenAction(c.Args(), c.GlobalString("keydir"), c.Bool("write"), scheme); err != nil {
 					fmt.Fprintln(os.Stderr, "Key generation failed:", err)
 					os.Exit(1)
 				}

crypto/boxed_message.go

@@ -3,13 +3,16 @@ package crypto
 import (
 	"encoding/base64"
 	"fmt"
-	"regexp"
 	"strconv"
+	"strings"
 )
 
-var messageParser = regexp.MustCompile("\\AEJ\\[(\\d):([A-Za-z0-9+=/]{44}):([A-Za-z0-9+=/]{32}):(.+)\\]\\z")
+const (
+	boxedMessagePrefix = "EJ["
+	boxedMessageSuffix = "]"
+)
 
-// boxedMessage dumps and loads the wire format for encrypted messages. The
+// boxedMessage dumps and loads the v1 wire format for encrypted messages. The
 // schema is fairly simple:
 //
 //	"EJ["
@@ -28,14 +31,26 @@ type boxedMessage struct {
 	Box             []byte
 }
 
-// IsBoxedMessage tests whether a value is formatted using the boxedMessage
-// format. This can be used to determine whether a string value requires
+// IsBoxedMessage tests whether a value is formatted using a supported boxed
+// message format. This can be used to determine whether a string value requires
 // encryption or is already encrypted.
 func IsBoxedMessage(data []byte) bool {
-	return messageParser.Find(data) != nil
+	version, fields, err := parseBoxedEnvelope(data)
+	if err != nil {
+		return false
+	}
+
+	switch version {
+	case SchemaVersionLegacy:
+		return len(fields) == 3 && len(fields[0]) == base64.StdEncoding.EncodedLen(32) && len(fields[1]) == base64.StdEncoding.EncodedLen(24) && fields[2] != ""
+	case SchemaVersionHybrid:
+		return len(fields) == 4 && len(fields[0]) == base64.StdEncoding.EncodedLen(32) && len(fields[1]) == base64.StdEncoding.EncodedLen(hybridMLKEMCiphertextSize) && len(fields[2]) == base64.StdEncoding.EncodedLen(24) && fields[3] != ""
+	default:
+		return false
+	}
 }
 
-// Dump dumps to the wire format
+// Dump dumps to the v1 wire format.
 func (b *boxedMessage) Dump() []byte {
 	pub := base64.StdEncoding.EncodeToString(b.EncrypterPublic[:])
 	nonce := base64.StdEncoding.EncodeToString(b.Nonce[:])
@@ -46,59 +61,60 @@ func (b *boxedMessage) Dump() []byte {
 	return []byte(str)
 }
 
-// Load restores from the wire format.
+// Load restores from the v1 wire format.
 func (b *boxedMessage) Load(from []byte) error {
-	var ssver, spub, snonce, sbox string
-	var err error
-
-	allMatches := messageParser.FindAllStringSubmatch(string(from), -1) // -> [][][]byte
-	if len(allMatches) != 1 {
-		return fmt.Errorf("invalid message format")
-	}
-	matches := allMatches[0]
-	if len(matches) != 5 {
-		return fmt.Errorf("invalid message format")
-	}
-
-	ssver = matches[1]
-	spub = matches[2]
-	snonce = matches[3]
-	sbox = matches[4]
-
-	b.SchemaVersion, err = strconv.Atoi(ssver)
+	version, fields, err := parseBoxedEnvelope(from)
 	if err != nil {
 		return err
 	}
+	if version != SchemaVersionLegacy || len(fields) != 3 {
+		return fmt.Errorf("invalid message format")
+	}
+	b.SchemaVersion = version
 
-	pub, err := base64.StdEncoding.DecodeString(spub)
+	pub, err := base64.StdEncoding.DecodeString(fields[0])
 	if err != nil {
 		return err
 	}
-	pubBytes := []byte(pub)
-	if len(pubBytes) != 32 {
+	if len(pub) != 32 {
 		return fmt.Errorf("public key invalid")
 	}
-	var public [32]byte
-	copy(public[:], pubBytes[0:32])
-	b.EncrypterPublic = public
+	copy(b.EncrypterPublic[:], pub)
 
-	nnc, err := base64.StdEncoding.DecodeString(snonce)
+	nnc, err := base64.StdEncoding.DecodeString(fields[1])
 	if err != nil {
 		return err
 	}
-	nonceBytes := []byte(nnc)
-	if len(nonceBytes) != 24 {
+	if len(nnc) != 24 {
 		return fmt.Errorf("nonce invalid")
 	}
-	var nonce [24]byte
-	copy(nonce[:], nonceBytes[0:24])
-	b.Nonce = nonce
+	copy(b.Nonce[:], nnc)
 
-	box, err := base64.StdEncoding.DecodeString(sbox)
+	box, err := base64.StdEncoding.DecodeString(fields[2])
 	if err != nil {
 		return err
 	}
 	b.Box = []byte(box)
 
 	return nil
 }
+
+func parseBoxedEnvelope(data []byte) (version int, fields []string, err error) {
+	message := string(data)
+	if !strings.HasPrefix(message, boxedMessagePrefix) || !strings.HasSuffix(message, boxedMessageSuffix) {
+		return 0, nil, fmt.Errorf("invalid message format")
+	}
+
+	body := strings.TrimSuffix(strings.TrimPrefix(message, boxedMessagePrefix), boxedMessageSuffix)
+	versionString, rest, ok := strings.Cut(body, ":")
+	if !ok || versionString == "" || rest == "" {
+		return 0, nil, fmt.Errorf("invalid message format")
+	}
+
+	version, err = strconv.Atoi(versionString)
+	if err != nil {
+		return 0, nil, err
+	}
+
+	return version, strings.Split(rest, ":"), nil
+}

crypto/crypto.go

@@ -1,13 +1,11 @@
-// Package crypto implements a simple convenience wrapper around
-// golang.org/x/crypto/nacl/box. It ultimately models a situation where you
-// don't care about authenticating the encryptor, so the nonce and encryption
-// public key are prepended to the encrypted message.
+// Package crypto implements ejson's encrypted value formats. Legacy v1 values
+// use golang.org/x/crypto/nacl/box. Hybrid v3 values use X25519 + ML-KEM-768 to
+// derive a shared key and XChaCha20-Poly1305 to encrypt the value.
 //
-// Shared key precomputation is used when encrypting but not when decrypting.
-// This is not an inherent limitation, but it would complicate the
-// implementation a little bit to do precomputation during decryption also.
-// If performance becomes an issue (highly unlikely), it's completely feasible
-// to add.
+// The legacy v1 format models a situation where you don't care about
+// authenticating the encryptor, so the nonce and encryption public key are
+// prepended to the encrypted message. Shared key precomputation is used when
+// encrypting v1 values but not when decrypting.
 package crypto
 
 import (