diff --git a/CLAUDE.md b/CLAUDE.md index 5e48ad6fb..bcb385b2b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -157,3 +157,15 @@ When making changes to the codebase: ## PR Reviews CodeRabbit reviews PRs automatically, but it has an hourly quota and runs out of org credits. If a PR shows a "Review limit reached" / "out of usage credits" message instead of an actual review, run the `/review` skill locally against the PR to get review feedback without waiting for the quota to refill. + +## PR Title and Description Maintenance + +When pushing additional commits to an existing PR, check whether the title and description still describe the change accurately. They often go stale during review iterations: a flag gets renamed, an API gets reshaped, an extra fix lands, etc. The PR description is what shows up in the squash-merge commit, so a stale title/body means a misleading entry in `git log` forever. + +Update them with `gh pr edit --title "..." --body "..."` whenever the scope shifts. Specifically watch for: + +- Flags, file names, or public APIs renamed in later commits but still referenced by their old name in the PR body. +- Bullet points in the "Summary" section that describe behavior the latest commits have changed or removed. +- The test-plan checklist getting out of date as new tests are added. + +When you edit a PR description you authored, keep the `(Written by Claude)` marker so reviewers still know the body wasn't human-authored. diff --git a/doc/bin/relay/cluster.md b/doc/bin/relay/cluster.md index 187f7ef38..64605b76e 100644 --- a/doc/bin/relay/cluster.md +++ b/doc/bin/relay/cluster.md @@ -5,96 +5,68 @@ description: Run multiple moq-relay instances across multiple hosts/regions # Clustering -Multiple relay instances can cluster for geographic distribution and improved latency. +Relays can be joined together to proxy announcements and subscriptions between each other. A viewer talks to whichever relay is closest; if their broadcast lives somewhere else in the cluster, the local relay fetches it from a neighbor and caches it. -## Overview +A broadcast carries a small hop list as it travels. Each relay it passes through adds itself to the list, which is how loops are caught and how the network picks the shortest path when there's more than one. -`moq-relay` uses a simple clustering scheme: +## Topology -1. **Root node** - A single relay (can serve public traffic) that tracks cluster membership -2. **Other nodes** - Accept internet traffic and consult the root for routing +Each relay lists the peers it wants to dial in `cluster.connect`. That's it; the topology is whatever you draw with those links. -When a relay publishes a broadcast, it advertises its `node` address to other relays via the root. +A simple chain works well when one region is the source and others are caches: -## Configuration +```text +eu-west <--- us-east <--- us-west +``` ```toml +# us-east.toml [cluster] -root = "https://root-relay.example.com" # Root node -node = "https://us-east.relay.example.com" # This node's address -``` - -### Cluster Arguments +connect = ["eu-west.example.com:4443"] -- `--cluster-root ` - Hostname/IP of the root node (omit to make this node the root) -- `--cluster-node ` - Hostname/IP of this instance (needs valid TLS cert) - -## How It Works +# us-west.toml +[cluster] +connect = ["us-east.example.com:4443"] +``` -1. Each relay connects to the root node on startup -2. When a publisher connects to any relay, that relay announces the broadcast -3. The root node tracks which relay has which broadcasts -4. When a subscriber connects, the relay queries the root to find the broadcast -5. Relays connect to each other to forward traffic +A publisher on `eu-west` reaches a viewer on `us-west` through `us-east`. If a second `us-west` viewer subscribes to the same broadcast, `us-east` already has it cached, so only one fetch crosses the Atlantic. A full mesh (every relay dialing every other) would skip the cache entirely and waste an outbound link per pair. -## Benefits +Pick the shape that matches your traffic. Linear chains are great for fanout; small N-way meshes are fine when latency matters more than dedup; mixed shapes work too. -- **Lower latency** - Users connect to nearest relay -- **Higher availability** - Redundancy across regions -- **Geographic distribution** - Serve global audiences +## Auto-discovery -## Example Topology +Listing every peer by hand can get tedious in larger clusters. Set `cluster.mesh` to this relay's own URL and connected peers will discover and dial it back automatically: -```text - ┌─────────────┐ - │ Root Node │ - │ (US-C) │ - └──────┬──────┘ - ┌───────────────┼───────────────┐ - │ │ │ - ┌──────┴──────┐ ┌──────┴──────┐ ┌──────┴──────┐ - │ US-East │ │ EU-West │ │ Asia-SE │ - │ Relay │ │ Relay │ │ Relay │ - └─────────────┘ └─────────────┘ └─────────────┘ +```toml +[cluster] +connect = ["us-east.example.com:4443"] +mesh = "us-west.example.com:4443" ``` -## Peer Authentication - -Cluster peers must authenticate to each other. Two options: - -### JWT token - -Each leaf reads a JWT from `cluster.token` (see [Authentication](/bin/relay/auth)) -and presents it to the root on connect. The token must grant cluster privileges -and full publish/subscribe access. - -### mTLS (recommended for new deployments) +Each node with `mesh` set creates a broadcast carrying its address, which other nodes pick up. `connect` is optional once gossip is running, but you still need at least one connection somewhere (either you dial a peer or a peer dials you) for the advertisement to flow. -Configure the root with `tls.root` pointing at the CA that signed the leaf -certificates. Leaves connect with a client certificate signed by that CA — -no JWT needed. The leaf's cluster node name is taken from the first DNS SAN on -its certificate, so identity is bound to the cert rather than self-declared. +A relay with `mesh` set and no `connect` is a passive rendezvous: it sits and waits for inbound connections, then helps everyone else find each other. -See [Authentication → mTLS Peer Authentication](/bin/relay/auth#mtls-peer-authentication) -for details. +## Authentication -## Current Limitations +Cluster peers must authenticate to each other: -- **Mesh topology** - All relays connect to all others -- **Not optimized for large clusters** - 3-5 nodes recommended -- **Single root node** - Future: multi-root for redundancy +- **mTLS** (recommended). Set `tls.root` to the CA that signed the cluster certificates. Inbound connections presenting a valid client cert are granted full access; outbound dials use `client.tls.cert` / `client.tls.key`. +- **JWT**. Each relay reads a token from `cluster.token` and presents it on outbound dials. The token needs broad enough scope to cover whatever paths the cluster carries. -## Production Example +See [Authentication](/bin/relay/auth) for the full setup. -The public CDN at `cdn.moq.dev` uses this clustering approach: +## Migration from older configs -- `usc.cdn.moq.dev` - US Central (root) -- `euc.cdn.moq.dev` - EU Central -- `sea.cdn.moq.dev` - Southeast Asia +`cluster.root` and `cluster.node` were both removed. If a config still sets either flag, the relay errors at startup with a message pointing at the replacements: -Clients use GeoDNS to connect to the nearest relay automatically. +| Old | New | +|---|---| +| `root = "rendezvous:4443"` + `node = "us-east:4443"` | `connect = ["rendezvous:4443"]` + `mesh = "us-east:4443"` | +| `root = "rendezvous:4443"` only | `mesh = "rendezvous:4443"` (passive rendezvous) | +| `node = "us-east:4443"` | `mesh = "us-east:4443"` | -## Next Steps +## Next steps - Deploy to [Production](/bin/relay/prod) - Set up [Authentication](/bin/relay/auth) diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index 3047b31cf..543fbb0bb 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -120,18 +120,18 @@ Clustering configuration for multi-relay deployments. ```toml [cluster] -# Address of the root relay to connect to -# Omit this to make this relay the root -connect = "root.relay.example.com:4443" +# Peers this relay dials. The topology is whatever you draw with these links. +connect = ["us-east.example.com:4443"] -# JWT token file for cluster authentication -token = "cluster.jwt" +# Optional. Set to this relay's own URL to advertise it so other peers find +# you automatically. +mesh = "us-west.example.com:4443" -# This relay's address, as reachable by other cluster nodes -node = "leaf1.relay.example.com:4443" +# JWT used for outbound cluster dials (alternative to mTLS). +token = "cluster.jwt" ``` -See [Clustering](/bin/relay/cluster) for deployment patterns. +See [Clustering](/bin/relay/cluster) for topology choices and the trade-off between hand-listed peers and gossip. ### \[client] diff --git a/rs/moq-relay/README.md b/rs/moq-relay/README.md index c84148805..14dc29f8f 100644 --- a/rs/moq-relay/README.md +++ b/rs/moq-relay/README.md @@ -58,22 +58,14 @@ HTTPS is currently not supported. ## Clustering -In order to scale MoQ, you will eventually need to run multiple moq-relay instances potentially in different regions. -This is called *clustering*, where the goal is that a user connects to the closest relay and they magically form a mesh behind the scenes. +Relays can be joined together to proxy announcements and subscriptions. A viewer talks to whichever relay is closest; if their broadcast lives elsewhere in the cluster, the local relay fetches it from a neighbor and caches it. Hop tracking on every broadcast keeps loops out and picks the shortest path when there's more than one. -**moq-relay** uses a simple clustering scheme using moq-lite. -This is both dog-fooding and a surprisingly ueeful way to distribute live metadata at scale. +- `--cluster-connect ` lists the peers this relay dials. Repeatable; defines the topology by hand. A simple chain like `eu-west <- us-east <- us-west` lets `us-east` cache and dedup the transatlantic fetches that fan out to many `us-west` viewers. +- `--cluster-mesh ` is optional. When set, this relay advertises its own URL to connected peers and dials any peers it learns about, so larger clusters don't need each node hand-configured. You still need at least one connection (in or out) so the advertisement has a path to flow. A relay with `--cluster-mesh` set and no `--cluster-connect` is a passive rendezvous. -We currently use a single "root" node that is used to discover members of the cluster and what broadcasts they offer. -This is a normal moq-relay instance, potentially serving public traffic, unaware of the fact that it's in charge of other relays. +`--cluster-root` and `--cluster-node` from earlier versions were removed. The relay errors at startup if either is set and points at the replacements. -The other moq-relay instances accept internet traffic and consult the root for routing. -They can then advertise their internal ip/hostname to other instances when publishing a broadcast. - -Cluster arguments: - -- `--cluster-root `: The hostname/ip of the root node. If missing, this node is a root. -- `--cluster-node `: The hostname/ip of this instance. There needs to be a corresponding valid TLS certificate, potentially self-signed. If missing, published broadcasts will only be available on this specific relay. +See [doc/bin/relay/cluster.md](https://github.com/moq-dev/moq/blob/main/doc/bin/relay/cluster.md) for the full walkthrough, including topology trade-offs and authentication. ## Authentication diff --git a/rs/moq-relay/src/cluster.rs b/rs/moq-relay/src/cluster.rs index d6ef4df66..1bdda3f02 100644 --- a/rs/moq-relay/src/cluster.rs +++ b/rs/moq-relay/src/cluster.rs @@ -1,16 +1,26 @@ -use std::path::PathBuf; +use std::{ + collections::HashSet, + path::PathBuf, + sync::{Arc, Mutex}, +}; use anyhow::Context; -use moq_net::{Origin, OriginConsumer, OriginProducer, Stats, Tier}; +use moq_net::{BroadcastProducer, Origin, OriginConsumer, OriginProducer, Path, Stats, Tier}; use url::Url; use crate::AuthToken; +/// Path prefix under which cluster nodes advertise their own URLs for gossip-style +/// peer discovery. +const MESH_PREFIX: &str = ".internal/origins"; + /// Configuration for relay clustering. /// -/// Each node runs a full mesh: every configured `--cluster-connect` peer is -/// dialed and kept open for the session's lifetime. Hop-based routing on -/// broadcasts prevents announcement loops. +/// [`Self::connect`] lists peers to dial. [`Self::mesh`] is optional: when set, this +/// relay advertises its own URL so other peers discover and dial it. Set both to +/// join an existing cluster; set mesh alone to act as a passive rendezvous. +/// +/// Hop-based routing on broadcasts prevents announcement loops regardless of topology. #[serde_with::serde_as] #[derive(clap::Args, Clone, Debug, serde::Serialize, serde::Deserialize, Default)] #[serde_with::skip_serializing_none] @@ -30,9 +40,24 @@ pub struct ClusterConfig { #[serde_as(as = "serde_with::OneOrMany<_>")] pub connect: Vec, + /// This relay's own externally-reachable URL. When set, the relay publishes its + /// address on the cluster origin so other peers can discover and dial it. Pair + /// with [`Self::connect`] to reach an initial peer who will gossip your address + /// onward, or set alone for passive rendezvous. + #[arg(id = "cluster-mesh", long = "cluster-mesh", env = "MOQ_CLUSTER_MESH")] + pub mesh: Option, + /// Use the token in this file when connecting to other nodes. #[arg(id = "cluster-token", long = "cluster-token", env = "MOQ_CLUSTER_TOKEN")] pub token: Option, + + /// Removed; present only to emit a migration error. Use [`Self::mesh`] instead. + #[arg(id = "cluster-node", long = "cluster-node", env = "MOQ_CLUSTER_NODE", hide = true)] + pub node: Option, + + /// Removed; present only to emit a migration error. Use [`Self::connect`] instead. + #[arg(id = "cluster-root", long = "cluster-root", env = "MOQ_CLUSTER_ROOT", hide = true)] + pub root: Option, } /// A relay cluster built around a single [`OriginProducer`]. @@ -107,23 +132,49 @@ impl Cluster { self.origin.with_root(&token.root)?.scope(&token.publish) } - /// Runs the cluster event loop, dialing the configured peers and keeping - /// each connection alive indefinitely with exponential backoff on failure. + /// Runs the cluster event loop. + /// + /// Modes are derived from config: standalone (no work) returns immediately; + /// passive rendezvous (`mesh` only) parks after publishing self-registration + /// and does not require a QUIC client; active (`connect` non-empty) dials + /// peers and, if `mesh` is also set, runs gossip discovery. /// - /// Completes once all dials have given up; a node with no peers (`connect` - /// empty) has no outbound work and returns immediately. Errors when peers - /// are configured but no client has been attached via + /// Bails when removed flags `cluster.root` / `cluster.node` are set, or when + /// `connect` is non-empty but no client was attached via /// [`with_client`](Self::with_client). pub async fn run(self) -> anyhow::Result<()> { - if self.config.connect.is_empty() { + if let Some(root) = &self.config.root { + anyhow::bail!( + "`cluster.root` / `--cluster-root` was removed (value: {root:?}). \ + Use `--cluster-connect ` to dial cluster peers, and \ + optionally `--cluster-mesh ` to gossip this relay's address \ + so other peers can discover and dial it. \ + See https://doc.moq.dev/bin/relay/cluster." + ); + } + if let Some(node) = &self.config.node { + anyhow::bail!( + "`cluster.node` / `--cluster-node` was renamed (value: {node:?}). \ + Use `--cluster-connect ` to dial cluster peers, and \ + optionally `--cluster-mesh ` to gossip this relay's address \ + so other peers can discover and dial it. \ + See https://doc.moq.dev/bin/relay/cluster." + ); + } + + let has_outbound = !self.config.connect.is_empty(); + let has_work = has_outbound || self.config.mesh.is_some(); + if !has_work { tracing::info!("no cluster peers configured; running standalone"); return Ok(()); } - anyhow::ensure!( - self.client.is_some(), - "cluster peers configured but no QUIC client attached (call Cluster::with_client)" - ); + if has_outbound { + anyhow::ensure!( + self.client.is_some(), + "cluster peers configured but no QUIC client attached (call Cluster::with_client)" + ); + } let token = match &self.config.token { Some(path) => std::fs::read_to_string(path) @@ -133,20 +184,105 @@ impl Cluster { None => String::new(), }; + // URLs we've already spawned a dial task for (static + gossip-discovered). + // Dedup only; we never abort entries based on gossip churn, since the + // "prefer shorter hop" logic in OriginProducer delivers reannounces as + // unannounce-then-announce pairs that would otherwise drive a tight loop. + let dialed: Arc>> = Arc::new(Mutex::new(HashSet::new())); let mut tasks = tokio::task::JoinSet::new(); + for peer in &self.config.connect { + Self::spawn_dial(&mut tasks, &dialed, self.clone(), peer.clone(), token.clone()); + } + + // Held in scope so the registration stays announced until `run` exits. + // Discovery is paired with it: a mesh-only relay (passive rendezvous) has + // nothing to discover, so we only run it when we also have an outbound peer. + let _self_registration: Option = if let Some(mesh) = self.config.mesh.as_deref() { + let path = Path::new(MESH_PREFIX).join(mesh); + let broadcast = self + .origin + .create_broadcast(&path) + .expect(".internal/origins is within the relay origin's root"); + tracing::info!(url = %mesh, %path, "advertising cluster mesh URL"); + + if has_outbound { + let this = self.clone(); + let token = token.clone(); + let dialed = dialed.clone(); + let self_url = mesh.to_owned(); + tasks.spawn(async move { + this.run_discovery(self_url, token, dialed).await; + }); + } + + Some(broadcast) + } else { + None + }; + + if tasks.is_empty() { + // Passive rendezvous: park to keep `_self_registration` alive. The + // process still exits via the other arms of `tokio::select!` in main. + std::future::pending::<()>().await + } + + while tasks.join_next().await.is_some() {} + Ok(()) + } + + /// Spawn a dial loop for `peer`. No-op if `peer` is already tracked. + fn spawn_dial( + tasks: &mut tokio::task::JoinSet<()>, + dialed: &Arc>>, + this: Self, + peer: String, + token: String, + ) { + if !dialed.lock().expect("dial set poisoned").insert(peer.clone()) { + return; + } + tasks.spawn(async move { + if let Err(err) = this.run_remote(&peer, token).await { + tracing::warn!(%err, %peer, "cluster peer connection ended"); + } + }); + } + + /// Watch `.internal/origins/*` for peer registrations and dial each newly-seen + /// URL. We deliberately don't abort dials on unannounce: the "prefer shorter + /// hop" path in OriginProducer delivers reannouncements as unannounce-then- + /// announce pairs whenever a closer copy of the same broadcast arrives, + /// which would otherwise drive a tight respawn loop. The dial loop reconnects + /// on its own; if a peer truly leaves, the loop just keeps backing off. + async fn run_discovery(self, self_url: String, token: String, dialed: Arc>>) { + let Some(mut consumer) = self.origin.consume().with_root(MESH_PREFIX) else { + tracing::warn!("could not scope cluster origin to {MESH_PREFIX}; discovery disabled"); + return; + }; + + while let Some((relative, announced)) = consumer.announced().await { + if announced.is_none() { + continue; + } + let peer = relative.as_str(); + if peer == self_url { + continue; + } + let peer = peer.to_owned(); + if !dialed.lock().expect("dial set poisoned").insert(peer.clone()) { + tracing::debug!(%peer, "discovered peer already tracked; skipping dial"); + continue; + } + tracing::info!(%peer, "discovered cluster peer; dialing"); let this = self.clone(); let token = token.clone(); - let peer = peer.clone(); - tasks.spawn(async move { + tokio::spawn(async move { if let Err(err) = this.run_remote(&peer, token).await { tracing::warn!(%err, %peer, "cluster peer connection ended"); } }); } - - while tasks.join_next().await.is_some() {} - Ok(()) } #[tracing::instrument("remote", skip_all, err, fields(%remote))] @@ -209,3 +345,130 @@ impl Cluster { session.closed().await.map_err(Into::into) } } + +#[cfg(test)] +mod tests { + use super::*; + use crate::Config; + + /// Setting `cluster.root` (the removed flag) at startup must surface a migration + /// message that names both the replacement flags. + #[tokio::test] + async fn cluster_root_errors_with_migration_message() { + let config = ClusterConfig { + root: Some("legacy-root.example.com:4443".to_string()), + ..Default::default() + }; + let err = Cluster::new(config).run().await.expect_err("should error"); + let msg = format!("{err}"); + assert!(msg.contains("cluster.root"), "missing cluster.root in: {msg}"); + assert!(msg.contains("--cluster-connect"), "missing --cluster-connect in: {msg}"); + assert!(msg.contains("--cluster-mesh"), "missing --cluster-mesh in: {msg}"); + } + + /// Setting `cluster.node` (the renamed flag) at startup must surface a migration + /// message that names both replacement flags. + #[tokio::test] + async fn cluster_node_errors_with_migration_message() { + let config = ClusterConfig { + node: Some("legacy-node.example.com:4443".to_string()), + ..Default::default() + }; + let err = Cluster::new(config).run().await.expect_err("should error"); + let msg = format!("{err}"); + assert!(msg.contains("cluster.node"), "missing cluster.node in: {msg}"); + assert!(msg.contains("--cluster-connect"), "missing --cluster-connect in: {msg}"); + assert!(msg.contains("--cluster-mesh"), "missing --cluster-mesh in: {msg}"); + } + + /// `cluster.root` parsed from TOML triggers the same migration error. + #[test] + fn cluster_root_toml_parses_then_errors() { + let toml = "[cluster]\nroot = \"legacy-root.example.com:4443\"\n"; + let dir = std::env::temp_dir().join("moq-relay-cluster-test"); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("cluster-root-toml.toml"); + std::fs::write(&path, toml).unwrap(); + + let args = vec![std::ffi::OsString::from("moq-relay"), std::ffi::OsString::from(&path)]; + let config = Config::parse_and_merge(args).expect("config load"); + assert_eq!(config.cluster.root.as_deref(), Some("legacy-root.example.com:4443")); + + let rt = tokio::runtime::Runtime::new().unwrap(); + let err = rt + .block_on(Cluster::new(config.cluster).run()) + .expect_err("should error"); + assert!(format!("{err}").contains("cluster.root")); + } + + /// `cluster.node` parsed from TOML triggers the same migration error. + #[test] + fn cluster_node_toml_parses_then_errors() { + let toml = "[cluster]\nnode = \"legacy-node.example.com:4443\"\n"; + let dir = std::env::temp_dir().join("moq-relay-cluster-test"); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("cluster-node-toml.toml"); + std::fs::write(&path, toml).unwrap(); + + let args = vec![std::ffi::OsString::from("moq-relay"), std::ffi::OsString::from(&path)]; + let config = Config::parse_and_merge(args).expect("config load"); + assert_eq!(config.cluster.node.as_deref(), Some("legacy-node.example.com:4443")); + + let rt = tokio::runtime::Runtime::new().unwrap(); + let err = rt + .block_on(Cluster::new(config.cluster).run()) + .expect_err("should error"); + assert!(format!("{err}").contains("cluster.node")); + } + + /// A relay configured with only `cluster.mesh` (passive rendezvous) must run + /// without a QUIC client, publish its self-registration on the cluster origin, + /// and keep that registration alive (i.e. not exit and drop the broadcast). + #[tokio::test(start_paused = true)] + async fn passive_rendezvous_runs_without_client_and_advertises_self() { + let cluster = Cluster::new(ClusterConfig { + mesh: Some("rendezvous.example.com:4443".to_string()), + ..Default::default() + }); + + // Snapshot a consumer on the cluster origin before run() takes ownership of + // `cluster` so we can later check that the registration was published. + let mut watcher = cluster.origin.consume(); + + let cluster_run = cluster.clone(); + let mut handle = tokio::spawn(async move { cluster_run.run().await }); + + // Give the runtime a moment to execute the synchronous setup work. + tokio::time::sleep(tokio::time::Duration::from_millis(10)).await; + + // The self-registration broadcast must be visible on the origin. + let (path, broadcast) = watcher.try_announced().expect("self-registration must be published"); + assert_eq!(path.as_str(), ".internal/origins/rendezvous.example.com:4443"); + assert!(broadcast.is_some()); + + // run() must NOT have returned: dropping the broadcast (via run returning) + // would unannounce the registration immediately. Use a short timeout to + // confirm we're still parked. + let still_running = tokio::time::timeout(tokio::time::Duration::from_millis(50), &mut handle) + .await + .is_err(); + assert!(still_running, "passive rendezvous run() should park, not return"); + + handle.abort(); + } + + /// `cluster.mesh` round-trips through TOML and CLI. + #[test] + fn cluster_mesh_round_trips() { + let toml = "[cluster]\nmesh = \"us-east.example.com:4443\"\nconnect = [\"root.example.com:4443\"]\n"; + let dir = std::env::temp_dir().join("moq-relay-cluster-test"); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("cluster-mesh-toml.toml"); + std::fs::write(&path, toml).unwrap(); + + let args = vec![std::ffi::OsString::from("moq-relay"), std::ffi::OsString::from(&path)]; + let config = Config::parse_and_merge(args).expect("config load"); + assert_eq!(config.cluster.mesh.as_deref(), Some("us-east.example.com:4443")); + assert_eq!(config.cluster.connect, vec!["root.example.com:4443".to_string()]); + } +}