Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,9 @@ jobs:
- name: Run clippy lints
run: cargo clippy --all-features -- -D warnings

- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0

- name: Check Rust-native e2e harness
run: make e2e-check

Expand Down
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,12 @@ kubectl --context kind-rustfs-e2e -n rustfs-system port-forward svc/rustfs-opera

Get a login token for the e2e Console. The Console login form expects a
Kubernetes ServiceAccount bearer token with permissions granted to the Console
ServiceAccount. After login, the Console stores it in an encrypted session
cookie; users do not pass this token on later API requests:
ServiceAccount. After login, the Console encrypts it in process and stores only
a random session ID in the browser cookie; users do not pass this token on later
API requests:

The Console must run as one process with a Recreate rollout. The Helm chart
enforces this; custom deployments must preserve both constraints.

```bash
TOKEN=$(kubectl --context kind-rustfs-e2e -n rustfs-system create token rustfs-operator-console --duration=24h)
Expand Down
2 changes: 1 addition & 1 deletion console-web/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Run the operator console HTTP API (e.g. `CONSOLE_COOKIE_SECURE=false cargo run -
Login uses a Kubernetes ServiceAccount bearer token. For a local e2e cluster,
generate one with `kubectl -n rustfs-system create token rustfs-operator-console --duration=24h`
and paste the printed token into the login form. After login, the backend stores
it in an encrypted `session` cookie.
the encrypted token in process and puts only a random session ID in the cookie.

## Build

Expand Down
2 changes: 2 additions & 0 deletions deploy/k8s-dev/console-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ metadata:
labels:
app.kubernetes.io/name: rustfs-operator-console
spec:
strategy:
type: Recreate
replicas: 1
selector:
matchLabels:
Expand Down
27 changes: 18 additions & 9 deletions deploy/rustfs-operator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -317,14 +317,23 @@ Serve the Console service under **one HTTPS host**:
`CONSOLE_COOKIE_SECURE=false` in `console.env`; do not use that setting for
production.

No CORS configuration is needed on the backend for this setup.

Console sessions are encrypted stateless cookies. Users paste a Kubernetes
ServiceAccount bearer token only during login; after validation, the Console
stores that token inside an encrypted `session` cookie for later API requests.
If you run multiple Console replicas, keep `console.jwtSecret` stable and shared
across all replicas. The chart reuses the existing generated Secret on upgrade
when `console.jwtSecret` is not set.
No CORS configuration is needed on the backend for this setup. The reverse
proxy must preserve the public Host/authority; if it rewrites the Host, add the
public Console origin to `CORS_ALLOWED_ORIGINS` so logout origin checks succeed.

Console sessions are stored in process. Users paste a Kubernetes ServiceAccount
bearer token only during login; after validation, the Console encrypts that
token in memory and stores only a random session ID in the browser cookie.
Logout removes the session immediately. The chart requires
`console.replicas=1` and uses a Recreate rollout; restarts and upgrades
invalidate existing sessions and require users to sign in again. Custom
deployments must preserve the same replica and rollout constraints.

When upgrading from an older release configured with multiple Console replicas,
set `console.replicas=1` first and expect brief downtime plus forced sign-in.
Before rolling back to a release without server-side sessions, scale the Console
Deployment to zero, perform the rollback, then restore one replica so the two
cookie formats never overlap.

### Backend CORS (when frontend is on a different host)

Expand All @@ -336,7 +345,7 @@ console:
- name: CORS_ALLOWED_ORIGINS
value: "https://ui.example.com"
# Required when the frontend and API are cross-site, so browsers send the
# encrypted session cookie on credentialed CORS requests.
# session cookie on credentialed CORS requests.
- name: CONSOLE_COOKIE_SAME_SITE
value: "None"
```
Expand Down
5 changes: 5 additions & 0 deletions deploy/rustfs-operator/templates/console-deployment.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
{{- if and .Values.console.enabled (ne (toString .Values.console.replicas) "1") -}}
{{- fail "console.replicas must be 1 because Console sessions are stored in process" -}}
{{- end -}}
{{- if .Values.console.enabled -}}
apiVersion: apps/v1
kind: Deployment
Expand All @@ -12,6 +15,8 @@ metadata:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
strategy:
type: Recreate
replicas: {{ .Values.console.replicas }}
selector:
matchLabels:
Expand Down
2 changes: 1 addition & 1 deletion deploy/rustfs-operator/templates/console-secret.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ metadata:
app.kubernetes.io/component: console
type: Opaque
data:
# Secret used to encrypt Console session cookies
# Secret used to encrypt in-process Console session data
# Generate with: openssl rand -base64 32
jwt-secret: {{ $jwtSecret | quote }}
{{- end }}
5 changes: 2 additions & 3 deletions deploy/rustfs-operator/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ console:
# Enable console deployment
enabled: true

# Number of console replicas
# Console sessions are stored in process, so this must remain 1.
replicas: 1

# Console server port
Expand All @@ -158,8 +158,7 @@ console:
# Log level for console (trace, debug, info, warn, error)
logLevel: info

# Secret used to encrypt Console session cookies.
# All Console replicas must use the same secret.
# Secret used to encrypt in-process Console session data.
# If not set, an existing chart Secret is reused.
# On first install, a random secret is generated.
# Generate with: openssl rand -base64 32
Expand Down
13 changes: 7 additions & 6 deletions docs/operator-user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ Common chart sections:
| `operator` | Operator Deployment replicas, image, resources, probes, metrics, scheduling, leader election, and tenant monitoring. |
| `sts` | Operator STS endpoint, service port, TokenReview audience, and TLS handling. |
| `serviceAccount` / `rbac` | Operator ServiceAccount and RBAC creation. |
| `console` | Operator Console backend/UI Deployment, service, session cookie secret, ingress, resources, and optional split frontend. |
| `console` | Operator Console backend/UI Deployment, service, session encryption secret, ingress, resources, and optional split frontend. |
| `namespace` | Namespace override for chart resources; defaults to the Helm release namespace. |
| `commonLabels` / `commonAnnotations` | Labels and annotations added to chart-managed resources. |

Expand All @@ -145,7 +145,7 @@ operator:

console:
enabled: true
replicas: 2
replicas: 1
jwtSecret: "<stable-base64-or-random-secret>"
ingress:
enabled: true
Expand All @@ -164,7 +164,8 @@ sts:
Notes:

- `operator.leaderElect` can be unset. The chart enables leader election automatically when `operator.replicas > 1`.
- Keep `console.jwtSecret` stable when running multiple Console replicas. If unset, the chart generates or reuses a Secret.
- Keep `console.replicas=1`. Console sessions are stored in process, and the chart uses a Recreate rollout so upgrades invalidate existing sessions without mixing incompatible cookie formats. Custom deployments must preserve both constraints.
- When upgrading from a release configured with multiple Console replicas, set `console.replicas=1` first; expect brief Console downtime and require users to sign in again. Before rolling back to a release without server-side sessions, scale the Console Deployment to zero, perform the rollback, then restore one replica so incompatible cookie formats never overlap.
- Keep `CONSOLE_COOKIE_SECURE` enabled for production HTTPS. Only disable it for local HTTP testing.
- `sts.tls.auto=true` lets the operator create the `sts-tls` Secret when missing.

Expand Down Expand Up @@ -574,15 +575,15 @@ console:
- host: console.example.com
```

The unified operator image serves both `/` and `/api/v1` from the Console service. No backend CORS configuration is needed for this mode.
The unified operator image serves both `/` and `/api/v1` from the Console service. No backend CORS configuration is needed for this mode. The reverse proxy must preserve the public Host/authority; if it rewrites the Host, add the public Console origin to `CORS_ALLOWED_ORIGINS` so logout origin checks succeed.

Console login uses a Kubernetes ServiceAccount bearer token. For the chart-managed Console ServiceAccount:

```bash
kubectl -n rustfs-system create token rustfs-operator-console --duration=24h
```

Paste the token into the login form. The Console stores the validated token in an encrypted session cookie.
Paste the token into the login form. The Console keeps the encrypted token in process and places only a random session ID in the browser cookie. Logging out removes that session immediately; restarting or upgrading the Console invalidates all existing sessions.

For local port-forward testing:

Expand Down Expand Up @@ -805,7 +806,7 @@ For the RustFS Tenant Console, use the Tenant admin credentials from `spec.creds
- Do not model hot/warm/cold tiers as pools inside one Tenant.
- Use separate Tenants for separate clusters, administrative boundaries, or performance isolation.
- Keep the Operator Console on HTTPS in production.
- Keep `console.jwtSecret` stable for multi-replica Console deployments.
- Keep `console.replicas=1`; Console restarts and upgrades intentionally require users to sign in again.
- Use `ServiceMonitor` and `PrometheusRule` only when Prometheus Operator is installed.
- Keep Tenant examples under version control, but never commit raw Secret values.
- Check `status.conditions` before debugging lower-level StatefulSets.
Expand Down
13 changes: 7 additions & 6 deletions docs/operator-user-guide.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ helm upgrade --install rustfs-operator deploy/rustfs-operator/ \
| `operator` | Operator Deployment 副本数、镜像、资源、探针、metrics、调度、leader election 和 Tenant monitor。 |
| `sts` | Operator STS 端点、Service 端口、TokenReview audience 和 TLS。 |
| `serviceAccount` / `rbac` | Operator ServiceAccount 和 RBAC 创建策略。 |
| `console` | Operator Console 后端/UI Deployment、Service、session cookie 密钥、Ingress、资源和可选独立前端。 |
| `console` | Operator Console 后端/UI Deployment、Service、session 加密密钥、Ingress、资源和可选独立前端。 |
| `namespace` | Chart 资源命名空间覆盖;默认使用 Helm release namespace。 |
| `commonLabels` / `commonAnnotations` | 添加到 Chart 管理资源上的统一 label 和 annotation。 |

Expand All @@ -147,7 +147,7 @@ operator:

console:
enabled: true
replicas: 2
replicas: 1
jwtSecret: "<stable-base64-or-random-secret>"
ingress:
enabled: true
Expand All @@ -166,7 +166,8 @@ sts:
配置说明:

- `operator.leaderElect` 可以不配置;当 `operator.replicas > 1` 时 Chart 会自动启用 leader election。
- 多副本 Console 部署需要保持 `console.jwtSecret` 稳定;不设置时 Chart 会生成或复用已有 Secret。
- 保持 `console.replicas=1`。Console 会话保存在进程内,Chart 使用 Recreate 升级策略,避免新旧 cookie 格式混跑;升级时现有会话会失效。自定义 Deployment 也必须保持这两个约束。
- 从多 Console 副本的旧版本升级前,先把 `console.replicas` 调整为 `1`;升级会有短暂 Console 不可用,用户需要重新登录。回滚到不支持服务端会话的旧版本前,先把 Console Deployment 缩容到零,完成回滚后再恢复一个副本,避免两种 cookie 格式同时在线。
- 生产环境应使用 HTTPS 并保持 `CONSOLE_COOKIE_SECURE` 启用。仅本地 HTTP 调试时才关闭。
- `sts.tls.auto=true` 时,Operator 会在缺失时创建 `sts-tls` Secret。

Expand Down Expand Up @@ -576,15 +577,15 @@ console:
- host: console.example.com
```

统一 Operator 镜像会通过同一个 Console Service 提供 `/` 和 `/api/v1`。该模式不需要后端 CORS 配置。
统一 Operator 镜像会通过同一个 Console Service 提供 `/` 和 `/api/v1`。该模式不需要后端 CORS 配置。反向代理必须保留公网 Host/authority;如果代理会改写 Host,请把公网 Console origin 加入 `CORS_ALLOWED_ORIGINS`,确保登出请求的 origin 校验通过。

Console 登录需要 Kubernetes ServiceAccount bearer token。Chart 管理的 Console ServiceAccount 可以这样生成短期 token:

```bash
kubectl -n rustfs-system create token rustfs-operator-console --duration=24h
```

将 token 粘贴到 Console 登录页。Console 会把验证后的 token 存入加密 session cookie
将 token 粘贴到 Console 登录页。Console 会把验证后的 token 加密保存在进程内,浏览器 cookie 仅保存随机 session ID。登出会立即删除该会话;Console 重启或升级会使全部现有会话失效

本地 port-forward 调试:

Expand Down Expand Up @@ -807,7 +808,7 @@ RustFS Tenant Console 登录失败时,应使用 `spec.credsSecret` 或 RustFS
- 不要把一个 Tenant 内的多个 pool 当作冷热分层。
- 独立集群、独立管理边界或独立性能隔离应使用多个 Tenant。
- 生产环境 Operator Console 使用 HTTPS。
- 多副本 Console 部署保持 `console.jwtSecret` 稳定
- 保持 `console.replicas=1`;Console 重启或升级后,用户需要重新登录
- 仅在安装 Prometheus Operator 后启用 `ServiceMonitor` 和 `PrometheusRule`。
- Tenant YAML 可以进入版本控制,但不要提交明文 Secret 值。
- 优先查看 `status.conditions`,再进一步排查 StatefulSet 和 Pod。
Expand Down
15 changes: 14 additions & 1 deletion e2e/src/framework/deploy.rs
Original file line number Diff line number Diff line change
Expand Up @@ -151,10 +151,13 @@ fn patch_images_and_tags(manifest: &str, image: &str, fallback: &str) -> String
#[cfg(test)]
mod tests {
use super::{
E2E_CONSOLE_WEB_IMAGE_TAG_DEFAULT, E2E_CONTROL_PLANE_DEPLOYMENTS,
CONSOLE_DEPLOYMENT, E2E_CONSOLE_WEB_IMAGE_TAG_DEFAULT, E2E_CONTROL_PLANE_DEPLOYMENTS,
E2E_OPERATOR_IMAGE_TAG_DEFAULT, patch_images_and_tags,
};

const CONSOLE_DEPLOYMENT_TEMPLATE: &str =
include_str!("../../../deploy/rustfs-operator/templates/console-deployment.yaml");

#[test]
fn patch_images_prefers_explicit_runtime_image_tags() {
let operator = patch_images_and_tags(
Expand Down Expand Up @@ -185,4 +188,14 @@ mod tests {
]
);
}

#[test]
fn console_session_deployments_are_single_replica_and_recreate() {
assert!(CONSOLE_DEPLOYMENT.contains("replicas: 1"));
assert!(CONSOLE_DEPLOYMENT.contains("strategy:\n type: Recreate"));
assert!(
CONSOLE_DEPLOYMENT_TEMPLATE.contains("ne (toString .Values.console.replicas) \"1\"")
);
assert!(CONSOLE_DEPLOYMENT_TEMPLATE.contains("strategy:\n type: Recreate"));
}
}
64 changes: 64 additions & 0 deletions e2e/tests/sts_manifest.rs
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,61 @@ fn helm_template_renders_sts_enabled_disabled_and_rejects_external_plaintext() {
assert!(external_stderr.contains("operator STS currently supports only ClusterIP"));
}

#[test]
fn console_session_deployments_enforce_single_recreate_process() {
let dev_manifest = std::fs::read_to_string("../deploy/k8s-dev/console-deployment.yaml")
.expect("k8s dev Console deployment exists");
let dev_deployment = find_yaml_document(&dev_manifest, "Deployment", "rustfs-operator-console")
.expect("k8s dev Console deployment is rendered");
assert_eq!(dev_deployment["spec"]["replicas"].as_i64(), Some(1));
assert_eq!(
dev_deployment["spec"]["strategy"]["type"].as_str(),
Some("Recreate")
);

let Some(default_render) = helm_template(&[]) else {
return;
};
assert!(
default_render.status.success(),
"default helm template should render successfully: {}",
String::from_utf8_lossy(&default_render.stderr)
);
let default_stdout = String::from_utf8(default_render.stdout).expect("helm stdout is utf8");
let deployment = find_yaml_document(&default_stdout, "Deployment", "rustfs-operator-console")
.expect("Helm Console deployment is rendered");
assert_eq!(deployment["spec"]["replicas"].as_i64(), Some(1));
assert_eq!(
deployment["spec"]["strategy"]["type"].as_str(),
Some("Recreate")
);

for replicas in ["2", "true"] {
let render = helm_template(&["--set", &format!("console.replicas={replicas}")])
.expect("helm remains available");
assert!(!render.status.success());
assert!(
String::from_utf8_lossy(&render.stderr).contains(
"console.replicas must be 1 because Console sessions are stored in process"
)
);
}

let disabled_render = helm_template(&[
"--set",
"console.enabled=false",
"--set",
"console.replicas=2",
])
.expect("helm remains available");
assert!(disabled_render.status.success());
let disabled_stdout =
String::from_utf8(disabled_render.stdout).expect("disabled helm stdout is utf8");
assert!(
find_yaml_document(&disabled_stdout, "Deployment", "rustfs-operator-console").is_none()
);
}

fn helm_template(args: &[&str]) -> Option<Output> {
if !helm_is_available() {
eprintln!("skipping helm template assertions: helm binary is not available");
Expand Down Expand Up @@ -214,3 +269,12 @@ fn assert_yaml_documents_parse(yaml: &str, name: &str) {
"{name} should contain at least one yaml document"
);
}

fn find_yaml_document(yaml: &str, kind: &str, name: &str) -> Option<Value> {
yaml.split("---")
.filter_map(|document| serde_yaml_ng::from_str::<Value>(document).ok())
.find(|document| {
document["kind"].as_str() == Some(kind)
&& document["metadata"]["name"].as_str() == Some(name)
})
}
26 changes: 26 additions & 0 deletions src/console/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ pub enum Error {
#[snafu(display("Conflict: {}", message))]
Conflict { message: String },

#[snafu(display("Too many requests: {}", message))]
TooManyRequests { message: String },

#[snafu(display("Action required: {}", message))]
ActionRequired {
status: StatusCode,
Expand Down Expand Up @@ -181,6 +184,14 @@ impl Error {
Vec::new(),
None,
),
Error::TooManyRequests { message } => (
StatusCode::TOO_MANY_REQUESTS,
"TooManyRequests".to_string(),
"RateLimitExceeded".to_string(),
message,
Vec::new(),
None,
),
Error::ActionRequired {
status,
code,
Expand Down Expand Up @@ -288,6 +299,21 @@ mod tests {
Ok(())
}

#[test]
fn too_many_requests_maps_to_stable_error_contract() {
let (status, response) = Error::TooManyRequests {
message: "Too many active Console sessions".to_string(),
}
.into_response_parts();

assert_eq!(status, StatusCode::TOO_MANY_REQUESTS);
assert_eq!(response.code, "TooManyRequests");
assert_eq!(response.reason, "RateLimitExceeded");
assert_eq!(response.message, "Too many active Console sessions");
assert!(response.next_actions.is_empty());
assert!(response.details.is_none());
}

#[test]
fn action_required_maps_to_stable_error_contract() -> std::result::Result<(), serde_json::Error>
{
Expand Down
Loading
Loading