diff --git a/content/en/blog/2026-08-04-blockstor-linstor-compatible-storage-for-kubernetes/blockstor-announcement.png b/content/en/blog/2026-08-04-blockstor-linstor-compatible-storage-for-kubernetes/blockstor-announcement.png new file mode 100644 index 00000000..60f645a0 Binary files /dev/null and b/content/en/blog/2026-08-04-blockstor-linstor-compatible-storage-for-kubernetes/blockstor-announcement.png differ diff --git a/content/en/blog/2026-08-04-blockstor-linstor-compatible-storage-for-kubernetes/index.md b/content/en/blog/2026-08-04-blockstor-linstor-compatible-storage-for-kubernetes/index.md new file mode 100644 index 00000000..55d75f96 --- /dev/null +++ b/content/en/blog/2026-08-04-blockstor-linstor-compatible-storage-for-kubernetes/index.md @@ -0,0 +1,152 @@ +--- +title: "Blockstor: a LINSTOR-compatible storage system for Kubernetes, written from scratch in Go" +slug: "blockstor-linstor-compatible-storage-for-kubernetes" +date: 2026-08-04 +author: "Cozystack Team" +description: "The Cozystack team has open-sourced Blockstor: LVM and ZFS backends, DRBD replication, and a LINSTOR-compatible REST API, so existing client tooling keeps working unchanged." +images: + - "blockstor-announcement.png" +article_types: + - "announcement" +topics: + - "storage" + - "linstor" + - "drbd" + - "platform" +--- + +{{< figure src="blockstor-announcement.png" alt="Blockstor — a LINSTOR-compatible storage system for Kubernetes, written from scratch in Go" width="720" >}} + +The Cozystack team has open-sourced Blockstor, a control plane for block storage in Kubernetes: LVM and ZFS as backends, replication over DRBD, and a LINSTOR-compatible REST API. The project lives in the cozystack organization and is developed as part of Cozystack, a platform accepted into the CNCF Sandbox. The license is Apache 2.0. + +The main thing that makes it worth a look: it is not a fork, and it is not a wrapper. Blockstor is written from scratch in Go, but it speaks the same REST API as LINSTOR — so all the client tooling you already run keeps working without a single change: the `linstor` CLI, linstor-csi, piraeus-operator, and the golinstor library. + +## Why Blockstor takes a different approach + +LINSTOR is a mature system, and it ran in production in Cozystack for years. We did not hit a functionality ceiling — we hit the model. + +The original controller is request-based: for most API calls it goes out to the nodes in real time and polls their state to assemble a response. That has two consequences. First, this design scales poorly. Second, with no reconciliation loop, automatic recovery from failures has to be bolted on from the outside. + +Blockstor is built the way Kubernetes operators are normally built: the desired state lives in CRDs, and a set of reconcilers on controller-runtime drives the cluster toward it. Three practical consequences follow: + +- No external database to back up and worry about +- No in-memory state to lose when the controller restarts +- No controller-side polling of nodes that can fall behind reality + +The satellites watch the API themselves and write the observed state back through Server-Side Apply, using separate field managers. Spec belongs to the controller, Status to the satellite, and that split is enforced strictly. + +## What it is made of + +Three components, all of them ordinary Kubernetes workloads: + +| Component | Role | +|---|---| +| `blockstor-controller` | A Deployment running the controller-runtime reconcilers | +| `blockstor-apiserver` | A stateless, LINSTOR-compatible REST front end, backed by CRDs. This is what `linstor`, CSI, and Piraeus talk to | +| `blockstor-satellite` | A DaemonSet: it brings up the DRBD, LUKS, and STORAGE layers on the node and calls `drbdadm`, `lvs`, `zfs`, and `cryptsetup` | + +The objects live in the `blockstor.cozystack.io/v1alpha1` group: Node, StoragePool, ResourceGroup, ResourceDefinition, Resource, Snapshot, PhysicalDevice, and ControllerConfig. The CRDs are designed as a public integration point, with schema-level validation and a safe multi-writer model for Status, so that GitOps tooling and monitoring can work with them directly. + +## What already works + +- Replicated DRBD volumes on top of LVM, LVM-thin, ZFS, ZFS-thin, and file backends +- A DRBD-free mode — a single replica, diskful or diskless +- LUKS encryption at the volume level; the layers stack as DRBD → LUKS → STORAGE +- Auto-placement with constraints: zones, node properties, and replica spreading +- TieBreaker and quorum policies — one of the most heavily tested parts of the system +- Snapshots: create, roll back, clone, and restore into a new resource +- Snapshot shipping within the cluster using `zfs send/recv` and thin-send-recv +- Online volume resize. Shrinking is disabled by default and requires an explicit `force=true` — here we are deliberately stricter than the original +- Creating pools from physical disks +- Replica rebalancing and migration: automatic evacuation from a departing node, automatic promotion to diskful, and recovery after split-brain +- Skipping the initial sync when a replica is added, by seeding the Generation Identifier. Adding a third replica to a multi-terabyte volume does not turn into a multi-hour resync +- mTLS on the API with hot certificate reload, Prometheus metrics, and images for amd64 and arm64 +- RWX — verified by an end-to-end test through linstor-csi and NFS-Ganesha + +## What is not there yet + +We would rather put this in the announcement than have you discover it on day three. + +The following are not implemented, and they return an honest `501 Not Implemented` rather than a silent 404: cross-cluster snapshot shipping, backups and the backup queue, schedules, remote backends such as S3, and the SPDK, NVMe-oF, OpenFlex, and Exos drivers. There is no Helm chart — installation goes through plain manifests. The version is still 0.x. + +The list of CLI behaviour differences from the original is maintained in public, along with a register of known issues and a write-up of the csi-sanity tests that fail. Put plainly: the project itself publishes the list of its own gaps. + +## Why you can trust this + +A storage control plane rewritten from scratch is a claim that needs proof, not promises. Our answer is tests. + +The implementation is 94,000 lines. The tests are 170,000 lines of Go and another 46,000 lines of shell. And these are not only unit tests: + +- 108 integration tests run the real `linstor` Python client against envtest on every PR +- Contract tests run real `drbdmeta` and `drbdadm` in Docker on top of loopback devices +- 89 end-to-end scenarios run on a Talos and QEMU rig with real DRBD +- 91 CLI matrix cells and 74 replay scenarios cover operator workflows +- A parity harness compares Blockstor's responses against a live upstream LINSTOR and fails CI on any divergence that is not on the accepted list + +Release v0.1.11 deserves a separate mention: it reproduced and closed 48 edge cases pulled from the linstor-server bug tracker itself — on a live rig, with the disputed cases settled by checking against a running upstream. + +## Compatibility and the legal side + +Blockstor returns `1.33.2+git=blockstor` for `linstor controller version` and implements the endpoints that linstor-csi and piraeus-operator actually call. Piraeus connects in external-controller mode — point it at the address of the Blockstor apiserver, and linstor-csi keeps working untouched. + +LINSTOR is distributed under the GPL and Blockstor under Apache 2.0, so no sources from the original were used. The project is a clean-room implementation: the compatibility types come from golinstor, an Apache 2.0 library, and no code is copied or generated from GPL sources. This is not a declaration but a checkable rule: on every PR, a license gate runs in CI that keeps GPL, AGPL, LGPL, and SSPL out of the runtime graph — including code generated from a GPL-licensed specification. + +And to be direct: LINSTOR, LINBIT, and DRBD are trademarks of LINBIT. Blockstor is an independent project, not affiliated with, endorsed by, or sponsored by LINBIT. We are grateful to LINBIT and to the DRBD, LINSTOR, and Piraeus communities: Blockstor speaks the LINSTOR API deliberately, precisely so that the ecosystem the community built keeps working. + +## How to try it + +One nice detail for anyone already living on LINSTOR: you can install Blockstor on the same nodes, next to a running LINSTOR. The TCP port ranges and the DRBD minor-number ranges are deliberately kept clear of the upstream ones, so you can try it without shutting anything down. + +The hosts need the DRBD 9 kernel module, drbd-utils, lvm2, and cryptsetup; for ZFS, the module and zfsutils-linux. On Talos those are the `siderolabs/drbd` and `siderolabs/zfs` extensions. + +```bash +kubectl apply -f config/crd/bases/ + +# then the manifests from stand/: controller, apiserver, satellite + +kubectl -n blockstor-system rollout status deploy/blockstor-controller +kubectl -n blockstor-system rollout status deploy/blockstor-apiserver +kubectl -n blockstor-system rollout status daemonset/blockstor-satellite +``` + +From there on, it is the ordinary `linstor`, unchanged: + +```bash +kubectl -n blockstor-system port-forward deploy/blockstor-apiserver 3370:3370 +export LS_CONTROLLERS=http://localhost:3370 + +linstor node create worker-1 10.0.0.11 +linstor physical-storage create-device-pool --pool-name data --storage-pool data zfs worker-1 /dev/sdb +linstor resource-group create mygroup --place-count 3 --storage-pool data +linstor volume-group create mygroup +linstor resource-group spawn mygroup myvolume 10G +linstor resource list +``` + +In the output you will see two UpToDate diskful replicas and one TieBreaker. + +## What is next, and where we would welcome help + +Blockstor has already been through the Cozystack end-to-end suite on a three-node rig: PVCs are served through an untouched CSI and untouched StorageClasses, including three-way DRBD replication. But the integration is still a proof of concept — it has proven substitutability, not the default. + +A tool for migrating an existing LINSTOR cluster in place is in the works: it moves the metadata into CRDs and adopts existing zvols, LVs, and running DRBD devices with no data copying and no resync, preserving minor numbers, node IDs, ports, and the DRBD shared secret. Where something cannot be carried over without guesswork, the tool refuses to guess and reports it. + +The project is young and was written mostly by one person — and that is exactly where outside hands help. These are the areas where help would matter most right now: + +- Testing outside Talos — Ubuntu and other distributions have not been checked yet +- A Helm chart — there is not one +- Verifying the pairing with ha-controller — compatibility is claimed but not covered by a test +- Grafana dashboards and alerts for Blockstor metrics +- The hardest and most valuable area — the DRBD kernel layer: filesystem assembly and attachment on the satellite, split-brain, and real synchronization. The project itself names this as its main remaining risk + +If you run DRBD in production, the most useful contribution starts with installing Blockstor alongside it and telling us what broke. + +Questions, bug reports, and "it doesn't work for me" belong in the repository issues or the chats below. A report on what did not work is worth more right now than a star on GitHub. + +## Join the community + +- [Blockstor on GitHub](https://github.com/cozystack/blockstor) +- [Cozystack on GitHub](https://github.com/cozystack/cozystack) +- Telegram [group](https://t.me/cozystack) +- Slack [group](https://kubernetes.slack.com/archives/C06L3CPRVN1) (Get invite at [https://slack.kubernetes.io](https://slack.kubernetes.io)) +- [Community Meeting Calendar](https://calendar.google.com/calendar?cid=ZTQzZDIxZTVjOWI0NWE5NWYyOGM1ZDY0OWMyY2IxZTFmNDMzZTJlNjUzYjU2ZGJiZGE3NGNhMzA2ZjBkMGY2OEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t) diff --git a/content/en/blog/2026-08-04-cozystack-1-6-talos-workers-tenant-sso-security-groups-hierarchical-quotas/cozystack-1-6-banner.png b/content/en/blog/2026-08-04-cozystack-1-6-talos-workers-tenant-sso-security-groups-hierarchical-quotas/cozystack-1-6-banner.png new file mode 100644 index 00000000..23944495 Binary files /dev/null and b/content/en/blog/2026-08-04-cozystack-1-6-talos-workers-tenant-sso-security-groups-hierarchical-quotas/cozystack-1-6-banner.png differ diff --git a/content/en/blog/2026-08-04-cozystack-1-6-talos-workers-tenant-sso-security-groups-hierarchical-quotas/index.md b/content/en/blog/2026-08-04-cozystack-1-6-talos-workers-tenant-sso-security-groups-hierarchical-quotas/index.md new file mode 100644 index 00000000..dc18e9e4 --- /dev/null +++ b/content/en/blog/2026-08-04-cozystack-1-6-talos-workers-tenant-sso-security-groups-hierarchical-quotas/index.md @@ -0,0 +1,187 @@ +--- +title: "Cozystack 1.6: Talos Workers, Tenant SSO, Security Groups, Hierarchical Quotas, and Safer etcd Upgrades" +slug: "cozystack-1-6-talos-workers-tenant-sso-security-groups-hierarchical-quotas" +date: 2026-08-04 +author: "Cozystack Team" +description: "Cozystack v1.6.0 moves tenant Kubernetes workers to Talos Linux, adds tenant-controlled OIDC, a SecurityGroup firewall API, hierarchical quotas, and in-place etcd-operator adoption." +images: + - "cozystack-1-6-banner.png" +article_types: + - "release" +topics: + - "platform" + - "kubernetes" + - "talos" + - "security" + - "storage" +--- + +{{< figure src="cozystack-1-6-banner.png" alt="Cozystack v1.6.0 release banner — Talos workers, tenant SSO, security groups, hierarchical quotas" width="720" >}} + +Cozystack v1.6.0 is now available. The release was published on July 22, 2026, and includes all fixes previously shipped in the v1.5.1, v1.5.2, and v1.5.3 patch releases. + +This release changes several important parts of the platform. Tenant Kubernetes workers now run Talos Linux instead of Ubuntu, tenants can enable OIDC authentication for Kubernetes and Grafana, and a new SecurityGroup API provides a safer interface for managing application network policies. + +Cozystack 1.6 also introduces hierarchical resource quotas, completes the etcd-operator v1alpha2 migration, expands Keycloak security and backup options, and makes application deletion consistently reclaim its storage. + +The upgrade surface is larger than usual. Operators should review the upgrade section before applying the release. + +## Talos Linux for tenant Kubernetes workers + +Tenant Kubernetes worker nodes no longer use Ubuntu and kubeadm. Cozystack now provisions them with Talos Linux through Cluster API Bootstrap Provider Talos. Workers boot from a Talos image delivered through CDI and use a single system disk managed by Talos instead of separate system and kubelet disks. + +Existing tenant clusters are migrated automatically. On the first reconciliation after the platform upgrade, the old worker machines are gradually replaced with Talos-based workers. + +This does not require tenants to recreate their clusters, but operators should expect a complete worker-pool rollout. Worker disks are reprovisioned, and container images must be downloaded again. + +MachineHealthCheck remediation is also enabled by default. Cluster API can now replace unhealthy workers automatically, with `maxUnhealthy` set to 50%. Operators who prefer the previous behaviour can temporarily set it to 0% while their fleets move to Talos. + +## OIDC single sign-on for tenant Kubernetes and Grafana + +Cozystack 1.6 introduces tenant-controlled OIDC authentication for managed Kubernetes clusters and individual Grafana instances. + +Each Kubernetes resource now supports three authentication modes: + +- `System` uses the platform Keycloak instance. +- `CustomConfig` accepts a tenant-provided Kubernetes authentication configuration. +- `None` keeps OIDC disabled. + +In `System` mode, tenants can assign admin or view access to individual users. Cozystack creates the required role bindings and exposes a ready-to-use kubeconfig containing the `kubectl oidc-login` configuration through the dashboard. + +Grafana uses the same model. Tenants can connect an instance to the platform Keycloak realm and assign Admin, Editor, or Viewer roles to users without requiring platform-level configuration. The local Grafana administrator credentials remain available as a break-glass access method. + +## SecurityGroup: a tenant-facing firewall API + +Cozystack now includes the namespace-scoped SecurityGroup resource under the `sdn.cozystack.io/v1alpha1` API. It lets tenants manage network access between their applications without direct access to Cilium resources. + +A SecurityGroup can include several managed applications. Cozystack labels their pods and generates the corresponding CiliumNetworkPolicy. Rules can reference other SecurityGroups, making it possible to describe access at the application-group level instead of working with individual pod selectors. + +For example, a tenant can allow a frontend group to connect to an API group while allowing only the API group to reach the database group. + +In v1.6, SecurityGroup rules add permitted traffic. An empty rule list does not create a default-deny policy, because connectivity is still calculated from all policies selecting the pods. Default-deny behaviour is planned separately. + +## Hierarchical resource quotas + +Resource quotas now follow the tenant hierarchy. + +Previously, every tenant quota was enforced only inside its own namespace. A tenant administrator could create a sub-tenant with a larger quota, or with no quota, and consume more resources than the parent tenant had been allocated. + +In Cozystack 1.6, a tenant quota represents the budget of the entire subtree below that tenant. A child tenant with its own quota reserves part of the parent's remaining budget. A child without a separate quota shares the parent's resource pool. A quota that exceeds the parent's available budget is rejected during admission. + +The controller also tracks aggregate usage across the subtree and maintains additional ResourceQuota objects to enforce the shared limit at runtime. + +Existing multi-level tenant structures should be reviewed before upgrading. Overcommitted tenant trees generate a `QuotaOvercommitted` event, and operators can configure a temporary rollout buffer for workloads already above the newly enforced limit. + +## etcd-operator v1alpha2 with in-place adoption + +Cozystack completes its migration to the new `etcd-operator.cozystack.io/v1alpha2` API. + +The new operator uses a membership-based lifecycle instead of managing etcd as a conventional StatefulSet. Its CRDs are now delivered through a separate `etcd-operator-crds` package so they can be installed before the controller. + +Existing etcd clusters are adopted in place. The migration rewrites the required ownership information and resources before the new operator starts managing the cluster. No etcd data move or pod restart is required. + +Before adoption, Cozystack takes a mandatory snapshot of every legacy cluster using the platform-managed backup infrastructure. The upgrade stops if the snapshot destination cannot be reached. This protects both standalone tenant etcd instances and the etcd clusters backing tenant Kubernetes control planes. + +The old per-application `backup.*` settings have been removed from the etcd module. Backups should now be configured through a BackupClass and the platform Etcd backup strategy. + +## More secure and manageable Keycloak deployments + +Keycloak receives several independent, opt-in improvements. + +An optional database proxy can encrypt selected database fields at the application level. Encryption keys can be provided directly or managed through Vault Transit, with Kubernetes and AppRole authentication supported for Vault connections. + +Operators can also expose the Keycloak administration console and Administration REST API through a separate hostname. This makes it possible to keep the public authentication endpoint accessible while placing administrative access behind a private ingress class or Gateway. + +The Keycloak PostgreSQL database can now be backed up to S3 through Barman. The platform login theme is also configurable through branding values, including a custom theme image. + +## Wildcard certificates across the tenant tree + +Cozystack 1.5 allowed operators to provide an existing wildcard certificate for platform services and the root tenant. Version 1.6 extends this model to child tenants. + +The platform controller replicates the certificate into every tenant namespace where TLS terminates. Tenant ingress controllers and Gateways can then use it automatically without cross-namespace Secret access or additional operator configuration. + +Operators can also let Cozystack request and manage a wildcard certificate through DNS-01. Using one wildcard certificate instead of issuing a separate certificate for every hostname helps avoid ACME rate limits on larger installations. + +## Stable releases now contain the exact tested artifacts + +The release pipeline has been redesigned around immutable image tags and explicit release-candidate promotion. + +A stable release is no longer rebuilt independently. The images tested as `vX.Y.Z-rc.N` are promoted by digest to `vX.Y.Z`, making the stable release byte-identical to the release candidate that passed end-to-end testing. + +Release tags are no longer force-moved, and the scheduled workflow that automatically created patch releases has been removed. Stable versions are now created only through an explicit RC-to-stable promotion process. + +## Application deletion now reclaims storage + +Deleting a managed application previously left some PVCs and generated Secrets behind. + +Cozystack 1.6 adds cleanup handling across the application catalog. Storage used by ClickHouse, Qdrant, OpenBao, monitoring, SeaweedFS, etcd, Harbor, and several other managed services is now reclaimed when the application is deleted. + +This fixes resource leaks, but it also changes deletion semantics. Deleting an application is now destructive. Operators and tenants should create backups or snapshots before removing workloads whose data may still be needed. + +## Also in v1.6.0 + +The dashboard now displays the external IP assigned to LoadBalancer services directly on the application Services tab. + +Kamaji now runs two controller replicas with soft anti-affinity. The release also removes its telemetry admission webhook, reducing admission latency for TenantControlPlane resources on multi-tenant installations. + +Velero moves to version 1.18.1. Multiple backups can now be processed concurrently, and restore data movers can use cache volumes instead of relying entirely on node ephemeral storage. + +The release also includes fixes for SeaweedFS upgrade naming, KubeVirt memory and connection leaks, Cilium networking and Gateway API behaviour, LINSTOR scheduler admission, and COSI BucketClaim reconciliation. + +## Platform components + +Several core components are updated in this release. + +Talos Linux moves from v1.13.0 to v1.13.6, including kernel updates that address the CVE-2026-53359 and CVE-2026-46113 KVM guest-to-host escape vulnerabilities. + +Other major updates include: + +- etcd-operator v0.5.2 with the new v1alpha2 API +- Cilium 1.19.5 +- KubeVirt 1.8.4 +- Velero 1.18.1 +- Vertical Pod Autoscaler 1.5.0 +- Harbor 2.15.1 +- Keycloak 26.6.3 +- LINSTOR 1.33.3 and linstor-csi v1.11.2 +- FoundationDB operator 2.30.0 +- HAMi 2.9.0 +- Percona MongoDB operator 1.22.0 +- OpenBao 2.5.1 +- csi-driver-nfs 4.13.3 + +Managed Kubernetes patch versions are updated to v1.32.13, v1.33.13, v1.34.9, and v1.35.6. Kubernetes v1.30 is no longer supported for tenant clusters. + +## Upgrade notes + +Cozystack 1.6 has the largest upgrade surface since v1.0. Several preconditions can stop the upgrade, so operators should review the complete release notes and run the provided checks before applying the new Platform Package. + +The most important items are: + +1. **Verify the etcd backup destination.** Legacy etcd clusters require a working platform backup target before they can be adopted by the new operator. +2. **Audit SeaweedFS installations.** Some clusters that were installed or upgraded through v1.5.x may require recovery of their workload naming before the chart can be rendered safely. +3. **Move tenant Kubernetes clusters away from v1.30.** Live resources are migrated to v1.31 automatically, but GitOps-managed resources must also be updated in Git. +4. **Check tenant StorageClasses.** A manually created StorageClass that has the same name as a propagated LINSTOR class can block the tenant CSI deployment. +5. **Plan for worker replacement.** Existing tenant worker pools will roll from Ubuntu to Talos Linux. Worker disks are recreated, and images are downloaded again. +6. **Review MachineHealthCheck settings.** Automated worker remediation is now enabled with a default `maxUnhealthy` value of 50%. +7. **Back up applications before deleting them.** Application deletion now removes associated storage instead of leaving PVCs behind. + +Clusters upgrading directly from v1.4.x must also meet the v1.5 requirement: Kubernetes 1.33 or newer is required for the management cluster and for tenant clusters using the Flux addon. + +## Thank you to all contributors + +Cozystack v1.6.0 was made possible by @androndo, @IvanHunters, @kvaps, @lexfrei, @lllamnyp, @mattia-eleuteri, @matthieu-robin, @myasnikovdaniil, @scooby87, @shreyaabaranwal, @sircthulhu, and @tym83. + +We are especially glad to welcome our first-time contributor, @shreyaabaranwal. + +## Release links + +- [Cozystack v1.6.0 release notes on GitHub](https://github.com/cozystack/cozystack/releases/tag/v1.6.0) +- [Full changelog from v1.5.0 to v1.6.0](https://github.com/cozystack/cozystack/compare/v1.5.0...v1.6.0) + +## Join the community + +- [Cozystack on GitHub](https://github.com/cozystack/cozystack) +- Telegram [group](https://t.me/cozystack) +- Slack [group](https://kubernetes.slack.com/archives/C06L3CPRVN1) (Get invite at [https://slack.kubernetes.io](https://slack.kubernetes.io)) +- [Community Meeting Calendar](https://calendar.google.com/calendar?cid=ZTQzZDIxZTVjOWI0NWE5NWYyOGM1ZDY0OWMyY2IxZTFmNDMzZTJlNjUzYjU2ZGJiZGE3NGNhMzA2ZjBkMGY2OEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t)