diff --git a/docs/content/docs/gitops/gitops-workflow.svg b/docs/content/docs/gitops/gitops-workflow.svg new file mode 100644 index 000000000..1235ba9dd --- /dev/null +++ b/docs/content/docs/gitops/gitops-workflow.svg @@ -0,0 +1,163 @@ + + + + + + + + + + + + + + + + + + + + + + + + + DEVELOP + + + + + + + + GITHUB + Write SQL files + migration or schema + + + + + + + + + + PR created + + + + + + CI + + + + + + + + + + Create Pull Request + Open PR with SQL changes + + + + + + triggers + + + + + + + + SQL Review + Check syntax, lint, enforce policy + + + + + + post + review + + + + + + + + + PR merged + + + + + + CD + + + + + + + + + + Merge to branch + Webhook notifies Bytebase + + + + + + triggers + + + + + + + + Create Release + Plan, rollout, and deploy + + + + + + apply + + + + + + + + + + + + Test + + + + + + + + + Prod + + + diff --git a/docs/docs.json b/docs/docs.json index 49ebdc09c..4606895b7 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -126,6 +126,7 @@ "group": "GitOps Workflow", "pages": [ "gitops/overview", + "gitops/get-started", "gitops/installation", { "group": "Migration-Based Workflow", diff --git a/docs/gitops/get-started.mdx b/docs/gitops/get-started.mdx new file mode 100644 index 000000000..ec6d1905c --- /dev/null +++ b/docs/gitops/get-started.mdx @@ -0,0 +1,190 @@ +--- +title: Get Started +--- + +This guide walks you through setting up a GitOps workflow with Bytebase. By the end, you'll have automated SQL review on every pull/merge request and automated database deployment on merge. + +## Prerequisites + +- A running Bytebase instance ([Cloud](https://hub.bytebase.com/) or [self-hosted](/get-started/self-host/deploy-with-docker)) +- A GitHub or GitLab repository +- At least one database registered in a Bytebase project + +## Step 1 — Set Up Authentication + +Your CI/CD pipeline needs to authenticate with the Bytebase API. We recommend [Workload Identity](/administration/workload-identity/overview) — it uses short-lived OIDC tokens so there are no secrets to store or rotate. Follow the setup guide for your platform: + +- [GitHub Actions Workload Identity](/administration/workload-identity/github-actions) +- [GitLab CI Workload Identity](/administration/workload-identity/gitlab-ci) + + +If you use a VCS platform that doesn't support Workload Identity (e.g., Azure DevOps, Bitbucket), you can use a [Service Account](/administration/service-account) instead. Create a service account in **IAM & Admin > Users & Groups** with the **GitOps Service Agent** role, then store the API key as a CI/CD secret in your platform. + + +## Step 2 — Configure Runners + +CI/CD workflows execute on **runners** — machines that run your pipeline jobs. The runner must have network access to your Bytebase instance. + +**How to choose:** +- If your Bytebase instance is publicly accessible (Bytebase Cloud or a public URL), use the default cloud-hosted runners provided by your VCS platform. No setup needed. +- If your Bytebase instance is in a private network (VPN, internal network, firewall-restricted), you need a self-hosted runner deployed in the same network so it can reach Bytebase. + + + + **Cloud-hosted runners** (default) — GitHub-hosted runners work out of the box when Bytebase is publicly accessible. Your workflows will use `runs-on: ubuntu-latest` and GitHub manages the infrastructure. + + **Self-hosted runners** — If Bytebase is behind a firewall or in a private network, install a GitHub Actions runner on a machine that can reach your Bytebase instance: + + 1. In your GitHub repository, go to **Settings > Actions > Runners > New self-hosted runner**. + 2. Follow the instructions to download and configure the runner on your server. + 3. In your workflow files, change `runs-on: ubuntu-latest` to `runs-on: self-hosted`. + + See [GitHub self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners) for detailed instructions. + + + **Shared runners** (default) — GitLab.com shared runners work out of the box when Bytebase is publicly accessible. No configuration needed. + + **Self-managed runners** — If Bytebase is behind a firewall or in a private network, install a GitLab Runner on a machine that can reach your Bytebase instance: + + 1. Install GitLab Runner on your server following the [official installation guide](https://docs.gitlab.com/runner/install/). + 2. Register the runner with your GitLab project using a project or group runner token. + 3. In your `.gitlab-ci.yml`, add a `tags` field to target your runner. + + If your runner has no internet access, you'll also need to pre-load the `bytebase/bytebase-action` Docker image. See the [GitLab tutorial](/tutorials/gitops-gitlab-workflow#self-hosted-gitlab-considerations) for instructions. + + + +## Step 3 — Generate Workflow Files + +In Bytebase, go to your project and open the **GitOps** page. Follow the steps to select your VCS platform, target databases, and migration file pattern. Download the generated workflow files and place them in your repository: + + + + - `.github/workflows/sql-review.yml` + - `.github/workflows/release.yml` + + + - `.gitlab-ci.yml` (includes both review and rollout stages) + + + +Commit the files and push to your `main` branch. + + +If you prefer to write the workflow files by hand, refer to the example repositories for a working setup: +- GitHub: [bytebase/example-gitops-github-flow](https://github.com/bytebase/example-gitops-github-flow) +- GitLab: [bytebase-sample/gitops-example](https://gitlab.com/bytebase-sample/gitops-example) + +Key environment variables to configure in the workflow files: +- **`BYTEBASE_URL`** — Your Bytebase instance URL +- **`BYTEBASE_SERVICE_ACCOUNT`** — Service account or workload identity email +- **`BYTEBASE_PROJECT`** — Project resource name (e.g., `projects/my-project`) +- **`BYTEBASE_TARGETS`** — Comma-separated database resource names (e.g., `instances/test/databases/mydb,instances/prod/databases/mydb`) +- **`FILE_PATTERN`** — Migration file glob pattern (e.g., `migrations/*.sql`) + + +## Step 4 — Run Your First Migration + +With authentication, runners, and workflow files in place, you're ready to run your first database migration through GitOps. + +### Create a migration file + +1. Create a new branch in your repository. + +2. Add a migration file under your migrations directory. The file name must start with a version number and may end with `_ddl` or `_dml` to indicate the change type (default is DDL): + + **`migrations/202503131500_create_table_t1_ddl.sql`** + + ```sql + CREATE TABLE t1 ( + id SERIAL PRIMARY KEY, + name TEXT + ); + ``` + + + Migration files must start with a version number. Two common patterns: + - **Timestamp-based:** `202503131500_create_table_t1_ddl.sql` + - **Semver-based:** `1.0.0_create_table_t1_ddl.sql` + + The suffix indicates the change type: + - `_ddl` — Schema changes (CREATE, ALTER, DROP) + - `_dml` — Data changes (INSERT, UPDATE, DELETE) + - No suffix — Defaults to DDL + + +3. Commit, push, and open a pull request (GitHub) or merge request (GitLab) against `main`. + +### SQL review runs automatically + +4. The SQL review workflow triggers and posts results back to your pull/merge request. You'll see a warning because the `name` column allows NULL. + + + + ![GitHub PR SQL Review](/content/docs/tutorials/gitops-github-workflow/gh-sql-review-warning.webp) + + + ![GitLab MR SQL Review](/content/docs/tutorials/gitops-gitlab-workflow/gl-sql-review-warning.webp) + + + +5. Fix the SQL based on the review feedback — add `NOT NULL` to the `name` column: + + ```sql + CREATE TABLE t1 ( + id SERIAL PRIMARY KEY, + name TEXT NOT NULL + ); + ``` + + Push the fix. The review re-runs and passes. + +### Merge triggers deployment + +6. Once SQL review passes, merge the pull/merge request. The release workflow triggers automatically — it creates a Bytebase release and rolls out the changes to your target databases. + +7. Verify the deployment in your CI/CD pipeline: + + + + Go to the **Actions** tab. You'll see the release workflow triggered by the merge. + + ![GitHub Actions workflow run](/content/docs/tutorials/gitops-github-workflow/gh-merge-run.webp) + + Click into the workflow run to see the job summary — build, deploy-to-test, and deploy-to-prod all completed. + + ![GitHub deploy automatic](/content/docs/tutorials/gitops-github-workflow/gh-deploy-automatic.webp) + + Expand the deploy-to-test stage to see the Bytebase release and rollout links. + + ![GitHub deploy details](/content/docs/tutorials/gitops-github-workflow/gh-deploy-to-test-expand.webp) + + + Go to **Build > Pipelines**. You'll see the release pipeline triggered by the merge. + + ![GitLab pipelines](/content/docs/tutorials/gitops-gitlab-workflow/gl-pipelines.webp) + + Click into the pipeline to see the stages — sql-review, deploy-to-test, and deploy-to-prod. + + ![GitLab pipeline stages](/content/docs/tutorials/gitops-gitlab-workflow/gl-pipelines-stages.webp) + + + +You now have a working GitOps pipeline — SQL review on every change, automated deployment on merge. + +## Next Steps + + + + Incremental migration scripts + + + Declarative schema management + + + Configure review rules + + + Full walkthrough with manual rollout and ChatOps + + diff --git a/docs/gitops/overview.mdx b/docs/gitops/overview.mdx index c92c5ce66..45661ab0a 100644 --- a/docs/gitops/overview.mdx +++ b/docs/gitops/overview.mdx @@ -2,181 +2,23 @@ title: Overview --- -## Goal +![GitOps Workflow](/content/docs/gitops/gitops-workflow.svg) -GitOps for database CI/CD brings the same rigor and automation used in application development to database schema management. Database schema changes are equivalent to API changes in microservices - they define contracts between services, and misalignment can break production systems. +Bytebase GitOps lets you manage Database as Code. Write migration or schema files, review them through pull requests, and let Bytebase handle SQL review, rollout, and deployment to your databases. -By treating databases as dependencies that require versioning, compatibility management, and progressive rollout strategies, schema changes gain the same safety and predictability as application code. +Your database schema lives in Git alongside your application code — same branching, same review process, same audit trail. -## Why Database Schema Changes Matter +## When to Use GitOps -Database schema changes are fundamentally similar to API changes: -- They define contracts between services -- Breaking changes can cause application failures -- They require versioning and compatibility considerations -- They benefit from progressive rollout and feature flagging - -Bytebase enables database-as-code workflows, allowing you to manage database changes through your version control system (VCS) with the same process and confidence as application code. - -## Core Capabilities - -### Database-as-Code Workflow - -- **Unified CI/CD Pipeline** - Integrate database changes seamlessly into your existing CI/CD workflows -- **Version Control Integration** - Manage schema changes alongside application code in your VCS -- **Automated Change Detection** - Intelligently detect and apply only necessary changes, ensuring idempotency -- **Progressive Deployment** - Roll out changes across environments (development → staging → production) with configurable automation and approval gates - -### Enterprise-Grade Safety - -- **SQL Review and Validation** - Automatic policy enforcement during pull requests -- **Rollback Capabilities** - Support for safe schema rollbacks when needed -- **Multi-Region Support** - Deploy to isolated regional databases with separate or unified Bytebase deployments -- **Batch Operations** - Apply changes across multiple databases consistently - -## GitOps Workflow - -Bytebase GitOps follows a streamlined 3-stage pipeline: - -``` -Develop → Review (PR/MR) → Release (Bytebase) -``` - -### 1. Develop - -Developers create SQL files in feature branches following your team's Git workflow (GitHub Flow, GitLab Flow, trunk-based, etc.). - -**What happens:** -- Write SQL migration or schema files -- Follow naming conventions for proper versioning -- Test changes locally when possible -- Commit to version control - -### 2. Review (Pull Request / Merge Request) - -When developers open a pull/merge request, automated validation ensures changes meet your organization's standards. - -**What happens:** -- **Automated SQL Review** - Validates syntax and enforces policies -- **Risk Assessment** - Identifies high-risk operations -- **Schema Compatibility** - Checks for breaking changes -- **Team Review** - Human review of schema changes alongside code - -### 3. Release (Bytebase) - -After PR/MR approval and merge, Bytebase orchestrates the deployment. This single stage encompasses **Plan → Rollout → Revision** tracking, all managed within Bytebase. - -**What happens in Bytebase:** - -1. **Release Creation** - Immutable package of SQL files linked to VCS commit -2. **Plan Generation** - Defines deployment strategy and target databases -3. **Rollout Execution** - Creates and executes tasks across environments -4. **Revision Tracking** - Records applied migrations to prevent duplicates - - -The Release stage in Bytebase involves several coordinated steps: - -- **Plan**: Defines which databases to target and rollout strategy -- **Rollout**: Executes the plan, creating stages for progressive deployment -- **Revision**: Tracks which migrations have been applied to each database - -For detailed information about these components, see: -- Plan creation and targeting strategies -- Rollout execution and approval workflows -- Revision-based idempotency - -These topics are covered in detail within each workflow documentation. - - -**Key Principle**: Bytebase automatically detects previously applied changes by checking revision history and skips them, ensuring safe re-deployment and idempotent operations across all environments. - -## Two Workflow Approaches - -Bytebase supports two complementary approaches for managing database schema changes. Choose the one that fits your team's needs, or use both for different scenarios. - -### Migration-Based Workflow - -The **imperative** approach where you write incremental change scripts that describe how to modify the schema. - -```sql --- migrations/001__create_users.sql -CREATE TABLE users (id INT PRIMARY KEY, name TEXT); - --- migrations/002__add_email.sql -ALTER TABLE users ADD COLUMN email TEXT; -``` - -**Best for:** -- Teams familiar with traditional database migration tools -- Complex changes requiring specific execution order -- Data migrations (INSERT, UPDATE, DELETE operations) -- All supported databases (MySQL, PostgreSQL, SQL Server, etc.) -- Situations requiring fine-grained control over each change - -**How it works:** Each migration file contains explicit DDL/DML statements. Files execute sequentially based on version numbers. Bytebase tracks which versions have been applied to prevent re-execution. - - - Complete guide to migration-based approach with examples and best practices - - -### State-Based Workflow (SDL) - -The **declarative** approach where you define the desired end-state of your database schema, and Bytebase automatically generates the migration DDL. - -```sql --- schema/public.sql (complete desired state) -CREATE TABLE public.users ( - id INTEGER PRIMARY KEY, - name TEXT NOT NULL, - email TEXT -); -``` - -**Best for:** -- Teams adopting infrastructure-as-code principles -- Pure schema changes (DDL only, no data operations) -- Simplified Git diffs showing schema evolution -- PostgreSQL databases (currently) -- Reducing complexity around migration ordering - -**How it works:** You maintain the complete schema definition. Bytebase compares your desired state with the current database state and automatically generates the necessary ALTER/DROP statements to reach the desired state. - - - Complete guide to state-based (SDL) approach with validation rules and examples - - -## Choosing Between Workflows - -| Aspect | Migration-Based | State-Based (SDL) | -|--------|-----------------|-------------------| -| **Approach** | Imperative (how to change) | Declarative (desired end state) | -| **Files** | Multiple migration scripts | Single schema definition per module | -| **Git History** | Shows individual changes | Shows current state + diffs | -| **Operations** | DDL + DML | DDL only | -| **Data Changes** | ✅ Supported | ❌ Not supported | -| **Database Support** | All databases | PostgreSQL only (currently) | -| **Learning Curve** | Familiar to DBAs | Familiar to DevOps/IaC users | -| **Control** | Explicit control over each step | Automatic diff generation | -| **Rollback** | Write reverse migration | Define previous state | - -**Can I use both?** Yes! You can use state-based workflow for schema structure and migration-based workflow for data operations in the same project. +- **Colocate database and application changes.** Your database schema lives in the same repository and goes through the same CI/CD pipeline as your application code. +- **Consistent multi-environment deployment.** A single merge fans out changes to all target databases — test, staging, production — with the same rollout and approval controls. +- **Faster iteration.** Developers use the same pull request workflow for both application and database changes — no context switching, no separate approval process. +- **AI agent friendly.** SQL files in Git are easy for AI coding agents to read, modify, and submit as pull requests. ## Next Steps - - Traditional approach with incremental changes - - - - Modern declarative schema management - - - - Set up VCS integration and runners - - - - Production-ready workflow patterns + + Set up GitOps with Bytebase step by step