Bump default serverless environment version to 5 by rugpanov · Pull Request #6378 · databricks/cli

rugpanov · 2026-08-25T13:10:48Z

Changes

Bumps the serverless environment_version generated by the bundle-init templates from 4 to 5.

libs/template/templates/default/library/versions.tmpl — the shared serverless_environment_version macro (drives default-python, pydabs, default-minimal, lakeflow-pipelines), plus a refresh of the stale "version 4 (and upcoming v5)" comment.
libs/template/templates/dbt-sql/library/versions.tmpl and default-scala/.../{{.project_name}}.job.yml.tmpl — these carry their own value; bumped to match.
Regenerated acceptance goldens under acceptance/bundle/templates/ and acceptance/pipelines/ (init + e2e).
Changelog fragment under .nextchanges/bundles/.

lakeflow-integrations already ships 5; default-sql uses SQL-warehouse compute and has no environment_version. The experimental/ssh tunnel keeps its own v4 default (a v5 driver-proxy bug, see 65b2901) and is intentionally untouched.

Why

Serverless environment version 5 is the current version, and lakeflow-integrations already generates it. Verified against python/serverless/serverless-v5/pyproject.toml in the databricks/environments repo: v5 stays on Python 3.12 (requires-python = "==3.12.*"), so the templates' databricks-connect requirement — the deliberately conservative >=15.4,<15.5 floor — is unchanged.

Tests

Template + pipelines acceptance goldens regenerated with -update; the resulting diff is exclusively environment_version: "4" → "5".
go test ./libs/template/... and the template/pipelines acceptance suites pass.
End-to-end: built the CLI, ran bundle init for default-python, default-scala, and dbt-sql — each generates environment_version: "5" and bundle validate succeeds against a live workspace. uv sync on the generated default-python project resolves databricks-connect==15.4.26 / pyspark==3.5.0 on Python 3.12 and imports cleanly.

This pull request and its description were written by Isaac.

Update the serverless environment version generated by the bundle templates from 4 to 5. Version 5 stays on Python 3.12, so the templates' databricks-connect requirement is unchanged (verified against the serverless-v5 environment in the databricks/environments repo). The shared default template body (default-python, pydabs, lakeflow-pipelines) is driven by the serverless_environment_version macro; dbt-sql and default-scala carry their own values and are bumped to match. Acceptance goldens are regenerated accordingly. Co-authored-by: Isaac <no-reply@databricks.com>

The `pipelines init` and `pipelines e2e` acceptance tests render the shared lakeflow-pipelines template, so their generated sample_job.job.yml goldens also move from environment_version "4" to "5". These live under acceptance/pipelines/ and were outside the bundle/templates test filter. Co-authored-by: Isaac <no-reply@databricks.com>

github-actions · 2026-08-25T13:29:46Z

Approval status: pending

`/acceptance/bundle/` - needs approval

21 files changed
Suggested: @denik
Also eligible: @janniklasrose, @lennartkats-db, @andrewnester, @shreyas-goenka, @pietern, @anton-107

`/acceptance/pipelines/` - needs approval

5 files changed
Suggested: @lennartkats-db
Also eligible: @kanterov, @jefferycheng1

`/libs/template/` - needs approval

Files: libs/template/templates/dbt-sql/library/versions.tmpl, libs/template/templates/default-scala/template/{{.project_name}}/resources/{{.project_name}}.job.yml.tmpl, libs/template/templates/default/library/versions.tmpl
Suggested: @denik
Also eligible: @janniklasrose, @lennartkats-db, @andrewnester, @shreyas-goenka, @pietern, @anton-107

General files (require maintainer)

Files: .nextchanges/bundles/serverless-environment-version-v5.md
Based on git history:

@denik -- recent work in .nextchanges/bundles/, acceptance/bundle/templates/default-python/serverless/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rclarey) can approve all areas.

See OWNERS for ownership rules.}

eng-dev-ecosystem-bot · 2026-08-25T13:44:56Z

Integration test report

Commit: 3f41ff3

Run: 32897570309

	Env	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
💚	aws linux	1	4	274	1184	5:23
💚	aws windows	1	4	276	1182	3:53
💚	azure linux	1	4	273	1184	4:32
💚	azure windows	1	4	275	1182	3:47
💚	gcp linux	1	4	274	1184	5:17
💚	gcp windows	1	4	276	1182	3:54

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/bundle/invariant/no_drift	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
3:50	aws windows	TestAccept
3:44	azure windows	TestAccept
3:17	gcp windows	TestAccept

lennartkats-db · 2026-08-25T16:06:13Z

 {{/* Python version specification for the template.
   * This needs to be compatible with the version of DB Connect and the serverless_environment_version.
   *
-   * Serverless environment version 4 (and upcoming v5), DBR 16, and DBR 17 all use Python 3.12.


Please also still mention 4 here (for maintainers)

Done — the comment now reads "Serverless environment versions 4 and 5, DBR 16, and DBR 17 all use Python 3.12."

lennartkats-db · 2026-08-25T16:10:13Z

We should probably bump this to the oldest possible DB Connect version that fully uses Python 3.12? Which might be v16?

DB Connect only offers forward compatibility, which is why we use a conservative ("oldest") version here

See also ohttps://github.com/Update templates to use serverless environment version 4 and matching Python version #3897 for version considerations.

lennartkats-db · 2026-08-25T16:11:04Z

   * See https://docs.databricks.com/release-notes/runtime/index.html for available versions.
   */}}
 {{define "latest_lts_dbr_version" -}}
  16.4.x-scala2.12


We should bump this as well but since the Python version is the same that could be done in a separate PR.

Agreed - I'll leave the latest_lts_dbr_version bump for a separate PR. It's independent of this change.

Per review feedback, move the templates' conservative DB Connect version from 15.4 (Python 3.11) to 16.4 — the oldest DB Connect that fully uses Python 3.12 — so local development matches the serverless v5 runtime. databricks-connect 16.4 wheels are requires-python ==3.12.*, so python_version_spec is pinned to >=3.12,<3.13 to keep generated projects installable. Comment also restored to mention env version 4 alongside 5. Regenerate affected pyproject.toml goldens; update changelog fragment. Co-authored-by: Isaac <no-reply@databricks.com>

yiweidai-db · 2026-08-26T03:43:28Z

    "pytest",
    "ruff",
    "databricks-dlt",
-    "databricks-connect>=15.4,<15.5",


[question]
environment_version 5 actually requires datbaricks-connect>=18.0 to work now.
Is updating the db-connect version for the Python version compatibility rather than the db-connect client <> server compatibility?

Good point — bumped to >=18.0,<18.1 to match v5's DB Connect 18 (client↔server compat). Verified: uv sync resolves databricks-connect==18.0.9, v5's shipped version. 18.0 is also Python 3.12, so the Python pin is unchanged.

yiweidai-db · 2026-08-26T03:52:58Z

@@ -0,0 +1 @@
+Bundle templates now generate serverless jobs and notebooks with `environment_version: "5"` instead of `"4"` (`default-python`, `default-scala`, `dbt-sql`, `pydabs`, and `lakeflow-pipelines`). For the Python templates, the generated `pyproject.toml` now pins `databricks-connect>=16.4,<16.5` and Python `>=3.12,<3.13` (previously `databricks-connect>=15.4,<15.5` and Python `>=3.10,<3.13`) so local development matches serverless environment version 5's Python 3.12 runtime.


[suggestion]

link the environment version 5 release notes to the change logs.

Suggested change

Bundle templates now generate serverless jobs and notebooks with `environment_version: "5"` instead of `"4"` (`default-python`, `default-scala`, `dbt-sql`, `pydabs`, and `lakeflow-pipelines`). For the Python templates, the generated `pyproject.toml` now pins `databricks-connect>=16.4,<16.5` and Python `>=3.12,<3.13` (previously `databricks-connect>=15.4,<15.5` and Python `>=3.10,<3.13`) so local development matches serverless environment version 5's Python 3.12 runtime.

Bundle templates are upgraded from serverless environment version 4 to [version 5](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/five) for generated serverless jobs and notebooks (`default-python`, `default-scala`, `dbt-sql`, `pydabs`, and `lakeflow-pipelines`). Python templates now pin `databricks-connect>=16.4,<16.5` and Python `>=3.12,<3.13` (previously `databricks-connect>=15.4,<15.5` and Python `>=3.10,<3.13`), aligning local development with environment version 5’s Python 3.12 runtime.

Done — applied (with the version bumped to 18.0).

yiweidai-db · 2026-08-26T03:53:57Z

 {{- end}}

 {{define "latest_lts_db_connect_version_spec" -}}
  >=15.4,<15.5


[question]

Do we need to upgrade this?

…e notes Per review, serverless environment version 5 ships DB Connect 18, so set the templates' DB Connect floor to >=18.0,<18.1 to match the runtime for client-server compatibility (18.0 also uses Python 3.12, so the Python pin is unchanged). Pinned to the 18.0 minor to avoid resolving a client newer than the v5 server. Also link the environment version 5 release notes from the changelog. Regenerate affected pyproject.toml goldens. Co-authored-by: Isaac <no-reply@databricks.com>

rugpanov and others added 2 commits August 25, 2026 16:10

rugpanov marked this pull request as ready for review August 25, 2026 13:29

lennartkats-db reviewed Aug 25, 2026

View reviewed changes

yiweidai-db reviewed Aug 26, 2026

View reviewed changes

Navigation Menu

Uh oh!

Bump default serverless environment version to 5 - #6378

Bump default serverless environment version to 5#6378
rugpanov wants to merge 4 commits into
mainfrom
serverless-env-version-v5

rugpanov commented Aug 25, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 25, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Aug 25, 2026 •

edited

Loading

Uh oh!

lennartkats-db Aug 25, 2026

Uh oh!

rugpanov Aug 25, 2026

Uh oh!

lennartkats-db Aug 25, 2026

Uh oh!

rugpanov Aug 25, 2026 •

edited

Loading

Uh oh!

lennartkats-db Aug 25, 2026

Uh oh!

rugpanov Aug 25, 2026 •

edited

Loading

Uh oh!

yiweidai-db Aug 26, 2026

Uh oh!

rugpanov Aug 26, 2026

Uh oh!

yiweidai-db Aug 26, 2026

Uh oh!

rugpanov Aug 26, 2026

Uh oh!

yiweidai-db Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -0,0 +1 @@
		Bundle templates now generate serverless jobs and notebooks with `environment_version: "5"` instead of `"4"` (`default-python`, `default-scala`, `dbt-sql`, `pydabs`, and `lakeflow-pipelines`). For the Python templates, the generated `pyproject.toml` now pins `databricks-connect>=16.4,<16.5` and Python `>=3.12,<3.13` (previously `databricks-connect>=15.4,<15.5` and Python `>=3.10,<3.13`) so local development matches serverless environment version 5's Python 3.12 runtime.

Uh oh!

Conversation

rugpanov commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

github-actions Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approval status: pending

/acceptance/bundle/ - needs approval

/acceptance/pipelines/ - needs approval

/libs/template/ - needs approval

General files (require maintainer)

Uh oh!

eng-dev-ecosystem-bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rugpanov Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rugpanov Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rugpanov commented Aug 25, 2026 •

edited

Loading

github-actions Bot commented Aug 25, 2026 •

edited

Loading

`/acceptance/bundle/` - needs approval

`/acceptance/pipelines/` - needs approval

`/libs/template/` - needs approval

eng-dev-ecosystem-bot commented Aug 25, 2026 •

edited

Loading

rugpanov Aug 25, 2026 •

edited

Loading

rugpanov Aug 25, 2026 •

edited

Loading