Switch default-python to use pyproject.toml & uv & dynamic_version in dev [hatchling backend] (#3042)

## Changes
- Update default-python template to use pyproject.toml and uv (with
hatchling).
- Get rid of dynamic version in setup.py and using dynamic_version
attribute on artifacts instead (only in dev target and only on
non-serverless clusters since serverless has fixed the issue with
versions not being updated in the platform). See
#2427
#2520

## Why
Modern Python is based around pyproject.toml, not setup.py. 

Using DAB-provided dynamic_version attribute allows keeping
pyproject.toml simple and does not constraint what build backend users
can choose.

Note, I've also considered using uv_build backend
#3004 but it seems too early at
the moment and hatchling is better tested with DABs.

## Tests
Existing tests.

Author: denik

NEXT_CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Release v0.258.0
 
 ### Notable Changes
+* Switch default-python template to use pyproject.toml + dynamic\_version in dev target. uv is now required. ([#3042](https://github.com/databricks/cli/pull/3042))
 
 ### Dependency updates
 * Upgraded TF provider to 1.84.0 ([#3151](https://github.com/databricks/cli/pull/3151))

acceptance/bundle/templates/default-python/classic/out.compare-vs-serverless.diff

@@ -1,3 +1,17 @@
+--- [TESTROOT]/bundle/templates/default-python/classic/../serverless/output/my_default_python/databricks.yml
++++ output/my_default_python/databricks.yml
+@@ -25,4 +25,11 @@
+       host: [DATABRICKS_URL]
+
++    presets:
++      # Set dynamic_version: true on all artifacts of type "whl".
++      # This makes "bundle deploy" add a timestamp to wheel's version before uploading,
++      # new wheel takes over the previous installation even if actual wheel version is unchanged.
++      # See https://docs.databricks.com/aws/en/dev-tools/bundles/settings
++      artifacts_dynamic_version: true
++
+   prod:
+     mode: production
 --- [TESTROOT]/bundle/templates/default-python/classic/../serverless/output/my_default_python/resources/my_default_python.job.yml
 +++ output/my_default_python/resources/my_default_python.job.yml
 @@ -17,4 +17,5 @@

acceptance/bundle/templates/default-python/classic/output/my_default_python/README.md

@@ -4,6 +4,8 @@ The 'my_default_python' project was generated by using the default-python templa
 
 ## Getting started
 
+0. Install UV: https://docs.astral.sh/uv/getting-started/installation/
+
 1. Install the Databricks CLI from https://docs.databricks.com/dev-tools/cli/databricks-cli.html
 
 2. Authenticate to your Databricks workspace, if you have not done so already:

acceptance/bundle/templates/default-python/classic/output/my_default_python/databricks.yml

@@ -4,6 +4,11 @@ bundle:
   name: my_default_python
   uuid: [UUID]
 
+artifacts:
+  python_artifact:
+    type: whl
+    build: uv build --wheel
+
 include:
   - resources/*.yml
   - resources/*/*.yml
@@ -19,6 +24,13 @@ targets:
     workspace:
       host: [DATABRICKS_URL]
 
+    presets:
+      # Set dynamic_version: true on all artifacts of type "whl".
+      # This makes "bundle deploy" add a timestamp to wheel's version before uploading,
+      # new wheel takes over the previous installation even if actual wheel version is unchanged.
+      # See https://docs.databricks.com/aws/en/dev-tools/bundles/settings
+      artifacts_dynamic_version: true
+
   prod:
     mode: production
     workspace:

acceptance/bundle/templates/default-python/classic/output/my_default_python/pyproject.toml

@@ -0,0 +1,41 @@
+[project]
+name = "my_default_python"
+version = "0.0.1"
+authors = [{ name = "[USERNAME]" }]
+requires-python = ">= 3.11"
+
+[project.optional-dependencies]
+dev = [
+    "pytest",
+
+    # Code completion support for DLT, also install databricks-connect
+    "databricks-dlt",
+
+    # databricks-connect can be used to run parts of this project locally.
+    # See https://docs.databricks.com/dev-tools/databricks-connect.html.
+    #
+    # Note, databricks-connect is automatically installed if you're using Databricks
+    # extension for Visual Studio Code
+    # (https://docs.databricks.com/dev-tools/vscode-ext/dev-tasks/databricks-connect.html).
+    #
+    # To manually install databricks-connect, uncomment the line below to install a version
+    # of db-connect that corresponds to the Databricks Runtime version used for this project.
+    # See https://docs.databricks.com/dev-tools/databricks-connect.html
+    # "databricks-connect>=15.4,<15.5",
+]
+
+[tool.pytest.ini_options]
+pythonpath = "src"
+testpaths = [
+    "tests",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/my_default_python"]
+
+[project.scripts]
+main = "my_default_python.main:main"

acceptance/bundle/templates/default-python/classic/output/my_default_python/pytest.ini

@@ -1 +0,0 @@
-__version__ = "0.0.1"

acceptance/bundle/templates/default-python/classic/output/my_default_python/requirements-dev.txt

@@ -1,6 +1,7 @@
 #!/usr/bin/env python3
 import sys
 import os
+import re
 import subprocess
 
 SERVERLESS = os.environ["SERVERLESS"] == "yes"
@@ -11,7 +12,7 @@
     sys.exit(f"SKIP_TEST SERVERLESS=yes but TEST_METASTORE_ID is empty in this env {CLOUD_ENV=}")
 
 BUILDING = "Building python_artifact"
-UPLOADING = "Uploading dist/"
+UPLOADING_WHL = re.compile("Uploading .*whl")
 STATE = "Updating deployment state"
 
 
@@ -21,7 +22,7 @@ def is_printable_line(line):
         return False
 
     # only shown when include_python=yes
-    if line.startswith(UPLOADING):
+    if UPLOADING_WHL.match(line):
         return False
 
     # not shown when all settings are equal to "no"
@@ -33,19 +34,19 @@ def is_printable_line(line):
 
 p = subprocess.run(sys.argv[1:], stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding="utf-8")
 try:
-    assert p.returncode == 0
+    assert p.returncode == 0, p.returncode
     assert p.stdout == ""
+    if INCLUDE_PYTHON:
+        assert BUILDING in p.stderr, BUILDING
+        assert UPLOADING_WHL.search(p.stderr), UPLOADING_WHL
+    else:
+        assert BUILDING not in p.stderr, BUILDING
+        assert not UPLOADING_WHL.search(p.stderr), UPLOADING_WHL
+
     for line in p.stderr.strip().split("\n"):
         if is_printable_line(line):
             print(line.strip())
 
-    if INCLUDE_PYTHON:
-        assert BUILDING in p.stderr
-        assert UPLOADING in p.stderr
-    else:
-        assert BUILDING not in p.stderr
-        assert UPLOADING not in p.stderr
-
 except:
     print(f"STDOUT: {len(p.stdout)} chars")
     if p.stdout: