[Python] Add volumes support #3383
Conversation
kanterov
requested review from
andrewnester,
anton-107,
denik,
pietern and
shreyas-goenka
as code owners
kanterov
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
| from dataclasses import replace | ||
|
|
||
| from databricks.bundles.core import volume_mutator | ||
| from databricks.bundles.catalog import Volume |
There was a problem hiding this comment.
JSON schema volumes are located in the "catalog" namespace. We need to decide whether to vendor them into a separate namespace (e.g. databricks.bundles.volumes) or co-locate them with "catalog" objects.
There was a problem hiding this comment.
@pietern I think databricks.bundles.volumes will be a better option. We should keep each resource contained. What do you think?
There was a problem hiding this comment.
I think following the schema makes it easier to navigate for someone already familiar with any of the SDKs, no?
What's the advantage of separate module per each resource?
There was a problem hiding this comment.
We are switching to a model where each service will have an independent set of classes. That is how SDK v1, which is replacing SDK v0, is built. So, if we want to be consistent with it, we should follow the same approach. It simplifies making major/breaking changes because they can be isolated to a particular namespace.
There was a problem hiding this comment.
+1 to that. Prefer separate namespace over using catalog now and having to retrofit one later when the SDK moves on. Mental model is also simpler, as the "boundary" here is the resource.
|
kanterov
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
kanterov
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
| from dataclasses import replace | ||
|
|
||
| from databricks.bundles.core import volume_mutator | ||
| from databricks.bundles.catalog import Volume |
There was a problem hiding this comment.
I think following the schema makes it easier to navigate for someone already familiar with any of the SDKs, no?
What's the advantage of separate module per each resource?
| from dataclasses import replace | ||
|
|
||
| from databricks.bundles.core import volume_mutator | ||
| from databricks.bundles.catalog import Volume |
There was a problem hiding this comment.
+1 to that. Prefer separate namespace over using catalog now and having to retrofit one later when the SDK moves on. Mental model is also simpler, as the "boundary" here is the resource.
kanterov
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
kanterov
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
kanterov
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
kanterov
force-pushed
the
add-volumes
branch
3 times, most recently
from
da1e280 to
dbd122c
Compare
kanterov
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
## Changes Add volume resource type support to Python ## Why It makes it possible to define volumes with Python code, similar to jobs and pipelines ## Tests Acceptance and unit tests
## Release v0.266.0 ### Notable Changes * Breaking change: DABs now return an error when paths are incorrectly defined relative to the job or pipeline definition location instead of the configuration file location. Previously, the CLI would show a warning and fallback to resolving the path relative to the resource location. Users must update their bundle configurations to define all relative paths relative to the configuration file where the path is specified. See more details here: ([#3225](#3225)) * Add support volumes in Python support ([#3383])(#3383)) ### Bundles * [Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines ([#3225](#3225)) * Add support for Lakebase synced database tables in DABs ([#3467](#3467)) * Rename Delta Live Tables to Lakeflow Declarative Pipelines in the default-python template ([#3476](#3476)). * Fixed bundle init not working on Standard tier ([#3496](#3496))
Changes
Add volume resource type support to Python
Why
It makes it possible to define volumes with Python code, similar to jobs and pipelines
Tests
Acceptance and unit tests