Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions NEXT_CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

* `bundle generate job` now downloads workspace files referenced by `spark_python_task`, rewriting them to a relative path like it already does for notebooks. Git-sourced files and cloud URIs are left untouched ([#5799](https://github.com/databricks/cli/pull/5799)).
* Fix permissions added to a job or pipeline by a Python (PyDABs) mutator failing to deploy with "must have exactly one owner"; the deploying identity is now set as owner, matching resources whose permissions are declared in YAML ([#5821](https://github.com/databricks/cli/pull/5821)).
* `bundle generate` now warns when the generated configuration file is not matched by any pattern in the `include` section of `databricks.yml` ([#XXXX](https://github.com/databricks/cli/pull/XXXX)).

### Dependency updates

Expand Down
8 changes: 8 additions & 0 deletions acceptance/bundle/generate/alert/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,11 @@
>>> [CLI] bundle generate alert --existing-id [ALERT_ID] --source-dir out/alert --config-dir out/resource
Alert configuration successfully saved to [TEST_TMP_DIR]/out/resource/test_alert.alert.yml
Serialized alert definition to [TEST_TMP_DIR]/out/alert/test_alert.dbalert.json
Warning: Generated configuration is not included in the bundle

The file out/resource/test_alert.alert.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- out/resource/*.yml

8 changes: 8 additions & 0 deletions acceptance/bundle/generate/app_not_yet_deployed/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,11 @@
>>> [CLI] bundle generate app --existing-app-name my-app --config-dir . --key out
Loading app 'my-app' configuration
App configuration successfully saved to out.app.yml
Warning: Generated configuration is not included in the bundle

The file out.app.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- *.yml

8 changes: 8 additions & 0 deletions acceptance/bundle/generate/app_subfolders/output.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
Loading app 'my_app' configuration
File successfully saved to out/sub/folder/1.py
App configuration successfully saved to out.app.yml
Warning: Generated configuration is not included in the bundle

The file out.app.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- *.yml

8 changes: 8 additions & 0 deletions acceptance/bundle/generate/dashboard/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,11 @@
>>> [CLI] bundle generate dashboard --existing-id [DASHBOARD_ID] --dashboard-dir out/dashboard --resource-dir out/resource
Writing dashboard to out/dashboard/test_dashboard.lvdash.json
Writing configuration to out/resource/test_dashboard.dashboard.yml
Warning: Generated configuration is not included in the bundle

The file out/resource/test_dashboard.dashboard.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- out/resource/*.yml

Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,23 @@
>>> [CLI] bundle generate dashboard --existing-path /path/to/dashboard --dashboard-dir out/dashboard --resource-dir out/resource
Writing dashboard to out/dashboard/this_is_a_test_dashboard.lvdash.json
Writing configuration to out/resource/this_is_a_test_dashboard.dashboard.yml
Warning: Generated configuration is not included in the bundle

The file out/resource/this_is_a_test_dashboard.dashboard.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- out/resource/*.yml


>>> [CLI] bundle generate dashboard --existing-path /path/to/dashboard --key custom_key --dashboard-dir out/dashboard --resource-dir out/resource
Writing dashboard to out/dashboard/custom_key.lvdash.json
Writing configuration to out/resource/custom_key.dashboard.yml
Warning: Generated configuration is not included in the bundle

The file out/resource/custom_key.dashboard.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- out/resource/*.yml

8 changes: 8 additions & 0 deletions acceptance/bundle/generate/designer_job/output.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,10 @@
File successfully saved to outnotebook.designer.ipynb
Job configuration successfully saved to out.job.yml
Warning: Generated configuration is not included in the bundle

The file out.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- *.yml

8 changes: 8 additions & 0 deletions acceptance/bundle/generate/genie_space/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,11 @@
>>> [CLI] bundle generate genie-space --existing-id [GENIE_SPACE_ID] --genie-space-dir out/genie_space --resource-dir out/resource
Writing genie space to out/genie_space/test_genie_space.geniespace.json
Writing configuration to out/resource/test_genie_space.genie_space.yml
Warning: Generated configuration is not included in the bundle

The file out/resource/test_genie_space.genie_space.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- out/resource/*.yml

8 changes: 8 additions & 0 deletions acceptance/bundle/generate/git_job/output.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,10 @@
Job is using Git source, skipping downloading files
Job configuration successfully saved to out.job.yml
Warning: Generated configuration is not included in the bundle

The file out.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- *.yml

5 changes: 5 additions & 0 deletions acceptance/bundle/generate/include_warning/databricks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
bundle:
name: include_warning

include:
- resources/*.yml
3 changes: 3 additions & 0 deletions acceptance/bundle/generate/include_warning/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

16 changes: 16 additions & 0 deletions acceptance/bundle/generate/include_warning/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@

>>> [CLI] bundle generate job --existing-job-id 1234 --config-dir resources --key included
Job is using Git source, skipping downloading files
Job configuration successfully saved to resources/included.job.yml

>>> [CLI] bundle generate job --existing-job-id 1234 --config-dir subdir --key excluded
Job is using Git source, skipping downloading files
Job configuration successfully saved to subdir/excluded.job.yml
Warning: Generated configuration is not included in the bundle

The file subdir/excluded.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- subdir/*.yml

11 changes: 11 additions & 0 deletions acceptance/bundle/generate/include_warning/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# A resource generated into a directory matched by the 'include' section
# is deployed, so no warning is emitted.
trace $CLI bundle generate job --existing-job-id 1234 --config-dir resources --key included

# A resource generated into a directory that no 'include' pattern matches is
# written to disk but silently ignored during deploy, so we warn about it.
trace $CLI bundle generate job --existing-job-id 1234 --config-dir subdir --key excluded

# The generated config files are not asserted on here; remove them so they are
# not reported as unexpected leftover files.
rm -r resources subdir
24 changes: 24 additions & 0 deletions acceptance/bundle/generate/include_warning/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
[[Server]]
Pattern = "GET /api/2.2/jobs/get"
Response.Body = '''
{
"job_id": 11223344,
"settings": {
"name": "gitjob",
"git_source": {
"git_url": "https://git.databricks.com",
"git_provider": "github",
"git_branch": "main",
"git_commit": "abcdef"
},
"tasks": [
{
"task_key": "test_task",
"notebook_task": {
"notebook_path": "some/test/notebook.py"
}
}
]
}
}
'''
8 changes: 8 additions & 0 deletions acceptance/bundle/generate/ipynb_job/output.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,10 @@
File successfully saved to outnotebook.ipynb
Job configuration successfully saved to out.job.yml
Warning: Generated configuration is not included in the bundle

The file out.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- *.yml

8 changes: 8 additions & 0 deletions acceptance/bundle/generate/job_nested_notebooks/output.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
File successfully saved to src/my_folder/my_notebook.py
File successfully saved to src/other_folder/other_notebook.py
Job configuration successfully saved to out.job.yml
Warning: Generated configuration is not included in the bundle

The file out.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- *.yml

=== old flattened files should be gone ===
src/my_notebook.py removed
src/other_notebook.py removed
Expand Down
8 changes: 7 additions & 1 deletion acceptance/bundle/generate/job_nested_notebooks/script
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,13 @@ mkdir -p src
echo "old" > src/my_notebook.py
echo "old" > src/other_notebook.py

$CLI bundle generate job --existing-job-id 1234 --config-dir . --key out --force --source-dir src 2>&1 | sort
# The concurrent downloader emits the "File successfully saved" lines in
# nondeterministic order, so sort just those; the remaining lines (including the
# multi-line include warning) are deterministic and kept in their natural order.
$CLI bundle generate job --existing-job-id 1234 --config-dir . --key out --force --source-dir src > out.gen.txt 2>&1
grep '^File successfully saved to' out.gen.txt | sort
grep -v '^File successfully saved to' out.gen.txt
rm out.gen.txt

echo "=== old flattened files should be gone ==="
test ! -f src/my_notebook.py && echo "src/my_notebook.py removed" || echo "src/my_notebook.py still exists"
Expand Down
8 changes: 8 additions & 0 deletions acceptance/bundle/generate/lakeflow_pipelines/output.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
Pipeline configuration successfully saved to out/config/out.pipeline.yml
Warning: Generated configuration is not included in the bundle

The file out/config/out.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- out/config/*.yml

File successfully saved to out/pipeline/explorations/1.py
File successfully saved to out/pipeline/transformations/1.py
File successfully saved to out/pipeline/transformations/2.py
8 changes: 8 additions & 0 deletions acceptance/bundle/generate/pipeline/output.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,10 @@
File successfully saved to outnotebook.py
Pipeline configuration successfully saved to out.pipeline.yml
Warning: Generated configuration is not included in the bundle

The file out.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- *.yml

8 changes: 8 additions & 0 deletions acceptance/bundle/generate/pipeline_and_deploy/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@
=== Generate bundle config from the pipelineFile successfully saved to src/notebook.py
File successfully saved to src/test.py
Pipeline configuration successfully saved to resources/out.pipeline.yml
Warning: Generated configuration is not included in the bundle

The file resources/out.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- resources/*.yml


=== Verify generated yaml has expected fields
=== Deploy the generated bundle
Expand Down
8 changes: 7 additions & 1 deletion acceptance/bundle/generate/pipeline_and_deploy/script
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,13 @@ cleanup() {
trap cleanup EXIT

title "Generate bundle config from the pipeline"
$CLI bundle generate pipeline --existing-pipeline-id "$PIPELINE_ID" --key out --config-dir resources --source-dir src --force 2>&1 | sort
# The concurrent downloader emits the "File successfully saved" lines in
# nondeterministic order, so sort just those; the remaining lines (including the
# multi-line include warning) are deterministic and kept in their natural order.
$CLI bundle generate pipeline --existing-pipeline-id "$PIPELINE_ID" --key out --config-dir resources --source-dir src --force > out.gen.txt 2>&1
grep '^File successfully saved to' out.gen.txt | sort
grep -v '^File successfully saved to' out.gen.txt
rm out.gen.txt

title "Verify generated yaml has expected fields"
cat resources/out.pipeline.yml | env -u MSYS_NO_PATHCONV contains.py "libraries:" "- notebook:" "path: ../src/notebook.py" "- file:" "path: ../src/test.py" > /dev/null
Expand Down
8 changes: 8 additions & 0 deletions acceptance/bundle/generate/pipeline_with_glob/output.txt
Original file line number Diff line number Diff line change
@@ -1 +1,9 @@
Pipeline configuration successfully saved to out.pipeline.yml
Warning: Generated configuration is not included in the bundle

The file out.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- *.yml

8 changes: 8 additions & 0 deletions acceptance/bundle/generate/pipeline_with_sql/output.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,10 @@
File successfully saved to outsql.sql
Pipeline configuration successfully saved to out.pipeline.yml
Warning: Generated configuration is not included in the bundle

The file out.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- *.yml

8 changes: 8 additions & 0 deletions acceptance/bundle/generate/python_job/output.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,10 @@
File successfully saved to outnotebook.py
Job configuration successfully saved to out.job.yml
Warning: Generated configuration is not included in the bundle

The file out.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- *.yml

8 changes: 8 additions & 0 deletions acceptance/bundle/generate/python_job_and_deploy/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,14 @@
>>> [CLI] bundle generate job --existing-job-id [JOB_ID] --key out --config-dir resources --source-dir src --force
File successfully saved to src/test_notebook.py
Job configuration successfully saved to resources/out.job.yml
Warning: Generated configuration is not included in the bundle

The file resources/out.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- resources/*.yml


=== Verify generated yaml has expected fields
=== Deploy the generated bundle
Expand Down
8 changes: 8 additions & 0 deletions acceptance/bundle/generate/spark_python_task_job/output.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,10 @@
File successfully saved to outjob.py
Job configuration successfully saved to out.job.yml
Warning: Generated configuration is not included in the bundle

The file out.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
so it will not be deployed. Add a matching entry to the 'include' section, for example:

include:
- *.yml

2 changes: 2 additions & 0 deletions cmd/bundle/generate/alert.go
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,8 @@ After generation, you can deploy this alert to other targets using:
cmdio.LogString(ctx, "Alert configuration successfully saved to "+filepath.ToSlash(configPath))
cmdio.LogString(ctx, "Serialized alert definition to "+filepath.ToSlash(alertPath))

warnIfNotIncluded(ctx, b, configPath)

return nil
}

Expand Down
2 changes: 2 additions & 0 deletions cmd/bundle/generate/app.go
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,8 @@ per target environment.`,

cmdio.LogString(ctx, "App configuration successfully saved to "+filename)

warnIfNotIncluded(ctx, b, filename)

if bind {
return deployment.BindResource(cmd, appKey, app.Name, true, false, true)
}
Expand Down
2 changes: 2 additions & 0 deletions cmd/bundle/generate/dashboard.go
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,8 @@ func (d *dashboard) saveConfiguration(ctx context.Context, b *bundle.Bundle, das
return err
}

warnIfNotIncluded(ctx, b, resourcePath)

return nil
}

Expand Down
2 changes: 2 additions & 0 deletions cmd/bundle/generate/genie_space.go
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,8 @@ func (g *genieSpace) saveConfiguration(ctx context.Context, b *bundle.Bundle, ge
return err
}

warnIfNotIncluded(ctx, b, resourcePath)

return nil
}

Expand Down
Loading