DLT init reusing lakeflow-pipelines template #2
Conversation
shreyas-goenka
left a comment
shreyas-goenka
left a comment
There was a problem hiding this comment.
Have we considered creating a new independent template for this? Being surgical here can be more complex than just having a new template.
For example: The Readme in lakeflow-pipelines template has a bunch of references to the databricks that you would probably like to change to dlt.
| "is_dlt": { | ||
| "skip_prompt_if": {}, | ||
| "type": "boolean", | ||
| "description": "DLT pipelines using this template", |
There was a problem hiding this comment.
| "description": "DLT pipelines using this template", | |
| "description": "Whether this template is initialized via the DLT CLI", |
| # This is a Databricks asset bundle definition for {{.project_name}}. | ||
| # See https://docs.databricks.com/dev-tools/bundles/index.html for documentation. | ||
| bundle: | ||
| {{$isDLT := eq .is_dlt true}}{{if $isDLT}}project{{else}}bundle{{end}}: |
There was a problem hiding this comment.
| {{$isDLT := eq .is_dlt true}}{{if $isDLT}}project{{else}}bundle{{end}}: | |
| {{- $isDLT := eq .is_dlt true}} | |
| {{if $isDLT}}project{{else}}bundle{{end}}: |
Please play a bit with the - here to make this a bit easier to read. The - chops a way all whitespace.
| include: | ||
| - resources/*.yml | ||
| - resources/*/*.yml | ||
| {{if $isDLT}}- ./*.yml{{else}}- resources/*.yml |
There was a problem hiding this comment.
The dlt template should include both
| # See also https://docs.databricks.com/dev-tools/bundles/deployment-modes.html. | ||
| mode: development | ||
| default: true | ||
| {{if $isDLT}}deploy_on_run: true{{end}} |
There was a problem hiding this comment.
This would leave a blank line when false; please experiment with {{- :)
| permissions: | ||
| {{if not $isDLT}}# We explicitly deploy to /Workspace/Users/{{user_name}} to make sure we only have a single copy. | ||
| root_path: /Workspace/Users/{{user_name}}/.bundle/${bundle.name}/${bundle.target}}{{end}} | ||
| {{if $isDLT}}owner: user@company.com{{else}}permissions: |
There was a problem hiding this comment.
Please keep in mind that the template won't yet work if you replace the root_path with the owner as I originally suggested. The CLI would first need to support that.
So you should probably sequence this change to be later.
alyssa-db
force-pushed
the
install-dlt-feat
branch
3 times, most recently
from
515956f to
1da3ff0
Compare
3 participants
Changes
New
dlt initcommand that generates a pipeline specific template.Adding new field to schema that allows for reusing lakeflow-pipelines template for DLT
Why
Allow for
dlt initcommand to reuse templateTests
Acceptance tests pending but locally generates 2 sets of files if using
databricks bundle initordlt init