Prefect and Django

[abkfenris]

I figured out how to get Prefect and Django to play together nicely after connecting a few threads between Slack discussions and issues.

The two most important bits:

• Call django.setup() before importing your models.
• Use Local storage and stored_as_script=True to make sure that that django.setup() is run.
  • You may be able to get around that restriction by running django.setup() and importing your models within your tasks.

Other hiccups that I encountered or solutions I found:

• There is a difference between what RUNNER, TENANT and USER tokens can do, and they aren't nested (a USER can't do everything a RUNNER token can do, nor the other way around).
  • To register your code you need a TENANT token (which can be exposed to your code as from the PREFECT__CLOUD__AUTH_TOKEN environment variable).
  • To run an agent you need a RUNNER token (which can be exposed as PREFECT__CLOUD__AGENT__AUTH_TOKEN)
  • I initially tried a RUNNER token for both, which actually let me register but then threw an error.
• I made a Django management command to integrate Prefect better with Django and my workflow. This way I can call one command python manage.py prefect --register --agent and it will register the latest version of all my flows and start up an agent to run them.

Here's a PR where I added Prefect to an existing Django project of mine. It also includes configuration changes for running the registration and agent command under docker-compose and Kubernetes.