Skip to content

DEVELOPMENT.md

Latest commit

 

History

History
247 lines (194 loc) · 11.3 KB

DEVELOPMENT.md

File metadata and controls

247 lines (194 loc) · 11.3 KB

Developing YNR locally

This is a guide to the tools and workflows that you'll need to use when changing and updating YNR on your development machine.

Installation

Install and test the development prerequisites as detailed in INSTALL.md. This will leave you with a known-good setup that's able to run the commands described in this guide.

Quick start

Restoring data

If you have access to the YNR prod AWS account, you can grab an anonymised dump from prod and restore it to a containerised database as follows:

  1. Start the database container: podman compose up -d dbpsql
  2. Restore the database dump: 
    AWS_PROFILE=ynr-prod ./scripts/get-prod-db.sh
  3. Apply any pending migrations: ./scripts/container.run.bash python manage.py migrate
  4. Ensure the cache table exists: ./scripts/container.manage-py.bash createcachetable
  5. Shut down the database container: podman compose down

Running the app

  1. Add or update any environment variables in .env as required.
  2. Start the compose stack: podman compose up -d
  3. (In a separate terminal) Start tailing the stack's logs: podman compose logs --follow (you can safely CTRL-C this process at any time).
  4. (If your active ynr/settings/... file does NOT include DEBUG = True)
    Collect the static assets: ./scripts/container.run.bash python manage.py collectstatic --no-input
  5. Browse to http://localhost:8080
  6. Changes made inside ynr/ will be immediately available to the app, which will be auto-reloaded.
  7. Remember to shut down the compose stack when you're done: podman compose down

Testing your changes

If the app is capable of being started

  1. Start the compose stack: podman compose up -d
  2. Run the test suite: ./scripts/container.pytest.bash
    • We can provide additional pytest options. For example, to run the test suite and stop on the first failure: ./scripts/container.pytest.bash -x
  3. Stop the compose stack: podman compose down

If the app can't be started, and you need to run the test suite to figure out why

  1. Start the compose stack's database server: podman compose up -d dbpsql
  2. Run the entire test suite in a new container (without starting the app): ./scripts/container.run.bash pytest
  3. Stop the compose stack: podman compose down

Running Django management commands

As detailed later in this guide, there are several different ways to run a command inside a frontend container. The method described here uses a dedicated script to invoke a Django management command inside a frontend container that's already running the webapp. (If you need to run a command but don't want to start the webapp, use the more general container.run.bash script instead).

Run a Django management command:

  1. Add or update any environment variables in .env as required.
  2. Start the compose stack: podman compose up -d
  3. Use the container.manage-py.bash script to invoke the command: 
    # ./scripts/container.manage-py.bash command-to-invoke --command-args command params
# e.g.
./scripts/container.manage-py.bash check
  4. Stop the compose stack: podman compose down

After you stop the compose stack, any files added or changed by the management command inside the ynr directory will be persisted directly on your machine. The same applies to any files mentioned in docker-compose.yml, in the frontend container's "volumes" section. Any changes the management command makes to files outside those locations will be lost when you stop the compose stack. Changes to the database are persisted in the database's data volume.

Working with Podman

YNR uses a container runtime called Podman. You can think of it as like Docker, but able to run without needing a persistent background daemon and without requiring root access. Podman provides the podman command which is intended to be CLI-compatible with much of the docker command.

You've also been asked to install podman-compose - a separate project that gives the podman command its podman compose subcommand. You shouldn't need to invoke podman-compose (with a hyphen) directly. The podman compose command works with the "compose stack" defined in docker-compose.yml, which comprises two services: the frontend webapp, and the Postgres dbpsql service.

Working on the webapp

Use podman compose up -d to start the compose stack, with the webapp exposed on localhost:8080. Changes you make inside the ynr/ directory are automatically reflected in the running app. Changes to other entries in the frontend's docker-compose.yml list of volumes that are bind-mounted from your local checkout of this repo are also immediately visible to the running app.

View the app (and DB) logs with podman compose logs --follow. It's a good idea to run this immediately after starting the stack.

Shut down the stack with podman compose down. This is always safe to run, even when the stack is already stopped. It deliberately leaves the database's data behind as a "volume", so that Postgres can access it the next time you start the stack. If you need to delete the database's contents completely, run podman compose down --volumes.

podman compose ... subcommands do provide the expected --help flag, but some of the docs aren't perfect. Here's a summary of the commands you might run:

Scheduled Tasks

In production, we run scheduled jobs with

If you want to set your dev copy up to run scheduled tasks:

  1. Run ./scripts/container.manage-py.bash setup_periodic_tasks. This will create DB records for the scheduled tasks we define in code. If we change the content of the @register_task decorator we need to re-run this to update the DB. Changing the code inside the decorated function doesn't require running setup_periodic_tasks.
  2. Set ENABLE_SCHEDULED_JOBS = True. This is disabled by default because we usually don't want our local copy ruuning scheduled tasks
Command                                                   Purpose Notes
podman compose up -d Start the entire stack. -d forks the action into the background, which is optional but strongly recommended.
podman compose up -d dbpsql Start only the named container in the stack.
podman compose down Stop any running containers in the stack.
podman compose down --volumes Stop any running containers in the stack and also destroy their persistent data. Anything bind-mounted from your local repo into the frontend webapp container is left untouched.
podman compose ps Display the status of containers in the stack.
podman volume ls List the persistent volumes that podman controls on your machine.
podman compose build Rebuild the webapp's frontend container image.
podman compose build --no-cache Rebuild the webapp's frontend container image from scratch. Takes several minutes to finish.
podman compose logs Display the last N stdout/stderr lines emitted by any running containers.
podman compose logs --follow Display the last N stdout/stderr lines emitted by any running containers, and then wait for more lines.
podman system reset Destroy everything that Podman controls. "Everything seems to have gone wrong, so I'll just start from scratch". It wipes out all containers, networks, images, volumes, etc ... so avoid this if possible!

Scripts

These executable scripts are available from the scripts directory.

Script Purpose Parameters
container.image.build.bash Builds the YNR container image $1 -- The named stage from container/build/Containerfile to build and tag (required)
$2, $3, ... -- Any parameters to pass to the underlying builder process (optional)
container.exec.bash Runs a command inside the already-running frontend container The unquoted command to run (required)
container.manage-py.bash Runs a Django management command inside the already-running frontend container The unquoted command to run (required)
container.pytest.bash Runs pytest inside the already-running frontend container Any parameters for Pytest (optional)
container.run.bash Runs a command inside a freshly-instantiated, ephemeral frontend container The unquoted command to run (required)

Rebuilding the application container image

You will need to rebuild the application's container image if you change any of the application's dependencies, across any of the packaging ecosystems it currently relies on:

  • container/build/system-packages: System / APT dependendencies
  • package{,-lock}.json: Node dependencies
  • requirements/*.txt: Python dependencies
  • .dockerignore: Container build-time file dependencies

The above list is presented in descending order of how slow a rebuild will be, if a particular package ecosystem's dependencies are changed. Changing a system dependency, for example, forces a longer rebuild than changing a Python dependency. You do not need to rebuild the application's container image if you only change files in the ynr/ directory. Changes to the YNR application are picked up automatically when using the compose stack locally (as described elsewhere in this guide).

The build process for the YNR application is encoded in container/build/Containerfile. This Docker-compatible file describes two image stages, prod and test, with test being built on top of prod. Locally, on your development machine, you will need to use the test stage.

Build the test stage using a build cache

./scripts/container.image.build.bash test

Build the test stage without a build cache

Avoiding the use of your local build cache significantly increases the time it takes to build the container image, but is sometimes useful when there's a problem with external dependencies (e.g. if a important update has been published for an APT package but it's not visible in the container's package index).

./scripts/container.image.build.bash test --no-cache

Podman [compose] fails

If there are problems using the podman compose (with space) command, first check you have podman-compose==v1.2.0 installed:

podman-compose --version

If you're still having problems, one option is to burn everything down and start again:

podman down
podman system migrate
podman system reset
podman system disable
systemctl stop podman.socket
systemctl disable podman.socket
systemctl --user start podman.socket
systemctl --user start podman

Once this is done, try running the podman compose commands again.