Skip to content

dataforgoodfr/14_VCMWaterWatch

Repository files navigation

VCM Water Watch

Objectif

Créer une plateforme collaborative et scientifique pour cartographier et analyser les risques de pollution de l’eau par les CVM/VCM dans les tuyaux en PVC installés dans les années 70/80 en Europe. La plateforme permettra de visualiser les risques connus et potentiels, d’identifier les manques de données, et de stimuler la contribution citoyenne et institutionnelle à la recherche.

Stack Components

  • NocoDB = database frontend
  • Next.js = web frontend
  • Python with Prefect and DuckDB = data transformation scripts

Local Development Environment

Prerequisites

Installation

docker-compose up

This will start:

  • A postgres container
  • A nocodb container
  • a default configuration in .env, pointing to the local instance

You will need to restore a postgres dump - ask tech lead for one, and the corresponding NocoDB password.

You can access NocoDB at http://localhost:8500.

Windows installation

When installing uv, just and docker, even if corresponding files are added to PATH, Git Bash may not read Windows path correctly. Thus, use:

nano ~/.bashrc

This will open the bashrc file in which you can write:

export PATH="path/to/your/required/file:$PATH"
# Example:
# export PATH="/c/ProgramData/chocolatey/bin:$PATH"

Then press Ctrl+O to write out, Enter to save, and Ctrl+X to exit. These lines will not be executed when restarting the shell, so type:

nano ~/.bash_profile

to open the bash_profile file, where you can write:

if [ -f ~/.bashrc ]; then
  . ~/.bashrc
fi

Then press Ctrl+O to write out, Enter to save, and Ctrl+X to exit. This will indicate that if bashrc file exists it should be run and will always be executed when starting a new Bash shell.

Deployment

Pushes to the main branch will be deployed automatically using Coolify. The service is accessible at vcm-watch.eu.

Pipeline Worker

A lightweight FastAPI service runs alongside the webapp to handle data pipeline triggers.

  • NocoDB webhooks → the worker receives webhook POSTs over the Docker internal network
  • PMTiles generation → triggered automatically when Country or DistributionZone records change
  • Entity images mirroring → triggered on Country/Team changes; downloads NocoDB image attachments to a shared volume so the webapp can serve them from stable, cache-friendly URLs instead of short-lived S3 signed URLs (see docs/architecture.md § Country Images)
  • Shared volumespmtiles-data at /public/pmtiles and images-data at /public/images, both mounted by worker and webapp; each volume is served by a dedicated Next.js route handler (/pmtiles/[...path] and /images/[entity]/[...path]) reading its directory via an env var (PM_TILES_DIR / EXPORT_IMAGES_DIR)
  • Webhook URL (configured in NocoDB): http://worker:3000/webhooks/nocodb

To run the exports manually (local dev):

just pipelines export-pmtiles
just pipelines export-country-images
just pipelines export-team-images
# or both image exports at once:
just pipelines export-all-images

Adding new webhook triggers

Edit pipelines/worker/app.py to map additional NocoDB table changes to pipeline flows.

Git Conventions

Branch Naming

Branches follow the pattern: <username>/<type>/<issue_number>-<description>

  • username: short name identifying the developer (e.g., nicg, juho)
  • type: one of feat, bug, task
  • issue_number: GitHub issue number, when one exists
  • description: short kebab-case summary

Examples:

  • nicg/feat/21-country-pmtiles
  • juho/bug/prevent-bug-image-no-signed-url
  • nicg/task/document_conventions

Commit Messages

When a commit is linked to a GitHub issue, use: #<issue_number> | <description>

For minor or infrastructure changes without an issue, a plain imperative description is acceptable.

Examples:

  • #24 | database-independant db_helper
  • #13 | BFF setup for distributionZones search in client component
  • Add github hook to run python unit tests

Side note for CLI users: since # is the default comment character in Git, to be able to do commit messages in the CLI you will need to do something like git config core.commentChar \;

Python Data Transformations

See Pipelines Documentation

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors