Skip to content

csse-uoft/ckanext-udc

BranchesTags

Repository files navigation

Tests Coverage Report

ckanext-udc

📊 View Coverage Report

Developer Manual

Testing Instructions

Config Schema

[   
    // The first level.
    {
        "title": "level 1 title",
        "name": "maturity_level_1",
        "fields": [
            {
                // Internal names, used in APIs. Special characters are not allowed.
                "name": "theme",

                // The Label of this field
                "label": "Domain / Topic",

                // The short description is under the field
                "short_description": "The theme.",

                // The long description is displayed when the cursor is hovered/clicked on the `quesion mark icon`
                "long_description": "The theme or topic of the package.",

                // The type of this field, default to "text".        
                // See "Supported field types" section for all supported types.
                "type": "text"
            },
            {
                // This clones the field that predefined in CKAN.
                // See "Supported CKAN Fields" section for all supported CKAN Fields
                "ckanField": "tags",

                // The short description is under the field
                "short_description": "The tags or keywords.",

                // The long description is displayed when the cursor is hovered/clicked on the `quesion mark icon`
                "long_description": "The tags or keywords of the package.",

            },
        ]
    },
    // The second level.
    {
        "title": "level 2 title",
        "name": "maturity_level_2",
        "fields": [
            {...}
        ]
    }
]

Supported CKAN Fields

Notes: short description and long description are not available for title, license, organization_and_visibility

  • title (*required): Title
  • description: Description
  • tags: Tags / Keywords
  • license_id: License
  • organization_and_visibility (*required): organization dropdown
  • source: url to the source
  • version: version number
  • author: Author/Creator
  • author_email
  • maintainer
  • maintainer_email
  • custom_fields: Custom CKAN Fields, key/value pairs

Supported field types

Notes: If type is not provided, it will default to text.

  • text
  • number
  • date
  • datetime
  • time
  • single_select 
    {
    "name": "access_diff_version",
    "label": "Can different versions of the data be accessed?",
    "type": "single_select",
    // A list of available options.
    // The "value" must be a string.
    "options": [
        {"text": "N/A", "value": ""},
        {"text": "Yes", "value": "true"},
        {"text": "No", "value": "false"}
    ]
}

Requirements

If your extension works across different versions you can add the following table:

Compatibility with core CKAN versions:

CKAN version Compatible?
2.6 and earlier not tested
2.7 not tested
2.8 not tested
2.9 not tested
2.10.X yes
2.11.X yes

Suggested values:

  • "yes"
  • "not tested" - I can't think of a reason why it wouldn't work
  • "not yet" - there is an intention to get it working
  • "no"

Installation

TODO: Add any additional install steps to the list below. For example installing any non-Python dependencies or adding any required config settings.

To Install NodeJS 20:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
# Reopen terminal or following the guide in the output to add 'nvm'

# Install NodeJS 20
nvm install 20

To install ckanext-udc:

  1. Activate your CKAN virtual environment, for example:

    . /usr/lib/ckan/default/bin/activate

  2. Clone the source and install it on the virtualenv

    cd /usr/lib/ckan/default/src/
git clone https://github.com/csse-uoft/ckanext-udc.git
cd ckanext-udc
pip install -e .
pip install -r requirements.txt

  3. Add udc udc_theme udc_import udc_import_other_portals udc_react to the ckan.plugins setting in your CKAN config file (by default the config file is located at /etc/ckan/default/ckan.ini).

    If you want to use the activity plugin, put it to the top (order matters).

    Adjust ckan.jobs.timeout to 36000 (10 hours)

  4. Compiling assets and translation files

    cd /usr/lib/ckan/default/src/ckanext-udc
python setup.py compile_catalog
ckan -c /etc/ckan/default/ckan.ini translation js
ckan -c /etc/ckan/default/ckan.ini asset build

  5. Install UDC-React Dependencies & Build

    cd /usr/lib/ckan/default/src/ckanext-udc/ckanext/udc_react/ckan-udc-react
npm install
npm run build

  6. Init DB

    # Use your own path to the ckan.ini
ckan -c /etc/ckan/default/ckan.ini udc initdb

  7. Restart CKAN. For example if you've deployed CKAN with Apache on Ubuntu:

    sudo supervisorctl reload

Scheduled imports

udc_import_other_portals scheduled imports are registered via rq-scheduler. Saving or updating an import config re-syncs the schedule automatically, but scheduled imports only run if both the CKAN worker and rqscheduler are running.

Local testing: worker + rqscheduler

Start the local development processes described in Run as a developer, including both the CKAN worker and rqscheduler.

Then, in the import UI, save an import config with a schedule that will fire soon, for example * * * * * for every minute, and confirm on the next tick that:

  • the worker logs show the scheduled job being picked up;
  • a new import log entry is created in the UDC import UI;
  • dataset changes appear as expected.

Notes:

  • The Redis URL above matches the current CKAN config at /etc/ckan/default/ckan.ini via ckan.redis.url. If your deployment uses a different Redis instance, set RQ_REDIS_URL to that value instead.
  • You do not need to restart rqscheduler after every cron change. Saving the import config re-syncs the scheduled job automatically.

Running the Scheduler as a Service on Ubuntu

On Ubuntu deployments, the simplest approach is to run rqscheduler under Supervisor alongside the CKAN worker.

  1. Make sure the extension and its Python requirements are installed in the CKAN virtualenv.

  2. Verify the command works manually first:

    source /usr/lib/ckan/default/bin/activate
RQ_REDIS_URL=redis://localhost:6379/0 rqscheduler --interval 60

  3. Add a Supervisor program entry for the scheduler, for example in /etc/supervisor/conf.d/ckan-rqscheduler.conf:

    [program:ckan-rqscheduler]
command=/bin/bash -lc 'source /usr/lib/ckan/default/bin/activate && RQ_REDIS_URL=redis://localhost:6379/0 rqscheduler --interval 60'
directory=/usr/lib/ckan/default/src
user=www-data
autostart=true
autorestart=true
startsecs=5
stopasgroup=true
killasgroup=true
stdout_logfile=/var/log/ckan/rqscheduler.log
stderr_logfile=/var/log/ckan/rqscheduler.err.log
environment=HOME="/var/lib/ckan"

  4. Reload Supervisor and start the service:

    sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start ckan-rqscheduler

  5. Check that it stays up and is emitting heartbeats:

    sudo supervisorctl status ckan-rqscheduler
tail -f /var/log/ckan/rqscheduler.log

Configure deployment server to support websocket connection and use Gevent for multi-tasking

  1. Reinstall uwsgi with SSL support

    sudo apt-get install libssl-dev

source /usr/lib/ckan/default/bin/activate

# Uninstall previous version of `uwsgi` if exists
pip uninstall uwsgi

## Manually build uwsgi with SSL support
# set necessary lib paths
export CFLAGS="-I/usr/include/openssl"
# aarch64-linux-gnu folder used for ARM architecture and may be different for your env
# use [apt-file list libssl-dev] to check lib folders (apt-file should be additionally installed)
export LDFLAGS="-L/usr/lib/aarch64-linux-gnu"
# activate SSL support
export UWSGI_PROFILE_OVERRIDE=ssl=true
# build uwsgi using pip (--no-use-wheel deprecated so used --no-binary instead)
# this command will install 2.0.20 version. Version may be changed or removed. It is not mandatory
pip install -I --no-binary=:all: --no-cache-dir uwsgi

# Check SSL support
uwsgi --help | grep https

  2. Add to /etc/ckan/default/ckan-uwsgi.ini

    Remove enable-threads and threads if exists, threading is not compitable with gevent.

    gevent          =  1000 # number of coroutine
http-websockets = true
gevent-monkey-patch = true
log-x-forwarded-for = true
processes = 6

  3. Update nginx config sudo nano /etc/nginx/sites-enabled/ckan

    Add the following after location / {...}

    location /socket.io/ {
    proxy_pass http://127.0.0.1:8080/socket.io/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Host $host;
}

  4. Copy wsgi.py to apply monkey patching

    cp /usr/lib/ckan/default/src/ckanext-udc/ckanext/udc/wsgi.py /etc/ckan/default/wsgi.py

Config settings

None at present

Run as a developer

For scheduled imports, run four processes locally: CKAN, the React frontend, one CKAN worker, and one rqscheduler process.

  • CKAN main procress 
    cd /etc/ckan/default
source /usr/lib/ckan/default/bin/activate
CKAN_INI=/etc/ckan/default/ckan.ini VITE_ORIGIN=http://<your-server-ip>:5173 WERKZEUG_DEBUG_PIN=223344 uwsgi --http :5000 --gevent 1000 --http-websockets --master --wsgi-file /etc/ckan/default/wsgi.py --callable application --py-autoreload=1
  • React Frontend 
    cd /usr/lib/ckan/default/src/ckanext-udc/ckanext/udc_react/ckan-udc-react/
npm run dev -- --host 0.0.0.0
  • CKAN worker process 
    source /usr/lib/ckan/default/bin/activate
WERKZEUG_DEBUG_PIN=223344 ckan -c /etc/ckan/default/ckan.ini jobs worker
  • rqscheduler process 
    source /usr/lib/ckan/default/bin/activate
RQ_REDIS_URL=redis://localhost:6379/0 rqscheduler --verbose --interval 5

Tests

To run the tests, do:

pytest --ckan-ini=test.ini

Releasing a new version of ckanext-udc

If ckanext-udc should be available on PyPI you can follow these steps to publish a new version:

  1. Update the version number in the setup.py file. See PEP 440 for how to choose version numbers.

  2. Make sure you have the latest version of necessary packages:

    pip install --upgrade setuptools wheel twine

  3. Create a source and binary distributions of the new version:

    python setup.py sdist bdist_wheel && twine check dist/*

    Fix any errors you get.

  4. Upload the source distribution to PyPI:

    twine upload dist/*

  5. Commit any outstanding changes:

    git commit -a
git push

  6. Tag the new release of the project on GitHub with the version number from the setup.py file. For example if the version number in setup.py is 0.0.1 then do:

    git tag 0.0.1
git push --tags

License

AGPL

About

No description, website, or topics provided.

Resources

Readme

License

AGPL-3.0 license
Activity
Custom properties

Stars

1 star

Watchers

3 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages