Production-readiness refactor: drop MinIO, make Crate IDs more robust, add async validation and offline cache

[alexhambley]

TLDR

This is a big refactor that strengthens the validation service by adding:

• An agnostic S3 storage layer;
• Reworking the "Crate ID" model;
• Adding consistent result/error types;
• Adding a faster metadata path;
• Fixing some major production concerns (retries, structured logging, health checks, offline validation);
• Removing reliance on minio-py SDK

PR

I've broken down the changes in more detail below:

Storage and RO-Crates

• Added a "vendor-neutral" StorageBackend abstraction-layer with a boto3 S3Backend (this should work with AWS / MinIO / RustFS / etc.) and an in-memory fake for tests. minio-py SDK is removed;
• Added stricter/safer Crate-ID validation (charset/length, no / or ..). This fixes the bugs/fragility with .zip-in-ID, by treating IDs as opaque;
• Added deterministic crate resolution by direct key checks and explicit zip vs directory, ambiguous (409) and not-found (404) raised;
• Set a canonical layout with separate crate and result prefixes.

Validation

• There is now a single ValidationOutcome type (valid/invalid/error) that replaces the old ValidationResult | str; one boundary to rocrate-validator;
• Metadata-only validation is now synchronous (removes the blocking result.get());
• Bumped rocrate-validator to 0.10.0 for (opt-in) offline validation. Cache warming at build, and --extra-profiles-path.

APIs

• ID endpoints take server-side credentials, so there are no storage secrets in request bodies/logs!
• Resolve-before-queue;
• The celery worker runs a staged fetch → validate → persist → webhook with retries and persisted error outcomes;
• Webhook delivery retries with backoff and surfaces any failures.

DevOps Stuff

• JSON logging with request IDs and secret redaction;
• /healthz and /readyz added to check the health and readiness for the storage + Redis broker ;
• I've tidied up dependencies. pyproject.toml + pip-tools locks, ruff lint/format, CI updated;
• There is now a single parameterised Dockerfile;
• Flask entrypoint is renamed cratey.py → wsgi.py for convention;
• Image and labels renamed to ro-crate-validation-service;
• The README.md has been rewritten and has some architecture/flow mermaid diagrams to help future external collaborators with understanding how the validator works;
• Tests are now organised properly to mirror app/.

Breaking Changes

• Environment variables are renamed;
• ID endpoints no longer accept minio_config / root_path.
• Status codes are clearer
• Storage layout is explicit: crates under crates/, results under validation-results/.
• Requires rocrate-validator >= 0.10.0

Closes #170

[elichad]

@alexhambley to make this slightly more manageable to review, are you able to pop the last three commits (particularly the last one because it touches many files) off into a separate PR that depends on this one?

[AnnieZQH]

I was able to bring the dev stack up locally and successfully tested the API endpoints. I've also read through the changes and the tests. The coverage report looks good.

The only issue I've found: the renamed image is not on eScienceLab's container registry yet, so using docker-compose.yml would throw an error.

[AnnieZQH]

This package does not exist yet.

[douglowe]

I've a few build comments, and some small swagger api comments.

I do wish that this PR was a series of separate PR's for each feature. I've not reviewed the async validation and offline cache changes yet, because I've been checking other changes.

I like the health checks - I think these are a good addition. And the making crate ID's more robust seems sensible. As does moving to more generic S3 libraries.

I am concerned about the removal of the S3 secrets from the API call. These were added at the explicit request of our project partners, because they access S3 services using ephemeral secrets. We should have a design conversation about this, to decide if this is something that needs to be supported or not.

Regarding the logging checks - I agree we should try to avoid recording secrets in the logs. This will be more difficult if we are allowing ephemeral secrets to be used, as we won't be able to directly grep for known secrets. But we could add field name redaction, or entropy / length checks, so that most secrets which might be passed are screened out.

[douglowe]

Within the RSE team we're moving to specifying the exact hash of the action we want to use, rather than only the version number (c.f.: https://github.com/UoMResearchIT/Actions/blob/20365fa75ab643043ec188ef0b67fd5e537d326c/reuse/action.yml#L67). Should we do the same here?

[douglowe]

Should we pin our pytest, requests, boto3 versions here? The most recent tests for this PR ran with pytest-9.1.0, requests-2.33.1, and boto3-1.43.29

[douglowe]

We can, and should, organise this RUN command better:

RUN <<EOF_PROFILES
if [ -n "$PROFILES_ARCHIVE_URL" ]; then
   mkdir -p /app/extra-profiles
   wget -O /tmp/profiles.tar.gz "$PROFILES_ARCHIVE_URL"
   tar -xzf /tmp/profiles.tar.gz
            -C /app/extra-profiles \
            --strip-components=3 \
            "rocrate-validator-${FIVE_SAFES_PROFILE_VERSION}/rocrate_validator/profiles/five-safes-crate"
   rm /tmp/profiles.tar.gz
fi
EOF_PROFILES

[douglowe]

Same as above, let's do this better:

RUN <<<EOF_PROFILES_PATH
if [ -n "$EXTRA_PROFILES_PATH" ]; then
    rocrate-validator cache warm --all-profiles \
            --extra-profiles-path "$EXTRA_PROFILES_PATH" --cache-path "$CACHE_PATH"
else
    rocrate-validator cache warm --all-profiles --cache-path "$CACHE_PATH"
fi
EOF_PROFILES_PATH

[douglowe]

Do we need the project name string here? (-p cratey_integration)

[douglowe]

This class was originally added because of a request from the Nottingham team. They manage access to MinIO using temporary access and secret keys - so requested that we have such a schema in the API for access to the object store. If we move these back into the docker compose file then we are locking each validator instance to only accessing a single S3 store, and potentially blocking compatibility of this tool with the 5S-TES stack.

Can we discuss this change with our collaborators, to make sure we are moving in the correct direction here?

[douglowe]

It's annoying me that the swagger docs have 2 post routes (a small problem I know, but I think it makes the swagger docs look less coherent).

We could combine these by adding the same tag to each of the APIBlueprint calls. e.g.:

post_routes_bp = APIBlueprint("post_routes", __name__, tag="Post_Routes")
...
minio_post_routes_bp = APIBlueprint("minio_post_routes", __name__, tag="Post_Routes")