Merge pull request #108 from NHSDigital/release_v09

Release v09

Author: stevenhsd

.mise.toml

@@ -1,4 +1,4 @@
 [tools]
 python="3.12"
-poetry="2.3.3"
+poetry="2.4.1"
 java="liberica-1.8.0"

CHANGELOG.md

@@ -1,5 +1,32 @@
+## v0.9.0 (2026-08-12)
+
+### Fix
+
+- alphanumeric and identifier type (#136)
+
+## v0.9.0rc0 (2026-07-16)
+
+### Fix
+
+- inherit is_mandatory with get_type_and_validators
+- adjusted the `is_type_complex` function to work with subclasses of 'simple' types
+
+### Refactor
+
+- upgrade pydantic to v2 (#107)
+
 ## v0.8.4 (2026-08-11)
 
+### Fix
+
+- ensure that format for times can be accessed and supplied in duckdb casting (#135)
+
+### Build
+
+- bump cryptography from 48.0.1 to 50.0.0 (#133)
+- bump pymdown-extensions from 10.21.3 to 11.0.1 (#134)
+
+
 ## v0.8.3 (2026-07-14)
 
 ### Fix

CONTRIBUTE.md CONTRIBUTING.mdCONTRIBUTE.md renamed to CONTRIBUTING.md

@@ -58,7 +58,7 @@ poetry install --with lint
 
 ## Installation for Development
 
-We are utilising Poetry for build dependency management and packaging. If you're on a system that has `Make` available, you can simply run `make install` to setup a local virtual environment with all the dependencies installed (this won't install Poetry for you).
+We are utilising Poetry for build dependency management and packaging. To install simply run `poetry install`. If you need to update any of the dependencies then you will need to run `poetry lock` before the install command. **Please always review the contents of the lock before installing. We have a `min-release-age` of `10` which means that packages will not be updated until they have been released for at least 10 days.
 
 ## Testing
 

Makefile

@@ -1,10 +1,5 @@
 activate = poetry run
 
-# dev
-install:
-	poetry lock
-	poetry install --with dev
-
 # dist
 wheel:
 	poetry build -f wheel

README.md

@@ -1,5 +1,5 @@
 <h1 style="display: flex; align-items: center; gap: 10px;">
-    <img src="https://github.com/NHSDigital/data-validation-engine/blob/616b55890306db4546177f7effac48ca241857ec/overrides/.icons/nhseng.svg" alt="" width="5%" height="100%" align="left">
+    <img src="https://raw.githubusercontent.com/NHSDigital/data-validation-engine/616b55890306db4546177f7effac48ca241857ec/overrides/.icons/nhseng.svg" alt="" width="5%" height="100%" align="left">
     Data Validation Engine
 </h1>
 
@@ -9,49 +9,27 @@
 [![CI Unit Tests](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_testing.yml/badge.svg)](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_testing.yml)
 [![CI Formatting & Linting](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_linting.yml/badge.svg)](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_linting.yml)
 
-The Data Validation Engine (DVE) is a configuration driven data validation library built and utilised by NHS England. Currently the package has been reverted from v1.0.0 release to a 0.x as we feel the package is not yet mature enough to be considered a 1.0.0 release. So please bear this in mind if reading through the commits and references to a v1+ release when on v0.x.
+The Data Validation Engine (DVE) is a configuration-driven data validation library created and used by NHS England. It lets users define validation rules once and apply them across multiple dataset collections - supporting consistent, accurate data checks.
 
-As mentioned above, the DVE is "configuration driven" which means the majority of development for you as a user will be building a JSON document to describe how the data will be validated. The JSON document is known as a `dischema` file and example files can be accessed [here](https://github.com/NHSDigital/data-validation-engine/tree/main/tests/testdata). If you'd like to learn more about JSON document and how to build one from scratch, then please read the documentation [here](https://nhsdigital.github.io/data-validation-engine/).
+__The DVE offers__:
 
-Once a dischema file has been defined, you are ready to use the DVE. The DVE is typically orchestrated based on four key "services". These are...
+- SQL configuration-based validations
+- Format normalization to Parquet for a unified data representation
+- Data modelling and typecasting
+- Business-rule validations executed on supported backends such as Spark and DuckDB, with the option to add custom backends
+- Deriving new fields and entities
+- Clear validation reporting, including summary insights and record-level error messages
 
-|    | Service | Purpose |
-| -- | ------- | ------- |
-| 1. | File Transformation | This service will take submitted files and turn them into stringified parquet file(s) to ensure that a consistent data structure can be passed through the other services. |
-| 2. | Data Contract | This service will validate and perform type casting against a stringified parquet file using [pydantic models](https://docs.pydantic.dev/1.10/). |
-| 3. | Business Rules | The business rules service will perform more complex validations such as comparisons between fields and tables, aggregations, filters etc to generate new entities. |
-| 4. | Error Reports | The error reports service will take all the errors raised in previous services and surface them into a readable format for a downstream users/service. Currently, this implemented to be an excel spreadsheet but could be reconfigured to meet other requirements/use cases. |
-
-If you'd like more detailed documentation around these services the please read the extended documentation [here](https://nhsdigital.github.io/data-validation-engine/).
-
-The DVE has been designed in a way that's modular and can support users who just want to utilise specific "services" from the DVE (i.e. just the file transformation + data contract). Additionally, the DVE is designed to support different backend implementations. As part of the base installation of DVE, you will find backend support for `Spark` and `DuckDB`. So, if you need a `MySQL` backend implementation, you can implement this yourself. Given our organisations requirements, it will be unlikely that we add anymore specific backend implementations into the base package beyond Spark and DuckDB. So, if you are unable to implement this yourself, I would recommend reading the guidance on [requesting new features and raising bug reports here](#requesting-new-features-and-raising-bug-reports).
-
-Additionally, if you'd like to contribute a new backend implementation into the base DVE package, then please look at the [Contributing](#Contributing) section.
+As mentioned above, the DVE is "configuration driven" which means the majority of development for you as a user will be building a JSON document to describe how the data will be validated. The JSON document is known as a `dischema` (data ingest schema) file and example files can be accessed [here ↗️](https://github.com/NHSDigital/data-validation-engine/tree/main/tests/testdata). If you'd like to learn more about JSON document and how to build one from scratch, then please read the documentation [here ↗️](https://nhsdigital.github.io/data-validation-engine/).
 
 ## Installation and usage
 
-The DVE is a Python package and can be installed using package managers such as [pip](https://pypi.org/project/pip/). As of the latest release we support Python 3.10 & 3.11, with Spark v3.4 and DuckDB v1.1. In the future we will be looking to upgrade the DVE to working on a higher versions of Python, DuckDB and Spark.
-
-If you're planning to use the Spark backend implementation, you will also need OpenJDK 11 installed.
-
-Python dependencies are listed in `pyproject.toml`.
-
-To install the DVE package you can simply install using a package manager such as [pip](https://pypi.org/project/pip/).
-
-```
-pip install data-validation-engine
-```
-
-*Note - Only versions >=0.6.2 are available on PyPi. For older versions please install directly from the git repo or build from source.*
-
-Once you have installed the DVE you are ready to use it. For guidance on how to create your dischema JSON document (configuration), please read the [documentation](https://nhsdigital.github.io/data-validation-engine/).
-
-Version 0.0.1 does support a working Python 3.7 installation. However, we will not be supporting any issues with that version of the DVE if you choose to use it. __Use at your own risk__.
+Please see the documentation [here ↗️](https://nhsdigital.github.io/data-validation-engine/user_guidance/install/).
 
 ## Requesting new features and raising bug reports
 **Before creating new issues, please check to see if the same bug/feature has been created already. Where a duplicate is created, the ticket will be closed and referenced to an existing issue.**
 
-If you have spotted a bug with the DVE then please raise an issue [here](https://github.com/nhsengland/Data-Validation-Engine/issues) using the "bug template". 
+If you have spotted a bug with the DVE then please raise an issue [here ↗️](https://github.com/nhsengland/Data-Validation-Engine/issues) using the "bug template". 
 
 If you have feature request then please follow the same process whilst using the "Feature request template".
 
@@ -63,7 +41,7 @@ Below is a list of features that we would like to implement or have been request
 | Uplift to Python 3.11                                                           | 0.2.0             | Yes       |
 | Uplift Pyspark to 3.5                                                           | 0.8.0             | Yes       |
 | Allow DVE to run on Python 3.12+                                                | 0.8.0             | Yes       |
-| Upgrade to Pydantic 2.0                                                         | 0.9.0             | No        |
+| Upgrade to Pydantic 2.0                                                         | 0.9.0             | Yes        |
 | Uplift Pyspark to 4.0+                                                          | TBA               | No        |
 | Polars upgrade to v1+ | TBA | No |
 | DuckDB upgrade to v1.5+ | TBA | No |
@@ -73,7 +51,7 @@ Below is a list of features that we would like to implement or have been request
 If you are interested in getting any of the unreleased features listed above available, then please read the [Contributing](#Contributing) section and then submit us a pull request.
 
 ## Contributing
-Please see guidance [here](https://github.com/NHSDigital/data-validation-engine/blob/main/CONTRIBUTE.md).
+Please see guidance [here ↗️](https://github.com/NHSDigital/data-validation-engine/blob/main/CONTRIBUTE.md).
 
 ## Legal
 This codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.

SECURITY.md

@@ -0,0 +1,23 @@
+# Security
+
+We take security and the protection of private data extremely seriously. If you believe you have found a vulnerability or other issue which has compromised or could compromise the security of any of our systems or private data managed by our systems, please do not hesitate to contact us using the method outlined below.
+
+## Table of contents
+
+- [Security](#security)
+- [Table of contents](#table-of-contents)
+- [Reporting a vulnerability](#reporting-a-vulnerability)
+- [General security enquiries](#general-security-enquiries)
+
+## Reporting a vulnerability
+
+If you believe you have found a security issue in this repository, please report it using GitHub's private vulnerability reporting:
+
+1. [Report a vulnerability](https://github.com/NHSDigital/data-validation-engine/security/advisories/new)
+2. Provide details of the issue and steps to reproduce
+
+This creates a private channel for discussion and allows us to coordinate a fix before any public disclosure.
+
+## General security enquiries
+
+If you have general enquiries regarding our cybersecurity, please reach out to us at [cybersecurity@nhs.net](cybersecurity@nhs.net)

docs/advanced_guidance/json_schemas.md

@@ -17,6 +17,12 @@ For autocomplete support in VS Code, you can alter the `.vscode/settings.json` a
             ],
             "url": "./json_schemas/dataset.schema.json"
         },
+        {
+            "fileMatch": [
+                "*_data_contract_error_details.json"
+            ],
+            "url": "./json_schemas/contract/error_details.schema.json"
+        },
         {
             "fileMatch": [
                 "*.rulestore.json",

docs/advanced_guidance/json_schemas/contract/components/base_entity.schema.json

@@ -1,5 +1,5 @@
 {
-    "$schema": "https://json-schema.org/draft-07/schema",
+    "$schema": "https://json-schema.org/draft/2020-12/schema",
     "$id": "data-ingest:contract/components/base_entity.schema.json",
     "title": "base_entity",
     "description": "A complete entity specification within data ingest",
@@ -10,7 +10,8 @@
             "description": "A mapping of field names to their Python types. These will either be strings representing Python types (if there are no argumements to the type), and field specification objects otherwise",
             "additionalProperties": {
                 "$ref": "field.schema.json"
-            }
+            },
+            "minProperties": 1
         },
         "aliases": {
             "description": "A mapping of field name to allowed field alias",
@@ -26,5 +27,7 @@
                 "type": "string"
             }
         }
-    }
+    },
+    
+    "required": ["fields"]
 }

docs/advanced_guidance/json_schemas/contract/components/contact_error_details.schema.json

@@ -1,5 +1,5 @@
 {
-    "$schema": "https://json-schema.org/draft-07/schema",
+    "$schema": "https://json-schema.org/draft/2020-12/schema",
     "$id": "data-ingest:contract/components/contract_error_details.schema.json",
     "title": "base_entity",
     "description": "A mapping of field names to the custom error code and message required if these fields were to fail validation during the data contract phase. For nested fields, these should be specified using struct '.' notation (eg. fieldA.fieldB.fieldC)",

docs/advanced_guidance/json_schemas/contract/components/entity.schema.json

@@ -1,5 +1,5 @@
 {
-    "$schema": "https://json-schema.org/draft-07/schema",
+    "$schema": "https://json-schema.org/draft/2020-12/schema",
     "$id": "data-ingest:contract/components/entity.schema.json",
     "title": "entity",
     "description": "A concrete entity specification within data ingest",