Skip to content

Commit fe57001

Browse files
MatthiasValvekensMatthiasValvekens
committed
Enable coverage tracking
- Only run coverage on ubuntu-latest with Python 3.12 (Cython line tracing seems to have some issues with Python 3.13, to be determined) - There are a few paradoxical red spots in the Cython coverage analysis (code that must run for other blocks to be green, but nevertheless showing up as red) but the analysis does give a good indication of where the blind spots are already. - Factor out test setup into its own action with different PKCS#11 platforms to target, to minimise divergence between tests.yml and coverage.yml. Some divergence is unavoidable due to the fact that the coverage run needs an instrumented build of the C extension, and we still want the regular CI run to work with settings that are somewhat close to the actual release build.
1 parent 20e3b48 commit fe57001

10 files changed

Lines changed: 396 additions & 51 deletions

File tree

‎.github/actions/test-setup/action.yml‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
name: test-setup
2+
author: Matthias Valvekens
3+
description: Perform set-up for python-pkcs11 CI
4+
inputs:
5+
os:
6+
description: OS to target
7+
required: true
8+
python-version:
9+
description: Python version to target
10+
required: true
11+
dependency-group:
12+
description: UV dependency group to install
13+
required: true
14+
pkcs11-platform:
15+
description: PKCS#11 platform to target
16+
required: true
17+
token-label:
18+
description: Label assigned to the token
19+
required: true
20+
token-user-pin:
21+
description: User PIN to configure on the token
22+
required: true
23+
token-so-pin:
24+
description: Security officer PIN to configure on the token
25+
required: true
26+
outputs:
27+
module:
28+
description: Path to PKCS#11 module
29+
value: ${{ steps.install-result.outputs.module }}
30+
module2:
31+
description: Path to alternative PKCS#11 module ('multi' only)
32+
value: ${{ steps.install-result.outputs.module2 }}
33+
runs:
34+
using: "composite"
35+
steps:
36+
- name: Setup Python
37+
uses: actions/setup-python@v5
38+
with:
39+
python-version: ${{ inputs.python-version }}
40+
- uses: ./.github/actions/install-softhsm
41+
if: inputs.pkcs11-platform == 'softhsm' || inputs.pkcs11-platform == 'multi'
42+
id: softhsm
43+
with:
44+
os: ${{ inputs.os }}
45+
token-label: ${{ inputs.token-label }}
46+
token-so-pin: ${{ inputs.token-so-pin }}
47+
token-user-pin: ${{ inputs.token-user-pin }}
48+
- uses: ./.github/actions/install-opencryptoki
49+
# only run opencryptoki tests on ubuntu
50+
# (macos and windows don't seem to be supported)
51+
if: inputs.pkcs11-platform == 'opencryptoki' || inputs.pkcs11-platform == 'multi'
52+
id: opencryptoki
53+
with:
54+
os: ${{ inputs.os }}
55+
token-label: ${{ inputs.token-label }}
56+
token-so-pin: ${{ inputs.token-so-pin }}
57+
token-user-pin: ${{ inputs.token-user-pin }}
58+
- name: Set module path
59+
id: install-result
60+
shell: bash
61+
run: |
62+
if [[ "$PLATFORM" == 'opencryptoki' ]]; then
63+
echo "module=${{ steps.opencryptoki.outputs.module }}" >> "$GITHUB_OUTPUT"
64+
elif [[ "$PLATFORM" == 'softhsm' ]]; then
65+
echo "module=${{ steps.softhsm.outputs.module }}" >> "$GITHUB_OUTPUT"
66+
elif [[ "$PLATFORM" == 'multi' ]]; then
67+
# NOTE: the 'multi' platform is only used for testing the code that
68+
# swaps between multiple PKCS#11 implementations. As such, any two
69+
# PKCS#11 implementations will do. If we add a 3rd platform
70+
# to the CI at a later stage that is faster to install than opencryptoki,
71+
# switching is an option.
72+
echo "module=${{ steps.softhsm.outputs.module }}" >> "$GITHUB_OUTPUT"
73+
echo "module2=${{ steps.opencryptoki.outputs.module }}" >> "$GITHUB_OUTPUT"
74+
else
75+
echo "$PLATFORM is not a valid PKCS#11 platform choice"
76+
exit 1
77+
fi
78+
env:
79+
PLATFORM: ${{ inputs.pkcs11-platform }}
80+
- name: Install uv
81+
uses: astral-sh/setup-uv@v4
82+
with:
83+
enable-cache: true
84+
python-version: ${{ inputs.python-version }}
85+
- name: Install testing dependencies
86+
shell: bash
87+
run: uv sync --no-dev --exact --group "${{ inputs.dependency-group }}"

‎.github/workflows/coverage.yml‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
name: Coverage
2+
on:
3+
pull_request: {}
4+
workflow_dispatch: {}
5+
env:
6+
UV_PYTHON_PREFERENCE: only-system
7+
UV_NO_SYNC: "1"
8+
PKCS11_TOKEN_LABEL: "TEST"
9+
PKCS11_TOKEN_PIN: "1234"
10+
PKCS11_TOKEN_SO_PIN: "5678"
11+
jobs:
12+
# For now, we run the coverage as a separate job.
13+
# At the time of writing, the latest version of Cython's line tracing
14+
# seems to lead to segfaults in Python 3.13 -> TODO: investigate
15+
pytest-coverage:
16+
runs-on: ubuntu-latest
17+
strategy:
18+
matrix:
19+
pkcs11-platform:
20+
- softhsm
21+
- opencryptoki
22+
steps:
23+
- name: Acquire sources
24+
uses: actions/checkout@v4
25+
- name: Arm coverage-only compiler directives
26+
# Unfortunately, it doesn't seem to be possible to pass directives
27+
# to Cython through environment variables: https://github.com/cython/cython/issues/3930
28+
# Doing it here is still better than introducing a non-declarative setup.py into the
29+
# build again.
30+
run: sed -i 's/#coverage#cython/#cython/g' pkcs11/*.pyx
31+
- uses: ./.github/actions/test-setup
32+
id: setup
33+
with:
34+
os: ubuntu-latest
35+
python-version: "3.12"
36+
dependency-group: coverage
37+
token-label: ${{ env.PKCS11_TOKEN_LABEL }}
38+
token-so-pin: ${{ env.PKCS11_TOKEN_SO_PIN }}
39+
token-user-pin: ${{ env.PKCS11_TOKEN_PIN }}
40+
pkcs11-platform: ${{ matrix.pkcs11-platform }}
41+
env:
42+
CFLAGS: "-DCYTHON_TRACE_NOGIL=1"
43+
EXT_BUILD_DEBUG: "1"
44+
- name: Run tests
45+
run: uv run pytest -v --cov=pkcs11 --cov-branch --cov-report=xml:${{ matrix.pkcs11-platform }}-coverage.xml
46+
env:
47+
PKCS11_MODULE: ${{ steps.setup.outputs.module }}
48+
- name: Stash coverage report
49+
uses: actions/upload-artifact@v4
50+
with:
51+
name: coverage-${{ strategy.job-index }}
52+
path: "*-coverage.xml"
53+
pytest-coverage-multilib:
54+
runs-on: ubuntu-latest
55+
steps:
56+
- name: Acquire sources
57+
uses: actions/checkout@v4
58+
- uses: ./.github/actions/test-setup
59+
id: setup
60+
with:
61+
os: ubuntu-latest
62+
pkcs11-platform: multi
63+
token-label: ${{ env.PKCS11_TOKEN_LABEL }}
64+
token-so-pin: ${{ env.PKCS11_TOKEN_SO_PIN }}
65+
token-user-pin: ${{ env.PKCS11_TOKEN_PIN }}
66+
python-version: "3.12"
67+
dependency-group: coverage
68+
- name: Run tests
69+
run: uv run pytest -v --cov=pkcs11 --cov-branch --cov-report=xml:multilib-coverage.xml tests/test_multilib.py
70+
env:
71+
PKCS11_MODULE: ${{ steps.setup.outputs.module }}
72+
PKCS11_MODULE2: ${{ steps.setup.outputs.module2 }}
73+
- name: Stash coverage report
74+
uses: actions/upload-artifact@v4
75+
with:
76+
name: coverage-multilib
77+
path: "*-coverage.xml"
78+
codecov-upload:
79+
permissions:
80+
actions: write
81+
contents: read
82+
runs-on: ubuntu-latest
83+
needs: [pytest-coverage]
84+
steps:
85+
# checkout necessary to ensure the uploaded report contains the correct paths
86+
- uses: actions/checkout@v4
87+
- name: Retrieve coverage reports
88+
uses: actions/download-artifact@v4
89+
with:
90+
pattern: coverage-*
91+
path: ./reports/
92+
- name: Upload all coverage reports to Codecov
93+
uses: codecov/codecov-action@v5
94+
with:
95+
token: ${{ secrets.CODECOV_TOKEN }}
96+
directory: ./reports/
97+
flags: unittests
98+
env_vars: OS,PYTHON
99+
name: codecov-umbrella

‎.github/workflows/tests.yml‎

Lines changed: 43 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ env:
1111
PKCS11_TOKEN_PIN: "1234"
1212
PKCS11_TOKEN_SO_PIN: "5678"
1313
jobs:
14-
run:
14+
tests:
1515
runs-on: ${{ matrix.os }}
1616
strategy:
1717
# Our test suite is pretty fast, so fail-fast: false allows for better troubleshooting.
@@ -27,46 +27,59 @@ jobs:
2727
- "3.11"
2828
- "3.12"
2929
- "3.13"
30+
pkcs11-platform:
31+
- softhsm
32+
- opencryptoki
33+
exclude:
34+
# only run opencryptoki tests on ubuntu
35+
# (macos and windows don't seem to be supported)
36+
- pkcs11-platform: opencryptoki
37+
os: windows-latest
38+
- pkcs11-platform: opencryptoki
39+
os: macos-latest
3040
steps:
3141
- name: Acquire sources
3242
uses: actions/checkout@v4
33-
34-
- name: Setup Python
35-
uses: actions/setup-python@v5
36-
with:
37-
python-version: ${{ matrix.python-version }}
38-
- uses: ./.github/actions/install-softhsm
39-
id: softhsm
43+
- uses: ./.github/actions/test-setup
44+
id: setup
4045
with:
4146
os: ${{ matrix.os }}
4247
token-label: ${{ env.PKCS11_TOKEN_LABEL }}
4348
token-so-pin: ${{ env.PKCS11_TOKEN_SO_PIN }}
4449
token-user-pin: ${{ env.PKCS11_TOKEN_PIN }}
45-
- uses: ./.github/actions/install-opencryptoki
46-
# only run opencryptoki tests on ubuntu
47-
# (macos and windows don't seem to be supported)
48-
if: matrix.os == 'ubuntu-latest'
49-
id: opencryptoki
50+
python-version: ${{ matrix.python-version }}
51+
pkcs11-platform: ${{ matrix.pkcs11-platform }}
52+
dependency-group: testing
53+
- name: Run tests
54+
run: uv run pytest -v
55+
env:
56+
PKCS11_MODULE: ${{ steps.setup.outputs.module }}
57+
multilib-tests:
58+
runs-on: ubuntu-latest
59+
strategy:
60+
fail-fast: false
61+
matrix:
62+
python-version:
63+
- "3.9"
64+
- "3.10"
65+
- "3.11"
66+
- "3.12"
67+
- "3.13"
68+
steps:
69+
- name: Acquire sources
70+
uses: actions/checkout@v4
71+
- uses: ./.github/actions/test-setup
72+
id: setup
5073
with:
51-
os: ${{ matrix.os }}
74+
os: ubuntu-latest
75+
pkcs11-platform: multi
5276
token-label: ${{ env.PKCS11_TOKEN_LABEL }}
5377
token-so-pin: ${{ env.PKCS11_TOKEN_SO_PIN }}
5478
token-user-pin: ${{ env.PKCS11_TOKEN_PIN }}
55-
- name: Install uv
56-
uses: astral-sh/setup-uv@v4
57-
with:
58-
enable-cache: true
5979
python-version: ${{ matrix.python-version }}
60-
- name: Install testing dependencies
61-
run: uv sync --no-dev --exact --group testing
62-
- name: Run tests with SoftHSM
63-
run: uv run pytest -v
64-
env:
65-
PKCS11_MODULE: ${{ steps.softhsm.outputs.module }}
66-
- name: Run tests with opencryptoki
67-
if: matrix.os == 'ubuntu-latest'
68-
run: uv run pytest -v
80+
dependency-group: testing
81+
- name: Run tests
82+
run: uv run pytest -v tests/test_multilib.py
6983
env:
70-
PKCS11_MODULE: ${{ steps.opencryptoki.outputs.module }}
71-
# For testing logic around swapping PKCS#11 libs
72-
PKCS11_MODULE2: ${{ steps.softhsm.outputs.module }}
84+
PKCS11_MODULE: ${{ steps.setup.outputs.module }}
85+
PKCS11_MODULE2: ${{ steps.setup.outputs.module2 }}

‎.gitignore‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,7 @@ __pycache__
66
/dist/
77
/docs/_build
88
/python_pkcs11.egg-info/
9-
/.eggs/
9+
/.eggs/
10+
.coverage
11+
*coverage.xml
12+
*.html

‎pkcs11/_pkcs11.pxd‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@ Definitions imported from PKCS11 C headers.
44

55
from cython.view cimport array
66

7-
from pkcs11.defaults import *
87
from pkcs11.exceptions import *
98

109
cdef extern from '../extern/cryptoki.h':
@@ -638,10 +637,11 @@ cdef inline CK_ULONG_buffer(length):
638637
return array(shape=(length,), itemsize=sizeof(CK_ULONG), format='L')
639638

640639

641-
cdef inline object map_rv_to_error(
642-
CK_RV rv
643-
):
640+
# Note: this `cdef inline` declaration doesn't seem to be consistently labelled
641+
# as executed by Cython's line tracing, so we flag it as nocover
642+
# to avoid noise in the metrics.
644643

644+
cdef inline object map_rv_to_error(CK_RV rv): # pragma: nocover
645645
if rv == CKR_ATTRIBUTE_TYPE_INVALID:
646646
exc = AttributeTypeInvalid()
647647
elif rv == CKR_ATTRIBUTE_VALUE_INVALID:

‎pkcs11/_pkcs11.pyx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
#!python
21
#cython: language_level=3
2+
#coverage#cython: linetrace=True
33
"""
44
High-level Python PKCS#11 Wrapper.
55

‎pyproject.toml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,12 +67,35 @@ archs = ["universal2"]
6767
[tool.setuptools.packages.find]
6868
include = ["pkcs11*"]
6969

70+
[tool.coverage.run]
71+
plugins = ["Cython.Coverage"]
72+
73+
[tool.coverage.report]
74+
exclude_lines = [
75+
"pragma: no cover",
76+
"pragma: nocover",
77+
"raise AssertionError",
78+
"raise NotImplementedError",
79+
"raise MemoryError",
80+
"raise TypeError",
81+
"TYPE_CHECKING",
82+
"^\\s*\\.\\.\\.",
83+
"noqa"
84+
]
85+
precision = 2
86+
7087
[dependency-groups]
7188
testing = [
7289
"cryptography>=44.0.0",
7390
"parameterized>=0.9.0",
7491
"pytest>=8.3.4",
7592
]
93+
coverage = [
94+
{ include-group = "testing" },
95+
"coverage>=7.9.1",
96+
"pytest-cov>=4.0,<6.3",
97+
"cython",
98+
]
7699
docs = [
77100
"sphinx>=7.4.7",
78101
"sphinx-rtd-theme>=3.0.2",

‎tests/test_multilib.py‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
"""
2+
PKCS#11 Slots and Tokens
3+
"""
4+
5+
import os
6+
import unittest
7+
8+
import pkcs11
9+
10+
from . import LIB
11+
12+
13+
@unittest.skipUnless("PKCS11_MODULE2" in os.environ, "Requires an additional PKCS#11 module")
14+
class MultilibTests(unittest.TestCase):
15+
def test_double_initialise_different_libs(self):
16+
lib1 = pkcs11.lib(LIB)
17+
lib2 = pkcs11.lib(os.environ["PKCS11_MODULE2"])
18+
self.assertIsNotNone(lib1)
19+
self.assertIsNotNone(lib2)
20+
self.assertIsNot(lib1, lib2)
21+
22+
slots1 = lib1.get_slots()
23+
slots2 = lib2.get_slots()
24+
25+
self.assertGreaterEqual(len(slots1), 1)
26+
self.assertGreaterEqual(len(slots2), 1)

0 commit comments

Comments
 (0)