CK_FUNCTION_LIST has incorrect ABI on Windows

[G1gg1L3s]

Hi! While playing with the library on Windows, I discovered an ABI bug which produced segfaults when a program used functions from C_GetFunctionList.

Compilation

To compile the library, I used mingw toolchain on linux with the following dockerfile (it includes woflssl and wolftpm as well, but it's not that related):

Details

FROM debian:bookworm-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
        git \
        ca-certificates \
        build-essential \
        autoconf \
        automake \
        libtool \
        pkg-config \
        mingw-w64

WORKDIR /src

RUN git clone --branch v5.9.2-stable https://github.com/wolfSSL/wolfssl.git
RUN git clone --branch v4.1.0 https://github.com/wolfSSL/wolfTPM.git
RUN git clone --branch v2.1.0-stable https://github.com/wolfSSL/wolfPKCS11.git

ENV HOST=x86_64-w64-mingw32 \
    PREFIX=/opt/wolf-win \
    TOOLCHAIN=/opt/mingw-w64-x86_64.cmake \
    WIN_CFLAGS="-DWIN32 -DMINGW -D_WIN32_WINNT=0x0600" \
    PKG_CONFIG_PATH=/opt/wolf-win/lib/pkgconfig

RUN cat > "$TOOLCHAIN" <<'EOF'
set(CMAKE_SYSTEM_NAME Windows)
set(CMAKE_SYSTEM_PROCESSOR x86_64)
set(TC x86_64-w64-mingw32)
set(CMAKE_C_COMPILER   ${TC}-gcc)
set(CMAKE_CXX_COMPILER ${TC}-g++)
set(CMAKE_RC_COMPILER  ${TC}-windres)
set(CMAKE_FIND_ROOT_PATH /usr/${TC} $ENV{PREFIX})
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY BOTH)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE BOTH)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE BOTH)
EOF

RUN apt-get install -y --no-install-recommends cmake

# --- 1. Build WolfSSL
RUN cd wolfssl && \
    ./autogen.sh && \
    ./configure \
        --host="$HOST" CC="$HOST-gcc" \
        --prefix="$PREFIX" \
        --enable-wolftpm \
        --enable-aescfb \
        --enable-rsapss \
        --enable-keygen \
        --enable-pwdbased \
        --enable-scrypt \
        --enable-debug-code-points \
        CFLAGS="$WIN_CFLAGS" \
        LDFLAGS="-static-libgcc" \
        C_EXTRA_FLAGS="-DWOLFSSL_PUBLIC_MP -DWC_RSA_DIRECT" \
        LIBS="-lws2_32" && \
    make -j"$(nproc)" && \
    make install


# --- 2. Build woflTPM
RUN cd wolfTPM && \
    ./autogen.sh && \
    ./configure \
        --host="$HOST" CC="$HOST-gcc" \
        --prefix="$PREFIX" \
        --enable-winapi \
        --disable-examples \
        CPPFLAGS="-I$PREFIX/include" \
        LDFLAGS="-L$PREFIX/lib -static-libgcc" \
        CFLAGS="$WIN_CFLAGS -DDEBUG_WOLFTPM" \
        LIBS="-lws2_32" && \
    make -j"$(nproc)" && \
    make install

# --- 3. Build wolfPKCS11
RUN mkdir wolfPKCS11/build && \
    cd wolfPKCS11/build && \
    cmake \
        -DCMAKE_TOOLCHAIN_FILE="$TOOLCHAIN" \
        -DCMAKE_INSTALL_PREFIX="$PREFIX" \
        -DCMAKE_PREFIX_PATH="$PREFIX" \
        -DCMAKE_BUILD_TYPE=Release \
        -DBUILD_SHARED_LIBS=ON \
        -DWOLFPKCS11_TPM=yes \
        -DWOLFPKCS11_DH=no \
        -DWOLFPKCS11_AESKEYWRAP=yes \
        -DWOLFPKCS11_EXAMPLES=no \
        -DWOLFPKCS11_TESTS=no \
        -DWOLFPKCS11_DEBUG=yes \
        -DCMAKE_SHARED_LINKER_FLAGS="-L$PREFIX/lib -L$PREFIX/bin -static-libgcc" \
        ..  && \
    cmake --build . && \
    cmake --install .

Note: I had to use cmake for wolfPKCS11 because it's was not possible to disable examples and tests, but they failed with the mingw toolchain.

The build passed, but the resulting libwolfpkcs11.dll doesn't work with some programs.

Specifically, when I used pkcs11-tool from OpenSC, it failed:

pksc11-tool.exe --module C:\tpm\libwolfpkcs11.dll -I
echo $LASTEXITCODE
-1073741819

The cause

I pinpointed issue to the C_GetFunctionList. It provides a pointer to the CK_FUNCTION_LIST, which is a structure with a version header and a list of functions. However, it seems like on Windows this structure is packed. For example, here is a byte dump of this structure from opensc-pkcs11.dll:

header
|
|     | address #1
|     |                       | address #2
02 14 dc be 8e 24 fe 7f 00 00 fc c0 8e 24 fe 7f 00 00 c8 be 8e 24 fe 7f 00 00 38 c2 8e 24 fe 7f
00 00 50 c2 8e 24 fe 7f 00 00 b8 c4 8e 24 fe 7f 00 00 f4 cc 8f 24 fe 7f 00 00 04 c7 8e 24 fe 7f
00 00 74 c7 8e 24 fe 7f 00 00 e0 c7 8e 24 fe 7f 00 00 58 d2 8e 24 fe 7f 00 00 ac d8 8e 24 fe 7f
00 00 f8 d5 8e 24 fe 7f 00 00 b0 d0 8e 24 fe 7f 00 00 58 d0 8e 24 fe 7f 00 00 f8 d0 8e 24 fe 7f
...

The version is just two bytes. This is what rust cryptoki expects on windows for example.

And here is what libwolfpkcs11.dll produces:

| Version
|                       | address #1
|                       |                       | address #2
02 28 00 00 00 00 00 00 e0 42 f2 62 fe 7f 00 00 10 46 f2 62 fe 7f 00 00 80 44 f2 62 fe 7f 00 00
e0 43 f2 62 fe 7f 00 00 40 47 f2 62 fe 7f 00 00 c0 48 f2 62 fe 7f 00 00 90 4a f2 62 fe 7f 00 00
30 4e f2 62 fe 7f 00 00 a0 51 f2 62 fe 7f 00 00 20 59 f2 62 fe 7f 00 00 00 5c f2 62 fe 7f 00 00
40 5e f2 62 fe 7f 00 00 b0 61 f2 62 fe 7f 00 00 f0 63 f2 62 fe 7f 00 00 20 65 f2 62 fe 7f 00 00
40 66 f2 62 fe 7f 00 00 c0 67 f2 62 fe 7f 00 00 40 69 f2 62 fe 7f 00 00 d0 6a f2 62 fe 7f 00 00
...

There is a padding after the version. Because of this, the function addresses are read incorrectly. For example, I had segfaults because C_Initialize (the first function after the header) had address 0x42e0000000000000.

MinGW and Visual Studio

I suspect that this is a problem not only for MinGW but also for Visual Studio builds, because I don't see any pack options for it as well. And the Windows CI just builds the project but doesn't run it, so it's not clear whether VS is affected from it.

Other ABIs?

I don't know much about Windows API, but from what I see in rust cryptoki bindings - every structure is expected to be packed. So, it seems like it would be not enough to add packed attribute only to the CK_FUNCTION_LIST on Windows.