Skip to content

fix(py_loader,rs_port): bootstrap Python home on Windows to prevent CPython init_fs_encoding failures#734

Draft
k5602 wants to merge 1 commit intometacall:developfrom
k5602:rs_port
Draft

fix(py_loader,rs_port): bootstrap Python home on Windows to prevent CPython init_fs_encoding failures#734
k5602 wants to merge 1 commit intometacall:developfrom
k5602:rs_port

Conversation

@k5602
Copy link
Copy Markdown
Contributor

@k5602 k5602 commented Mar 27, 2026

Description

This PR fixes Windows CI instability in the Rust port caused by Python embedded initialization failing with:

  • Fatal Python error: init_fs_encoding
  • ModuleNotFoundError: No module named 'encodings'

The fix ensures Python home/bootstrap paths are resolved consistently when running cargo test against installed MetaCall binaries on Windows.

Root cause

rs_port tests were running against installed MetaCall artifacts (from the install script) where runtime Python bootstrap context was not guaranteed at process startup. In this context, embedded Python could initialize without a valid PYTHONHOME, causing stdlib bootstrap failure.

What changed

1) Python loader configuration (proper loader-side fix)

  • Added source/loaders/py_loader/data/py_loader.json.in.
  • Added python_home to py_loader configuration.
  • Wired py_loader CMake to use this custom template.
  • Added Windows search_paths generation for development/install configs.

Files:

  • source/loaders/py_loader/data/py_loader.json.in
  • source/loaders/py_loader/CMakeLists.txt

2) Python loader runtime bootstrap

  • Added Windows-only initialization hook in py_loader that:
    • reads python_home from loader config,
    • respects user-provided PYTHONHOME if already set,
    • sets PYTHONHOME before Py_InitializeEx(0).

File:

  • source/loaders/py_loader/source/py_loader_impl.c

3) Rust port compatibility fallback (for older installed packages)

  • Added Windows runtime fallback in rs_port init path:
    • if PYTHONHOME is unset, infer it from METACALL_PYTHONHOME and runtime layout,
    • prefer .../runtimes/python when present.
  • Exposed METACALL_PYTHONHOME from metacall-sys build metadata on Windows.

Files:

  • source/ports/rs_port/src/init.rs
  • source/ports/rs_port/sys/src/lib.rs

Why this approach

  • Fixes the issue at the loader level (correct long-term boundary).
  • Keeps compatibility with currently installed/distributed Windows artifacts via Rust-side fallback.
  • Avoids fragile workflow-only environment hacks.

Validation

  • Reproduced and analyzed failing Windows CI behavior (inlines_test before fix).
  • Verified post-fix behavior where:
    • inlines_test passes,
    • full Rust test suite completes successfully.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update
  • Documentation update

Checklist

  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • My changes generate no new warnings.
  • I have added tests/screenshots (if any) that prove my fix is effective or that my feature works.
  • I have tested the tests implicated (if any) by my own code and they pass.
  • If my change is significant or breaking, I have passed all tests with ./docker-compose.sh test and attached the output.
  • I have tested my code with OPTION_BUILD_ADDRESS_SANITIZER or ./docker-compose.sh test-address-sanitizer and OPTION_TEST_MEMORYCHECK.
  • I have tested my code with OPTION_BUILD_THREAD_SANITIZER or ./docker-compose.sh test-thread-sanitizer.
  • I have tested with Helgrind in case my code works with threading.
  • I have run make clang-format in order to format my code and my code follows the style guidelines.

@viferga
Copy link
Copy Markdown
Member

viferga commented Mar 27, 2026

it's good but I'm not sure if this is the approach we should do, have you seen the Linux distributable that has a profile file for storing all env vars? I was thinking about that

@viferga viferga marked this pull request as draft March 27, 2026 08:23
@viferga viferga marked this pull request as ready for review March 27, 2026 11:25
@viferga
Copy link
Copy Markdown
Member

viferga commented Mar 30, 2026

I reviewed the code and it's actually very good. We should investigate anyway how are they installed, my original idea can be combined with yours, which was finding the PYTHONPATH etc based on the libpython.so path. Because they can be relinked dynamically.

@viferga
Copy link
Copy Markdown
Member

viferga commented Mar 30, 2026

When it runs in python.exe, do we assume PYTHONPATH is correct?

@viferga
Copy link
Copy Markdown
Member

viferga commented Apr 13, 2026

I'm leaving this as draft until we design this:
#760

@viferga viferga marked this pull request as draft April 13, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants