diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000000..4c166970c0 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,80 @@ +# PROJECT KNOWLEDGE BASE + +## OVERVIEW +GraalPy (GraalVM Python) implementation: Java (Truffle) + C (CPython C-API compatibility) + Python stdlib/overrides, built and tested via the `mx` build tool. + +## STRUCTURE +```text +./ +├── graalpython/ # Core sources + stdlib + tests (multi-language) +│ ├── com.oracle.graal.python/ # Main Java implementation (Truffle AST, runtime, builtins) +│ ├── com.oracle.graal.python.cext/ # C-API (headers + C sources + adapted CPython modules) +│ ├── com.oracle.graal.python.test/ # Python-level + Java-level tests and runners +│ ├── lib-graalpython/ # GraalPy-specific Python modules/patches +│ └── lib-python/ # Vendored/adapted CPython stdlib + CPython tests +├── mx.graalpython/ # `mx` suite + helper commands (build/test/bench/bisect) +├── scripts/ # Dev utilities (launcher, formatting hooks, codegen) +├── .github/workflows/ # GitHub Actions entrypoints +├── ci/ # CI definitions (jsonnet/libsonnet) +├── docs/ # User + contributor docs +├── benchmarks/ # Benchmark harnesses +└── mxbuild/ # GENERATED build artifacts (ignore for code navigation) +``` + +## WHERE TO LOOK +| Task | Location | Notes | +|------|----------|-------| +| Build / run | `docs/contributor/CONTRIBUTING.md`, `mx.graalpython/` | This repo is `mx`-first (not a typical Maven/Gradle-only build). | +| Java runtime & Truffle nodes | `graalpython/com.oracle.graal.python/src/com/oracle/graal/python/{runtime,nodes,builtins}` | Main interpreter implementation. | +| C-API / native extensions | `graalpython/com.oracle.graal.python.cext/{include,src,modules}` | Mirrors CPython naming; many files are adapted from CPython. | +| Python stdlib overrides | `graalpython/lib-graalpython/` | GraalPy-specific modules executed at startup and/or used by builtins. | +| Vendored stdlib + CPython tests | `graalpython/lib-python/3/` | Large; treat as upstream-ish unless you are explicitly changing stdlib/tests. | +| Python-level tests | `graalpython/com.oracle.graal.python.test/src/tests/` | Includes tagged tests + C-API tests. Runner: `.../src/runner.py`. | +| CI pipelines | `.github/workflows/`, `ci.jsonnet`, `ci/` | Workflows typically drive `mx` gates/tags. | +| Launchers / helper scripts | `scripts/python.sh`, `scripts/*` | `python.sh` is the local launcher wrapper. | + +## CONVENTIONS (DEVIATIONS) +- `mx` is the primary build/test entrypoint; suite definition lives in `mx.graalpython/suite.py`. +- `black` is configured but intentionally **disabled** for this repo (root `pyproject.toml`); line length is 120 and version locked to `23` for consistency. +- Formatting/linting is enforced via **pre-commit** with repo-specific hooks (Eclipse formatter, checkstyle, copyright), and pylint only on `mx.graalpython/*.py`. +- Large generated/build outputs exist in-tree (`mxbuild/`, `*.dist/`); do not use them as the source of truth when navigating code. + +## ANTI-PATTERNS (THIS PROJECT) +- **Do not** base edits or reviews on `mxbuild/**` or `*.dist/**` outputs; change sources under `graalpython/**`, `mx.graalpython/**`, etc. +- Security reports: follow `SECURITY.md` (do **NOT** file public issues for vulnerabilities). +- C-API: heed CPython-style invariants in headers (e.g. `ceval.h`: **NEVER** nest `Py_BEGIN_ALLOW_THREADS` blocks; avoid mixing PyMem/PyObject allocators with `malloc`). +- Interop: foreign `executable` / `instantiable` objects are **never** called with keyword arguments (see `docs/user/Interoperability.md`). + +## UNIQUE STYLES / GOTCHAS +- Builtins: implemented in Java classes annotated with `@CoreFunctions`; individual operations are Nodes annotated with `@Builtin` (see contributing guide). +- Many operations have shared equivalents under `com.oracle.graal.python.lib`; prefer using/adding shared lib nodes instead of duplicating patterns. +- Parser work may require regenerating golden files (see `docs/contributor/CONTRIBUTING.md`). + +## COMMANDS + +* Import dependent suites / download deps + `mx sforceimport` +* Build and run + `mx python-jvm` + `mx python -c 'print(42)'` +* Common local testing + * Run cpyext tests + `mx graalpytest cpyext` + * Rerun a specific failing test + `mx graalpytest TEST-SELECTOR` + * Run JUnit tests + `mx python-gate --tags python-junit` +* Style / formatting + `mx python-style --fix` + `mx python-gate --tags style` + +## NOTES +- When searching for implementation, prefer `graalpython/com.oracle.graal.python/src/...` over vendored `lib-python` unless you are intentionally modifying upstream stdlib/tests. +- If you see very large files under `com.oracle.graal.python.cext/modules/_sqlite/` or `expat/`, treat them as upstream imports/adaptations (patch carefully). + +## PULL REQUESTS AND JIRA TICKETS + +We use Jira and Bitbucket, and each PR should reference a Jira ticket with the form [GR-XXXX] where XXXX is the ticket number. +When asked to open pull requests, agents should ask for the Jira ticket number. +When asked to create a ticket, the `ol-cli jira` tool can be used to create a ticket for the "Python" component. +When asked to create, run gates on, or check on the builds previously run on a pull request, use the `ol-cli bitbucket` tool. diff --git a/ci/AGENTS.md b/ci/AGENTS.md new file mode 100644 index 0000000000..f10f0c1aeb --- /dev/null +++ b/ci/AGENTS.md @@ -0,0 +1,18 @@ +# ci/ — CI DEFINITIONS + +## OVERVIEW +Jsonnet/libsonnet CI definitions consumed by GitHub Actions matrix generation. + +## WHERE TO LOOK +| Task | Location | Notes | +|------|----------|-------| +| Main pipelines | `../ci.jsonnet`, `python-gate.libsonnet` | Defines gates/tags executed in CI. | +| Shared constants | `constants.libsonnet` | Shared settings used across CI definitions. | +| Helpers | `utils.libsonnet` | Common functions/macros. | + +## CONVENTIONS +- GitHub workflows typically call `ci-matrix-gen.yml`, which evaluates `ci.jsonnet` and these libraries. + +## ANTI-PATTERNS +- Don’t encode secrets or environment-specific paths here. +- Keep changes compatible with matrix generation (small diffs, deterministic output). diff --git a/docs/AGENTS.md b/docs/AGENTS.md new file mode 100644 index 0000000000..ef0201d8ee --- /dev/null +++ b/docs/AGENTS.md @@ -0,0 +1,19 @@ +# docs/ — DOCUMENTATION + +## OVERVIEW +Contributor and user documentation; many repo-wide rules live here. + +## WHERE TO LOOK +| Task | Location | Notes | +|------|----------|-------| +| Build/test instructions | `contributor/CONTRIBUTING.md` | Primary “how to build/run gates” guide. | +| Runtime internals | `contributor/IMPLEMENTATION_DETAILS.md` | Interpreter design details. | +| Interop rules | `user/Interoperability.md` | Keyword-arg restrictions for foreign calls, etc. | +| Native extensions | `user/Native-Extensions.md` | Guidance on native packages and tooling. | +| Standalone apps | `user/Python-Standalone-Applications.md` | Notes on packaging and limitations. | + +## CONVENTIONS +- Prefer updating docs here rather than duplicating instructions in random READMEs. + +## ANTI-PATTERNS +- Security: follow `SECURITY.md` at repo root; do not instruct users to file public vulnerability issues. diff --git a/graalpython/AGENTS.md b/graalpython/AGENTS.md new file mode 100644 index 0000000000..ba87f467cf --- /dev/null +++ b/graalpython/AGENTS.md @@ -0,0 +1,24 @@ +# graalpython/ — SOURCE + STDLIB + TESTS + +## OVERVIEW +Main implementation tree: Java (Truffle interpreter), C (CPython C-API compatibility), and Python (stdlib overlays + tooling/tests). + +## WHERE TO LOOK +| Task | Location | Notes | +|------|----------|-------| +| Java interpreter | `com.oracle.graal.python/src/com/oracle/graal/python/{runtime,nodes,builtins}` | Most core behavior lives here. | +| Shared Java lib nodes | `com.oracle.graal.python/src/com/oracle/graal/python/lib` | Prefer adding/reusing lib nodes for common operations. | +| C-API headers + runtime | `com.oracle.graal.python.cext/include`, `.../src` | CPython-like naming; follow CPython invariants. | +| Adapted native modules | `com.oracle.graal.python.cext/modules/` | Many files are upstream-derived; patch carefully. | +| GraalPy stdlib overlays | `lib-graalpython/` | Python files executed at startup / for builtins. | +| Vendored CPython stdlib/tests | `lib-python/3/` | Treat as upstream-ish unless explicitly changing it. | +| GraalPy tests | `com.oracle.graal.python.test/src/tests/` | Python-level tests + tag files. | +| Parser components | `com.oracle.graal.python.pegparser*` | Parser implementation + golden files tests. | + +## CONVENTIONS / GOTCHAS +- This subtree contains both “source of truth” code and vendored/upstream-ish imports; keep patches minimal in `lib-python/` and large C module imports. +- Some large headers/databases (unicode tables) are generated; avoid editing them by hand unless you also update the generator pipeline. + +## ANTI-PATTERNS +- Don’t use `mxbuild/**` outputs to understand behavior; always navigate `.../src/...` trees. +- C-API: never mix `PyMem_*` / `PyObject_*` allocators with platform `malloc` family. diff --git a/graalpython/com.oracle.graal.python.cext/AGENTS.md b/graalpython/com.oracle.graal.python.cext/AGENTS.md new file mode 100644 index 0000000000..2fec21bb92 --- /dev/null +++ b/graalpython/com.oracle.graal.python.cext/AGENTS.md @@ -0,0 +1,20 @@ +# com.oracle.graal.python.cext/ — CPYTHON C-API COMPAT + +## OVERVIEW +C headers and native code implementing CPython C-API compatibility plus adapted builtin extension modules. + +## WHERE TO LOOK +| Task | Location | Notes | +|------|----------|-------| +| Public C-API headers | `include/` | Mirrors CPython header structure. | +| Core C-API impl | `src/` | CPython-like file naming (`unicodeobject.c`, `typeobject.c`, ...). | +| Adapted extension modules | `modules/` | Large upstream-derived modules (e.g., `_sqlite`). | +| Embedded/third-party imports | `expat/`, `modules/_sqlite/sqlite/` | Treat as upstream; patch minimally. | + +## CONVENTIONS / GOTCHAS +- Follow CPython invariants in headers (examples: do not nest `Py_BEGIN_ALLOW_THREADS`; “never mix allocators” warnings). +- Large unicode databases/headers are often generated (e.g., `unicodename_db.h`, `unicodedata_db.h`): avoid hand edits. + +## ANTI-PATTERNS +- **NEVER** nest `Py_BEGIN_ALLOW_THREADS` blocks (see `include/ceval.h`). +- Never mix `PyMem_*` / `PyObject_*` allocators with platform `malloc` family. diff --git a/graalpython/com.oracle.graal.python.test/AGENTS.md b/graalpython/com.oracle.graal.python.test/AGENTS.md new file mode 100644 index 0000000000..2969e8ca18 --- /dev/null +++ b/graalpython/com.oracle.graal.python.test/AGENTS.md @@ -0,0 +1,19 @@ +# com.oracle.graal.python.test/ — TEST SUITES + +## OVERVIEW +Python-level tests, tag files, and harnesses used by `mx python-gate` and `mx graalpytest`. + +## WHERE TO LOOK +| Task | Location | Notes | +|------|----------|-------| +| Python tests | `src/tests/` | Main Python tests; includes C-API related tests. | +| Tagged/unittest tiers | `src/tests/unittest_tags/` | Tag files select which stdlib tests should pass. | +| Test runner | `src/runner.py` | Harness for executing Python tests. | +| Test data | `testData/` | Golden files and fixtures. | + +## CONVENTIONS +- Tests are typically run via `mx python-gate --tags ...` rather than invoking pytest directly. +- Use `mx [-d] graalpytest TEST-SELECTOR` to rerun a failing test or selection. + +## ANTI-PATTERNS +- Don’t delete failing tests; fix root causes or update tag selections when behavior intentionally changes. diff --git a/graalpython/com.oracle.graal.python/AGENTS.md b/graalpython/com.oracle.graal.python/AGENTS.md new file mode 100644 index 0000000000..9567d4e747 --- /dev/null +++ b/graalpython/com.oracle.graal.python/AGENTS.md @@ -0,0 +1,32 @@ +# com.oracle.graal.python/ — CORE JAVA IMPLEMENTATION + +## OVERVIEW +Truffle-based Python runtime: AST nodes, builtins, compiler/bytecode support, interop, and shared runtime services. + +## STRUCTURE +```text +com.oracle.graal.python/ +└── src/com/oracle/graal/python/ + ├── runtime/ # Context, POSIX emulation, state, interop runtime glue + ├── nodes/ # Truffle AST nodes + bytecode nodes + ├── builtins/ # Builtin modules/types; @CoreFunctions + @Builtin nodes + ├── lib/ # Shared lib nodes/utilities used across builtins/nodes + ├── compiler/ # Bytecode compiler + DSL tooling + └── util/ # Shared helpers/data structures +``` + +## WHERE TO LOOK +| Task | Location | Notes | +|------|----------|-------| +| Add/fix builtin | `.../builtins/**` | Modules/classes via `@CoreFunctions`; ops via `@Builtin` Nodes. | +| Cross-cutting ops | `.../lib/**` | Prefer adding/reusing lib nodes instead of duplicating patterns. | +| Interop behavior | `.../nodes/interop`, `.../runtime/interop` | Foreign call rules + conversions. | +| Bytecode execution | `.../nodes/bytecode/**` | Root nodes and bytecode interpreter pieces. | + +## CONVENTIONS +- Code style enforced via pre-commit Eclipse formatter + checkstyle; don’t hand-format Java. +- Keep naming/layout close to CPython where practical (helps cross-referencing). + +## ANTI-PATTERNS +- Don’t edit generated sources under `mxbuild/**` or distribution outputs; edit `src/**`. +- Avoid re-implementing common helpers in builtins; check `.../lib/**` first. diff --git a/graalpython/lib-graalpython/AGENTS.md b/graalpython/lib-graalpython/AGENTS.md new file mode 100644 index 0000000000..e9bedf6f6d --- /dev/null +++ b/graalpython/lib-graalpython/AGENTS.md @@ -0,0 +1,17 @@ +# lib-graalpython/ — GRAALPY STDLIB OVERLAYS + +## OVERVIEW +Python modules/patches that implement or tweak stdlib behavior for GraalPy; some execute at startup. + +## WHERE TO LOOK +| Task | Location | Notes | +|------|----------|-------| +| Overlay modules | `modules/` | Python modules provided/overridden by GraalPy. | +| Patches | `patches/` | Patch files applied to external packages on pip installation to fix compatibility issues. | + +## CONVENTIONS / GOTCHAS +- If a file name matches a builtin module name, it can run in that module context during startup (see contributor docs). +- Keep changes aligned with corresponding Java builtins (`com.oracle.graal.python.builtins`) where relevant. + +## ANTI-PATTERNS +- Don’t fork upstream stdlib behavior unnecessarily; prefer minimal overlays. diff --git a/graalpython/lib-python/AGENTS.md b/graalpython/lib-python/AGENTS.md new file mode 100644 index 0000000000..f10810025d --- /dev/null +++ b/graalpython/lib-python/AGENTS.md @@ -0,0 +1,17 @@ +# lib-python/ — VENDORED CPYTHON STDLIB + TESTS + +## OVERVIEW +Vendored/adapted CPython standard library and CPython test suite used for compatibility validation. + +## WHERE TO LOOK +| Task | Location | Notes | +|------|----------|-------| +| Stdlib sources | `3/` | CPython stdlib tree. | +| CPython tests | `3/test/` | Large upstream test suite; GraalPy runs selected tests via tag files. | +| Local lint configs | `3/test/.ruff.toml`, `3/test/**/mypy.ini` | Primarily for the vendored tests area. | + +## CONVENTIONS +- Treat this tree as upstream-ish: changes should be minimal, well-justified, and ideally correspond to upstream fixes. + +## ANTI-PATTERNS +- Don’t apply sweeping refactors or formatting changes here; it makes rebasing/upstream sync harder. diff --git a/mx.graalpython/AGENTS.md b/mx.graalpython/AGENTS.md new file mode 100644 index 0000000000..0e36665f40 --- /dev/null +++ b/mx.graalpython/AGENTS.md @@ -0,0 +1,21 @@ +# mx.graalpython/ — MX BUILD TOOLING + +## OVERVIEW +`mx` suite definition and Python utilities used for building, testing, benchmarking, and CI orchestration. + +## WHERE TO LOOK +| Task | Location | Notes | +|------|----------|-------| +| Suite definition | `suite.py` | Defines projects, imports, distributions, version metadata. | +| mx commands | `mx_graalpython.py` | Implements custom `mx` commands (e.g., test runners). | +| Import helpers | `mx_graalpython_import.py` | `mx sforceimport`-related logic. | +| Bench tooling | `mx_graalpython_benchmark.py`, `mx_graalpython_python_benchmarks.py` | Performance harness integration. | +| Bisect tooling | `mx_graalpython_bisect.py` | Benchmark/test bisect support. | + +## CONVENTIONS +- Pylint is enforced (via pre-commit) only for `mx.graalpython/*.py`. +- Treat `suite.py` as the authoritative dependency/version map; CI workflows typically call into these mx gates. + +## ANTI-PATTERNS +- Don’t re-implement build/test flows in ad-hoc scripts when an `mx` command already exists. +- Avoid adding heavyweight dependencies here; this code runs in many CI contexts.