[experiment] python re-implement

[AlexandreYang]

What does this PR do?

Motivation

Testing

Checklist

• Tests added/updated
• Documentation updated (if applicable)

[AlexandreYang]

Plan: Replace gpython with a Custom Pure-Go Python Interpreter

Context

The python builtin currently uses github.com/go-python/gpython — a pure-Go Python 3.4 interpreter — as its execution engine. The goal is to remove that dependency entirely and rewrite the interpreter layer from scratch in pure Go, while keeping the same user-facing interface (python -c, script files, stdin) and the same security sandbox (blocked dangerous ops, read-only open(), AllowedPaths enforcement).

The existing test suite has 129 scenario tests and a Go unit test suite that together validate:

• Full Python 3 syntax: classes, generators, try/except, comprehensions, lambdas
• stdlib modules: sys, math, os (read-only), binascii, string, time
• Security sandbox: os.system/popen/exec blocked, write-mode open blocked, tempfile/glob ImportError
• I/O: sandboxed open(), readline(), readlines(), with-statement, stdin/stdout/stderr
• Error propagation: SyntaxError, RuntimeError, sys.exit(N), tracebacks to stderr

---

Recommended Approach: Custom Pure-Go Python Interpreter

Why not Starlark (github.com/google/starlark-go):

• No class keyword (breaks ~30 tests: classes, inheritance, data structures)
• No try/except (breaks ~20 error-handling tests)
• No yield/generators (breaks 4 tests)
• No import X syntax (breaks ALL tests that use sys, math, os, etc.)
• Would require removing >40% of the existing test suite

Why a custom interpreter is the right call:

• Preserves nearly all 129 existing tests unchanged
• Gives full control over the security sandbox
• Removes all external Python-related dependencies
• The user explicitly said "the re-implementation can be a complex endeavour, this is fine, do it!"

---

Implementation Scope

The interpreter implements the Python 3 subset actually used by the tests. Out-of-scope: decorators, multiple-inheritance MRO edge cases, async/await, metaclasses, the full CPython stdlib.

Files to create (in builtins/internal/pyruntime/)

File	Description	Est. lines
lexer.go	Tokenizer: keywords, operators, indentation (INDENT/DEDENT), string/number literals	~600
ast.go	AST node types for all statements and expressions	~400
parser.go	Recursive-descent parser: all statement forms, operator precedence	~1800
types.go	Python object model: Int, Float, Str, Bytes, Bool, None, List, Dict, Set, Tuple, Function, Class, Instance, Generator, Exception	~1000
eval.go	Tree-walking evaluator: scopes, exceptions, generators, context managers	~2500
builtins_py.go	Built-in functions: print, len, range, enumerate, zip, sorted, map, filter, sum, min, max, abs, chr, ord, bin, hex, oct, type, isinstance, repr, str, int, float, bool, list, dict, set, tuple, open (sandboxed)	~700
modules.go	Standard modules: sys (argv, exit, stdin, stdout, stderr), math, os (read-only subset), string, binascii, time	~600
sandbox.go	Security layer: blocked os functions, write-mode open rejection, blocked imports (tempfile, glob)	~300
pyruntime.go	Entry point: Run(ctx, RunOpts) int, same signature as current	~200

Total estimate: ~8,100 lines (the current pyruntime.go is ~700 lines using gpython; the difference is the interpreter itself)

Files to modify

File	Change
builtins/internal/pyruntime/pyruntime.go	Replace entirely (same Run() API, new pure-Go implementation)
builtins/python/python.go	No change — already delegates to pyruntime.Run()
go.mod	Remove github.com/go-python/gpython entry
go.sum	Remove gpython hashes (go mod tidy)
SHELL_FEATURES.md	Update Python description (remove "gpython", "Python 3.4")
builtins/tests/python/python_fuzz_test.go	Update gpython-specific comment in FuzzPythonSource
analysis/symbols_builtins_test.go	Update any gpython-specific allowlist entries

---

Python features to implement

Core language:

• All literals: int (decimal, hex, octal, binary), float, complex, string (all quote forms + raw), bytes, bool, None, ellipsis
• Operators: arithmetic, bitwise, comparison, boolean, in/not in, is/is not
• Assignments: simple, augmented (+=, etc.), tuple unpacking, starred assignment (a, *b, c = ...)
• Statements: if/elif/else, for/in, while, break, continue, pass, del, return, yield, yield from, raise, try/except/else/finally, with, assert, global, nonlocal, import, from-import, class, def
• Comprehensions: list, dict, set, generator expression
• Lambda expressions
• Slicing (a[1:3:2])
• Attribute access, subscript
• Starred calls (*args, **kwargs)
• Class definitions: single inheritance, __init__, methods, __str__, __repr__, __enter__/__exit__, __iter__/__next__
• Generators: yield, yield from, send(), StopIteration

Exception handling:

• All standard exception types: BaseException, Exception, ValueError, TypeError, KeyError, IndexError, AttributeError, NameError, ZeroDivisionError, IOError, OSError, FileNotFoundError, PermissionError, StopIteration, RuntimeError, ImportError, MemoryError, SystemExit, SyntaxError, AssertionError, NotImplementedError
• Exception chaining
• Custom exception classes (subclassing Exception)

Built-in functions (28+): print, len, range, enumerate, zip, sorted, reversed, map, filter, sum, min, max, abs, chr, ord, bin, hex, oct, type, isinstance, issubclass, repr, str, int, float, bool, list, dict, set, tuple, open (sandboxed), hash, id, iter, next, callable, getattr, setattr, hasattr, delattr, dir, vars, all, any, round, divmod, pow

Modules:

• sys: argv, exit, stdin, stdout, stderr, version, platform, path (empty)
• math: floor, ceil, sqrt, log, log2, log10, exp, sin, cos, tan, asin, acos, atan, atan2, pi, e, inf, nan, fabs, factorial, gcd, isnan, isinf, isfinite, degrees, radians, hypot
• os: listdir (AllowedPaths), getcwd, path.join, path.dirname, path.basename, path.exists, path.isfile, path.isdir, path.splitext, getenv, environ (read-only), sep, linesep, devnull
• string: whitespace, ascii_letters, ascii_lowercase, ascii_uppercase, digits, hexdigits, octdigits, printable, punctuation
• binascii: hexlify, unhexlify, b2a_hex, a2b_hex
• time: time, sleep (limited), monotonic

Security sandbox (same as current):

• Blocked os functions: system, popen, remove, unlink, mkdir, makedirs, rmdir, removedirs, rename, renames, replace, link, symlink, chmod, chown, chroot, execl/le/lp/lpe, execv/ve/vp/vpe, _exit, fork, forkpty, kill, killpg, popen2-4, spawnl/le/lp/lpe/v/ve/vp/vpe, startfile, truncate, write, putenv, unsetenv, walk (removed)
• open() write modes rejected (PermissionError)
• tempfile and glob imports blocked (ImportError)
• File reads capped at 1 MiB
• Source code capped at 1 MiB

---

Test strategy

• Existing 129 scenario tests: should all pass unchanged
• Existing Go unit tests: update only the gpython-specific comment in FuzzPythonSource
• The interpreter runs in a goroutine; select on ctx.Done() for context cancellation
• Memory limits: maxSourceBytes = 1 MiB, maxReadBytes = 1 MiB per file.read() call
• Traceback format: Traceback (most recent call last):\n File "name", line N\nExceptionType: msg

---

Verification

make fmt
go build ./...
go test ./builtins/... ./tests/... -timeout 120s
RSHELL_BASH_TEST=1 go test ./tests/ -run TestShellScenariosAgainstBash -timeout 120s  # skip; scenarios have skip_assert_against_bash: true
go run ./cmd/rshell --allow-all-commands -c 'python -c "print(\"hello\")"'
go run ./cmd/rshell --allow-all-commands -c 'help' | grep python

[AlexandreYang]

TODO: create symbols_python_builtins.go checks?