Apple Silicon Metal backend (ft serve on macOS arm64)

[jasonkneen]

Summary

Runs FreeToken on Apple Silicon without CUDA. ft serve on Darwin routes to the Metal backend (MLX or llama.cpp) before torch is imported. install.sh takes the mlx-lm path on macOS arm64.

Why

ft serve currently imports the CUDA launcher and crashes in a Metal venv (ModuleNotFoundError: torch). There are no macOS wheels on PyPI. This branch is the installable CLI path for M1–M4.

Verified on this machine

• Native arm64 venv, mlx-lm 0.31.3, MLX GPU
• ft serve --model mlx-community/Qwen3-0.6B-4bit — /health (maintenance: serving), /v1/models, chat completions
• tests/server/test_serve_macos.py, tests/server/test_metal_backend.py, tests/daemon/test_serve_command_platform.py — 48 passed locally (no CUDA)

Notes

• Does not add a desktop .dmg. The download site PR installs this CLI.
• Until this lands on main, macOS install falls back to jasonkneen/FreeToken@feat/apple-metal-backend.

[Copilot]

🟡 Changes recommended

Critical lifecycle, routing, torch-free startup, shutdown, and access-control issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds Apple Silicon Metal serving via MLX or llama.cpp, with macOS routing, installation, proxying, shell support, documentation, and tests.

Changes:

• Adds standalone and integrated Metal backend support.
• Updates CLI, daemon, installer, packaging, and shell behavior for macOS arm64.
• Adds Metal tests, documentation, and benchmark support.

File summaries

File	Summary
tests/server/test_serve_macos.py	Tests Darwin CLI routing.
tests/server/test_metal_backend.py	Tests Metal backend and proxy behavior.
tests/daemon/test_serve_command_platform.py	Tests platform-specific daemon commands.
setup.py	Skips CUDA extensions on macOS.
scripts/start-metal.sh	Adds Metal startup helper.
README.md	Documents Apple Silicon usage.
python/freetoken/utils/logger.py	Provides logging fallback behavior.
python/freetoken/shell/tui.py	Adds shell model switching.
python/freetoken/shell/client.py	Adds model discovery and loading APIs.
python/freetoken/shell/__init__.py	Routes local shell engines on macOS.
python/freetoken/server/metal.py	Implements Metal lifecycle and proxying.
python/freetoken/server/metal_main.py	Adds the standalone Metal entrypoint.
python/freetoken/server/launch.py	Integrates Metal backend selection.
python/freetoken/server/args.py	Adds backend and Metal-port arguments.
python/freetoken/server/api_server.py	Adds backend-specific route installation.
python/freetoken/server/__init__.py	Makes server launch imports lazy.
python/freetoken/daemon/serve_manager.py	Selects Metal commands on Darwin.
python/freetoken/cli.py	Adds serve-metal and Darwin routing.
pyproject.toml	Makes CUDA dependencies Linux-only.
install.sh	Adds Apple Silicon installation flow.
docs/quickstart.md	Documents Metal quickstarts.
docs/install.md	Documents macOS installation.
docs/cli.md	Documents serve-metal.
benchmarks/bench_decode_moe.py	Adds Metal benchmark support.

Review details

Suppressed comments (9)

python/freetoken/cli.py:36

• The Darwin branch forwards ft serve arguments unchanged, but the Metal parser only recognizes --shell; --shell-mode is therefore reported as an ignored CUDA flag. ft serve --model ... --shell-mode now starts only the API and never attaches the interactive shell, unlike the existing Linux behavior. Translate this flag to --shell or accept it in the Metal entrypoint.

    if sys.platform == "darwin":
        from freetoken.server.metal_main import main

        return main(argv)

python/freetoken/daemon/serve_manager.py:131

• The new Darwin daemon command launches a Metal parent that owns a separate upstream child. On macOS osproc.signal_group() cannot inspect /proc and falls back to signaling only the parent PID; if the parent requires escalation and receives SIGKILL, its finally cleanup cannot terminate the upstream, which can keep serving and hold the port after ft daemon stop. Use portable process-group signaling or an external child cleanup path for Darwin.

    subcommand = "serve-metal" if sys.platform == "darwin" else "serve"
    argv = [python, "-m", "freetoken.cli", subcommand, "--model", model, "--port", str(port), *args]

python/freetoken/server/metal.py:68

• _claim_port closes its socket before the caller launches the upstream, so the selected port is not reserved. Two Metal servers starting concurrently can both observe the same free port and one will fail to bind; keep a reservation through Popen or retry on bind failure.

        return _claim_port(preferred) or _scan_free_port()

python/freetoken/server/metal.py:378

• raise_for_status() runs inside the iterator after StreamingResponse has already sent its HTTP 200 response headers. If the upstream rejects a streamed request with 4xx/5xx, clients receive a successful-but-truncated 200 stream instead of the upstream error. The proxy needs to obtain and translate the upstream status before committing the streaming response.

    async with httpx.AsyncClient(timeout=None) as client:
        async with client.stream(
            "POST", f"{upstream_base_url}{path}", content=body, headers=headers
        ) as resp:
            resp.raise_for_status()
            async for chunk in resp.aiter_bytes():

python/freetoken/server/metal.py:413

• Only POST /v1/messages is registered here. The native API and Anthropic clients expose POST /v1/messages/count_tokens; standalone Metal consequently returns FastAPI's generic 404 for clients that preflight token counts, despite the entrypoint documenting the same Anthropic surface. Add a compatible handler or narrow the advertised API contract.

    @app.post("/v1/messages")
    async def proxy_messages(request: Request):
        return await _forward(request, get_backend)

python/freetoken/server/metal_main.py:70

• The standalone command's help says auto prefers MLX, but it calls the shared resolver whose order is CUDA, MLX, then llama. On a host with a usable CUDA installation, ft serve-metal --backend auto resolves to cuda and then launch_metal_backend() rejects it as unsupported. Use a Metal-only resolver for this entrypoint.

    backend = resolve_backend(args.backend)

python/freetoken/server/metal_main.py:41

• The Metal CLI has no --served-model-name, and parse_known_args() later drops that documented ft serve option as if it were CUDA-only. The proxy consequently publishes the raw model path rather than the native default basename or an explicitly requested alias, breaking clients configured with the documented model ID. Accept and propagate this option into model listing and responses.

    p.add_argument("--model", required=True, help="Model path or HF id for the Metal backend.")

python/freetoken/shell/init.py:85

• This list treats boolean flags as value-taking flags. For example, ft shell --model M --moe-cache-auto --backend llama consumes --backend as the value of --moe-cache-auto, leaving only llama to pass through; the Metal entrypoint then defaults to MLX (when installed) instead of honoring the requested backend. Keep boolean Metal-unsupported flags separate from flags that consume a value.

    value_flags = {f for f in _ENGINE_FLAGS + _METAL_UNKNOWN_FLAGS}
    model = None
    passthrough: list[str] = []
    i = 0
    while i < len(argv):
        arg = argv[i]
        if arg.startswith("--") and "=" in arg:
            name, _, value = arg.partition("=")
            if name in value_flags:
                if name in _ENGINE_FLAGS:
                    model = value
            else:
                passthrough.append(arg)
            i += 1
        elif arg in value_flags:
            value = argv[i + 1] if i + 1 < len(argv) else None
            if arg in _ENGINE_FLAGS:
                model = value
            i += 2 if value is not None else 1

scripts/start-metal.sh:82

• An explicit --port in "$@" appears after the script's generated --port "$PORT" and therefore overrides the actual server port, while readiness and the pre-check continue using the old PORT. scripts/start-metal.sh model --port 2000 consequently serves on 2000 but polls 1919 and can time out or attach to the wrong process. Parse the effective port before launching or reject duplicate port options.

"$ROOT/.venv/bin/ft" serve-metal --model "$MODEL" --port "$PORT" "$@" &
SERVER_PID=$!

• Files reviewed: 24/24 changed files
• Comments generated: 18
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[gdevenyi]

Why would this benefit when METAL has unified memory?

[jasonkneen]

Why would this benefit when METAL has unified memory?

UMA removes the discrete VRAM/PCIe boundary, so the CUDA-style offload story isn’t the Metal benefit. The Metal backend uses MLX/llama.cpp; FreeToken adds serving, model lifecycle, API compatibility and shell tooling. UMA still has finite capacity and bandwidth, so working-set and sparse-execution choices still matter.