diff --git a/.agents/skills/mt5-httpapi/SKILL.md b/.agents/skills/mt5-httpapi/SKILL.md index 92ce88b..2dec246 100644 --- a/.agents/skills/mt5-httpapi/SKILL.md +++ b/.agents/skills/mt5-httpapi/SKILL.md @@ -324,6 +324,56 @@ curl -H "Authorization: Bearer $MT5_API_TOKEN" "$MT5_API_URL/history/deals?from= Deal fields: `type` (0=buy, 1=sell), `entry` (0=opening, 1=closing), `profit` (0 for entries, realized P&L for exits). +### Chart Deployments + +Deploy Expert Advisors to charts over HTTP — no RDP, no terminal restart. +Stage `.ex5` + `.set` files, declare deployments, and a resident loader EA +inside the terminal reconciles charts to match. The API holds desired state; +the loader reports observed truth. A deployment only flips to `running` once +the loader confirms the expert is live on a chart. + +**Setup:** automatic via `[StartUp] Expert=` on boot. No manual attach needed. +Disable per terminal with `chartctl: false`. + +Endpoint reference: + +| Method | Endpoint | Description | +| ------ | -------- | ----------- | +| `POST` / `GET` / `DELETE` | `/experts` `/experts/` | Stage, list, remove EA `.ex5` | +| `POST` / `GET` | `/sets` `/sets/` | Stage, list, inspect `.set` (returns parsed inputs) | +| `POST` / `GET` | `/deployments` | Create or list deployments | +| `GET` / `PATCH` / `DELETE` | `/deployments/` | Inspect, pause/resume, change set, delete | +| `POST` | `/deployments/reconcile` | Force immediate reconcile | +| `GET` | `/charts` | Live chart/EA inventory | +| `GET` | `/loader` | Loader EA status and version | +| `POST` | `/charts//screenshot` | Capture chart as PNG | +| `POST` | `/charts//close` | Close a chart by id | + +```bash +# Stage artifacts +curl -F "expert=@HappyGoldScalp.ex5" "$MT5_API_URL/experts" +curl -F "set=@gold-m5.set" "$MT5_API_URL/sets" + +# Deploy +curl -X POST "$MT5_API_URL/deployments" \ + -H "Authorization: Bearer $MT5_API_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"expert":"HappyGoldScalp.ex5","set":"gold-m5.set","symbol":"XAUUSD","timeframe":"M5"}' + +# Check status +curl -H "Authorization: Bearer $MT5_API_TOKEN" "$MT5_API_URL/deployments" + +# Pause / resume / delete +curl -X PATCH "$MT5_API_URL/deployments/dep_a1b2c3" \ + -H "Authorization: Bearer $MT5_API_TOKEN" \ + -d '{"enabled":false}' +curl -X DELETE "$MT5_API_URL/deployments/dep_a1b2c3" \ + -H "Authorization: Bearer $MT5_API_TOKEN" +``` + +Full protocol: [`docs/chart-control-protocol.md`](docs/chart-control-protocol.md). +WebRequest allowlist provisioning: `GET/PUT /webrequest`, `POST /webrequest/apply`. + ### Backtest Run MT5 Strategy Tester via the API. Two-stage workflow: build the INI from a diff --git a/.gitignore b/.gitignore index 373b247..5299c1d 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,9 @@ mt5installers/* assets/experts/* !assets/experts/.gitkeep !assets/experts/MT5SystemWarmup.mq5 +!assets/experts/MT5ChartLoader.mq5 +!assets/experts/include +!assets/experts/include/ChartControl.mqh assets/sets/* !assets/sets/.gitkeep config/config.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 44bc9a1..891ffb0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ The project follows [Semantic Versioning](https://semver.org/): patch = bug fixe --- +## [Unreleased] — Chart Deployments (chartctl) + +Remote EA deployment: attach Expert Advisors to charts with set files over the HTTP API, no RDP and no terminal restart. + ## [v4.12.1] — 2026-08-08 Documentation. No code changed. @@ -203,145 +207,43 @@ to prove what it is. ### Added -- **`make test-mcpunifier`** — an end-to-end test for the MCP unifier, which had no automated coverage: `scripts/lint.sh` only reaches `.ps1` and `.sh` files, so nothing verified `mcpunifier/`. `scripts/test-mcpunifier.sh` builds the unifier image, stands it up beside a stub terminal on a scratch network, and asserts seven behaviours — health, the 25-tool surface, `list_terminals` reporting every configured terminal, a live terminal routing to its own port, a configured-but-down terminal failing only the calls that name it, a broker/account pair that is not configured being refused rather than routed, and the service staying healthy after both failures. -- Every resource the harness creates carries one name prefix and is removed by an `EXIT` trap, so a pass, a failure and an interrupt all leave nothing behind. It also sweeps that prefix on entry, so a run killed outright — where the trap never fires — is cleaned up by the next invocation. On a readiness timeout it dumps each container's state and last log lines *before* tearing down, since the teardown would otherwise destroy the only evidence of why the run failed. -- Fixtures are written under the gitignored `.data/` rather than `/tmp`: a bind-mount source is resolved on the docker daemon's filesystem, so a `mktemp -d` directory that exists only in the caller's namespace would be bound as an empty dir and the service would start with no configuration. +- **`make test-mcpunifier`** — an end-to-end test for the MCP unifier, which had no automated coverage. ## [v4.9.1] — 2026-07-28 ### Fixed -- **A missing `mcpunifier` container no longer stops nginx from starting and takes every other route down with it.** v4.9.0 generated `location /mcp/` with a literal `proxy_pass http://mcpunifier:6600/`. nginx resolves a literal upstream hostname while *parsing* the config, so on a deployment without that container nginx aborts with `host not found in upstream "mcpunifier"` — and every per-terminal route plus the whole REST API returns 502 behind it. Because `docker-compose.yml` is gitignored, pulling v4.9.0 delivered the new `scripts/config_helper.py` without the service it referenced, so the next restart broke the stack. -- The upstream now routes through a variable (`set $mcp_upstream …;` then `proxy_pass $mcp_upstream;`) with an explicit `resolver`, which defers the lookup to request time. nginx starts whether or not the container exists, every terminal route serves normally, and only `/mcp/` returns 502 until the unifier is running — which makes the service genuinely optional, as it needs to be. +- **A missing `mcpunifier` container no longer stops nginx from starting and takes every other route down with it.** ## [v4.9.0] — 2026-07-28 ### Added -- **Unified MCP endpoint at `/mcp`, spanning every configured terminal.** Each terminal already had its own MCP server at `///mcp`, and a session was permanently bound to whichever one it connected to — an MCP session has a fixed tool catalog, so there was no per-call slot to name a terminal. The new endpoint exposes the same 24 tools, each taking `broker` and `account` (plus optional `instance`), so one session can drive every terminal. -- **`list_terminals`**, reporting each configured terminal's broker, account, instance and whether it is a live or demo account. Every other tool refuses a broker/account pair that is not configured and answers with the valid list, rather than routing to something plausible but wrong. -- **`mcpunifier` service** (`mcpunifier/`, `Dockerfile.mcpunifier`) — a Linux container running beside the Windows VM. It reads the same `config/config.yaml` that generates the nginx routing, so it cannot route somewhere nginx does not, and reaches each terminal directly on that terminal's own port. `nginx` proxies `/mcp/` to it. - -### Notes - -- **Nothing existing changes.** The per-terminal `///mcp` endpoints and the whole REST surface are untouched. Which endpoint a client reaches depends only on the URL it is pointed at: a root URL gets the unified tools, a `//` URL gets that terminal's existing tools. -- **No startup coupling and no shared failure.** The unifier never waits on a terminal — the routing table is static and is not re-probed. A terminal that is down fails only the calls naming it and leaves the rest usable; successful responses carry the `terminal` key that answered. `/health` reports whether the unifier can route, never a terminal's state. -- The unified endpoint is gated by the same bearer token as the REST API. The service runs as a non-root user with a read-only root filesystem and all Linux capabilities dropped, and mounts `config/config.yaml` read-only. -- `scripts/config_helper.py` now refuses to generate nginx config for a broker literally named `mcp`, which would otherwise shadow the unified route. +- **Unified MCP endpoint at `/mcp`, spanning every configured terminal.** +- **`list_terminals`**, reporting each configured terminal's broker, account, instance and whether it is a live or demo account. +- **`mcpunifier` service** (`mcpunifier/`, `Dockerfile.mcpunifier`). ### Changed -- **Docs and plugin manifests now describe both MCP endpoints.** The README's MCP section, the Claude Code manifest's `api_url` prompt, and the OpenClaw bridge's `MT5_API_URL` all previously described the base URL as terminal-scoped only, which was the entire truth before this release and is now half of it. Each states that the server root reaches every terminal while a `//` path pins one, so the value a user is prompted for no longer steers them into single-terminal mode without mentioning the alternative. +- **Docs and plugin manifests now describe both MCP endpoints.** ## [v4.8.4] — 2026-07-27 ### Fixed -- The README's Codex subsection under `## Agent integrations` stopped after `codex plugin marketplace add psyb0t/agents` and never told the reader how to actually install the plugin. Added the missing command, `codex plugin add mt5-httpapi@psyb0t`. -- Clarified that skill invocation differs by install path: a marketplace-installed skill invokes as `$mt5-httpapi:mt5-httpapi`, while a skill Codex picks up automatically from a repo's own `.agents/skills/` invokes as plain `$mt5-httpapi`. +- The README's Codex subsection under `## Agent integrations` stopped after `codex plugin marketplace add psyb0t/agents` and never told the reader how to actually install the plugin. ## [v4.8.3] — 2026-07-27 ### Added -- **Codex plugin manifest** (`.agents/.codex-plugin/plugin.json`) — points `skills` at `.agents/skills/`, so mt5-httpapi installs as a distribution channel via `codex plugin marketplace add psyb0t/agents`. The skill itself already worked in Codex with zero files (native `.agents/skills/` scanning); this only adds discovery. -- **`## Agent integrations` README section** with copy-pasteable install commands for Claude Code, Codex, and the OpenClaw skill + MCP-bridge plugin, linked from the Table of Contents. - -### Fixed - -- The README's prior Claude Code install snippet pointed `claude plugin marketplace add` at this repo directly, which has no `marketplace.json` and would fail. It now points at the shared catalog, `psyb0t/agents`. +- **Codex plugin manifest** (`.agents/.codex-plugin/plugin.json`). +- **`## Agent integrations` README section** with copy-pasteable install commands. ### Removed -- Deleted the stray `.claude-plugin/marketplace.json` from this repo — marketplaces register by name and collide across repos; the catalog now lives solely in `psyb0t/agents`. - -## [v4.8.2] — 2026-07-27 - -### Added - -- Added a GitHub Actions CI status badge to the README. - -## [v4.8.1] — 2026-07-27 - -### Added - -- Added self-hosted version and license badges; wired a badges job into pipeline.yml. - -## [v4.8.0] — 2026-07-26 - -MCP interface reworked from a single generic passthrough to dedicated, typed tools. - -### Changed - -- **`/mcp` now exposes ~24 dedicated typed tools** grouped by family (market data, account, positions, orders, history, terminal, backtest) instead of the lone generic `request` passthrough. Each tool has typed params + a description the agent reads — e.g. `create_order(symbol, type, volume, price?, sl?, tp?)`, `get_rates(symbol, timeframe, count?)`, `close_position(ticket, volume?)` — so the tool schema IS the documentation. Order/position mutation tools carry an explicit irreversible-live-account note. A generic `request` + `endpoints` catalog remain as a fallback for routes without a dedicated tool. Every tool still runs the same handler + auth + MT5 locking as a real HTTP call (in-process). README + skill + plugin docs updated. -- Submitting a backtest (`POST /backtest`) is **not** exposed as a tool — that route takes a multipart file upload; `get_backtest` polls status/report/log/tail, and new runs are submitted via the REST API. - -## [v4.7.0] — 2026-07-26 - -New MCP interface — the API is now also driveable over the Model Context Protocol. - -### Added - -- **MCP server mounted at `/mcp`** (streamable-HTTP), in the same process as the REST API on every terminal. Three tools mirror the whole REST surface: `ping` (lock-free liveness), `endpoints` (the route catalog), and `request(method, path, query, body)` — call any REST endpoint, running the exact same handler + auth + MT5 locking as a real HTTP request. Same bearer auth as REST (empty `api_token` = auth off; a configured token requires `Authorization: Bearer ` on `/mcp` too). See `mt5api/mcp_server.py`. -- **`@psyb0t/mt5-httpapi` ClawHub plugin** (`.agents/plugins/mt5-httpapi/`) — a stdio↔HTTP MCP bridge (`mcp-remote`) so an OpenClaw/MCP agent can drive a running terminal. Point `MT5_API_URL` at the terminal's base (+ `MT5_API_TOKEN` if auth is on); the reachable endpoint is `$MT5_API_URL/mcp/`. CI publishes it to ClawHub alongside the skill. -- README and the `mt5-httpapi` skill gain an **MCP interface** section. - -### Note - -- mt5api is a Flask/WSGI app; the ASGI MCP app is bridged in via `a2wsgi` behind `/mcp` (there's a `TODO` to migrate mt5api to FastAPI and drop the bridge). New runtime deps `mcp` + `a2wsgi` are installed by `scripts/start.bat` on boot and tracked in `requirements-api.txt`. No REST endpoint or trading-path change. - -## [v4.6.0] — 2026-07-26 - -Hotfix for a boot-blocking regression introduced in v4.5.0, plus a `make lint` / `make format` gate so that class of bug cannot reach the VM again. - -### Fixed - -- **v4.5.0's `scripts/acquire_lock.ps1` deadlocked every boot.** The file contained em-dashes in comments *and in string literals*, with no UTF-8 BOM. Windows PowerShell 5.1 reads `.ps1` as ANSI, so those bytes were mangled, the string literals terminated early, and the script died with `Unexpected token` / `The hash literal was incomplete`. The script is now pure ASCII, which needs no BOM to stay stable. -- **A failing lock helper was indistinguishable from a held lock.** `acquire_lock.ps1` used exit 1 for "another instance holds the lock" — the same code PowerShell returns for a parse error. So the syntax error above made `start.bat` conclude the lock was taken and exit, on every boot, which is the exact deadlock the lock rewrite was meant to remove. Exit codes are now distinct: `0` acquired, `10` held, anything else means the helper itself failed. On that third case `scripts/start.bat` and `scripts/install.bat` log a warning and fall back to a plain `mkdir` lock, so a broken helper can degrade single-instance safety but can never block boot. -- `scripts/event-log-tailer.ps1`: em-dashes in comments replaced with ASCII (same mojibake hazard); `Append-Full` renamed to `Add-FullLogLine` (`Add` is an approved PowerShell verb); its `catch {}` no longer swallows silently — a failed `full.log` append is now reported into `windows-events.log`, which is not the contended file that just failed. - -### Added - -- **`make lint`** — lints every tracked-or-new script in a throwaway Docker image (built, run, `docker rmi`'d, repo mounted read-only), mirroring how `make test` works. Six checks: a self-test of its own non-ASCII detector, the `.ps1` ASCII gate, a `.ps1` parse check, PSScriptAnalyzer, shellcheck (warning and above), and shfmt. `Dockerfile.lint` + `scripts/lint.sh`. - - The detector self-test exists because a checker that silently stops detecting is worse than no checker — the same failure mode as the healthcheck fixed in v4.5.0. It verifies the pattern still flags a real em-dash and still passes pure ASCII, and fails the whole run if it cannot tell them apart. - - Files are selected with `git ls-files --cached --others --exclude-standard`, so brand-new scripts are covered while gitignored local scratch is not. The vendored `scripts/defender-remover/` tree is excluded. -- **`make format`** — applies shfmt in place. Delegates to `scripts/lint.sh --format` so it shares file selection with `make lint`; when the two had separate lists, `format` skipped untracked files that `lint` still flagged and the gate could never go green. - -### Changed - -- Applied shfmt formatting to `run.sh`, `test.sh`, `scripts/rotate-logs.sh`, and `tests/real/run.sh`. Whitespace and layout only — no behavior change. In `test.sh` this expands single-line function bodies (`pass() { echo …; PASS=…; }`) onto separate lines, which is most of the diff. -- README's Make Targets list now includes `lint`, `format`, and `test` (`test` had been missing). - -## [v4.5.0] — 2026-07-26 - -Boot-lock and reboot hardening for the Windows VM, a critical healthcheck false-positive fix, and a `make test` build fix. Also adds third-party license notices for the vendored Windows Defender removal tool. - -### Fixed - -- **Healthcheck reported dead terminals as healthy.** `scripts/healthcheck.sh` probed each terminal with `curl … -w '%{http_code}' … || echo 000`. curl already prints `000` on a failed connection, so the `|| echo 000` fallback appended a second one — yielding `000000`, which compared unequal to `000` and marked the port UP. A full outage could sit behind a green Docker healthcheck indefinitely. The probe now whitelists a valid HTTP status shape (`[1-5][0-9][0-9]`) and fails closed; any real status, including 4xx/5xx, proves the process is listening. -- **Reboot-orphaned boot locks deadlocked the stack.** `%SHARED%\start.running` (and install.bat's lock) live on the host-mounted volume and survive a VM reboot. The auto-reboot task fires `shutdown /r /t 0 /f` with no grace period and can land mid-run, stranding the lock so every later boot bailed on the orphan forever. New `scripts/acquire_lock.ps1` stamps each lock with the OS boot time, so a lock from a previous boot is provably ownerless and is cleared automatically; a live same-boot instance still blocks. `scripts/start.bat` and `scripts/install.bat` acquire through it and release via a single `release_lock`. -- **`make test` was dead on a clean checkout.** `Dockerfile.test` COPYed `config/requirements.txt`, which had been retired and is gitignored, so the build failed with `"/config/requirements.txt": not found`. Tests now install from the tracked `requirements-api.txt`, additionally COPY `scripts/config_helper.py` (loaded by `tests/test_terminal_instances.py`), and skip the live-deployment `tests/real/` suite in the default offline run. - -### Added - -- `scripts/reboot.bat` — the single reboot path for the VM. Writes `rebooting.flag` and releases both lock dirs in one place, replacing three separate inline flag+shutdown+rmdir sequences that had to be kept in sync. -- `requirements-api.txt` — tracked source of truth for the mt5api HTTP server's Python dependencies (replacing the retired, gitignored `config/requirements.txt`). `scripts/start.bat` installs the same set inline on every boot. Retains the documented `numpy<2` pin — the MetaTrader5 `5.0.5735` wheel is built against numpy 1.x, and under numpy 2.x `order_send` fails with `(-2, 'Unnamed arguments not allowed')`. - -### Changed - -- Renamed the boot entrypoint `start-mt5.bat` → `start.bat` and its log `start-mt5.log` → `start.log`; README file-tree and log references updated to match. - -### Licensing - -- Added `THIRD_PARTY.md` and `scripts/defender-remover/LICENSE` (GPL-3.0). `scripts/defender-remover/` is a verbatim vendored copy of the third-party windows-defender-remover tool, which is GPL-3.0-licensed; the rest of mt5-httpapi stays WTFPL. Documents the licensing of what the repo actually distributes. - -## [v4.4.3] — 2026-07-26 - -Docs: hardened the `mt5-httpapi` agent skill with explicit destructive-operation guardrails and an auth/exfil-style warning. Renamed the safety section to `## Security & safety`, spelled out that trade/order/position mutations are irreversible with no client-side auto-retry, and made the "empty `api_token` = unauthenticated" warning more explicit. No behavior, endpoint, or API change. - -## [v4.4.2] — 2026-07-25 +- Deleted the stray `.claude-plugin/marketplace.json` from this repo. -CI: switch the ClawHub skill publish to `clawhub-publish.yml` directly — the `clawhub-skills-publish-workflow.yml` shim was removed upstream. No trading-path or API change. ## [v4.3.1] — 2026-05-17 diff --git a/README.md b/README.md index 7e70747..b6d2349 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,7 @@ Run multiple brokers, accounts, and cloned terminal instances at the same time. - **Built-in technical analysis** — [wickworks](https://github.com/psyb0t/docker-wickworks) calculates indicators and SMC primitives server-side without exposing its own port. - **MCP for the robot overlords** — one endpoint per terminal or one unified endpoint for the whole pile, plus Claude Code, Codex, and OpenClaw integrations. - **Optional multi-VM fuckery** — spread terminals across VMs or NUMA nodes, then reach them through Tailscale or Cloudflare Tunnel if you need to. +- **Chart Deployments** for remote EA deployment: stage `.ex5` + `.set` files, declare a deployment (symbol + timeframe), and a resident loader EA inside the terminal reconciles charts automatically — no RDP or manual attach needed. ## Requirements @@ -129,6 +130,7 @@ The old README became a massive wall of API shit, so the details now live in sep | Copy working curl and Go examples instead of guessing | [Clients and examples](docs/clients-and-examples.md) | | Operate the bastard: Make targets, ports, remote access, concurrency, and logs | [Operations](docs/operations.md) | | Split terminals across several Windows VMs or NUMA nodes | [Multi-VM setup](docs/multi-vm-setup.md) | +| Deploy EAs to charts remotely over HTTP | [Chart Deployments (chartctl)](docs/chart-control-protocol.md) | ## API at a glance @@ -153,6 +155,23 @@ curl -H "Authorization: Bearer $MT5_API_TOKEN" \ That is just the hello-world shit. The [REST API docs](docs/rest-api.md) link to every endpoint and response shape. +**Chart Deployments** endpoints (live-mode terminals only; gated by `chartctl.enabled`): + +```bash +# Stage an expert and set file, then deploy +curl -H "Authorization: Bearer $MT5_API_TOKEN" -F "expert=@MyEA.ex5" "$MT5_API_URL/experts" +curl -H "Authorization: Bearer $MT5_API_TOKEN" -F "set=@params.set" "$MT5_API_URL/sets" +curl -H "Authorization: Bearer $MT5_API_TOKEN" -X POST "$MT5_API_URL/deployments" \ + -H "Content-Type: application/json" \ + -d '{"expert":"MyEA.ex5","set":"params.set","symbol":"EURUSD","timeframe":"M15"}' + +# Verify — status flips to "running" once the loader confirms attach +curl -H "Authorization: Bearer $MT5_API_TOKEN" "$MT5_API_URL/deployments" +curl -H "Authorization: Bearer $MT5_API_TOKEN" "$MT5_API_URL/loader" +``` + +Full protocol contract at [`docs/chart-control-protocol.md`](docs/chart-control-protocol.md). + ## Development ```bash diff --git a/assets/autoit/AutoIt3_x64.exe b/assets/autoit/AutoIt3_x64.exe new file mode 100644 index 0000000..43d81f5 Binary files /dev/null and b/assets/autoit/AutoIt3_x64.exe differ diff --git a/assets/autoit/inspect_options.au3 b/assets/autoit/inspect_options.au3 new file mode 100644 index 0000000..8380279 --- /dev/null +++ b/assets/autoit/inspect_options.au3 @@ -0,0 +1,124 @@ +; inspect_options.au3 +; Non-destructive inspector for MT5's Tools->Options dialog. NO #includes +; (an undefined include function pops a blocking error dialog). Finds the MT5 +; window containing (the login), opens Options (Ctrl+O), walks +; every tab, and logs the VISIBLE controls (ClassNN, pos, text) so we can +; identify the Expert Advisors tab index + the WebRequest checkbox/list, then +; cancels with Esc. + +Opt("WinTitleMatchMode", 2) +Opt("SendKeyDelay", 15) + +Global $gLog = -1 + +Func LogW($s) + If $gLog <> -1 Then FileWrite($gLog, $s & @CRLF) +EndFunc + +Func DumpVisible($hWin, $tag) + LogW("--- " & $tag & " ---") + Local $cl = WinGetClassList($hWin) + Local $arr = StringSplit(StringStripCR($cl), @LF) + Local $seen = "|" + For $i = 1 To $arr[0] + Local $c = $arr[$i] + If $c = "" Then ContinueLoop + If StringInStr($seen, "|" & $c & "|") > 0 Then ContinueLoop + $seen &= $c & "|" + For $n = 1 To 80 + Local $ctrl = $c & $n + Local $pos = ControlGetPos($hWin, "", $ctrl) + If @error Then ExitLoop + Local $vis = ControlCommand($hWin, "", $ctrl, "IsVisible", "") + If $vis = 1 Then + Local $txt = ControlGetText($hWin, "", $ctrl) + LogW(" " & $ctrl _ + & " xywh=" & $pos[0] & "," & $pos[1] & "," & $pos[2] & "," & $pos[3] _ + & " text='" & StringLeft(StringReplace($txt, @CRLF, " "), 70) & "'") + EndIf + Next + Next +EndFunc + +; ---- args ---- +; 3 args: match, pid, logpath. 2 args (legacy): match, logpath. +If $CmdLine[0] < 2 Then Exit 10 +Global $match = $CmdLine[1] +Global $pid = 0 +Global $logpath +If $CmdLine[0] >= 3 Then + $pid = Int($CmdLine[2]) + $logpath = $CmdLine[3] +Else + $logpath = $CmdLine[2] +EndIf + +$gLog = FileOpen($logpath, 2) +If $gLog = -1 Then Exit 11 +LogW("=== inspect_options match=" & $match & " pid=" & $pid & " ===") + +; ---- find + activate MT5 (visible windows only; by owner pid when known — +; same-login terminal clones share the exact same window title) ---- +Local $wl = WinList() +Local $hMT5 = 0 +For $i = 1 To $wl[0][0] + Local $h = $wl[$i][1] + If $wl[$i][0] = "" Then ContinueLoop + If BitAND(WinGetState($h), 2) = 0 Then ContinueLoop + If $pid > 0 And WinGetProcess($h) <> $pid Then ContinueLoop + If StringInStr($wl[$i][0], $match) > 0 Then $hMT5 = $h +Next +If $hMT5 = 0 Then + LogW("RESULT=FAIL reason=mt5_window_not_found") + FileClose($gLog) + Exit 2 +EndIf +WinActivate($hMT5) +Sleep(600) +LogW("mt5='" & WinGetTitle($hMT5) & "' win_pid=" & WinGetProcess($hMT5)) + +; ---- open Options ---- +Send("^o") +Local $hOpt = WinWait("Options", "", 10) +If $hOpt <> 0 And $pid > 0 And WinGetProcess($hOpt) <> $pid Then $hOpt = 0 +If $hOpt = 0 Then + LogW("RESULT=FAIL reason=options_not_found") + FileClose($gLog) + Exit 3 +EndIf +WinActivate($hOpt) +Sleep(300) +LogW("options='" & WinGetTitle($hOpt) & "'") +LogW("classlist=" & StringReplace(WinGetClassList($hOpt), @LF, " | ")) + +; ---- find tab control, walk tabs ---- +Local $tabClass = "" +Local $cls = StringSplit(StringStripCR(WinGetClassList($hOpt)), @LF) +For $i = 1 To $cls[0] + If StringInStr($cls[$i], "SysTabControl32") > 0 Then + $tabClass = $cls[$i] + ExitLoop + EndIf +Next +LogW("tabClass='" & $tabClass & "'") + +If $tabClass <> "" Then + Local $cnt = ControlCommand($hOpt, "", $tabClass, "GetItemCount", "") + LogW("tab_count=" & $cnt) + If $cnt >= 1 Then + For $t = 1 To $cnt + ControlCommand($hOpt, "", $tabClass, "CurrentTab", $t) + Sleep(250) + DumpVisible($hOpt, "tab#" & $t) + Next + Else + DumpVisible($hOpt, "single") + EndIf +Else + DumpVisible($hOpt, "no-tabctl") +EndIf + +LogW("RESULT=OK") +Send("{ESC}") +FileClose($gLog) +Exit 0 diff --git a/assets/autoit/selftest.au3 b/assets/autoit/selftest.au3 new file mode 100644 index 0000000..f544dc1 --- /dev/null +++ b/assets/autoit/selftest.au3 @@ -0,0 +1,60 @@ +; selftest.au3 [] +; Minimal, NO external includes. Confirms AutoIt runs at all, reports whether +; it's elevated, and whether it can find + drive the (elevated) MT5 window. +; With 3 args the second is the terminal64.exe pid (0 = unknown) — same-login +; terminal clones share the exact same window title, so only the pid pins one. + +Opt("WinTitleMatchMode", 2) +Opt("SendKeyDelay", 15) + +Local $match = ($CmdLine[0] >= 1) ? $CmdLine[1] : "?" +Local $pid = 0 +Local $logpath = @ScriptDir & "\selftest.log" +If $CmdLine[0] >= 3 Then + $pid = Int($CmdLine[2]) + $logpath = $CmdLine[3] +ElseIf $CmdLine[0] = 2 Then + $logpath = $CmdLine[2] +EndIf + +Local $h = FileOpen($logpath, 2) +If $h = -1 Then Exit 11 +FileWrite($h, "started args=" & $CmdLine[0] & " match=" & $match & " pid=" & $pid & @CRLF) +FileWrite($h, "IsAdmin=" & IsAdmin() & @CRLF) + +Local $wl = WinList() +FileWrite($h, "windows_total=" & $wl[0][0] & @CRLF) +Local $hMT5 = 0 +For $i = 1 To $wl[0][0] + If $wl[$i][0] <> "" And StringInStr($wl[$i][0], $match) > 0 Then + FileWrite($h, " match_win='" & $wl[$i][0] & "' visible=" _ + & (BitAND(WinGetState($wl[$i][1]), 2) > 0 ? 1 : 0) _ + & " pid=" & WinGetProcess($wl[$i][1]) & @CRLF) + If BitAND(WinGetState($wl[$i][1]), 2) = 0 Then ContinueLoop + If $pid > 0 And WinGetProcess($wl[$i][1]) <> $pid Then ContinueLoop + $hMT5 = $wl[$i][1] + EndIf +Next +If $hMT5 = 0 Then + FileWrite($h, "RESULT=FAIL reason=no_mt5_window" & @CRLF) + FileClose($h) + Exit 2 +EndIf + +WinActivate($hMT5) +Sleep(600) +FileWrite($h, "active_title='" & WinGetTitle("[ACTIVE]") & "'" & @CRLF) + +Send("^o") +Local $hOpt = WinWait("Options", "", 8) +If $hOpt <> 0 And $pid > 0 And WinGetProcess($hOpt) <> $pid Then $hOpt = 0 +If $hOpt = 0 Then + FileWrite($h, "RESULT=FAIL reason=options_did_not_open (input blocked? UIPI/elevation)" & @CRLF) + FileClose($h) + Exit 3 +EndIf +FileWrite($h, "options_opened title='" & WinGetTitle($hOpt) & "'" & @CRLF) +Send("{ESC}") +FileWrite($h, "RESULT=OK" & @CRLF) +FileClose($h) +Exit 0 diff --git a/assets/autoit/set_webrequest.au3 b/assets/autoit/set_webrequest.au3 new file mode 100644 index 0000000..8d97db3 --- /dev/null +++ b/assets/autoit/set_webrequest.au3 @@ -0,0 +1,232 @@ +; set_webrequest.au3 [] +; Adds the URLs in (one per line) to MT5's Tools->Options->Expert +; Advisors "Allow WebRequest for listed URL" list, for the terminal whose +; window title contains (the login). NO #includes (an undefined +; include function pops a blocking error dialog). +; +; (optional, 0 = unknown) is the terminal64.exe process id. Cloned +; terminals of the same account have IDENTICAL window titles, so the login is +; ambiguous — the pid is the only thing that pins the right terminal. When +; given, both the main window and the Options dialog are matched by owner pid. +; +; MT5's URL list is a SysListView32 with a greyed "add new URL like ..." row at +; the bottom; double-clicking it opens an inline edit. We type the URL + Enter, +; which commits and produces a fresh add-row below. Then OK. + +Opt("WinTitleMatchMode", 2) +Opt("SendKeyDelay", 20) +Opt("SendKeyDownDelay", 5) +Opt("MouseCoordMode", 2) ; coords relative to the control for ControlClick + +Global $gLog = -1 +Func LogW($s) + If $gLog <> -1 Then FileWrite($gLog, $s & @CRLF) +EndFunc + +; Scan Button1..25 for the first VISIBLE control whose text contains $needle. +Func FindButtonByText($hWin, $needle) + For $n = 1 To 25 + Local $ctrl = "Button" & $n + ControlGetPos($hWin, "", $ctrl) + If @error Then ContinueLoop + If ControlCommand($hWin, "", $ctrl, "IsVisible", "") <> 1 Then ContinueLoop + If StringInStr(ControlGetText($hWin, "", $ctrl), $needle) > 0 Then Return $ctrl + Next + Return "" +EndFunc + +Func DumpControls($hWin, $tag) + LogW(" [" & $tag & "] classlist=" & StringReplace(WinGetClassList($hWin), @LF, " | ")) +EndFunc + +; Log the current listview rows (so we can verify what MT5 actually holds). +Func DumpList($hWin, $tag) + Local $cnt = ControlListView($hWin, "", "SysListView321", "GetItemCount") + Local $s = "" + For $r = 0 To $cnt - 1 + $s &= "[" & ControlListView($hWin, "", "SysListView321", "GetText", $r, 0) & "] " + Next + LogW(" list(" & $tag & ") count=" & $cnt & " items=" & $s) + Return $cnt +EndFunc + +; Visible top-level window for this terminal: by owner pid when known +; (title as tie-break among the pid's windows), else by title substring. +; WinList() alone is NOT enough — it returns hidden windows, and same-login +; terminal clones share the exact same title. +Func FindMainWindow($match, $pid) + Local $wl = WinList() + Local $best = 0 + For $i = 1 To $wl[0][0] + Local $h = $wl[$i][1] + Local $title = $wl[$i][0] + If $title = "" Then ContinueLoop + If BitAND(WinGetState($h), 2) = 0 Then ContinueLoop ; visible only + If $pid > 0 Then + If WinGetProcess($h) <> $pid Then ContinueLoop + If StringInStr($title, $match) > 0 Then Return $h + If $best = 0 Then $best = $h + Else + If StringInStr($title, $match) > 0 Then $best = $h + EndIf + Next + Return $best +EndFunc + +; Focus the terminal and open Tools->Options (Ctrl+O), retrying, and only +; accept an Options window owned by our pid (another terminal's dialog, or +; any window with "Options" in its title, must not be driven). +Func OpenOptions($hMT5, $pid) + For $try = 1 To 3 + WinActivate($hMT5) + If WinWaitActive($hMT5, "", 3) = 0 Then + LogW(" activate attempt " & $try & " failed (active='" & WinGetTitle("[ACTIVE]") & "')") + ContinueLoop + EndIf + Send("^o") + Local $t = TimerInit() + While TimerDiff($t) < 5000 + Local $hOpt = WinWait("Options", "", 1) + If $hOpt <> 0 Then + If $pid = 0 Or WinGetProcess($hOpt) = $pid Then Return $hOpt + LogW(" ignoring foreign Options window (pid=" & WinGetProcess($hOpt) & ")") + EndIf + WEnd + LogW(" ctrl+o attempt " & $try & ": no Options dialog") + Next + Return 0 +EndFunc + +; ---- args ---- +; 4 args: match, pid, urlfile, logpath. 3 args (legacy caller): match, +; urlfile, logpath with pid unknown. +If $CmdLine[0] < 3 Then Exit 10 +Global $match = $CmdLine[1] +Global $pid = 0 +Global $urlfile, $logpath +If $CmdLine[0] >= 4 Then + $pid = Int($CmdLine[2]) + $urlfile = $CmdLine[3] + $logpath = $CmdLine[4] +Else + $urlfile = $CmdLine[2] + $logpath = $CmdLine[3] +EndIf + +$gLog = FileOpen($logpath, 2) +If $gLog = -1 Then Exit 11 +LogW("=== set_webrequest match=" & $match & " pid=" & $pid & " ===") + +; ---- read urls ---- +Global $raw = FileRead($urlfile) +Global $urls = StringSplit(StringStripCR($raw), @LF) +Local $n_urls = 0 +For $i = 1 To $urls[0] + If StringStripWS($urls[$i], 3) <> "" Then $n_urls += 1 +Next +LogW("urls_in_file=" & $n_urls) + +; ---- find + activate MT5 ---- +Local $hMT5 = FindMainWindow($match, $pid) +If $hMT5 = 0 Then + LogW("RESULT=FAIL reason=mt5_window_not_found") + FileClose($gLog) + Exit 2 +EndIf +LogW("mt5='" & WinGetTitle($hMT5) & "' win_pid=" & WinGetProcess($hMT5)) + +; ---- open Options ---- +Local $hOpt = OpenOptions($hMT5, $pid) +If $hOpt = 0 Then + LogW("RESULT=FAIL reason=options_not_found") + FileClose($gLog) + Exit 3 +EndIf +WinActivate($hOpt) +Sleep(400) + +; ---- ensure the Expert Advisors tab is active (WebRequest checkbox present) ---- +Local $cb = FindButtonByText($hOpt, "WebRequest") +Local $tries = 0 +While $cb = "" And $tries < 12 + Send("^{TAB}") ; cycle property-sheet tabs + Sleep(250) + $cb = FindButtonByText($hOpt, "WebRequest") + $tries += 1 +WEnd +If $cb = "" Then + LogW("RESULT=FAIL reason=webrequest_checkbox_not_found") + Send("{ESC}") + FileClose($gLog) + Exit 4 +EndIf +LogW("checkbox=" & $cb & " text='" & ControlGetText($hOpt, "", $cb) & "'") + +; ---- ensure the checkbox is checked (real click triggers MT5's enable logic) ---- +If ControlCommand($hOpt, "", $cb, "IsChecked", "") <> 1 Then + ControlClick($hOpt, "", $cb) + Sleep(300) + LogW("checkbox now checked=" & ControlCommand($hOpt, "", $cb, "IsChecked", "")) +Else + LogW("checkbox already checked") +EndIf + +; ---- the URL list ---- +Local $lp = ControlGetPos($hOpt, "", "SysListView321") +If @error Then + LogW("RESULT=FAIL reason=listview_not_found") + Send("{ESC}") + FileClose($gLog) + Exit 5 +EndIf +LogW("list xywh=" & $lp[0] & "," & $lp[1] & "," & $lp[2] & "," & $lp[3]) +Local $rowH = 17 + +; ---- clear existing entries (a PUT sets the full list) ---- +; Select the first data row and press Delete, repeatedly. The greyed "add new +; URL" row can't be deleted, so extra iterations are harmless no-ops. +DumpList($hOpt, "before-clear") +For $k = 1 To 40 + Local $before = ControlListView($hOpt, "", "SysListView321", "GetItemCount") + If $before <= 0 Then ExitLoop + ControlClick($hOpt, "", "SysListView321", "left", 1, Int($lp[2] / 2), 10) + Sleep(60) + Send("{DELETE}") + Sleep(90) + If ControlListView($hOpt, "", "SysListView321", "GetItemCount") >= $before Then ExitLoop +Next +DumpList($hOpt, "after-clear") + +; ---- add each url ---- +Local $added = 0 +Local $idx = 0 +For $i = 1 To $urls[0] + Local $u = StringStripWS($urls[$i], 3) + If $u = "" Then ContinueLoop + ; add-row is the last row: relative Y grows by rowH per existing entry + Local $ry = 10 + ($idx * $rowH) + If $ry > $lp[3] - 4 Then $ry = $lp[3] - 8 ; clamp into the control + ControlClick($hOpt, "", "SysListView321", "left", 2, Int($lp[2] / 2), $ry) + Sleep(250) + If $i = 1 Then DumpControls($hOpt, "after-first-dblclick") + ; type the URL into whatever inline edit appeared, then commit + Send("^a") ; select any placeholder text + Send($u, 1) ; raw send (URLs contain / : . which are literal in raw mode) + Sleep(120) + Send("{ENTER}") + Sleep(300) + LogW("typed[" & $idx & "] ry=" & $ry & " url=" & $u) + $added += 1 + $idx += 1 +Next +DumpList($hOpt, "final") + +; ---- confirm with OK ---- +Local $ok = FindButtonByText($hOpt, "OK") +If $ok = "" Then $ok = "Button8" +ControlClick($hOpt, "", $ok) +Sleep(400) + +LogW("RESULT=OK added=" & $added) +FileClose($gLog) +Exit 0 diff --git a/assets/binaries.lock.json b/assets/binaries.lock.json index aacddfd..ba53824 100644 --- a/assets/binaries.lock.json +++ b/assets/binaries.lock.json @@ -21,6 +21,17 @@ "source": "https://www.sordum.org/downloads/?power-run=", "signature": "malformed", "note": "REPACKED, NOT PRISTINE. Arrived vendored inside the defender-remover toolkit rather than from Sordum directly. Its certificate directory is not a well-formed WIN_CERTIFICATE (declared length 776284822, revision 0xc496, type 14951 against the required 0x200/2) and 512 bytes trail it, so the Authenticode signature cannot validate. Its hash matches no Sordum release: official v1.6.0.0 x64 is da77bc401ef0d7b8e23be3a9387660172aea176cd9d1248034130811d29942c9 (934464 bytes) and current v1.9.0.0 x64 is b305126c2881073a53073bd1782d0649764d198562e02fc02395fc07637dcec5. Not known to be malicious — repacking is normal for that toolkit — but it is unverifiable. Replacing it with an official Sordum download would let this entry move to signature=valid." + }, + { + "path": "assets/autoit/AutoIt3_x64.exe", + "sha256": "5d69a932a077fee044b193c28e84564143f5c7e51079ab48e88fef74ab0b77b7", + "size": 1107552, + "product": "AutoIt v3 Script", + "version": "3.3.18.8", + "vendor": "Jonathan Bennett & AutoIt Team", + "source": "https://www.autoitscript.com/site/autoit/downloads/", + "signature": "valid", + "note": "AutoIt interpreter used for GUI automation of the WebRequest allowlist dialog on the Windows VM. Version and product name read from the binary's own version resource (FileVersion 3.3.18.8, Copyright 1999-2025); Authenticode digest verified by scripts/verify_binaries.py against the signing certificate, not against a separately downloaded upstream copy." } ] } diff --git a/assets/experts/MT5ChartLoader.mq5 b/assets/experts/MT5ChartLoader.mq5 new file mode 100644 index 0000000..b8c109b --- /dev/null +++ b/assets/experts/MT5ChartLoader.mq5 @@ -0,0 +1,62 @@ +//+------------------------------------------------------------------+ +//| MT5ChartLoader.mq5 | +//| Reference chart-deployment loader for mt5-httpapi chartctl. | +//| | +//| Attach this to ANY single chart in the terminal (it doesn't | +//| matter which symbol/timeframe — it manages other charts, not | +//| its own). It reconciles the terminal to the desired deployments | +//| written by the API and reports live state back. | +//| | +//| It never trades. It only opens/closes charts and applies | +//| templates. Safe to run on a live account. | +//| | +//| This EA is intentionally thin: all logic lives in the portable | +//| include ChartControl.mqh so you can drop the same capability | +//| into your own resident EA (e.g. an account tracker) instead of | +//| running this standalone. See docs/chart-control-protocol.md. | +//+------------------------------------------------------------------+ +#property copyright "mt5-httpapi" +#property link "https://github.com/psyb0t/mt5-httpapi" +#property version "1.00" +#property strict + +#include + +input int InpLoopSeconds = 1; // reconcile timer period (seconds) + +CChartControl ctl; + +//+------------------------------------------------------------------+ +int OnInit() +{ + // true: if another loader already owns the terminal, close this chart + // and vanish (mt5start.ini [StartUp] re-attaches us on every launch; + // this keeps that idempotent instead of accumulating loader charts). + if(!ctl.Init(true)) + return INIT_FAILED; + EventSetTimer(InpLoopSeconds < 1 ? 1 : InpLoopSeconds); + Comment("MT5ChartLoader active — chartctl loader\n", + ctl.IsOwner() ? "role: OWNER" : "role: passive (another loader owns the mutex)"); + return INIT_SUCCEEDED; +} + +//+------------------------------------------------------------------+ +void OnTimer() +{ + ctl.Tick(); +} + +//+------------------------------------------------------------------+ +void OnDeinit(const int reason) +{ + EventKillTimer(); + ctl.Deinit(); + Comment(""); +} + +//+------------------------------------------------------------------+ +//| No trading. OnTick is intentionally empty; the loader is timer- | +//| driven so it works on weekends and on disconnected symbols. | +//+------------------------------------------------------------------+ +void OnTick() {} +//+------------------------------------------------------------------+ diff --git a/assets/experts/include/ChartControl.mqh b/assets/experts/include/ChartControl.mqh new file mode 100644 index 0000000..a2df55c --- /dev/null +++ b/assets/experts/include/ChartControl.mqh @@ -0,0 +1,787 @@ +//+------------------------------------------------------------------+ +//| ChartControl.mqh | +//| Chart Control Protocol v1 — reference implementation | +//| | +//| Drop this into a resident EA to make it the terminal's chart | +//| deployment loader. It reconciles the terminal's charts to the | +//| desired state written by mt5-httpapi's chartctl endpoints. | +//| | +//| Usage inside your EA: | +//| #include | +//| CChartControl ctl; | +//| int OnInit(){ if(!ctl.Init()) return INIT_FAILED; | +//| EventSetTimer(1); return INIT_SUCCEEDED; } | +//| void OnTimer(){ ctl.Tick(); } | +//| void OnDeinit(const int r){ ctl.Deinit(); } | +//| | +//| Contract & file formats: docs/chart-control-protocol.md | +//+------------------------------------------------------------------+ +#property strict + +#define CHARTCTL_PROTOCOL 1 +#define CHARTCTL_VERSION "1.0.2" +#define CHARTCTL_DIR "chartctl" // under MQL5\Files\ +#define CHARTCTL_MUTEX_GV "chartctl_loader_owner" +#define CHARTCTL_ID_INPUT "__chartctl_id" + +//--- one desired deployment +struct ChartCtlDeployment +{ + string id; + string expert; // expert short name (CHART_EXPERT_NAME match target) + string templ; // template path for ChartApplyTemplate; leading \ = relative to \MQL5 (e.g. \Files\chartctl\dep_x.tpl) + string symbol; + string timeframe; + bool enabled; +}; + +//--- what we observed on one chart +struct ChartCtlChart +{ + long chart_id; + string symbol; + string timeframe; + string expert; + bool expert_enabled; + string deployment_id; // parsed from the chart's __chartctl_id if present +}; + +//+------------------------------------------------------------------+ +class CChartControl +{ +private: + bool m_owner; // did we win the single-loader mutex? + long m_applied_revision; // last desired revision we reconciled + long m_last_revision_seen; + datetime m_started; + // Per-deployment error slots (parallel arrays keyed by deployment id). + // A single shared slot let one deployment's error mask another's. + string m_err_ids[]; + string m_err_codes[]; + string m_err_details[]; + datetime m_err_times[]; // drives the failed-attach retry cooldown + + //--- file helpers ------------------------------------------------- + bool ReadFile(const string relpath, string &out); + bool WriteFileAtomic(const string relpath, const string content); + void DeleteFileSafe(const string relpath); + + //--- json (minimal, tailored to our own compact output) ---------- + string JsonStr(const string key, const string s, const string json); + long JsonNum(const string key, const string json); + bool ExtractDeployments(const string json, ChartCtlDeployment &out[]); + string JsonEscape(const string s); + + //--- reconcile ---------------------------------------------------- + void ScanCharts(ChartCtlChart &out[]); + long FindChartFor(const string dep_id, ChartCtlChart &charts[]); + long FindAdoptableChart(const ChartCtlDeployment &dep, ChartCtlChart &charts[]); + bool StampChart(const long cid, const string dep_id); + bool AttachDeployment(const ChartCtlDeployment &dep); + void DetachChart(const long chart_id); + ENUM_TIMEFRAMES TF(const string s); + void RecordError(const string id, const string code, const string detail); + void ClearError(const string id); + bool InRetryCooldown(const string id); + + //--- observed + command output ----------------------------------- + void WriteObserved(ChartCtlDeployment &desired[], ChartCtlChart &charts[]); + void HandleCommand(); + +public: + CChartControl(void); + // close_own_chart_on_duplicate: standalone loaders pass true so a + // second copy (e.g. re-fired by mt5start.ini [StartUp] on every + // launch) closes its own chart and vanishes instead of idling. + // EAs that embed this module MUST leave it false — closing the chart + // would kill the host EA (e.g. an account tracker) too. + bool Init(const bool close_own_chart_on_duplicate=false); + void Tick(void); + void Deinit(void); + bool IsOwner(void) const { return m_owner; } +}; + +//+------------------------------------------------------------------+ +CChartControl::CChartControl(void) +{ + m_owner = false; + m_applied_revision = -1; + m_last_revision_seen = -1; + m_started = 0; +} + +//+------------------------------------------------------------------+ +//| Claim the single-loader mutex via a terminal GlobalVariable. | +//+------------------------------------------------------------------+ +bool CChartControl::Init(const bool close_own_chart_on_duplicate) +{ + m_started = TimeCurrent(); + + // If another loader already holds the mutex and is fresh, step aside. + if(GlobalVariableCheck(CHARTCTL_MUTEX_GV)) + { + datetime held = (datetime)GlobalVariableGet(CHARTCTL_MUTEX_GV); + // Treat a mutex touched within 120s as a live owner. + if(TimeCurrent() - held < 120) + { + m_owner = false; + if(close_own_chart_on_duplicate) + { + // Standalone duplicate (e.g. [StartUp] re-fired on relaunch): + // remove ourselves entirely so charts never accumulate. + Print("ChartControl: live owner exists; closing own chart."); + ChartClose(ChartID()); + return true; // unloading anyway; don't fail the host + } + Print("ChartControl: another loader owns the mutex; standing down."); + return true; // do NOT fail the host EA — just stay passive + } + } + GlobalVariableSet(CHARTCTL_MUTEX_GV, (double)TimeCurrent()); + GlobalVariableTemp(CHARTCTL_MUTEX_GV); // auto-clears if terminal exits + m_owner = true; + PrintFormat("ChartControl v%s active (owner). dir=MQL5\\Files\\%s", + CHARTCTL_VERSION, CHARTCTL_DIR); + return true; +} + +//+------------------------------------------------------------------+ +void CChartControl::Tick(void) +{ + if(!m_owner) + { + // Passive mode: try to reclaim if the previous owner is gone. + if(!GlobalVariableCheck(CHARTCTL_MUTEX_GV)) + Init(); + return; + } + + // Refresh mutex heartbeat. + GlobalVariableSet(CHARTCTL_MUTEX_GV, (double)TimeCurrent()); + + // Always answer commands (screenshot etc.), even without desired change. + HandleCommand(); + + string desired_json; + ChartCtlDeployment desired[]; + if(ReadFile(CHARTCTL_DIR + "\\desired.json", desired_json)) + { + long rev = JsonNum("revision", desired_json); + ExtractDeployments(desired_json, desired); + + // Reconcile every pass (cheap) — attach missing, detach orphans. + ChartCtlChart charts[]; + ScanCharts(charts); + + // 1) Attach / repair enabled deployments. + for(int i = 0; i < ArraySize(desired); i++) + { + if(!desired[i].enabled) + continue; + long cid = FindChartFor(desired[i].id, charts); + if(cid >= 0) + continue; + // Adopt before opening: an unowned chart already running this + // exact expert/symbol/timeframe is almost certainly a previous + // incarnation of this deployment whose comment stamp was lost + // (comments do NOT reliably survive terminal restarts). Claiming + // it instead of opening a fresh chart is what stops duplicates + // from accumulating one-per-reboot. + cid = FindAdoptableChart(desired[i], charts); + if(cid >= 0) + { + if(StampChart(cid, desired[i].id)) + { + ClearError(desired[i].id); + PrintFormat("ChartControl: adopted chart %I64d for %s (%s %s)", + cid, desired[i].id, desired[i].symbol, + desired[i].timeframe); + } + else + RecordError(desired[i].id, "STAMP_FAILED", + "adoption stamp on chart " + + IntegerToString(cid) + " did not read back"); + continue; + } + if(InRetryCooldown(desired[i].id)) + continue; // recent failure — don't hammer ChartOpen every pass + AttachDeployment(desired[i]); + } + + // 2) Detach charts we own whose deployment is gone or disabled. + ScanCharts(charts); // rescan after possible attaches + for(int c = 0; c < ArraySize(charts); c++) + { + if(charts[c].deployment_id == "") + continue; // not ours — never touch + bool wanted = false; + for(int d = 0; d < ArraySize(desired); d++) + if(desired[d].id == charts[c].deployment_id && desired[d].enabled) + { wanted = true; break; } + if(!wanted) + DetachChart(charts[c].chart_id); + } + + m_applied_revision = rev; + ScanCharts(charts); + WriteObserved(desired, charts); + } + else + { + // No desired file yet — still publish liveness + inventory. + ChartCtlChart charts[]; + ScanCharts(charts); + ChartCtlDeployment none[]; + WriteObserved(none, charts); + } +} + +//+------------------------------------------------------------------+ +void CChartControl::Deinit(void) +{ + if(m_owner && GlobalVariableCheck(CHARTCTL_MUTEX_GV)) + GlobalVariableDel(CHARTCTL_MUTEX_GV); +} + +//+------------------------------------------------------------------+ +//| Attach: select symbol, open chart, apply template, verify. | +//+------------------------------------------------------------------+ +bool CChartControl::AttachDeployment(const ChartCtlDeployment &dep) +{ + if(!SymbolSelect(dep.symbol, true)) + { + RecordError(dep.id, "SYMBOL_NOT_FOUND", + "SymbolSelect failed for " + dep.symbol); + return false; + } + + long cid = ChartOpen(dep.symbol, TF(dep.timeframe)); + if(cid == 0) + { + RecordError(dep.id, "CHART_OPEN_FAILED", + "ChartOpen failed err=" + IntegerToString(GetLastError())); + return false; + } + + if(!ChartApplyTemplate(cid, dep.templ)) + { + RecordError(dep.id, "TEMPLATE_APPLY_FAILED", + "ChartApplyTemplate(" + dep.templ + ") err=" + + IntegerToString(GetLastError())); + ChartClose(cid); + return false; + } + + // Verify the expert actually attached within ~10s. + for(int i = 0; i < 40; i++) + { + ChartRedraw(cid); + // CHART_EXPERT_NAME is NULL (not "") when no expert is attached, + // and NULL != "" is true in MQL5 — test length, or the very first + // iteration false-passes and an expert-less chart reports running. + string en = ChartGetString(cid, CHART_EXPERT_NAME); + if(StringLen(en) > 0) + { + if(!StampChart(cid, dep.id)) + { + // Without the stamp we could never re-identify the chart and + // would open a duplicate next pass — better to fail visibly. + RecordError(dep.id, "STAMP_FAILED", + "expert attached but CHART_COMMENT stamp did not " + "read back; closing chart"); + ChartClose(cid); + return false; + } + ClearError(dep.id); + PrintFormat("ChartControl: attached %s on %s %s (chart %I64d)", + en, dep.symbol, dep.timeframe, cid); + return true; + } + Sleep(250); + } + + // Leaving the chart open here leaks an expert-less chart per pass (the + // expert may still load later, but then adoption reclaims a closed-and- + // reopened one just as well). Close what we opened. + ChartClose(cid); + RecordError(dep.id, "EXPERT_NOT_ATTACHED", + "template applied but CHART_EXPERT_NAME empty after 10s; " + "GetLastError=" + IntegerToString(GetLastError())); + return false; +} + +//+------------------------------------------------------------------+ +//| Stamp attribution into the chart comment and verify it stuck. | +//| ChartSetString is asynchronous — the write is only queued — so | +//| read it back (with retries) before trusting it. | +//+------------------------------------------------------------------+ +bool CChartControl::StampChart(const long cid, const string dep_id) +{ + string want = "chartctl:" + dep_id; + for(int i = 0; i < 12; i++) + { + ChartSetString(cid, CHART_COMMENT, want); + ChartRedraw(cid); + Sleep(250); + if(ChartGetString(cid, CHART_COMMENT) == want) + return true; + } + PrintFormat("ChartControl: CHART_COMMENT stamp failed on chart %I64d (%s)", + cid, dep_id); + return false; +} + +//+------------------------------------------------------------------+ +//| An unowned chart matching a deployment's expert+symbol+timeframe | +//| (a prior incarnation whose stamp was lost, or a verify-timeout | +//| chart whose expert loaded late). | +//+------------------------------------------------------------------+ +long CChartControl::FindAdoptableChart(const ChartCtlDeployment &dep, + ChartCtlChart &charts[]) +{ + for(int i = 0; i < ArraySize(charts); i++) + { + if(charts[i].deployment_id != "") + continue; // owned by another deployment + if(!charts[i].expert_enabled) + continue; + if(charts[i].expert != dep.expert) + continue; + if(charts[i].symbol != dep.symbol) + continue; + if(charts[i].timeframe != "PERIOD_" + dep.timeframe) + continue; + return charts[i].chart_id; + } + return -1; +} + +//+------------------------------------------------------------------+ +void CChartControl::DetachChart(const long chart_id) +{ + PrintFormat("ChartControl: detaching chart %I64d", chart_id); + ChartClose(chart_id); +} + +//+------------------------------------------------------------------+ +//| Enumerate all open charts and classify ownership by the | +//| __chartctl_id we baked into each deployment template. | +//+------------------------------------------------------------------+ +void CChartControl::ScanCharts(ChartCtlChart &out[]) +{ + ArrayResize(out, 0); + long cid = ChartFirst(); + int guard = 0; + while(cid >= 0 && guard < 1000) + { + guard++; + ChartCtlChart c; + c.chart_id = cid; + c.symbol = ChartSymbol(cid); + c.timeframe = EnumToString(ChartPeriod(cid)); + c.expert = ChartGetString(cid, CHART_EXPERT_NAME); + c.expert_enabled = (c.expert != ""); + c.deployment_id = ""; // attribution below + + // Attribution is by the chart comment we set at attach time + // (ChartSetString CHART_COMMENT = "chartctl:"). We cannot read + // a foreign expert's inputs from MQL5, which is why the comment — + // not the template's __chartctl_id input — is the marker. The + // comment does NOT reliably survive a terminal restart (observed + // live 2026-07-16: one duplicate chart accumulated per reboot), so + // reconcile also adopts unowned exact-match charts (see + // FindAdoptableChart) instead of trusting this alone. + string cmt = ChartGetString(cid, CHART_COMMENT); + int p = StringFind(cmt, "chartctl:"); + if(p >= 0) + c.deployment_id = StringSubstr(cmt, p + 9); + + int n = ArraySize(out); + ArrayResize(out, n + 1); + out[n] = c; + + cid = ChartNext(cid); + } +} + +//+------------------------------------------------------------------+ +long CChartControl::FindChartFor(const string dep_id, ChartCtlChart &charts[]) +{ + for(int i = 0; i < ArraySize(charts); i++) + if(charts[i].deployment_id == dep_id && charts[i].expert_enabled) + return charts[i].chart_id; + return -1; +} + +//+------------------------------------------------------------------+ +//| Command channel: one-shot ops that produce artifacts. | +//+------------------------------------------------------------------+ +void CChartControl::HandleCommand(void) +{ + string body; + if(!ReadFile(CHARTCTL_DIR + "\\command.json", body)) + return; + + string cmd_id = JsonStr("command_id", "", body); + string action = JsonStr("action", "", body); + if(cmd_id == "") + { + DeleteFileSafe(CHARTCTL_DIR + "\\command.json"); + return; + } + + string result = "{"; + result += "\"command_id\":\"" + JsonEscape(cmd_id) + "\","; + + if(action == "screenshot") + { + long cid = JsonNum("chart_id", body); + int w = (int)JsonNum("width", body); if(w <= 0) w = 1280; + int h = (int)JsonNum("height", body); if(h <= 0) h = 720; + string fname = "shots\\" + cmd_id + ".png"; + // ChartScreenShot writes under MQL5\Files\. + if(ChartScreenShot((long)cid, CHARTCTL_DIR + "\\" + fname, w, h)) + result += "\"status\":\"ok\",\"file\":\"" + cmd_id + ".png\""; + else + result += "\"status\":\"error\",\"error_code\":\"SCREENSHOT_FAILED\"," + + "\"error_detail\":\"err=" + + IntegerToString(GetLastError()) + "\""; + } + else if(action == "reconcile") + { + m_applied_revision = -1; // force a full reconcile next pass + result += "\"status\":\"ok\""; + } + else if(action == "close_chart") + { + long cid = JsonNum("chart_id", body); + if(cid == ChartID()) + result += "\"status\":\"error\",\"error_code\":\"CLOSE_REFUSED\"," + + "\"error_detail\":\"refusing to close the loader's own chart\""; + else if(ChartClose(cid)) + result += "\"status\":\"ok\""; + else + result += "\"status\":\"error\",\"error_code\":\"CLOSE_FAILED\"," + + "\"error_detail\":\"err=" + + IntegerToString(GetLastError()) + "\""; + } + else + { + result += "\"status\":\"error\",\"error_code\":\"UNKNOWN_ACTION\"," + + "\"error_detail\":\"" + JsonEscape(action) + "\""; + } + result += "}"; + + WriteFileAtomic(CHARTCTL_DIR + "\\command_result.json", result); + DeleteFileSafe(CHARTCTL_DIR + "\\command.json"); +} + +//+------------------------------------------------------------------+ +//| Write observed.json — the API's window into terminal truth. | +//+------------------------------------------------------------------+ +void CChartControl::WriteObserved(ChartCtlDeployment &desired[], + ChartCtlChart &charts[]) +{ + string j = "{"; + j += "\"protocol\":" + IntegerToString(CHARTCTL_PROTOCOL) + ","; + j += "\"loader\":{"; + j += "\"name\":\"" + JsonEscape(MQLInfoString(MQL_PROGRAM_NAME)) + "\","; + j += "\"version\":\"" + CHARTCTL_VERSION + "\","; + j += "\"last_loop\":\"" + TimeToString(TimeGMT(), TIME_DATE|TIME_SECONDS) + "\","; + j += "\"applied_revision\":" + IntegerToString(m_applied_revision); + j += "},"; + j += "\"terminal\":{\"auto_trading\":" + + (string)(TerminalInfoInteger(TERMINAL_TRADE_ALLOWED) ? "true" : "false") + + "},"; + + // charts[] + j += "\"charts\":["; + for(int i = 0; i < ArraySize(charts); i++) + { + if(i) j += ","; + j += "{"; + j += "\"chart_id\":" + IntegerToString(charts[i].chart_id) + ","; + j += "\"symbol\":\"" + JsonEscape(charts[i].symbol) + "\","; + j += "\"timeframe\":\"" + JsonEscape(charts[i].timeframe) + "\","; + j += "\"expert\":\"" + JsonEscape(charts[i].expert) + "\","; + j += "\"expert_enabled\":" + (string)(charts[i].expert_enabled ? "true" : "false") + ","; + j += "\"deployment_id\":\"" + JsonEscape(charts[i].deployment_id) + "\""; + j += "}"; + } + j += "],"; + + // deployments[] status + j += "\"deployments\":["; + int written = 0; + for(int d = 0; d < ArraySize(desired); d++) + { + long cid = FindChartFor(desired[d].id, charts); + string status = (cid >= 0) ? "running" + : (desired[d].enabled ? "pending" : "paused"); + if(written) j += ","; + j += "{\"id\":\"" + JsonEscape(desired[d].id) + "\","; + j += "\"status\":\"" + status + "\""; + if(cid >= 0) j += ",\"chart_id\":" + IntegerToString(cid); + j += "}"; + written++; + } + j += "],"; + + // errors[] — one entry per failing deployment, cleared on its success + j += "\"errors\":["; + for(int e = 0; e < ArraySize(m_err_ids); e++) + { + if(e) j += ","; + j += "{\"id\":\"" + JsonEscape(m_err_ids[e]) + "\","; + j += "\"status\":\"failed\","; + j += "\"code\":\"" + JsonEscape(m_err_codes[e]) + "\","; + j += "\"detail\":\"" + JsonEscape(m_err_details[e]) + "\"}"; + } + j += "]"; + + j += "}"; + WriteFileAtomic(CHARTCTL_DIR + "\\observed.json", j); +} + +//+------------------------------------------------------------------+ +void CChartControl::RecordError(const string id, const string code, + const string detail) +{ + int slot = -1; + for(int i = 0; i < ArraySize(m_err_ids); i++) + if(m_err_ids[i] == id) { slot = i; break; } + if(slot < 0) + { + slot = ArraySize(m_err_ids); + ArrayResize(m_err_ids, slot + 1); + ArrayResize(m_err_codes, slot + 1); + ArrayResize(m_err_details, slot + 1); + ArrayResize(m_err_times, slot + 1); + } + m_err_ids[slot] = id; + m_err_codes[slot] = code; + m_err_details[slot] = detail; + m_err_times[slot] = TimeCurrent(); + PrintFormat("ChartControl ERROR [%s] %s: %s", id, code, detail); +} + +void CChartControl::ClearError(const string id) +{ + for(int i = 0; i < ArraySize(m_err_ids); i++) + { + if(m_err_ids[i] != id) + continue; + int last = ArraySize(m_err_ids) - 1; + m_err_ids[i] = m_err_ids[last]; + m_err_codes[i] = m_err_codes[last]; + m_err_details[i] = m_err_details[last]; + m_err_times[i] = m_err_times[last]; + ArrayResize(m_err_ids, last); + ArrayResize(m_err_codes, last); + ArrayResize(m_err_details, last); + ArrayResize(m_err_times, last); + return; + } +} + +// A deployment that just failed to attach gets a 60s cooldown so the +// loader doesn't churn ChartOpen/ChartClose on every reconcile pass. +bool CChartControl::InRetryCooldown(const string id) +{ + for(int i = 0; i < ArraySize(m_err_ids); i++) + if(m_err_ids[i] == id) + return (TimeCurrent() - m_err_times[i]) < 60; + return false; +} + +//+------------------------------------------------------------------+ +//| File helpers — everything under MQL5\Files\ (FILE_COMMON off). | +//+------------------------------------------------------------------+ +bool CChartControl::ReadFile(const string relpath, string &out) +{ + int h = FileOpen(relpath, FILE_READ | FILE_TXT | FILE_ANSI); + if(h == INVALID_HANDLE) + return false; + out = ""; + while(!FileIsEnding(h)) + out += FileReadString(h); + FileClose(h); + return true; +} + +bool CChartControl::WriteFileAtomic(const string relpath, const string content) +{ + string tmp = relpath + ".tmp"; + int h = FileOpen(tmp, FILE_WRITE | FILE_TXT | FILE_ANSI); + if(h == INVALID_HANDLE) + { + PrintFormat("ChartControl: cannot open %s for write (err=%d)", + tmp, GetLastError()); + return false; + } + FileWriteString(h, content); + FileClose(h); + // FileMove with rewrite flag = atomic-ish replace on Windows. Without + // FILE_REWRITE the move needs the pre-delete to have worked, and that + // fails with 5020 whenever the API side has the file open for a read. + if(!FileMove(tmp, 0, relpath, FILE_REWRITE)) + { + PrintFormat("ChartControl: FileMove %s->%s failed (err=%d)", + tmp, relpath, GetLastError()); + return false; + } + return true; +} + +void CChartControl::DeleteFileSafe(const string relpath) +{ + if(FileIsExist(relpath)) + FileDelete(relpath); +} + +//+------------------------------------------------------------------+ +//| Minimal JSON readers for our own compact, predictable output. | +//| NOT a general parser — only the shapes chartctl produces. | +//+------------------------------------------------------------------+ +string CChartControl::JsonStr(const string key, const string def, + const string json) +{ + string needle = "\"" + key + "\""; + int p = StringFind(json, needle); + if(p < 0) return def; + int colon = StringFind(json, ":", p + StringLen(needle)); + if(colon < 0) return def; + int q1 = StringFind(json, "\"", colon + 1); + if(q1 < 0) return def; + // find unescaped closing quote + int i = q1 + 1; + string val = ""; + while(i < StringLen(json)) + { + ushort ch = StringGetCharacter(json, i); + if(ch == '\\') + { + if(i + 1 < StringLen(json)) + val += ShortToString(StringGetCharacter(json, i + 1)); + i += 2; + continue; + } + if(ch == '"') + break; + val += ShortToString(ch); + i++; + } + return val; +} + +long CChartControl::JsonNum(const string key, const string json) +{ + string needle = "\"" + key + "\""; + int p = StringFind(json, needle); + if(p < 0) return 0; + int colon = StringFind(json, ":", p + StringLen(needle)); + if(colon < 0) return 0; + int i = colon + 1; + string num = ""; + while(i < StringLen(json)) + { + ushort ch = StringGetCharacter(json, i); + if((ch >= '0' && ch <= '9') || ch == '-') + num += ShortToString(ch); + else if(num != "") + break; + i++; + } + return (long)StringToInteger(num); +} + +//+------------------------------------------------------------------+ +//| Extract the deployments[] array from desired.json. | +//| Each object: {id,expert,template,symbol,timeframe,enabled}. | +//+------------------------------------------------------------------+ +bool CChartControl::ExtractDeployments(const string json, + ChartCtlDeployment &out[]) +{ + ArrayResize(out, 0); + int arr = StringFind(json, "\"deployments\""); + if(arr < 0) return false; + int i = StringFind(json, "[", arr); + if(i < 0) return false; + + int depth = 0; + int obj_start = -1; + for(; i < StringLen(json); i++) + { + ushort ch = StringGetCharacter(json, i); + if(ch == '{') + { + if(depth == 0) obj_start = i; + depth++; + } + else if(ch == '}') + { + depth--; + if(depth == 0 && obj_start >= 0) + { + string obj = StringSubstr(json, obj_start, i - obj_start + 1); + ChartCtlDeployment d; + d.id = JsonStr("id", "", obj); + d.expert = JsonStr("expert", "", obj); + d.templ = JsonStr("template", "", obj); + d.symbol = JsonStr("symbol", "", obj); + d.timeframe = JsonStr("timeframe", "", obj); + // enabled is a bare JSON bool; desired.json emits it compactly. + d.enabled = (StringFind(obj, "\"enabled\": false") < 0 + && StringFind(obj, "\"enabled\":false") < 0); + if(d.id != "") + { + int n = ArraySize(out); + ArrayResize(out, n + 1); + out[n] = d; + } + obj_start = -1; + } + } + else if(ch == ']' && depth == 0) + break; + } + return true; +} + +string CChartControl::JsonEscape(const string s) +{ + string out = s; + StringReplace(out, "\\", "\\\\"); + StringReplace(out, "\"", "\\\""); + StringReplace(out, "\n", " "); + StringReplace(out, "\r", " "); + return out; +} + +//+------------------------------------------------------------------+ +ENUM_TIMEFRAMES CChartControl::TF(const string s) +{ + if(s == "M1") return PERIOD_M1; + if(s == "M2") return PERIOD_M2; + if(s == "M3") return PERIOD_M3; + if(s == "M4") return PERIOD_M4; + if(s == "M5") return PERIOD_M5; + if(s == "M6") return PERIOD_M6; + if(s == "M10") return PERIOD_M10; + if(s == "M12") return PERIOD_M12; + if(s == "M15") return PERIOD_M15; + if(s == "M20") return PERIOD_M20; + if(s == "M30") return PERIOD_M30; + if(s == "H1") return PERIOD_H1; + if(s == "H2") return PERIOD_H2; + if(s == "H3") return PERIOD_H3; + if(s == "H4") return PERIOD_H4; + if(s == "H6") return PERIOD_H6; + if(s == "H8") return PERIOD_H8; + if(s == "H12") return PERIOD_H12; + if(s == "D1") return PERIOD_D1; + if(s == "W1") return PERIOD_W1; + if(s == "MN1") return PERIOD_MN1; + return PERIOD_H1; +} +//+------------------------------------------------------------------+ diff --git a/config/config.yaml.example b/config/config.yaml.example index 79bddb6..bc4d295 100644 --- a/config/config.yaml.example +++ b/config/config.yaml.example @@ -25,6 +25,19 @@ tailscale: auth_key: "" # leave empty to disable the tailscale sidecar entirely login_server: "" # Headscale login server URL; empty = Tailscale cloud +# Chart Deployments (chartctl) — EA hosting primitives. Stage .ex5/.set +# artifacts and declare "run expert X with set Y on symbol S timeframe T" +# as desired state; a resident loader EA (assets/experts/MT5ChartLoader.mq5, +# or your own EA using assets/experts/include/ChartControl.mqh) reconciles +# the terminal's charts to it. Live-mode terminals only; backtest-mode +# terminals ignore this entirely. See docs/chart-control-protocol.md. +chartctl: + enabled: true # false = chartctl endpoints return 404 + reconcile_hint_interval: "5s" # written into desired.json for the loader + observed_stale_after: "60s" # observed.json older than this = loader stale + command_timeout: "30s" # wait for screenshot/one-shot loader replies + max_upload_bytes: 16777216 # 16 MB cap on .ex5/.set uploads + # Extra pip packages installed in the VM on every boot. # MetaTrader5, flask, waitress, and flask-compress are always installed. requirements: [] diff --git a/docs/chart-control-protocol.md b/docs/chart-control-protocol.md new file mode 100644 index 0000000..5df07b0 --- /dev/null +++ b/docs/chart-control-protocol.md @@ -0,0 +1,226 @@ +# Chart Control Protocol v1 + +How mt5-httpapi's **Chart Deployments** feature attaches Expert Advisors to +charts remotely, keeps them running, and reports what's actually live — +without RDP, without restarting terminals, and without any orchestrator. + +MT5 has no SDK call to attach an EA to a chart. The only programmatic path +is `ChartApplyTemplate()` from MQL5 code already running inside the +terminal. So the API and a small **resident loader EA** cooperate over a +handful of JSON files in the terminal's `MQL5\Files\chartctl\` sandbox. + +The protocol is intentionally file-based so that: + +- it needs zero WebRequest whitelist entries (works on locked-down terminals), +- it's trivially debuggable over RDP during rollout, and +- **any** EA can implement it — the bundled `MT5ChartLoader`, or your own + resident utility EA (e.g. an account tracker) that adopts + `ChartControl.mqh`. + +--- + +## Roles + +| Side | Writes | Reads | +|------|--------|-------| +| **API** (mt5-httpapi) | `desired.json`, `command.json`, generated `.tpl` files | `observed.json`, `command_result.json` | +| **Loader EA** (in terminal) | `observed.json`, `command_result.json`, screenshots | `desired.json`, `command.json` | + +The API owns *desired state*. The loader owns *observed truth*. Neither +writes the other's files. Success is defined as **observed converging on +desired**, never as "the API copied some files." + +--- + +## Files (all under `MQL5\Files\chartctl\`) + +### `desired.json` — API → loader + +```json +{ + "protocol": 1, + "revision": 42, + "updated_at": "2026-07-09T12:00:00Z", + "reconcile_interval": 5, + "deployments": [ + { + "id": "dep_a1b2c3", + "expert": "HappyGoldScalp", + "template": "\\Files\\chartctl\\dep_a1b2c3.tpl", + "symbol": "XAUUSD", + "timeframe": "M5", + "enabled": true + } + ] +} +``` + +`revision` is a monotonic counter; the loader can skip a full parse when it +hasn't changed. `template` is passed verbatim to `ChartApplyTemplate()`; +the leading backslash makes MT5 resolve it against `\MQL5` — the +only search root that doesn't depend on which EX5 hosts the loader +(paths without a leading backslash resolve relative to the calling EX5's +own folder, and the `templates\` GUI directory is never searched). The +API generates one `.tpl` per deployment, written into the +`MQL5\Files\chartctl\` protocol directory. + +### `observed.json` — loader → API (rewritten every reconcile pass, ~5s) + +```json +{ + "protocol": 1, + "loader": { "name": "MT5ChartLoader", "version": "1.0.0", + "last_loop": "2026-07-09 12:00:03", "applied_revision": 42 }, + "terminal": { "auto_trading": true }, + "charts": [ + { "chart_id": 133039117, "symbol": "XAUUSD", "timeframe": "PERIOD_M5", + "expert": "HappyGoldScalp", "expert_enabled": true, + "deployment_id": "dep_a1b2c3" } + ], + "deployments": [ + { "id": "dep_a1b2c3", "status": "running", "chart_id": 133039117 } + ], + "errors": [] +} +``` + +`applied_revision == desired.revision` **and** deployment `status: running` +is the only definition of a converged deployment. The API's +`GET /deployments` merges the two files and derives per-deployment status +(`pending → running → degraded → failed → paused`). + +### `command.json` / `command_result.json` — one-shot imperatives + +For operations that produce an artifact rather than converge state +(currently `screenshot`, `close_chart`, and a `reconcile` nudge). One +command in flight; the loader writes the result keyed by `command_id` and +deletes the command. + +`close_chart` closes an arbitrary chart by `chart_id` (from the observed +charts list) — the cleanup escape hatch for charts the loader cannot +attribute. The loader refuses (`CLOSE_REFUSED`) to close its own chart. + +--- + +## Loader responsibilities + +Every pass (timer-driven, ~1s), the owning loader: + +1. Reads `desired.json`; if `revision` changed, reconciles. +2. **Reconcile** = diff desired deployments against actual charts: + - Enabled deployment with no matching chart → first try to **adopt** an + unowned chart already running the exact expert + symbol + timeframe + (stamp it `chartctl:`); only if none exists, `SymbolSelect` → + `ChartOpen` → `ChartApplyTemplate` → verify `CHART_EXPERT_NAME` + within 10s → stamp `chartctl:` into the chart comment. Stamps + are verified by read-back (`ChartSetString` is asynchronous); a + failed attach closes the chart it opened and backs off 60s. + - A chart it owns (comment starts `chartctl:`) whose deployment is gone + or disabled → `ChartClose`. + - Charts it doesn't own are **reported but never touched** by + reconcile (an explicit `close_chart` command can close them). +3. Writes `observed.json`. +4. Answers any pending `command.json`. + +### Rules + +- The loader **never calls trade functions**. Chart lifecycle only. +- Exactly **one** loader per terminal, guarded by a terminal + `GlobalVariable` mutex (`chartctl_loader_owner`). A second loader detects + the live mutex and stays passive, reclaiming only if the owner vanishes. +- All file writes are atomic (temp + `FileMove`). +- Attribution is by the `chartctl:` chart comment the loader sets at + attach time. The comment does **not** reliably survive a terminal + restart (observed live: MT5's saved profile restores the chart and + expert but drops the comment, leaking one duplicate chart per reboot), + which is why reconcile adopts exact-match unowned charts instead of + trusting the comment alone. + +--- + +## Self-healing + +Three layers converge a terminal back to desired state after any restart +(including the optional `reboot_interval` VM reboots): + +1. **MT5 native chart restoration** brings back charts + attached experts + from the last saved profile — often everything, for free. +2. **Loader reconciliation** repairs whatever native restoration missed + (crash before profile save, chart closed by hand, failed `OnInit`). +3. **Watchdog** (`monitor.py`) logs loudly if the terminal is alive but the + loader's `last_loop` goes stale — the one state layers 1–2 can't fix + alone. + +--- + +## Implementing the protocol in your own EA + +The whole loader is a portable include. In your resident EA: + +```mql5 +#include +CChartControl ctl; + +int OnInit() { if(!ctl.Init()) return INIT_FAILED; + EventSetTimer(1); return INIT_SUCCEEDED; } +void OnTimer(){ ctl.Tick(); } +void OnDeinit(const int r){ ctl.Deinit(); } +``` + +That's it — your account tracker (or any always-on EA) becomes the loader, +so one resident EA does telemetry *and* chart deployment instead of two. The +mutex makes running both your EA and the standalone `MT5ChartLoader` +degrade safely. See `assets/experts/MT5ChartLoader.mq5` for the reference +glue and `assets/experts/include/ChartControl.mqh` for the implementation. + +--- + +## Bootstrapping the loader + +**Zero-touch (default).** Provisioning does everything — no RDP, no manual +attach, fully API/config-driven: + +1. On VM boot, `start.bat` runs `compile-chartctl-loader.bat`, which copies + `ChartControl.mqh` + `MT5ChartLoader.mq5` into every broker base + terminal, compiles with that base's MetaEditor64, and propagates the + `.ex5` into every existing terminal instance (new instances inherit it + from the base copy). +2. `config_helper.py` writes a `[StartUp] Expert=Advisors\MT5ChartLoader` + section into each terminal's generated `mt5start.ini` — so the terminal + attaches the loader itself at every launch. The startup chart symbol + honors the terminal's `symbol_suffix` (e.g. `EURUSD.r`). +3. The `[StartUp]` line re-fires on every launch (including the periodic + `reboot_interval` reboots); the GlobalVariable mutex makes this + idempotent — a duplicate loader closes its own chart and vanishes, so + charts never accumulate. + +Both steps honor the same gating as the API: live-mode terminals only, +`chartctl.enabled` globally, per-terminal `chartctl: false` opts out. + +**Manual (fallback / existing fleets).** Drag `MT5ChartLoader` onto any +chart once over RDP, or fold `ChartControl.mqh` into a resident EA you +already deploy. + +--- + +## Endpoint summary + +| Method | Path | Purpose | +|--------|------|---------| +| `POST` | `/experts` | Upload `.ex5` (multipart) | +| `GET` | `/experts` | List staged experts | +| `DELETE` | `/experts/` | Remove staged expert (refused if in use) | +| `POST` | `/sets` | Upload `.set` (returns parsed inputs) | +| `GET` | `/sets`, `/sets/` | List / inspect set files | +| `POST` | `/deployments` | Declare a deployment (EA+set+symbol+TF) | +| `GET` | `/deployments`, `/deployments/` | Desired ⋈ observed status | +| `PATCH` | `/deployments/` | Change set file or pause/resume | +| `DELETE` | `/deployments/` | Remove a deployment | +| `POST` | `/deployments/reconcile` | Force the loader to re-reconcile | +| `GET` | `/charts` | Live chart/EA inventory from the terminal | +| `GET` | `/loader` | Loader presence/version/liveness | +| `POST` | `/charts//screenshot` | PNG of a chart | +| `POST` | `/charts//close` | Close a chart (incl. unattributed ones) | + +All routes sit behind the terminal's existing per-account route prefix and +bearer-token auth. Nothing here touches the MT5 SDK lock. diff --git a/examples/chartctl/deploy.sh b/examples/chartctl/deploy.sh new file mode 100755 index 0000000..e95a6f8 --- /dev/null +++ b/examples/chartctl/deploy.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Chart Deployments — example workflow +# Stage an expert + set, deploy to a chart, verify, then tear down. +# +# Usage: +# export MT5_API_URL=http://localhost:8888/yourbroker/yourlogin +# export MT5_API_TOKEN=your-api-token +# ./deploy.sh +# +# Example: +# ./deploy.sh ~/EAs/MyScalp.ex5 ~/EAs/eurusd-m5.set EURUSD M5 + +MT5_API_URL="${MT5_API_URL:?Set MT5_API_URL}" +MT5_API_TOKEN="${MT5_API_TOKEN:-}" +AUTH=() +if [ -n "$MT5_API_TOKEN" ]; then + AUTH=(-H "Authorization: Bearer $MT5_API_TOKEN") +fi + +EXPERT_PATH="${1:?Usage: deploy.sh }" +SET_PATH="${2:?}" +SYMBOL="${3:?}" +TIMEFRAME="${4:?}" +EXPERT_NAME=$(basename "$EXPERT_PATH") +SET_NAME=$(basename "$SET_PATH") + +echo "=== Step 1: Stage expert ===" +curl -sS "${AUTH[@]}" -F "expert=@$EXPERT_PATH" "$MT5_API_URL/experts" | head -c 200 +echo + +echo "=== Step 2: Stage set file ===" +curl -sS "${AUTH[@]}" -F "set=@$SET_PATH" "$MT5_API_URL/sets" | head -c 200 +echo + +echo "=== Step 3: Create deployment ===" +DEPLOY_JSON=$(curl -sS -X POST "${AUTH[@]}" \ + -H "Content-Type: application/json" \ + "$MT5_API_URL/deployments" \ + -d "{\"expert\":\"$EXPERT_NAME\",\"set\":\"$SET_NAME\",\"symbol\":\"$SYMBOL\",\"timeframe\":\"$TIMEFRAME\"}") +echo "$DEPLOY_JSON" | head -c 300 +echo +DEPLOY_ID=$(echo "$DEPLOY_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])" 2>/dev/null || echo "") + +if [ -z "$DEPLOY_ID" ]; then + echo "Failed to extract deployment id from response." + exit 1 +fi + +echo "=== Step 4: Poll until running (up to 60s) ===" +for i in $(seq 1 12); do + sleep 5 + STATUS=$(curl -sS "${AUTH[@]}" "$MT5_API_URL/deployments/$DEPLOY_ID" \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('status','unknown'))" 2>/dev/null || echo "unknown") + echo " Attempt $i: status=$STATUS" + if [ "$STATUS" = "running" ]; then + echo "=== Deployment running! ===" + break + fi +done + +echo "=== Step 5: Verify via /charts ===" +curl -sS "${AUTH[@]}" "$MT5_API_URL/charts" | python3 -m json.tool | head -30 + +echo "=== Step 6: Take a screenshot ===" +CHART_ID=$(curl -sS "${AUTH[@]}" "$MT5_API_URL/charts" \ + | python3 -c "import sys,json; charts=json.load(sys.stdin).get('charts',[]); print([c['id'] for c in charts if c.get('deployment_id')=='$DEPLOY_ID'][0])" 2>/dev/null || echo "") +if [ -n "$CHART_ID" ]; then + curl -sS "${AUTH[@]}" "$MT5_API_URL/charts/$CHART_ID/screenshot" -o "chartctl-${SYMBOL}-${TIMEFRAME}.png" + echo "Screenshot saved to chartctl-${SYMBOL}-${TIMEFRAME}.png" +fi + +echo "=== Done ===" +echo "Deployment $DEPLOY_ID is running. Clean up with:" +echo " curl -X DELETE ${AUTH[@]} \"$MT5_API_URL/deployments/$DEPLOY_ID\"" diff --git a/mt5api/chartctl/__init__.py b/mt5api/chartctl/__init__.py new file mode 100644 index 0000000..96ea4a7 --- /dev/null +++ b/mt5api/chartctl/__init__.py @@ -0,0 +1,14 @@ +"""Chart Deployments (chartctl) — EA deployment primitives. + +Generic, orchestrator-agnostic capability: stage .ex5/.set artifacts, +declare desired deployments (expert + set + symbol + timeframe), and let a +resident loader EA reconcile the terminal's charts to that desired state +over a file protocol inside the MQL5 sandbox. + +Protocol contract: docs/chart-control-protocol.md +Reference loader: assets/experts/MT5ChartLoader.mq5 (+ ChartControl.mqh) + +Nothing in this package touches the MT5 SDK — every operation is plain +file I/O against TERMINAL_DIR, so no handler here ever queues behind the +process-wide MT5 lock. +""" diff --git a/mt5api/chartctl/autoit_webrequest.py b/mt5api/chartctl/autoit_webrequest.py new file mode 100644 index 0000000..a2da97f --- /dev/null +++ b/mt5api/chartctl/autoit_webrequest.py @@ -0,0 +1,240 @@ +"""Apply the WebRequest allowlist by driving MT5's Options dialog with AutoIt. + +Why not just write the file? On the dockur-VM terminal build the allowlist is +NOT stored in (or read back from) ``common.ini`` — it lives in the machine-bound +``MQL5\\experts.dat`` and does not survive a terminal restart even when set in +MT5's own Options dialog. So file injection can't provision it there. Instead we +set it the way a user would: open Tools -> Options -> Expert Advisors and add the +URLs. WebRequest then works immediately in-session (verified with a probe EA: +``ret=200`` on an allowed URL). Because MT5 drops the list on restart, this is a +re-appliable operation (a dedicated call, plus an optional boot re-apply). + +The heavy lifting is a portable AutoIt interpreter + script shipped under +``assets/autoit/`` (live-mounted into the VM). MT5 runs elevated (``-Verb RunAs``) +and Windows UIPI blocks a non-elevated process from sending it input — but the API +process itself already runs elevated here (verified: AutoIt reports ``IsAdmin=1``), +so a plain, blocking ``subprocess.run`` inherits that elevation, drives MT5 fine, +and lets us capture the exit code. A ``use_runas`` path (async ``Start-Process +-Verb RunAs`` + log polling) is kept as a fallback for non-elevated deployments. +""" +from __future__ import annotations + +import contextlib +import os +import re +import subprocess +import time + +from mt5api.config import ACCOUNT, ASSETS_DIR, BROKER, INI_FILE, INSTANCE, TERMINAL_DIR +from mt5api.logger import log + +AUTOIT_DIR = os.path.join(ASSETS_DIR, "autoit") +AUTOIT_EXE = os.path.join(AUTOIT_DIR, "AutoIt3_x64.exe") + +_LOG_NAME = "webrequest_autoit.log" +_URLS_NAME = "webrequest_apply_urls.txt" + +# Machine-wide mutex serializing GUI automation across every terminal's API +# process on this host (see _gui_lock). +_GUI_MUTEX_NAME = "Global\\mt5_httpapi_webrequest_autoit" +_GUI_LOCK_WAIT_MS = 180_000 # 3 min — comfortably longer than any real apply + + +@contextlib.contextmanager +def _gui_lock(wait_ms: int = _GUI_LOCK_WAIT_MS): + """Serialize GUI automation across all terminal API processes on this host. + + Desktop input (window focus + the keyboard) is a single shared resource, so + two AutoIt applies running at once would steal focus from each other and leak + keystrokes into the wrong terminal — even producing a RESULT=OK that typed a + URL into the wrong window. Each terminal runs its own API *process*, so an + in-process ``threading.Lock`` is not enough; we take a named Windows kernel + mutex, which is machine-wide. It is crash-safe: if a holder dies, the kernel + hands ownership to the next waiter (WAIT_ABANDONED), so there is no stale + lock. On wait-timeout we proceed anyway (URLs are needed and 3 min means + something is wedged, not merely busy). No-op off Windows.""" + if os.name != "nt": + yield True + return + import ctypes + from ctypes import wintypes + + k = ctypes.windll.kernel32 + k.CreateMutexW.restype = wintypes.HANDLE + k.CreateMutexW.argtypes = (wintypes.LPVOID, wintypes.BOOL, wintypes.LPCWSTR) + k.WaitForSingleObject.argtypes = (wintypes.HANDLE, wintypes.DWORD) + handle = k.CreateMutexW(None, False, _GUI_MUTEX_NAME) + owned = False + try: + if handle: + res = k.WaitForSingleObject(handle, wait_ms) + owned = res in (0x0, 0x80) # WAIT_OBJECT_0 / WAIT_ABANDONED + if not owned: + log.warning("WebRequest GUI lock: wait timed out (%d ms); proceeding", wait_ms) + else: + log.warning("WebRequest GUI lock: CreateMutex failed; proceeding unlocked") + yield owned + finally: + if handle: + if owned: + k.ReleaseMutex(handle) + k.CloseHandle(handle) + + +def _resolve_script(script: str) -> str: + """Only run a plain-named .au3 that actually exists in AUTOIT_DIR (the + read-only, repo-controlled mount) — no path separators, no traversal.""" + if os.path.basename(script) != script or not script.lower().endswith(".au3"): + raise ValueError(f"bad script name: {script}") + path = os.path.join(AUTOIT_DIR, script) + if not os.path.exists(path): + raise ValueError(f"script not found: {script}") + return path + + +def available() -> bool: + """True only on Windows with the AutoIt interpreter present (i.e. the VM). + The .exe ships in the repo for all platforms, but only runs under Windows, + so gate on the OS too — elsewhere the caller uses the common.ini fallback.""" + return os.name == "nt" and os.path.exists(AUTOIT_EXE) + + +def _window_match() -> str: + """A token guaranteed to appear in this terminal's window title: the login + (read lock-free from mt5start.ini). Falls back to the broker name.""" + try: + with open(INI_FILE, "r", encoding="utf-8", errors="ignore") as f: + for line in f: + if line.strip().lower().startswith("login="): + val = line.split("=", 1)[1].strip() + if val: + return val + except OSError: + pass + return BROKER + + +def _terminal_pid() -> int: + """PID of THIS terminal's terminal64.exe, resolved by executable path. + + The window title (login) is ambiguous — cloned terminals of the same + account have identical titles — so the pid is what actually pins the right + window for the AutoIt script. WMI via PowerShell, because it can read exe + paths of elevated processes. Returns 0 if not found (script falls back to + title matching).""" + if ACCOUNT: + path_filter = f"*\\{BROKER}\\{ACCOUNT}\\{INSTANCE}\\*" + else: + path_filter = f"*\\{BROKER}\\*" + ps_cmd = ( + "Get-WmiObject Win32_Process -Filter \"Name='terminal64.exe'\" " + "| Where-Object { $_.ExecutablePath -like '" + path_filter + "' } " + "| Select-Object -First 1 -ExpandProperty ProcessId" + ) + try: + result = subprocess.run( + ["powershell", "-Command", ps_cmd], + capture_output=True, text=True, timeout=30, + ) + pid = int(result.stdout.strip()) + return pid if pid > 0 else 0 + except (ValueError, subprocess.SubprocessError, OSError): + log.warning("WebRequest: terminal pid lookup failed; falling back to title match") + return 0 + + +def _config_path(name: str) -> str: + return os.path.join(TERMINAL_DIR, "Config", name) + + +def _wait_result(logpath: str, timeout: float) -> tuple[str, str]: + """Poll the AutoIt log for a ``RESULT=`` line. Returns (status, log).""" + deadline = time.time() + timeout + while time.time() < deadline: + time.sleep(1) + try: + with open(logpath, "r", encoding="utf-8", errors="ignore") as f: + txt = f.read() + except OSError: + continue + m = re.search(r"RESULT=(\w+)", txt) + if m: + return m.group(1), txt + try: + with open(logpath, "r", encoding="utf-8", errors="ignore") as f: + txt = f.read() + except OSError: + txt = "" + return "TIMEOUT", txt + + +def _run_script( + script: str, extra_args: list[str], timeout: float, use_runas: bool = False +) -> tuple[str, str]: + if not available(): + raise RuntimeError(f"AutoIt interpreter not found at {AUTOIT_EXE}") + script_path = _resolve_script(script) + + logpath = _config_path(_LOG_NAME) + try: + os.remove(logpath) + except OSError: + pass + + match = _window_match() + pid = _terminal_pid() + # AutoIt argv: match, pid, extra..., logpath + au3_args = [match, str(pid), *extra_args, logpath] + log.info("AutoIt: launching %s (match=%s, pid=%d, use_runas=%s)", + script, match, pid, use_runas) + + # Hold the machine-wide GUI mutex for the whole run so no other terminal's + # apply steals focus mid-type. + with _gui_lock(): + dbg = "" + if use_runas: + # MT5 runs elevated; UIPI blocks a non-elevated process from sending + # it input. Launch elevated (async — no exit code) and poll the log. + ps_args = ",".join("'%s'" % a for a in [script_path, *au3_args]) + ps = ( + f"Start-Process '{AUTOIT_EXE}' -ArgumentList {ps_args} " + "-Verb RunAs -WindowStyle Hidden" + ) + subprocess.Popen(["powershell", "-Command", ps]) + status, txt = _wait_result(logpath, timeout) + else: + # Direct, blocking — captures exit code + stderr for diagnostics. + cmd = [AUTOIT_EXE, script_path, *au3_args] + try: + proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout) + dbg = f"[rc={proc.returncode} stderr={proc.stderr.strip()[:300]}]" + except subprocess.TimeoutExpired: + dbg = "[rc=TIMEOUT]" + try: + with open(logpath, "r", encoding="utf-8", errors="ignore") as f: + txt = f.read() + except OSError: + txt = "" + m = re.search(r"RESULT=(\w+)", txt) + status = m.group(1) if m else "NORESULT" + + txt = (txt + "\n" + dbg).strip() + log.info("AutoIt: %s -> RESULT=%s %s", script, status, dbg) + if status not in ("OK",): + log.warning("AutoIt %s log tail:\n%s", script, txt[-800:]) + return status, txt + + +def apply_urls(urls: list[str], timeout: float = 120, use_runas: bool = False) -> tuple[str, str]: + """Set the given allowlist in the running terminal via the Options dialog. + Returns (status, autoit_log). status == 'OK' on success.""" + urlfile = _config_path(_URLS_NAME) + os.makedirs(os.path.dirname(urlfile), exist_ok=True) + with open(urlfile, "w", encoding="utf-8") as f: + f.write("\n".join(urls)) + return _run_script("set_webrequest.au3", [urlfile], timeout, use_runas) + + +def run_named(script: str, timeout: float = 60, use_runas: bool = False) -> tuple[str, str]: + """Dev helper: run an arbitrary repo-shipped .au3 (e.g. the inspector).""" + return _run_script(script, [], timeout, use_runas) diff --git a/mt5api/chartctl/command.py b/mt5api/chartctl/command.py new file mode 100644 index 0000000..6a79700 --- /dev/null +++ b/mt5api/chartctl/command.py @@ -0,0 +1,78 @@ +"""Imperative one-shot command channel (screenshot, forced refresh). + +Deploy/stop are NOT commands — they are desired-state edits handled by +registry.py. This channel exists only for operations that produce a +side-effect artifact rather than converge state. + +Protocol: API writes command.json {command_id, action, ...}; loader +executes, writes command_result.json {command_id, status, ...}, deletes +command.json. API polls for the result up to a timeout. One command in +flight at a time, serialized by a lock (matching the loader's +one-command contract). +""" +import json +import os +import secrets +import threading +import time + +from mt5api.chartctl import paths +from mt5api.config import CHARTCTL_COMMAND_TIMEOUT_SECONDS + +_LOCK = threading.Lock() + + +class LoaderTimeout(TimeoutError): + pass + + +class LoaderBusy(RuntimeError): + pass + + +def run_command(action: str, payload: dict | None = None, + timeout: float | None = None) -> dict: + """Write a command, block until its result arrives or timeout.""" + timeout = timeout or CHARTCTL_COMMAND_TIMEOUT_SECONDS + if not _LOCK.acquire(blocking=False): + raise LoaderBusy("another command is in flight") + try: + paths.ensure_dirs() + if os.path.exists(paths.COMMAND_PATH): + # Stale command from a crashed request: clear it. + _try_remove(paths.COMMAND_PATH) + _try_remove(paths.COMMAND_RESULT_PATH) + + command_id = "cmd_" + secrets.token_hex(4) + body = {"command_id": command_id, "action": action} + body.update(payload or {}) + paths.atomic_write_text(paths.COMMAND_PATH, + json.dumps(body, indent=2, sort_keys=True)) + + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + result = _read_result() + if result and result.get("command_id") == command_id: + _try_remove(paths.COMMAND_RESULT_PATH) + return result + time.sleep(0.25) + _try_remove(paths.COMMAND_PATH) + raise LoaderTimeout( + f"loader did not answer '{action}' within {timeout:.0f}s") + finally: + _LOCK.release() + + +def _read_result() -> dict | None: + try: + with open(paths.COMMAND_RESULT_PATH, "r", encoding="utf-8") as handle: + return json.load(handle) + except (OSError, ValueError): + return None + + +def _try_remove(path: str) -> None: + try: + os.remove(path) + except OSError: + pass diff --git a/mt5api/chartctl/paths.py b/mt5api/chartctl/paths.py new file mode 100644 index 0000000..15995ec --- /dev/null +++ b/mt5api/chartctl/paths.py @@ -0,0 +1,82 @@ +"""Directory layout and filename safety for chartctl. + +All chartctl writes are confined to four roots under TERMINAL_DIR: + + MQL5/Experts/Uploaded/ staged .ex5 (shared with the backtest feature) + chartctl/sets/ staged .set parameter files + chartctl/registry.json API-side desired-state registry + MQL5/Files/chartctl/ the EA-visible protocol directory, including + the generated per-deployment .tpl files — + ChartApplyTemplate only resolves paths under + the MQL5 dir (leading backslash) or relative + to the calling EX5, so templates must live here + +Every externally supplied filename passes safe_name() — same contract as +the backtest handler's _safe_basename, factored here so both artifact +endpoints and the tpl builder share one guard with one test matrix. +""" +import os +import re + +from mt5api.config import TERMINAL_DIR, ASSETS_DIR + +EXPERTS_DIR = os.path.join(TERMINAL_DIR, "MQL5", "Experts", "Uploaded") +SETS_DIR = os.path.join(TERMINAL_DIR, "chartctl", "sets") +REGISTRY_PATH = os.path.join(TERMINAL_DIR, "chartctl", "registry.json") +PROTOCOL_DIR = os.path.join(TERMINAL_DIR, "MQL5", "Files", "chartctl") +TEMPLATES_DIR = PROTOCOL_DIR +SCREENSHOTS_DIR = os.path.join(PROTOCOL_DIR, "shots") + +HOST_EXPERTS_DIR = os.path.join(ASSETS_DIR, "experts") +HOST_SETS_DIR = os.path.join(ASSETS_DIR, "sets") + +DESIRED_PATH = os.path.join(PROTOCOL_DIR, "desired.json") +OBSERVED_PATH = os.path.join(PROTOCOL_DIR, "observed.json") +COMMAND_PATH = os.path.join(PROTOCOL_DIR, "command.json") +COMMAND_RESULT_PATH = os.path.join(PROTOCOL_DIR, "command_result.json") + +# Windows-reserved characters plus anything that could smuggle a path. +_BAD_CHARS = re.compile(r'[<>:"/\\|?*\x00-\x1f]') + + +def ensure_dirs() -> None: + for d in (EXPERTS_DIR, SETS_DIR, PROTOCOL_DIR, TEMPLATES_DIR, + SCREENSHOTS_DIR, os.path.dirname(REGISTRY_PATH)): + os.makedirs(d, exist_ok=True) + + +def safe_name(name: str, field: str, required_ext: str | None = None) -> str: + """Validate an externally supplied filename. Returns the bare name. + + Rejects: empty, path separators, traversal, drive prefixes, UNC, + Windows-reserved characters, hidden dotfiles, and (optionally) a + wrong extension. Raises ValueError with a client-facing message. + """ + name = (name or "").strip() + if not name: + raise ValueError(f"{field}: filename is required") + if name != os.path.basename(name) or name in (".", ".."): + raise ValueError(f"{field}: path components are not allowed") + if _BAD_CHARS.search(name): + raise ValueError(f"{field}: illegal characters in filename") + if name.startswith("."): + raise ValueError(f"{field}: hidden files are not allowed") + if ".." in name: + raise ValueError(f"{field}: traversal sequences are not allowed") + if required_ext and not name.lower().endswith(required_ext): + raise ValueError(f"{field}: must end in {required_ext}") + return name + + +def atomic_write_text(path: str, text: str, encoding: str = "utf-8") -> None: + tmp = f"{path}.tmp" + with open(tmp, "w", encoding=encoding, newline="\r\n") as handle: + handle.write(text) + os.replace(tmp, path) + + +def atomic_write_bytes(path: str, data: bytes) -> None: + tmp = f"{path}.tmp" + with open(tmp, "wb") as handle: + handle.write(data) + os.replace(tmp, path) diff --git a/mt5api/chartctl/registry.py b/mt5api/chartctl/registry.py new file mode 100644 index 0000000..a910724 --- /dev/null +++ b/mt5api/chartctl/registry.py @@ -0,0 +1,274 @@ +"""Deployment registry: the API-side source of truth for desired state. + +Follows the backtest jobs.py pattern — persistent JSON, in-memory +write-through cache behind a lock, survives API restarts. Every mutation +bumps a monotonic `revision` and rewrites the EA-facing desired.json. + +Status model (derived, never stored as truth): + pending declared, loader has not confirmed it yet + running observed.json shows the expert live on a chart + degraded previously running, currently missing (reconciliation active) + failed loader reported a terminal error for this deployment + paused enabled=false in desired state +""" +from __future__ import annotations + +import json +import os +import secrets +import threading +import time +from datetime import datetime, timezone + +from mt5api.chartctl import paths +from mt5api.chartctl.tpl_builder import tpl_relative_name +from mt5api.config import ( + CHARTCTL_OBSERVED_STALE_SECONDS, + CHARTCTL_RECONCILE_HINT_SECONDS, +) +from mt5api.logger import log + +_LOCK = threading.Lock() +_STATE: dict | None = None # {"revision": int, "deployments": {id: {...}}} + +PROTOCOL_VERSION = 1 + + +def _now_iso() -> str: + return datetime.now(timezone.utc).replace(microsecond=0, tzinfo=None).isoformat() + "Z" + + +def new_id() -> str: + return "dep_" + secrets.token_hex(4) + + +def _empty_state() -> dict: + return {"revision": 0, "deployments": {}} + + +def _load_locked() -> dict: + global _STATE + if _STATE is not None: + return _STATE + if os.path.exists(paths.REGISTRY_PATH): + try: + with open(paths.REGISTRY_PATH, "r", encoding="utf-8") as handle: + _STATE = json.load(handle) + except (OSError, ValueError) as exc: + log.error("chartctl registry unreadable (%s) — starting empty; " + "corrupt file preserved as .bad", exc) + try: + os.replace(paths.REGISTRY_PATH, paths.REGISTRY_PATH + ".bad") + except OSError: + pass + _STATE = _empty_state() + else: + _STATE = _empty_state() + _STATE.setdefault("revision", 0) + _STATE.setdefault("deployments", {}) + return _STATE + + +def _persist_locked(state: dict) -> None: + paths.ensure_dirs() + paths.atomic_write_text(paths.REGISTRY_PATH, + json.dumps(state, indent=2, sort_keys=True)) + _write_desired_locked(state) + + +def _write_desired_locked(state: dict) -> None: + desired = { + "protocol": PROTOCOL_VERSION, + "revision": state["revision"], + "updated_at": _now_iso(), + "reconcile_interval": CHARTCTL_RECONCILE_HINT_SECONDS, + "deployments": [ + { + "id": dep["id"], + "expert": dep["expert_name"], + "template": tpl_relative_name(dep["id"]), + "symbol": dep["symbol"], + "timeframe": dep["timeframe"], + "enabled": dep.get("enabled", True), + } + for dep in sorted(state["deployments"].values(), + key=lambda d: d["created_at"]) + ], + } + paths.atomic_write_text(paths.DESIRED_PATH, + json.dumps(desired, indent=2, sort_keys=True)) + + +def list_deployments() -> list[dict]: + with _LOCK: + state = _load_locked() + return [dict(d) for d in state["deployments"].values()] + + +def get_deployment(dep_id: str) -> dict | None: + with _LOCK: + state = _load_locked() + dep = state["deployments"].get(dep_id) + return dict(dep) if dep else None + + +def add_deployment(*, expert_file: str, expert_name: str, set_file: str | None, + symbol: str, timeframe: str, enabled: bool = True) -> dict: + with _LOCK: + state = _load_locked() + for other in state["deployments"].values(): + if (other.get("enabled", True) and enabled + and other["symbol"].upper() == symbol.upper() + and other["timeframe"].upper() == timeframe.upper()): + raise DuplicateChart( + f"enabled deployment {other['id']} already targets " + f"{symbol} {timeframe}") + dep = { + "id": new_id(), + "expert_file": expert_file, + "expert_name": expert_name, + "set_file": set_file, + "symbol": symbol, + "timeframe": timeframe, + "enabled": enabled, + "created_at": _now_iso(), + "updated_at": _now_iso(), + } + state["deployments"][dep["id"]] = dep + state["revision"] += 1 + _persist_locked(state) + log.info("chartctl deployment %s created: %s %s %s (rev=%d)", + dep["id"], expert_name, symbol, timeframe, state["revision"]) + return dict(dep) + + +def update_deployment(dep_id: str, **changes) -> dict: + with _LOCK: + state = _load_locked() + dep = state["deployments"].get(dep_id) + if dep is None: + raise KeyError(dep_id) + dep.update(changes) + dep["updated_at"] = _now_iso() + state["revision"] += 1 + _persist_locked(state) + log.info("chartctl deployment %s updated (%s) rev=%d", + dep_id, ", ".join(changes), state["revision"]) + return dict(dep) + + +def remove_deployment(dep_id: str) -> dict: + with _LOCK: + state = _load_locked() + dep = state["deployments"].pop(dep_id, None) + if dep is None: + raise KeyError(dep_id) + state["revision"] += 1 + _persist_locked(state) + log.info("chartctl deployment %s removed (rev=%d)", + dep_id, state["revision"]) + return dep + + +def expert_in_use(expert_file: str) -> bool: + with _LOCK: + state = _load_locked() + return any(d["expert_file"] == expert_file + for d in state["deployments"].values()) + + +def current_revision() -> int: + with _LOCK: + return _load_locked()["revision"] + + +def rewrite_desired() -> None: + """Force-regenerate desired.json from the registry (bumps revision so + the loader re-reads even if content is identical — used by + /deployments/reconcile).""" + with _LOCK: + state = _load_locked() + state["revision"] += 1 + _persist_locked(state) + + +class DuplicateChart(ValueError): + pass + + +# ── Observed-state reading & merge ────────────────────────────────── + +def read_observed() -> tuple[dict | None, bool]: + """Return (observed_dict_or_None, is_stale).""" + try: + with open(paths.OBSERVED_PATH, "r", encoding="utf-8") as handle: + observed = json.load(handle) + except (OSError, ValueError): + return None, True + stale = True + try: + mtime = os.path.getmtime(paths.OBSERVED_PATH) + stale = (time.time() - mtime) > CHARTCTL_OBSERVED_STALE_SECONDS + except OSError: + pass + return observed, stale + + +def loader_alive(observed: dict | None, stale: bool) -> bool: + return bool(observed) and not stale and bool(observed.get("loader")) + + +def merged_view() -> dict: + """The GET /deployments payload: desired ⋈ observed.""" + observed, stale = read_observed() + obs_by_id: dict[str, dict] = {} + err_by_id: dict[str, dict] = {} + if observed: + for entry in observed.get("deployments") or []: + if entry.get("id"): + obs_by_id[entry["id"]] = entry + for entry in observed.get("errors") or []: + if entry.get("id"): + err_by_id[entry["id"]] = entry + + applied_revision = (observed or {}).get("loader", {}).get("applied_revision") + revision = current_revision() + items = [] + all_converged = True + for dep in list_deployments(): + obs = obs_by_id.get(dep["id"]) + err = err_by_id.get(dep["id"]) + status = _derive_status(dep, obs, err, stale) + if status not in ("running", "paused"): + all_converged = False + items.append({ + "id": dep["id"], + "desired": dep, + "observed": obs, + "error": err, + "status": status, + }) + + return { + "revision": revision, + "applied_revision": applied_revision, + "converged": all_converged and applied_revision == revision and not stale, + "observed_stale": stale, + "loader": (observed or {}).get("loader"), + "deployments": items, + } + + +def _derive_status(dep: dict, obs: dict | None, err: dict | None, + stale: bool) -> str: + if not dep.get("enabled", True): + return "paused" + if err and (obs is None or obs.get("status") != "running"): + return "failed" + if obs and obs.get("status") == "running" and not stale: + return "running" + if obs and obs.get("status") == "running" and stale: + return "degraded" + if obs and obs.get("status"): + return str(obs["status"]) + return "pending" diff --git a/mt5api/chartctl/setparse.py b/mt5api/chartctl/setparse.py new file mode 100644 index 0000000..8a87408 --- /dev/null +++ b/mt5api/chartctl/setparse.py @@ -0,0 +1,63 @@ +"""Parse MT5 .set parameter files into structured inputs. + +The mirror of backtest/set_builder.py (JSON -> .set); this goes .set -> +structured list. MT5 saves .set as UTF-16-LE with BOM; hand-written ones +are often UTF-8 or ASCII. Grammar per line: + + Name=value plain input + Name=value||start||step||stop||Y input with optimization metadata + ; comment + +The optimization tail is irrelevant for live deployment — the leading +value is what the terminal applies — but we keep it so clients can +round-trip and diff files losslessly. +""" + + +def _decode(data: bytes) -> str: + # Never blind-try utf-16: the codec "succeeds" on any even-length + # ASCII input by pairing bytes into CJK garbage, which then parses to + # zero inputs — and the deployment silently runs on EA defaults. + # Decide by BOM (MT5 exports carry one), then by embedded NULs + # (BOM-less UTF-16), then plain 8-bit. + if data[:2] in (b"\xff\xfe", b"\xfe\xff"): + return data.decode("utf-16") + if b"\x00" in data: + try: + return data.decode("utf-16-le") + except (UnicodeDecodeError, UnicodeError): + pass + for enc in ("utf-8-sig", "utf-8"): + try: + return data.decode(enc) + except UnicodeDecodeError: + continue + return data.decode("latin-1") + + +def parse_set_bytes(data: bytes) -> list[dict]: + return parse_set_text(_decode(data)) + + +def parse_set_text(text: str) -> list[dict]: + """Return [{name, value, optimize?, start?, step?, stop?}, ...].""" + inputs: list[dict] = [] + for raw_line in text.splitlines(): + line = raw_line.strip().lstrip("\ufeff") + if not line or line.startswith(";") or line.startswith("#"): + continue + if "=" not in line: + continue + name, _, rhs = line.partition("=") + name = name.strip() + if not name: + continue + parts = rhs.split("||") + entry: dict = {"name": name, "value": parts[0].strip()} + if len(parts) >= 5: + entry["start"] = parts[1].strip() + entry["step"] = parts[2].strip() + entry["stop"] = parts[3].strip() + entry["optimize"] = parts[4].strip().upper() == "Y" + inputs.append(entry) + return inputs diff --git a/mt5api/chartctl/tpl_builder.py b/mt5api/chartctl/tpl_builder.py new file mode 100644 index 0000000..e0cadbe --- /dev/null +++ b/mt5api/chartctl/tpl_builder.py @@ -0,0 +1,100 @@ +"""Generate per-deployment MT5 chart templates (.tpl). + +A minimal template whose block carries the expert path, flags, +and the full list translated from a parsed .set file. Applying +the template to a chart attaches the expert with those inputs — the only +programmatic attach path MT5 offers (via ChartApplyTemplate from MQL5). + +Attribution: the loader sets a chart comment `chartctl:` at attach +time — that's the authoritative cross-restart marker (MT5 persists the +comment with the saved chart). We ALSO stamp the id into the template's +`description` field and a reserved __chartctl_id input purely for human +forensics / grepping raw .tpl files; the loader does not depend on being +able to read another expert's inputs (MQL5 can't), so the comment is the +real mechanism. + +Encoding: modern terminal builds save templates as UTF-16-LE with BOM +and CRLF line endings; older builds accept the same. We always emit +UTF-16-LE + BOM. Every template carries a comment header with the +generator version and the terminal build it was generated for, so a +misbehaving attach can be forensically matched to an encoding profile. + +Known-risk note (spec §8): value encoding for enums/booleans has +build-specific quirks. Values are passed through verbatim from the .set +file, which is the safest policy: the .set was produced by the same +terminal family that will consume the template. Golden-file tests pin +the exact bytes per build. +""" +from mt5api.chartctl.paths import TEMPLATES_DIR, atomic_write_bytes + +import os + +GENERATOR_VERSION = "1.0.0" + +# EA flags observed in terminal-saved templates: allow live trading + +# allow DLL confirmations off + enabled. 343 = common "enabled, algo +# trading allowed" profile seen across builds; kept as a constant so a +# build-specific override is one line. +EXPERT_FLAGS = 343 + +# Reserved input name the loader EA reads for attribution. Harmless for +# the target expert: MT5 ignores unknown inputs in templates. +ID_INPUT = "__chartctl_id" + + +def build_tpl_text(*, deployment_id: str, expert_name: str, + expert_rel_path: str, inputs: list[dict], + terminal_build: int | None = None) -> str: + """Compose template text. expert_rel_path like 'Experts\\Uploaded\\X.ex5'.""" + lines = [ + "", + f"description=chartctl:{deployment_id} gen={GENERATOR_VERSION}" + + (f" build={terminal_build}" if terminal_build else ""), + "shift=1", + "autoscroll=1", + "ohlc=1", + "one_click=0", + "", + f"name={expert_name}", + f"path={expert_rel_path}", + f"flags={EXPERT_FLAGS}", + "expertmode=1", + "", + f"{ID_INPUT}={deployment_id}", + ] + for entry in inputs: + lines.append(f"{entry['name']}={entry['value']}") + lines += [ + "", + "", + "", + "", + ] + return "\r\n".join(lines) + + +def tpl_filename(deployment_id: str) -> str: + return f"{deployment_id}.tpl" + + +def tpl_relative_name(deployment_id: str) -> str: + """The name the loader passes to ChartApplyTemplate. The leading + backslash makes MT5 resolve it against \\MQL5 — the only + host-EA-independent root ChartApplyTemplate searches (without it the + path is relative to the calling EX5's folder, which varies per host + EA and yields err 5019 file-not-found).""" + return f"\\Files\\chartctl\\{tpl_filename(deployment_id)}" + + +def write_tpl(*, deployment_id: str, expert_name: str, expert_rel_path: str, + inputs: list[dict], terminal_build: int | None = None) -> str: + text = build_tpl_text( + deployment_id=deployment_id, + expert_name=expert_name, + expert_rel_path=expert_rel_path, + inputs=inputs, + terminal_build=terminal_build, + ) + path = os.path.join(TEMPLATES_DIR, tpl_filename(deployment_id)) + atomic_write_bytes(path, b"\xff\xfe" + text.encode("utf-16-le")) + return path diff --git a/mt5api/chartctl/webrequest.py b/mt5api/chartctl/webrequest.py new file mode 100644 index 0000000..620d136 --- /dev/null +++ b/mt5api/chartctl/webrequest.py @@ -0,0 +1,208 @@ +"""WebRequest allowlist manager for a single terminal. + +MT5's WebRequest allowed-URL list lives ONLY in ``/Config/common.ini`` +(``[Experts] WebRequest=1`` + ``WebRequestUrl=``), a UTF-16LE file that MT5 +reads at terminal startup. There is no live-reload, so applying a change requires a +terminal restart, and ``start.bat`` deletes ``common.ini`` on every boot — so the +authoritative desired state is kept in a sibling ``webrequest.json`` that survives +reboots and is re-applied to ``common.ini`` at two points: + + * boot — ``scripts/config_helper.py`` write_ini, right after start.bat deletes + common.ini and before the terminal launches; + * runtime — ``mt5client.restart_terminal``, after the terminal is killed and + before it is relaunched (MT5 rewrites common.ini on exit, so the + write must happen while the terminal is down). + +First adoption migrates from whatever the terminal currently has (decode the +existing ``WebRequestUrl=`` blob) so manually-configured URLs are preserved. + +The blob codec (fully reverse-engineered) lives in +``scripts/webrequest_allowlist_codec.py`` and is loaded here by path so it stays a +single dependency-free source of truth, importable from both the app and the boot +helper. +""" +from __future__ import annotations + +import importlib.util +import json +import os + +DESIRED_FILENAME = "webrequest.json" +_BOM = b"\xff\xfe" + +# --- load the standalone codec by path (no package coupling) --- +_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +_CODEC_PATH = os.path.join(_REPO_ROOT, "scripts", "webrequest_allowlist_codec.py") +_spec = importlib.util.spec_from_file_location("webrequest_allowlist_codec", _CODEC_PATH) +_codec = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(_codec) +encode_urls = _codec.encode_urls +decode_blob = _codec.decode_blob + + +# ── paths ──────────────────────────────────────────────────────────── +def config_dir(terminal_dir: str) -> str: + return os.path.join(terminal_dir, "Config") + + +def common_ini_path(cfg_dir: str) -> str: + return os.path.join(cfg_dir, "common.ini") + + +def desired_path(cfg_dir: str) -> str: + return os.path.join(cfg_dir, DESIRED_FILENAME) + + +# ── url hygiene ────────────────────────────────────────────────────── +def clean_urls(urls) -> list[str]: + """Trim, drop blanks/dupes, keep only http(s) URLs without the ';' delimiter.""" + out: list[str] = [] + if not isinstance(urls, (list, tuple)): + return out + for u in urls: + if not isinstance(u, str): + continue + u = u.strip() + if not u or ";" in u or any(ord(c) < 0x20 for c in u): + continue + low = u.lower() + if not (low.startswith("http://") or low.startswith("https://")): + continue + if u not in out: + out.append(u) + return out + + +# ── desired-state store (survives reboots) ─────────────────────────── +def load_desired(cfg_dir: str) -> list[str] | None: + """Return the persisted URL list, or None if this terminal has none set.""" + path = desired_path(cfg_dir) + if not os.path.exists(path): + return None + try: + with open(path, "r", encoding="utf-8") as f: + data = json.load(f) + except (OSError, ValueError): + return None + urls = data.get("urls") if isinstance(data, dict) else data + return clean_urls(urls) if isinstance(urls, list) else None + + +def save_desired(cfg_dir: str, urls: list[str]) -> None: + os.makedirs(cfg_dir, exist_ok=True) + path = desired_path(cfg_dir) + tmp = path + ".tmp" + with open(tmp, "w", encoding="utf-8") as f: + json.dump({"urls": clean_urls(urls)}, f, indent=2) + os.replace(tmp, path) + + +# ── common.ini (UTF-16LE + BOM + CRLF) ─────────────────────────────── +def _read_lines(path: str) -> list[str]: + if not os.path.exists(path): + return [] + try: + with open(path, "rb") as f: + raw = f.read() + except OSError: + return [] + text = raw.decode("utf-16", errors="ignore") # strips BOM + return text.replace("\r\n", "\n").replace("\r", "\n").split("\n") + + +def _write_lines(path: str, lines: list[str]) -> None: + os.makedirs(os.path.dirname(path), exist_ok=True) + text = "\r\n".join(lines) + data = _BOM + text.encode("utf-16-le") + tmp = path + ".tmp" + with open(tmp, "wb") as f: + f.write(data) + os.replace(tmp, path) + + +def read_current_urls(cfg_dir: str) -> list[str]: + """Decode the URL list from an existing common.ini (migrate-from-current).""" + blob = "" + in_experts = False + for line in _read_lines(common_ini_path(cfg_dir)): + s = line.strip() + if s.startswith("[") and s.endswith("]"): + in_experts = s.lower() == "[experts]" + continue + if in_experts and "=" in s and s.split("=", 1)[0].strip().lower() == "webrequesturl": + blob = s.split("=", 1)[1].strip() + break + if not blob: + return [] + try: + return clean_urls(decode_blob(blob)) + except Exception: + return [] + + +def effective_urls(cfg_dir: str) -> list[str]: + """Desired state if set, else whatever the terminal currently holds.""" + desired = load_desired(cfg_dir) + return desired if desired is not None else read_current_urls(cfg_dir) + + +def write_common_ini(cfg_dir: str, urls: list[str]) -> None: + """Set ``[Experts] WebRequest=1`` + ``WebRequestUrl=`` in common.ini, + preserving every other line/section. Creates the file (and [Experts]) if + absent. Encoding stays UTF-16LE+BOM+CRLF, as MT5 writes it.""" + urls = clean_urls(urls) + kv = {"WebRequest": "1", "WebRequestUrl": encode_urls(urls)} + lines = _read_lines(common_ini_path(cfg_dir)) + out: list[str] = [] + in_experts = False + written: set[str] = set() + has_experts = any(l.strip().lower() == "[experts]" for l in lines) + + def flush_missing(): + for key, val in kv.items(): + if key not in written: + out.append(f"{key}={val}") + written.add(key) + + for line in lines: + s = line.strip() + if s.startswith("[") and s.endswith("]"): + if in_experts: + flush_missing() + in_experts = s.lower() == "[experts]" + out.append(line) + continue + if in_experts and "=" in s: + key = s.split("=", 1)[0].strip() + for real in kv: + if key.lower() == real.lower(): + if real not in written: + out.append(f"{real}={kv[real]}") + written.add(real) + break + else: + out.append(line) + continue + out.append(line) + + if in_experts: + flush_missing() + if not has_experts: + out.append("[Experts]") + flush_missing() + # drop trailing blank lines then keep exactly one terminator via join + while out and out[-1] == "": + out.pop() + _write_lines(common_ini_path(cfg_dir), out) + + +def apply_from_desired(terminal_dir: str) -> int | None: + """Regenerate common.ini from the desired file. Returns URL count, or None + if there is no desired state (in which case common.ini is left untouched). + Safe to call while the terminal is stopped.""" + cfg_dir = config_dir(terminal_dir) + desired = load_desired(cfg_dir) + if desired is None: + return None + write_common_ini(cfg_dir, desired) + return len(desired) diff --git a/mt5api/config.py b/mt5api/config.py index 1338f4c..e92be85 100644 --- a/mt5api/config.py +++ b/mt5api/config.py @@ -280,6 +280,29 @@ def load_terminal_config(): TERMINAL_DIR = os.path.dirname(TERMINAL_PATH) INI_FILE = os.path.join(TERMINAL_DIR, "mt5start.ini") + +# Chart Deployments (chartctl) — live-mode only feature. Global default from +# config.yaml `chartctl:` block; per-terminal `chartctl: false` in terminals[] +# overrides it. Backtest-mode terminals never enable it: there is no running +# terminal64.exe to manage charts on. +_chartctl_cfg = load_yaml_config().get("chartctl") or {} +_chartctl_terminal_override = _terminal_config.get("chartctl") +_chartctl_global_enabled = bool(_chartctl_cfg.get("enabled", True)) +CHARTCTL_ENABLED = ( + MODE == "live" + and _chartctl_global_enabled + and (_chartctl_terminal_override is not False) +) +CHARTCTL_RECONCILE_HINT_SECONDS = parse_duration_to_seconds( + str(_chartctl_cfg.get("reconcile_hint_interval") or "5s") +) or 5 +CHARTCTL_OBSERVED_STALE_SECONDS = parse_duration_to_seconds( + str(_chartctl_cfg.get("observed_stale_after") or "60s") +) or 60 +CHARTCTL_COMMAND_TIMEOUT_SECONDS = parse_duration_to_seconds( + str(_chartctl_cfg.get("command_timeout") or "30s") +) or 30 +CHARTCTL_MAX_UPLOAD_BYTES = int(_chartctl_cfg.get("max_upload_bytes") or 16 * 1024 * 1024) IDENTITY = make_identity(BROKER, ACCOUNT, INSTANCE) LOG_DIR = os.path.join(BASE_DIR, "logs") FULL_LOG = os.path.join(LOG_DIR, "full.log") diff --git a/mt5api/handlers/chartctl.py b/mt5api/handlers/chartctl.py new file mode 100644 index 0000000..8dad4bc --- /dev/null +++ b/mt5api/handlers/chartctl.py @@ -0,0 +1,394 @@ +"""Chart Deployments REST handlers. + +Lock-free by design: nothing here touches the MT5 SDK, so these routes +never queue behind the process-wide MT5 lock (mt5client.py). Everything +is file I/O against TERMINAL_DIR plus the loader-EA file protocol. +""" +import hashlib +import os + +from flask import jsonify, request, send_file + +from mt5api.chartctl import command as cmd +from mt5api.chartctl import paths, registry +from mt5api.chartctl.setparse import parse_set_bytes +from mt5api.chartctl.tpl_builder import write_tpl +from mt5api.config import CHARTCTL_MAX_UPLOAD_BYTES +from mt5api.logger import log + +_VALID_TIMEFRAMES = frozenset({ + "M1", "M2", "M3", "M4", "M5", "M6", "M10", "M12", "M15", "M20", "M30", + "H1", "H2", "H3", "H4", "H6", "H8", "H12", "D1", "W1", "MN1", +}) + + +def _err(status: int, code: str, message: str): + return jsonify({"error": message, "code": code}), status + + +def _sha256(path: str) -> str: + digest = hashlib.sha256() + with open(path, "rb") as handle: + for chunk in iter(lambda: handle.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _list_dir(directory: str, ext: str, source: str) -> list[dict]: + items = [] + if not os.path.isdir(directory): + return items + for name in sorted(os.listdir(directory)): + if not name.lower().endswith(ext): + continue + full = os.path.join(directory, name) + if not os.path.isfile(full): + continue + stat = os.stat(full) + items.append({ + "name": name, + "size": stat.st_size, + "sha256": _sha256(full), + "modified_at": int(stat.st_mtime), + "source": source, + }) + return items + + +def _read_upload(field: str, required_ext: str) -> tuple[str, bytes]: + upload = request.files.get(field) + if upload is None or not upload.filename: + raise ValueError(f"Missing form file: {field}") + name = paths.safe_name(upload.filename, field, required_ext) + data = upload.stream.read(CHARTCTL_MAX_UPLOAD_BYTES + 1) + if len(data) > CHARTCTL_MAX_UPLOAD_BYTES: + raise ValueError( + f"{field}: file exceeds {CHARTCTL_MAX_UPLOAD_BYTES} bytes") + if not data: + raise ValueError(f"{field}: file is empty") + return name, data + + +# ── Artifacts: experts ─────────────────────────────────────────────── + +def upload_expert(): + try: + name, data = _read_upload("expert", ".ex5") + except ValueError as exc: + return _err(400, "BAD_REQUEST", str(exc)) + paths.ensure_dirs() + dest = os.path.join(paths.EXPERTS_DIR, name) + new_hash = hashlib.sha256(data).hexdigest() + overwrite = (request.args.get("overwrite", "false").lower() == "true") + if os.path.exists(dest) and not overwrite: + if _sha256(dest) == new_hash: + return jsonify({"name": name, "sha256": new_hash, + "skipped": True}) + return _err(409, "EXISTS", + f"{name} exists with different content; " + "pass ?overwrite=true to replace") + paths.atomic_write_bytes(dest, data) + log.info("chartctl expert staged: %s (%d bytes, %s)", + name, len(data), new_hash[:12]) + return jsonify({"name": name, "sha256": new_hash, "size": len(data)}), 201 + + +def list_experts(): + return jsonify({ + "experts": _list_dir(paths.EXPERTS_DIR, ".ex5", "uploaded") + + _list_dir(paths.HOST_EXPERTS_DIR, ".ex5", "host"), + }) + + +def delete_expert(name): + try: + name = paths.safe_name(name, "expert", ".ex5") + except ValueError as exc: + return _err(400, "BAD_REQUEST", str(exc)) + if os.path.exists(os.path.join(paths.HOST_EXPERTS_DIR, name)) and \ + not os.path.exists(os.path.join(paths.EXPERTS_DIR, name)): + return _err(403, "HOST_ASSET", "host-managed assets are read-only") + target = os.path.join(paths.EXPERTS_DIR, name) + if not os.path.exists(target): + return _err(404, "ARTIFACT_NOT_FOUND", f"{name} is not staged") + if registry.expert_in_use(name): + return _err(409, "IN_USE", + f"{name} is referenced by an existing deployment") + os.remove(target) + return jsonify({"deleted": name}) + + +# ── Artifacts: sets ────────────────────────────────────────────────── + +def upload_set(): + try: + name, data = _read_upload("set", ".set") + inputs = parse_set_bytes(data) + except ValueError as exc: + return _err(400, "BAD_REQUEST", str(exc)) + paths.ensure_dirs() + dest = os.path.join(paths.SETS_DIR, name) + paths.atomic_write_bytes(dest, data) + log.info("chartctl set staged: %s (%d inputs)", name, len(inputs)) + return jsonify({"name": name, + "sha256": hashlib.sha256(data).hexdigest(), + "inputs": inputs}), 201 + + +def list_sets(): + return jsonify({ + "sets": _list_dir(paths.SETS_DIR, ".set", "uploaded") + + _list_dir(paths.HOST_SETS_DIR, ".set", "host"), + }) + + +def get_set(name): + try: + name = paths.safe_name(name, "set", ".set") + except ValueError as exc: + return _err(400, "BAD_REQUEST", str(exc)) + path = _resolve_set(name) + if path is None: + return _err(404, "ARTIFACT_NOT_FOUND", f"{name} is not staged") + with open(path, "rb") as handle: + data = handle.read() + return jsonify({"name": name, "inputs": parse_set_bytes(data)}) + + +def _resolve_set(name: str) -> str | None: + for base in (paths.SETS_DIR, paths.HOST_SETS_DIR): + candidate = os.path.join(base, name) + if os.path.isfile(candidate): + return candidate + return None + + +def _resolve_expert(name: str) -> str | None: + for base in (paths.EXPERTS_DIR, paths.HOST_EXPERTS_DIR): + candidate = os.path.join(base, name) + if os.path.isfile(candidate): + return candidate + return None + + +# ── Deployments ────────────────────────────────────────────────────── + +def _terminal_build() -> int | None: + """Best-effort build stamp for the template header. Never blocks: + peeks at cached terminal info without taking the MT5 lock.""" + try: + from mt5api import mt5client + info = getattr(mt5client, "LAST_TERMINAL_INFO", None) + if info and getattr(info, "build", None): + return int(info.build) + except Exception: # noqa: BLE001 — stamp is cosmetic, never fail on it + pass + return None + + +def _materialize_tpl(dep: dict) -> None: + inputs: list[dict] = [] + if dep.get("set_file"): + set_path = _resolve_set(dep["set_file"]) + if set_path is None: + raise ValueError(f"set file {dep['set_file']} disappeared") + with open(set_path, "rb") as handle: + inputs = parse_set_bytes(handle.read()) + # Optimization tails are meaningless on a live chart. + inputs = [{"name": i["name"], "value": i["value"]} for i in inputs] + write_tpl( + deployment_id=dep["id"], + expert_name=dep["expert_name"], + expert_rel_path=f"Experts\\Uploaded\\{dep['expert_file']}", + inputs=inputs, + terminal_build=_terminal_build(), + ) + + +def create_deployment(): + body = request.get_json(silent=True) or {} + try: + expert_file = paths.safe_name(body.get("expert", ""), "expert", ".ex5") + set_file = None + if body.get("set"): + set_file = paths.safe_name(body["set"], "set", ".set") + symbol = str(body.get("symbol", "")).strip() + timeframe = str(body.get("timeframe", "")).strip().upper() + enabled = bool(body.get("enabled", True)) + if not symbol: + raise ValueError("symbol is required") + if timeframe not in _VALID_TIMEFRAMES: + raise ValueError(f"timeframe must be one of " + f"{sorted(_VALID_TIMEFRAMES)}") + except ValueError as exc: + return _err(400, "BAD_REQUEST", str(exc)) + + expert_path = _resolve_expert(expert_file) + if expert_path is None: + return _err(404, "ARTIFACT_NOT_FOUND", + f"expert {expert_file} is not staged — upload it first") + if expert_path.startswith(paths.HOST_EXPERTS_DIR): + # Host asset: mirror into Uploaded/ so the terminal can load it. + paths.ensure_dirs() + with open(expert_path, "rb") as handle: + paths.atomic_write_bytes( + os.path.join(paths.EXPERTS_DIR, expert_file), handle.read()) + if set_file and _resolve_set(set_file) is None: + return _err(404, "ARTIFACT_NOT_FOUND", + f"set {set_file} is not staged — upload it first") + + expert_name = expert_file[:-4] if expert_file.lower().endswith(".ex5") \ + else expert_file + try: + dep = registry.add_deployment( + expert_file=expert_file, expert_name=expert_name, + set_file=set_file, symbol=symbol, timeframe=timeframe, + enabled=enabled) + except registry.DuplicateChart as exc: + return _err(409, "DUPLICATE_CHART", str(exc)) + + try: + _materialize_tpl(dep) + except ValueError as exc: + registry.remove_deployment(dep["id"]) + return _err(500, "TPL_GENERATION_FAILED", str(exc)) + + return jsonify({"id": dep["id"], "status": "pending", + "deployment": dep}), 202 + + +def list_deployments(): + return jsonify(registry.merged_view()) + + +def get_deployment(dep_id): + view = registry.merged_view() + for item in view["deployments"]: + if item["id"] == dep_id: + item["revision"] = view["revision"] + item["observed_stale"] = view["observed_stale"] + return jsonify(item) + return _err(404, "NOT_FOUND", f"deployment {dep_id} does not exist") + + +def patch_deployment(dep_id): + body = request.get_json(silent=True) or {} + changes: dict = {} + try: + if "set" in body: + changes["set_file"] = ( + paths.safe_name(body["set"], "set", ".set") + if body["set"] else None) + if changes["set_file"] and _resolve_set(changes["set_file"]) is None: + return _err(404, "ARTIFACT_NOT_FOUND", + f"set {changes['set_file']} is not staged") + if "enabled" in body: + changes["enabled"] = bool(body["enabled"]) + except ValueError as exc: + return _err(400, "BAD_REQUEST", str(exc)) + if not changes: + return _err(400, "BAD_REQUEST", + "nothing to change: pass 'set' and/or 'enabled'") + try: + dep = registry.update_deployment(dep_id, **changes) + except KeyError: + return _err(404, "NOT_FOUND", f"deployment {dep_id} does not exist") + if "set_file" in changes: + try: + _materialize_tpl(dep) + except ValueError as exc: + return _err(500, "TPL_GENERATION_FAILED", str(exc)) + return jsonify({"id": dep_id, "deployment": dep}) + + +def delete_deployment(dep_id): + try: + dep = registry.remove_deployment(dep_id) + except KeyError: + return _err(404, "NOT_FOUND", f"deployment {dep_id} does not exist") + # Template file is left on disk until the loader confirms detach; the + # loader clears the chart because the deployment vanished from + # desired.json. Cleanup of orphaned .tpl files happens lazily here. + tpl = os.path.join(paths.TEMPLATES_DIR, f"{dep_id}.tpl") + try: + os.remove(tpl) + except OSError: + pass + return jsonify({"deleted": dep_id, "was": dep}) + + +def reconcile(): + registry.rewrite_desired() + return jsonify({"revision": registry.current_revision()}), 202 + + +# ── Observation ────────────────────────────────────────────────────── + +def charts(): + observed, stale = registry.read_observed() + return jsonify({ + "loader_alive": registry.loader_alive(observed, stale), + "observed_stale": stale, + "loader": (observed or {}).get("loader"), + "auto_trading": (observed or {}).get("terminal", {}).get("auto_trading"), + "charts": (observed or {}).get("charts", []), + }) + + +def loader_status(): + observed, stale = registry.read_observed() + alive = registry.loader_alive(observed, stale) + payload = { + "alive": alive, + "observed_stale": stale, + "loader": (observed or {}).get("loader"), + "desired_revision": registry.current_revision(), + "applied_revision": + (observed or {}).get("loader", {}).get("applied_revision"), + } + if not alive: + payload["hint"] = ( + "No live loader detected. Attach MT5ChartLoader (bundled under " + "assets/experts/) to any chart, or add ChartControl.mqh to your " + "own resident EA — see docs/chart-control-protocol.md.") + return jsonify(payload) + + +def close_chart(chart_id): + try: + result = cmd.run_command("close_chart", {"chart_id": int(chart_id)}) + except ValueError: + return _err(400, "BAD_REQUEST", "chart_id must be an int") + except cmd.LoaderBusy as exc: + return _err(409, "LOADER_BUSY", str(exc)) + except cmd.LoaderTimeout as exc: + return _err(504, "LOADER_TIMEOUT", str(exc)) + if result.get("status") != "ok": + return _err(502, result.get("error_code", "LOADER_ERROR"), + result.get("error_detail", "loader reported failure")) + return jsonify({"closed": int(chart_id)}) + + +def screenshot(chart_id): + try: + result = cmd.run_command("screenshot", { + "chart_id": int(chart_id), + "width": int(request.args.get("width", 1280)), + "height": int(request.args.get("height", 720)), + }) + except ValueError: + return _err(400, "BAD_REQUEST", "chart_id/width/height must be ints") + except cmd.LoaderBusy as exc: + return _err(409, "LOADER_BUSY", str(exc)) + except cmd.LoaderTimeout as exc: + return _err(504, "LOADER_TIMEOUT", str(exc)) + if result.get("status") != "ok": + return _err(502, result.get("error_code", "LOADER_ERROR"), + result.get("error_detail", "loader reported failure")) + filename = paths.safe_name(result.get("file", ""), "screenshot") + png = os.path.join(paths.SCREENSHOTS_DIR, filename) + if not os.path.isfile(png): + return _err(502, "LOADER_ERROR", + "loader reported a screenshot that does not exist") + response = send_file(png, mimetype="image/png") + return response diff --git a/mt5api/handlers/webrequest.py b/mt5api/handlers/webrequest.py new file mode 100644 index 0000000..f03bb9f --- /dev/null +++ b/mt5api/handlers/webrequest.py @@ -0,0 +1,100 @@ +"""WebRequest allowlist endpoints. + +GET /webrequest -> current effective allowlist for this terminal. +PUT /webrequest -> set/patch the allowlist and apply it now. +POST /webrequest/apply -> re-apply the current allowlist (boot / manual hook, + since the VM terminal drops the list on restart). + +The apply mechanism is chosen at runtime. Inside the dockur VM the allowlist is +set by driving MT5's Options dialog with AutoIt (the only thing that persists it +in-session there — see chartctl/autoit_webrequest.py). Elsewhere (a bare-metal +terminal where ``common.ini`` IS the WebRequest store) it falls back to writing +``common.ini`` and restarting the terminal. + +Dedicated call (not a deployment field): URLs are only needed by the minority of +EAs that use WebRequest. First use migrates from whatever the terminal already +has so manually-configured URLs are preserved. +""" +from flask import jsonify, request + +from mt5api.chartctl import autoit_webrequest as autoit +from mt5api.chartctl import webrequest as wr +from mt5api.config import TERMINAL_DIR +from mt5api.mt5client import restart_terminal, session + + +def _cfg_dir(): + return wr.config_dir(TERMINAL_DIR) + + +def get_webrequest(): + return jsonify({"urls": wr.effective_urls(_cfg_dir())}) + + +def _resolve_urls(body, cfg_dir): + """Return (urls, error). Full replace via 'urls', or patch via 'add'/'remove'.""" + if "urls" in body: + return wr.clean_urls(body.get("urls")), None + if "add" in body or "remove" in body: + current = wr.effective_urls(cfg_dir) # migrate-from-current on first use + remove = set(wr.clean_urls(body.get("remove", []))) + new_urls = [u for u in current if u not in remove] + for u in wr.clean_urls(body.get("add", [])): + if u not in new_urls: + new_urls.append(u) + return new_urls, None + return None, "provide 'urls', or 'add'/'remove'" + + +def _apply(urls, use_runas=False): + """Apply ``urls`` to the running terminal. Returns (ok, detail).""" + if autoit.available(): + status, _log = autoit.apply_urls(urls, use_runas=use_runas) + return status == "OK", f"autoit:{status}" + # bare-metal fallback: write common.ini from desired, then restart. + with session(): + ok = restart_terminal() + return ok, "restart" if ok else "restart-failed" + + +def _use_runas(): + return request.args.get("runas", "0") == "1" + + +def put_webrequest(): + body = request.get_json(silent=True) + if not isinstance(body, dict): + return jsonify({"success": False, "error": "JSON body required"}), 400 + + cfg_dir = _cfg_dir() + new_urls, err = _resolve_urls(body, cfg_dir) + if err: + return jsonify({"success": False, "error": err}), 400 + + wr.save_desired(cfg_dir, new_urls) + ok, detail = _apply(new_urls, _use_runas()) + if not ok: + return jsonify( + {"success": False, "error": f"apply failed ({detail})", "urls": new_urls} + ), 500 + return jsonify({"success": True, "urls": new_urls, "applied_via": detail}) + + +def apply_webrequest(): + """Re-apply the current desired allowlist (idempotent). Boot/manual hook.""" + # dev: ?script= runs a repo-shipped AutoIt script (VM only), e.g. + # inspect_options.au3. ?runas=0 launches it non-elevated for diagnostics. + dev_script = request.args.get("script") + if dev_script and autoit.available(): + status, txt = autoit.run_named(dev_script, use_runas=_use_runas()) + return jsonify({"script": dev_script, "status": status, "log": txt}) + + urls = wr.effective_urls(_cfg_dir()) + if not urls: + return jsonify({"success": True, "urls": [], "note": "nothing to apply"}) + ok, detail = _apply(urls, _use_runas()) + if not ok: + return jsonify( + {"success": False, "error": f"apply failed ({detail})", "urls": urls} + ), 500 + return jsonify({"success": True, "urls": urls, "applied_via": detail}) diff --git a/mt5api/main.py b/mt5api/main.py index 91372ac..1094756 100644 --- a/mt5api/main.py +++ b/mt5api/main.py @@ -38,6 +38,52 @@ RETRY_INTERVAL = 30 +# Seconds to let the terminal GUI settle before re-applying the WebRequest +# allowlist via AutoIt on boot (see _reapply_webrequest_once). +WEBREQUEST_BOOT_DELAY = 25 +_webrequest_reapplied = threading.Event() + + +def _reapply_webrequest_once(): + """Re-apply this terminal's desired WebRequest allowlist via AutoIt after a + (re)start. MT5 on the VM drops the list every restart, so the API re-sets it + once the terminal GUI is up. No-op unless AutoIt is present (the VM) AND a + desired allowlist has been configured. Runs in a daemon thread so it never + blocks startup; guarded to run only once per process.""" + if _webrequest_reapplied.is_set(): + return + _webrequest_reapplied.set() + + def _work(): + try: + from mt5api.chartctl import autoit_webrequest as autoit + from mt5api.chartctl import webrequest as wr + from mt5api.config import TERMINAL_DIR + + if not autoit.available(): + return + urls = wr.effective_urls(wr.config_dir(TERMINAL_DIR)) + if not urls: + return + # Let charts/loader finish attaching, plus a per-terminal stagger + # (from the port) so multiple terminals on one host don't all pile + # onto the machine-wide GUI mutex at once. + time.sleep(WEBREQUEST_BOOT_DELAY + (PORT % 10) * 3) + for attempt in range(1, 4): # brief retry: GUI may still be busy + status, _log = autoit.apply_urls(urls) + log.info( + "Boot WebRequest re-apply attempt %d: %d url(s) -> %s", + attempt, len(urls), status, + ) + if status == "OK": + break + time.sleep(15) + except Exception: + log.exception("Boot WebRequest re-apply failed") + + threading.Thread(target=_work, daemon=True).start() + + # --- MCP interface ----------------------------------------------------------- # TODO: change to fastapi or smth. This WSGI<->ASGI bridge is a stopgap. mt5api # is a Flask/WSGI app (served by waitress), but the MCP server is an ASGI @@ -121,6 +167,7 @@ def _background_init(): connected = False if connected: log.info("MT5 connected on attempt %d.", attempt) + _reapply_webrequest_once() return log.warning("MT5 not ready, retrying in %ds...", RETRY_INTERVAL) time.sleep(RETRY_INTERVAL) @@ -181,6 +228,7 @@ def main(): if connected: log.info("MT5 connected.") + _reapply_webrequest_once() else: log.warning( "MT5 not ready yet, retrying every %ds in background...", diff --git a/mt5api/mt5client.py b/mt5api/mt5client.py index 9f5baeb..4f3c498 100644 --- a/mt5api/mt5client.py +++ b/mt5api/mt5client.py @@ -289,12 +289,20 @@ def init_mt5(login=None, password=None, server=None): return result +# Last successful terminal_info snapshot, cached lock-free for cosmetic +# readers (e.g. chartctl template build stamp). Never authoritative. +LAST_TERMINAL_INFO = None + + def ensure_initialized(): """Probe + reconnect helper. Caller must hold the MT5 lock.""" + global LAST_TERMINAL_INFO try: info = m(mt5.terminal_info, _timeout=15) except MT5Timeout: info = None + if info is not None: + LAST_TERMINAL_INFO = info if info is None: log.warning("Terminal not responding, attempting full init...") account = get_first_account() @@ -412,6 +420,18 @@ def restart_terminal(): if not killed: log.warning("No terminal process found, launching fresh.") + # Apply the WebRequest allowlist while the terminal is down. MT5 rewrites + # common.ini on exit, so this must happen after the kill and before launch. + # No-op unless this terminal has a desired allowlist set. + try: + from mt5api.chartctl import webrequest as _wr + + applied = _wr.apply_from_desired(TERMINAL_DIR) + if applied is not None: + log.info("Applied WebRequest allowlist (%d URL(s)) to common.ini", applied) + except Exception: + log.exception("Failed to apply WebRequest allowlist; continuing restart") + today = date.today().strftime("%Y%m%d") journal_log = os.path.join(TERMINAL_DIR, "logs", f"{today}.log") offset = 0 diff --git a/mt5api/server.py b/mt5api/server.py index 1aca624..486881e 100644 --- a/mt5api/server.py +++ b/mt5api/server.py @@ -4,7 +4,7 @@ from flask import Flask, abort, g, request from flask_compress import Compress from mt5api.backtest import handler as backtest_handler -from mt5api.config import API_TOKEN +from mt5api.config import API_TOKEN, CHARTCTL_ENABLED from mt5api.handlers import account, history, orders, positions, symbols, terminal from mt5api.logger import log @@ -94,6 +94,39 @@ def _end_request(response): app.get("/history/orders")(history.get_orders) app.get("/history/deals")(history.get_deals) +# ── Chart Deployments (chartctl) ───────────────────────────────── +# Lock-free EA deployment primitives. Gated: live mode + config enabled. +if CHARTCTL_ENABLED: + from mt5api.handlers import chartctl + + app.post("/experts")(chartctl.upload_expert) + app.get("/experts")(chartctl.list_experts) + app.delete("/experts/")(chartctl.delete_expert) + + app.post("/sets")(chartctl.upload_set) + app.get("/sets")(chartctl.list_sets) + app.get("/sets/")(chartctl.get_set) + + app.post("/deployments")(chartctl.create_deployment) + app.get("/deployments")(chartctl.list_deployments) + app.post("/deployments/reconcile")(chartctl.reconcile) + app.get("/deployments/")(chartctl.get_deployment) + app.patch("/deployments/")(chartctl.patch_deployment) + app.delete("/deployments/")(chartctl.delete_deployment) + + app.get("/charts")(chartctl.charts) + app.get("/loader")(chartctl.loader_status) + app.post("/charts//screenshot")(chartctl.screenshot) + app.post("/charts//close")(chartctl.close_chart) + + # WebRequest allowlist — dedicated call. Applied via AutoIt (VM) or a + # common.ini rewrite + restart (bare metal). /apply re-applies on demand. + from mt5api.handlers import webrequest + + app.get("/webrequest")(webrequest.get_webrequest) + app.put("/webrequest")(webrequest.put_webrequest) + app.post("/webrequest/apply")(webrequest.apply_webrequest) + # ── Backtest ───────────────────────────────────────────────────── app.post("/backtest/build-ini")(backtest_handler.build_ini_route) app.post("/backtest/build-set")(backtest_handler.build_set_route) diff --git a/run.sh b/run.sh index dbe4b46..1e64df8 100755 --- a/run.sh +++ b/run.sh @@ -60,10 +60,10 @@ cp "${DIR}/scripts/reboot.bat" "${DIR}/data/shared/scripts/reboot.bat" cp "${DIR}/scripts/acquire_lock.ps1" "${DIR}/data/shared/scripts/acquire_lock.ps1" cp "${DIR}/scripts/api_runner.bat" "${DIR}/data/shared/scripts/api_runner.bat" cp "${DIR}/scripts/compile-warmup-ea.bat" "${DIR}/data/shared/scripts/compile-warmup-ea.bat" - cp "${DIR}/scripts/check_health.py" "${DIR}/data/shared/scripts/check_health.py" cp "${DIR}/scripts/config_helper.py" "${DIR}/data/shared/scripts/config_helper.py" - +cp "${DIR}/scripts/compile-chartctl-loader.bat" "${DIR}/data/shared/scripts/compile-chartctl-loader.bat" +cp "${DIR}/scripts/webrequest_allowlist_codec.py" "${DIR}/data/shared/scripts/webrequest_allowlist_codec.py" cp "${DIR}/scripts/event-log-tailer.ps1" "${DIR}/data/shared/scripts/event-log-tailer.ps1" cp "${DIR}/scripts/healthcheck.sh" "${DIR}/data/shared/scripts/healthcheck.sh" chmod +x "${DIR}/data/shared/scripts/healthcheck.sh" diff --git a/scripts/compile-chartctl-loader.bat b/scripts/compile-chartctl-loader.bat new file mode 100644 index 0000000..cddd72c --- /dev/null +++ b/scripts/compile-chartctl-loader.bat @@ -0,0 +1,87 @@ +@echo off +setlocal enabledelayedexpansion + +rem ════════════════════════════════════════════════════════════════ +rem compile-chartctl-loader.bat +rem +rem Zero-touch bootstrap for the chartctl loader EA: +rem 1. Copies ChartControl.mqh into MQL5\Include\ of every broker +rem base terminal and MT5ChartLoader.mq5 into MQL5\Experts\Advisors\. +rem 2. Compiles the loader with each base's MetaEditor64. +rem 3. Copies the compiled .ex5 (+ include + source) into every +rem already-provisioned terminal instance dir, so existing +rem terminals pick it up without re-provisioning. New instances +rem inherit it automatically via the base xcopy. +rem +rem Combined with the [StartUp] Expert= line that config_helper.py +rem writes into mt5start.ini for live chartctl terminals, the loader +rem attaches itself at terminal launch — no RDP, no manual step. +rem +rem Modeled on compile-warmup-ea.bat (same MetaEditor invocation). +rem ════════════════════════════════════════════════════════════════ + +set "SHARED=C:\Users\Docker\Desktop\Shared" +set "ASSETS=C:\Users\Docker\Desktop\Assets" +if not exist "%ASSETS%\experts" set "ASSETS=%SHARED%\assets" + +set "SRC_EA=%ASSETS%\experts\MT5ChartLoader.mq5" +set "SRC_INC=%ASSETS%\experts\include\ChartControl.mqh" +set "COMPILE_LOG=%SHARED%\logs\compile-chartctl-loader.log" + +if not exist "%SRC_EA%" ( + echo ERROR: source not found: %SRC_EA% + exit /b 1 +) +if not exist "%SRC_INC%" ( + echo ERROR: include not found: %SRC_INC% + exit /b 1 +) + +set FOUND=0 +set FAILED=0 + +for /d %%B in ("%SHARED%\terminals\*") do ( + if exist "%%~fB\base\MetaEditor64.exe" ( + set /a FOUND+=1 + set "BASE=%%~fB\base" + echo [%%~nB] compiling loader in base... + + if not exist "!BASE!\MQL5\Include" mkdir "!BASE!\MQL5\Include" + if not exist "!BASE!\MQL5\Experts\Advisors" mkdir "!BASE!\MQL5\Experts\Advisors" + copy /Y "%SRC_INC%" "!BASE!\MQL5\Include\ChartControl.mqh" >nul + copy /Y "%SRC_EA%" "!BASE!\MQL5\Experts\Advisors\MT5ChartLoader.mq5" >nul + + "!BASE!\MetaEditor64.exe" /compile:"!BASE!\MQL5\Experts\Advisors\MT5ChartLoader.mq5" /inc:"!BASE!\MQL5" /log:"%COMPILE_LOG%" >nul 2>&1 + + if exist "!BASE!\MQL5\Experts\Advisors\MT5ChartLoader.ex5" ( + echo [%%~nB] OK: MT5ChartLoader.ex5 + + rem Propagate into every provisioned instance of this broker: + rem terminals\\\\ layout, skip \base. + for /d %%A in ("%%~fB\*") do ( + if /i not "%%~nxA"=="base" ( + for /d %%I in ("%%~fA\*") do ( + if exist "%%~fI\terminal64.exe" ( + if not exist "%%~fI\MQL5\Include" mkdir "%%~fI\MQL5\Include" + if not exist "%%~fI\MQL5\Experts\Advisors" mkdir "%%~fI\MQL5\Experts\Advisors" + copy /Y "!BASE!\MQL5\Experts\Advisors\MT5ChartLoader.ex5" "%%~fI\MQL5\Experts\Advisors\" >nul + copy /Y "%SRC_INC%" "%%~fI\MQL5\Include\ChartControl.mqh" >nul + copy /Y "%SRC_EA%" "%%~fI\MQL5\Experts\Advisors\MT5ChartLoader.mq5" >nul + echo [%%~nB] propagated to %%~nA\%%~nI + ) + ) + ) + ) + ) else ( + set /a FAILED+=1 + echo [%%~nB] ERROR: compile produced no .ex5 — see %COMPILE_LOG% + ) + ) +) + +if %FOUND%==0 ( + echo ERROR: no broker base terminals found under %SHARED%\terminals + exit /b 1 +) +if %FAILED% gtr 0 exit /b 1 +exit /b 0 diff --git a/scripts/config_helper.py b/scripts/config_helper.py index 50ecfc0..b5ffbbb 100644 --- a/scripts/config_helper.py +++ b/scripts/config_helper.py @@ -165,9 +165,11 @@ def main(): elif cmd == "write_ini": if len(sys.argv) < 5: - print("Usage: config_helper.py write_ini ", file=sys.stderr) + print("Usage: config_helper.py write_ini [instance] [mode]", file=sys.stderr) sys.exit(1) broker, account, outpath = sys.argv[2], sys.argv[3], sys.argv[4] + instance = sys.argv[5] if len(sys.argv) > 5 else "default" + mode = (sys.argv[6] if len(sys.argv) > 6 else "live").lower() accounts = cfg.get("accounts", {}) b = accounts.get(broker, {}) creds = b.get(account) if account else next(iter(b.values()), None) if b else None @@ -179,9 +181,51 @@ def main(): ini += "KeepPrivate=0\nAutoTrading=1\nNewsEnable=0\n" ini += "[Experts]\nAllowLiveTrading=1\nAllowDllImport=1\nEnabled=1\n" ini += "[Email]\nEnable=0\n" + + # Chart Deployments loader bootstrap: auto-attach MT5ChartLoader at + # terminal launch via [StartUp]. Gated exactly like the API's + # CHARTCTL_ENABLED: live mode + global chartctl.enabled (default + # true) + no per-terminal `chartctl: false` override. The loader's + # GlobalVariable mutex makes the re-fire on every launch idempotent + # (a duplicate closes its own chart and exits). + chartctl_cfg = cfg.get("chartctl") or {} + chartctl_on = bool(chartctl_cfg.get("enabled", True)) + term_override = None + term_suffix = "" + for t in cfg.get("terminals", []): + if (t.get("broker") == broker and str(t.get("account")) == str(account) + and (t.get("instance") or "default") == instance): + term_override = t.get("chartctl") + term_suffix = t.get("symbol_suffix") or "" + break + if mode == "live" and chartctl_on and term_override is not False: + ini += "[StartUp]\n" + ini += "Expert=Advisors\\MT5ChartLoader\n" + ini += f"Symbol=EURUSD{term_suffix}\n" + ini += "Period=H1\n" + with open(outpath, "w", encoding="utf-8") as f: f.write(ini) + # WebRequest allowlist boot-seed: start.bat deletes Config/common.ini + # every boot, so re-emit it here (after the delete, before launch) from + # the persistent per-terminal desired file. No-op when this terminal has + # no allowlist set. Gated exactly like the loader [StartUp] block above. + if mode == "live" and chartctl_on and term_override is not False: + try: + sys.path.insert(0, _SHARED_DIR) + from mt5api.chartctl import webrequest as wr + cfg_dir = os.path.join(os.path.dirname(os.path.abspath(outpath)), "Config") + urls = wr.load_desired(cfg_dir) + if urls is not None: + wr.write_common_ini(cfg_dir, urls) + except Exception as exc: # non-fatal: never block terminal launch + print(f"WARN: WebRequest allowlist seed failed: {exc}", file=sys.stderr) + + elif cmd == "chartctl_enabled": + chartctl_cfg = cfg.get("chartctl") or {} + print("1" if bool(chartctl_cfg.get("enabled", True)) else "0") + elif cmd == "nginx_conf": if len(sys.argv) < 3: print("Usage: config_helper.py nginx_conf ", file=sys.stderr) diff --git a/scripts/start.bat b/scripts/start.bat index b880b21..5a3b5d1 100644 --- a/scripts/start.bat +++ b/scripts/start.bat @@ -1,388 +1,418 @@ -@echo off -setlocal enabledelayedexpansion -set SHARED=C:\Users\Docker\Desktop\Shared -set SCRIPTS=%SHARED%\scripts -set CONFIG=%SHARED%\config -set BROKERS=%SHARED%\terminals -set LOGDIR=%SHARED%\logs -set INSTALL_LOG=%LOGDIR%\install.log -set PIP_LOG=%LOGDIR%\pip.log -set START_LOG=%LOGDIR%\start.log -set FULL_LOG=%LOGDIR%\full.log -set "PYDIR=C:\Program Files\Python312" -set "PATH=%PYDIR%;%PYDIR%\Scripts;%PATH%" -set "LOCKDIR=%SHARED%\start.running" - -mkdir "%LOGDIR%" 2>nul -rmdir "%FULL_LOG%.lock" 2>nul - -:: ── Boot-scoped lock (only one start.bat instance per boot) ────── -:: A bare `mkdir %LOCKDIR%` used to deadlock the stack permanently. %LOCKDIR% -:: lives on the host-mounted %SHARED% volume, so it survives a VM reboot; the -:: MT5AutoReboot task fires `shutdown /r /t 0 /f` with no grace period and can -:: land anywhere inside this script's run (which legitimately spans from -:: seconds to over an hour). Killed mid-run, the lock outlived the process and -:: every later boot bailed on the orphan forever. -:: -:: acquire_lock.ps1 stamps the lock with the OS boot time, so a lock from a -:: previous boot is provably ownerless and gets cleared automatically. -:: Exit 0 = acquired, exit 1 = a live instance from THIS boot holds it. -:: -:: Deliberately does NOT consume %SHARED%\rebooting.flag — install.bat (called -:: below) owns that flag for its own lock, and eating it here would break it. -:: Tempfile rather than `for /f` because errorlevel after a for/f loop is the -:: loop body's exit code, not the invoked command's -- the lock verdict would -:: be silently lost. Same reason the API-token block below uses a tempfile. -set "LOCK_OUT=%TEMP%\mt5_lock_result.txt" -del "%LOCK_OUT%" 2>nul -powershell -NoProfile -ExecutionPolicy Bypass -File "%SCRIPTS%\acquire_lock.ps1" -LockDir "%LOCKDIR%" > "%LOCK_OUT%" 2>&1 -set "LOCK_EC=!errorlevel!" -if exist "%LOCK_OUT%" ( - for /f "usebackq delims=" %%A in ("%LOCK_OUT%") do ( - echo [%date% %time%] [start] lock: %%A >> "%FULL_LOG%" - echo [%date% %time%] lock: %%A >> "%START_LOG%" - ) -) -del "%LOCK_OUT%" 2>nul -rem Exit 10 means a live instance from THIS boot holds the lock -- bail. -if !LOCK_EC! equ 10 ( - echo [%date% %time%] start.bat already running this boot, exiting. - echo [%date% %time%] start.bat already running this boot, exiting. >> "%START_LOG%" - echo [%date% %time%] [start] start.bat already running this boot, exiting. >> "%FULL_LOG%" - exit /b 0 -) -rem Any other non-zero means acquire_lock.ps1 ITSELF failed (parse error, -rem missing cmdlet, unwritable mount). Do NOT treat that as "held" -- that is -rem precisely what deadlocked the stack when a syntax error in the helper made -rem PowerShell exit 1 and every boot concluded the lock was taken. Fall back to -rem the plain mkdir lock so boot still proceeds with single-instance safety. -if !LOCK_EC! neq 0 ( - echo [%date% %time%] WARN acquire_lock.ps1 failed ^(exit !LOCK_EC!^), falling back to mkdir lock >> "%START_LOG%" - echo [%date% %time%] [start] WARN acquire_lock.ps1 failed ^(exit !LOCK_EC!^), falling back to mkdir lock >> "%FULL_LOG%" - mkdir "%LOCKDIR%" 2>nul - if !errorlevel! neq 0 ( - echo [%date% %time%] fallback lock held, exiting. >> "%START_LOG%" - echo [%date% %time%] [start] fallback lock held, exiting. >> "%FULL_LOG%" - exit /b 0 - ) -) - -call :log "%START_LOG%" "====== Boot ======" -call :log "%INSTALL_LOG%" "====== Boot ======" - -:: ── Run install ────────────────────────────────────────────────── -call :log "%START_LOG%" "Running install.bat..." -call "%SCRIPTS%\install.bat" -if !errorlevel! equ 3 ( - call :log "%START_LOG%" "Reboot scheduled by install.bat, stopping." - call :release_lock - exit /b 0 -) -if !errorlevel! neq 0 ( - call :log "%START_LOG%" "ERROR: install.bat failed (exit code !errorlevel!)" - call :release_lock - exit /b 1 -) -call :log "%START_LOG%" "install.bat done." - -:: ── Pip install ────────────────────────────────────────────────── -:: Install base deps first (pyyaml required for config_helper.py below). -:: numpy<2 pin: MetaTrader5 5.0.5735 was built against numpy 1.x and breaks -:: silently with numpy 2.x — reads still work but order_send fails immediately -:: with (-2, 'Unnamed arguments not allowed'). Drop the pin once MetaQuotes -:: ships a numpy-2-compatible wheel. -:: -:: Each pip command writes to a per-call temp file so we can detect whether -:: anything ACTUALLY got installed/upgraded (presence of "Successfully -:: installed" in pip's output). If so, the python processes already running -:: from the previous boot are stale → reboot to pick up the new libs. -set "PIP_TMP=%TEMP%\mt5-pip-%RANDOM%-%RANDOM%.txt" -set "PIP_CHANGED=0" - -call :log "%START_LOG%" "Installing pip packages..." -call :log "%PIP_LOG%" "Installing pip packages..." -rem MCP v2 removed mcp.server.fastmcp; keep this synchronized with requirements-api.txt. -"%PYDIR%\python.exe" -m pip install pyyaml MetaTrader5 "numpy<2" flask waitress flask-compress psutil "mcp==1.28.0" a2wsgi > "%PIP_TMP%" 2>&1 -set "PIP_EC=!errorlevel!" -type "%PIP_TMP%" >> "%PIP_LOG%" -findstr /C:"Successfully installed" "%PIP_TMP%" >nul 2>&1 && set "PIP_CHANGED=1" -del "%PIP_TMP%" 2>nul -if !PIP_EC! neq 0 ( - call :log "%START_LOG%" "ERROR: pip install (base) failed (exit code !PIP_EC!), aborting." - call :log "%PIP_LOG%" "ERROR: pip install (base) failed" - call :release_lock - exit /b 1 -) -:: Extra packages from config.yaml requirements list. -:: NOTE: no `usebackq` — with usebackq, single-quoted strings are LITERAL, -:: not commands. Without usebackq, ('cmd') executes the command. This is -:: the same pattern install.bat uses for the `ports` lookup. -for /f "delims=" %%R in ('"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" requirements 2^>nul') do ( - "%PYDIR%\python.exe" -m pip install "%%R" > "%PIP_TMP%" 2>&1 - type "%PIP_TMP%" >> "%PIP_LOG%" - findstr /C:"Successfully installed" "%PIP_TMP%" >nul 2>&1 && set "PIP_CHANGED=1" - del "%PIP_TMP%" 2>nul -) -call :log "%START_LOG%" "pip done." -call :log "%PIP_LOG%" "pip done." - -:: Routed through reboot.bat so the flag write + lock release happen in one -:: place. Previously this inlined its own flag+shutdown+rmdir sequence, which -:: was correct but meant three separate reboot implementations to keep in sync. -if "!PIP_CHANGED!"=="1" ( - call :log "%START_LOG%" "pip changed packages -> rebooting so api_runners pick up new libs" - call "%SCRIPTS%\reboot.bat" pip-changed - exit /b 0 -) - -:: ── Start Windows event log tailer (background) ──────────────── -:: Streams Warning/Error/Critical from System + Application logs into -:: %LOGDIR%\windows-events.log so OOM kills, BSODs, terminal64 crashes, -:: etc. show up alongside the API logs. Single-instance via lock file -:: inside the script. -call :log "%START_LOG%" "Starting Windows event log tailer..." -start "Win Event Tailer" /B powershell -NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File "%SCRIPTS%\event-log-tailer.ps1" - -:: ── Kill lingering MT5 terminals ──────────────────────────────── -call :log "%START_LOG%" "Killing lingering MT5 terminals..." -tasklist /fi "imagename eq terminal64.exe" 2>nul | find /i "terminal64.exe" >nul && ( - taskkill /f /im terminal64.exe >nul 2>&1 - timeout /t 2 /nobreak >nul -) - -:: ── Verify config.yaml exists ─────────────────────────────────── -if not exist "%CONFIG%\config.yaml" ( - call :log "%START_LOG%" "ERROR: config.yaml not found! Copy config/config.yaml.example and re-run." - call :release_lock - exit /b 1 -) - -:: ── Parse config.yaml terminals once ──────────────────────────── -set "TERM_LIST=%TEMP%\mt5_terminals.txt" -"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" terminals > "%TERM_LIST%" 2>"%TEMP%\mt5_parse_err.txt" -if !errorlevel! neq 0 ( - call :log "%START_LOG%" "ERROR: Failed to parse config.yaml:" - type "%TEMP%\mt5_parse_err.txt" >> "%START_LOG%" - del "%TERM_LIST%" 2>nul - call :release_lock - exit /b 1 -) - -:: ── Periodic auto-reboot scheduled task ───────────────────────── -:: MT5 terminals share a desktop with DWM, and DWM/VirtIO-GPU crashes -:: under sustained load wedge the SDK pipe (terminal64.exe stops -:: responding to GDI/IPC). Cheapest mitigation: hard-reboot every N -:: minutes to flush GPU/desktop state before it rots. -:: Configured via config.yaml reboot_interval (minutes). 0 = disabled. -:: Default: 30. /f on schtasks is idempotent -- overwrites existing task. -:: -:: The task calls reboot.bat, NOT `shutdown` directly. Calling shutdown here -:: was the root cause of the permanent-deadlock outage: it killed start.bat -:: mid-run without writing rebooting.flag and without releasing -:: %SHARED%\start.running, and that orphaned lock (living on the host mount) -:: blocked every subsequent boot. reboot.bat always does both. -:: -:: No inner quotes in /tr -- schtasks quote-escaping is fragile, and %SCRIPTS% -:: has no spaces (C:\Users\Docker\Desktop\Shared\scripts). -set "REBOOT_INTERVAL=30" -"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" reboot_interval > "%SHARED%\mt5_ri.tmp" 2>nul -for /f "usebackq delims=" %%V in ("%SHARED%\mt5_ri.tmp") do set "REBOOT_INTERVAL=%%V" -del "%SHARED%\mt5_ri.tmp" 2>nul -if "!REBOOT_INTERVAL!"=="0" ( - schtasks /delete /tn "MT5AutoReboot" /f >nul 2>&1 - call :log "%START_LOG%" "Auto-reboot disabled (reboot_interval=0)." -) else ( - schtasks /create /tn "MT5AutoReboot" /tr "cmd.exe /c %SCRIPTS%\reboot.bat scheduled" /sc minute /mo !REBOOT_INTERVAL! /ru "SYSTEM" /rl HIGHEST /f >nul 2>&1 - if !errorlevel! equ 0 ( - call :log "%START_LOG%" "MT5AutoReboot task ensured (every !REBOOT_INTERVAL! min)." - ) else ( - call :log "%START_LOG%" "WARN: failed to create MT5AutoReboot task (errorlevel !errorlevel!)." - ) -) - -:: ── Launch MT5 terminals ───────────────────────────────────────── -call :log "%START_LOG%" "Launching MT5 terminals..." -set TERM_COUNT=0 -for /f "usebackq delims=" %%L in ("%TERM_LIST%") do ( - call :launch_terminal %%L - if !errorlevel! neq 0 ( - call :log "%START_LOG%" "ERROR: Failed to launch terminal, aborting." - del "%TERM_LIST%" 2>nul - call :release_lock - exit /b 1 - ) - set /a TERM_COUNT+=1 -) - -if !TERM_COUNT! equ 0 ( - call :log "%START_LOG%" "ERROR: No terminals configured in config.yaml" - del "%TERM_LIST%" 2>nul - call :release_lock - exit /b 1 -) - -call :log "%START_LOG%" "Launched !TERM_COUNT! terminal(s), waiting 30s to initialize..." -timeout /t 30 /nobreak >nul - -:: ── Load API token from config.yaml (optional) ────────────────── -:: Tempfile path is more robust than `for /f`'s subshell+quoting dance — -:: any python crash, pyyaml fallback install, or stdout buffering quirk -:: showed up as "API_TOKEN empty" through the for/f path. -set "API_TOKEN=" -set "TOKEN_TMP=%TEMP%\mt5_api_token.txt" -del "%TOKEN_TMP%" 2>nul -"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" api_token > "%TOKEN_TMP%" 2>nul -if exist "%TOKEN_TMP%" set /p API_TOKEN=<"%TOKEN_TMP%" -del "%TOKEN_TMP%" 2>nul -if defined API_TOKEN ( - call :log "%START_LOG%" "API token loaded." -) else ( - call :log "%START_LOG%" "WARNING: api_token empty in config.yaml, API running without auth." -) - -:: ── Launch API processes (all background) ──────────────────────── -call :log "%START_LOG%" "Launching API processes..." -for /f "usebackq delims=" %%L in ("%TERM_LIST%") do ( - call :launch_api_bg %%L -) -del "%TERM_LIST%" 2>nul -call :release_lock -call :log "%START_LOG%" "All !TERM_COUNT! API(s) running in background." - -:: ── Foreground: status + health monitor ────────────────────────── -:status_loop -cls -echo. -echo ===================================================== -echo MT5 HTTP API RUNNING -- %DATE% %TIME% -echo ===================================================== -echo. -"%PYDIR%\python.exe" "%SCRIPTS%\check_health.py" -echo. -timeout /t 60 /nobreak >nul -goto status_loop - -:: ══════════════════════════════════════════════════════════════════ -:launch_terminal -:: %1=broker %2=account %3=instance %4=port %5=utc_offset %6=mode (live|backtest) -set "LT_BROKER=%~1" -set "LT_ACCOUNT=%~2" -set "LT_INSTANCE=%~3" -set "LT_PORT=%~4" -set "LT_MODE=%~6" -if "!LT_INSTANCE!"=="" set "LT_INSTANCE=default" -if "!LT_MODE!"=="" set "LT_MODE=live" -set "LT_BASEDIR=%BROKERS%\!LT_BROKER!\base" -set "LT_DIR=%BROKERS%\!LT_BROKER!\!LT_ACCOUNT!\!LT_INSTANCE!" - -if not exist "!LT_BASEDIR!\terminal64.exe" ( - call :log "%START_LOG%" "ERROR: No base install for !LT_BROKER! at !LT_BASEDIR!" - exit /b 1 -) - -if not exist "!LT_DIR!\terminal64.exe" ( - call :log "%START_LOG%" "Copying !LT_BROKER!\base to !LT_BROKER!\!LT_ACCOUNT!\!LT_INSTANCE!..." - xcopy "!LT_BASEDIR!\*" "!LT_DIR!\" /E /I /H /Y /Q >nul 2>&1 - if !errorlevel! neq 0 ( - call :log "%START_LOG%" "ERROR: xcopy failed for !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE!" - exit /b 1 - ) -) - -del "!LT_DIR!\Config\settings.ini" 2>nul -del "!LT_DIR!\Config\common.ini" 2>nul - -call :write_ini "!LT_DIR!" "!LT_BROKER!" "!LT_ACCOUNT!" - -rem Save journal log size before launch so we only check NEW content -for /f "delims=" %%D in ('python -c "from datetime import date;print(date.today().strftime('%%Y%%m%%d'))"') do set "LT_LOGDATE=%%D" -set "LT_LOGFILE=!LT_DIR!\logs\!LT_LOGDATE!.log" -set LT_LOGSIZE=0 -if exist "!LT_LOGFILE!" ( - for %%A in ("!LT_LOGFILE!") do set LT_LOGSIZE=%%~zA -) - -if /i "!LT_MODE!"=="backtest" ( - call :log "%START_LOG%" " !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! mode=backtest -- portable dir prepared, terminal NOT launched (tester will spawn it on demand)." - exit /b 0 -) - -call :log "%START_LOG%" "Starting terminal: !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! (port !LT_PORT!) [log offset !LT_LOGSIZE!]" -powershell -Command "Start-Process '!LT_DIR!\terminal64.exe' -ArgumentList '/portable','/config:\"!LT_DIR!\mt5start.ini\"' -Verb RunAs -WindowStyle Normal" - -rem Wait for 'started for' in journal log (for /L avoids goto inside call) -set LT_STARTED=0 -for /L %%N in (1,1,120) do ( - if !LT_STARTED! equ 0 ( - python -c "import sys;f=open(sys.argv[1],'rb');f.seek(int(sys.argv[2]));d=f.read().decode('utf-16-le',errors='ignore');f.close();sys.exit(0 if 'started for' in d else 1)" "!LT_LOGFILE!" !LT_LOGSIZE! 2>nul - if !errorlevel! equ 0 ( - set LT_STARTED=1 - ) else ( - call :log "%START_LOG%" " Waiting for !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! to start (%%N)..." - timeout /t 5 /nobreak >nul - ) - ) -) -if !LT_STARTED! equ 0 ( - call :log "%START_LOG%" "ERROR: !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! failed to start after 10 minutes" - exit /b 1 -) -call :log "%START_LOG%" " !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! started." -exit /b 0 - -:: ══════════════════════════════════════════════════════════════════ -:launch_api_bg -set "LA_BROKER=%~1" -set "LA_ACCOUNT=%~2" -set "LA_INSTANCE=%~3" -set "LA_PORT=%~4" -set "LA_OFFSET=%~5" -set "LA_MODE=%~6" -if "!LA_INSTANCE!"=="" set "LA_INSTANCE=default" -if "!LA_OFFSET!"=="" set "LA_OFFSET=0" -if "!LA_MODE!"=="" set "LA_MODE=live" - -call :log "%START_LOG%" "Starting API (bg): !LA_BROKER!/!LA_ACCOUNT!/!LA_INSTANCE! on port !LA_PORT! (utc_offset=!LA_OFFSET! mode=!LA_MODE!)" -if "!LA_INSTANCE!"=="default" ( - start "MT5 API !LA_BROKER!/!LA_ACCOUNT!" cmd /c ""%SCRIPTS%\api_runner.bat" !LA_BROKER! !LA_ACCOUNT! !LA_INSTANCE! !LA_PORT! !API_TOKEN! !LA_OFFSET! !LA_MODE!" -) else ( - start "MT5 API !LA_BROKER!/!LA_ACCOUNT!/!LA_INSTANCE!" cmd /c ""%SCRIPTS%\api_runner.bat" !LA_BROKER! !LA_ACCOUNT! !LA_INSTANCE! !LA_PORT! !API_TOKEN! !LA_OFFSET! !LA_MODE!" -) -exit /b 0 - -:: ══════════════════════════════════════════════════════════════════ -:write_ini -set "WI_DIR=%~1" -set "WI_BROKER=%~2" -set "WI_ACCOUNT=%~3" -set "WI_CFG=!WI_DIR!\mt5start.ini" -"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" write_ini "!WI_BROKER!" "!WI_ACCOUNT!" "!WI_CFG!" >> "%START_LOG%" 2>&1 -if errorlevel 1 ( - call :log "%START_LOG%" "WARNING: Could not write ini for !WI_BROKER!/!WI_ACCOUNT!, using defaults" - echo [Common]> "!WI_CFG!" - echo KeepPrivate=0>> "!WI_CFG!" - echo AutoTrading=1>> "!WI_CFG!" - echo NewsEnable=0>> "!WI_CFG!" - echo [Experts]>> "!WI_CFG!" - echo AllowLiveTrading=1>> "!WI_CFG!" - echo AllowDllImport=1>> "!WI_CFG!" - echo Enabled=1>> "!WI_CFG!" - echo [Email]>> "!WI_CFG!" - echo Enable=0>> "!WI_CFG!" -) -exit /b 0 - -:: ══════════════════════════════════════════════════════════════════ -:release_lock -:: /s /q is REQUIRED: the lock dir contains acquire_lock.ps1's boot.id stamp, -:: so a bare `rmdir` fails on a non-empty directory and would leave the lock -:: behind -- reintroducing the deadlock this whole mechanism removes. -rmdir /s /q "%LOCKDIR%" 2>nul -exit /b 0 - -:: ══════════════════════════════════════════════════════════════════ -:log -echo [%date% %time%] %~2 -echo [%date% %time%] %~2 >> "%~1" -echo [%date% %time%] [start] %~2 >> "%FULL_LOG%" -exit /b 0 +@echo off +setlocal enabledelayedexpansion +set SHARED=C:\Users\Docker\Desktop\Shared +set SCRIPTS=%SHARED%\scripts +set CONFIG=%SHARED%\config +set BROKERS=%SHARED%\terminals +set LOGDIR=%SHARED%\logs +set INSTALL_LOG=%LOGDIR%\install.log +set PIP_LOG=%LOGDIR%\pip.log +set START_LOG=%LOGDIR%\start.log +set FULL_LOG=%LOGDIR%\full.log +set "PYDIR=C:\Program Files\Python312" +set "PATH=%PYDIR%;%PYDIR%\Scripts;%PATH%" +set "LOCKDIR=%SHARED%\start.running" + +mkdir "%LOGDIR%" 2>nul +rmdir "%FULL_LOG%.lock" 2>nul + +:: ── Boot-scoped lock (only one start.bat instance per boot) ────── +:: A bare `mkdir %LOCKDIR%` used to deadlock the stack permanently. %LOCKDIR% +:: lives on the host-mounted %SHARED% volume, so it survives a VM reboot; the +:: MT5AutoReboot task fires `shutdown /r /t 0 /f` with no grace period and can +:: land anywhere inside this script's run (which legitimately spans from +:: seconds to over an hour). Killed mid-run, the lock outlived the process and +:: every later boot bailed on the orphan forever. +:: +:: acquire_lock.ps1 stamps the lock with the OS boot time, so a lock from a +:: previous boot is provably ownerless and gets cleared automatically. +:: Exit 0 = acquired, exit 1 = a live instance from THIS boot holds it. +:: +:: Deliberately does NOT consume %SHARED%\rebooting.flag — install.bat (called +:: below) owns that flag for its own lock, and eating it here would break it. +:: Tempfile rather than `for /f` because errorlevel after a for/f loop is the +:: loop body's exit code, not the invoked command's -- the lock verdict would +:: be silently lost. Same reason the API-token block below uses a tempfile. +set "LOCK_OUT=%TEMP%\mt5_lock_result.txt" +del "%LOCK_OUT%" 2>nul +powershell -NoProfile -ExecutionPolicy Bypass -File "%SCRIPTS%\acquire_lock.ps1" -LockDir "%LOCKDIR%" > "%LOCK_OUT%" 2>&1 +set "LOCK_EC=!errorlevel!" +if exist "%LOCK_OUT%" ( + for /f "usebackq delims=" %%A in ("%LOCK_OUT%") do ( + echo [%date% %time%] [start] lock: %%A >> "%FULL_LOG%" + echo [%date% %time%] lock: %%A >> "%START_LOG%" + ) +) +del "%LOCK_OUT%" 2>nul +rem Exit 10 means a live instance from THIS boot holds the lock -- bail. +if !LOCK_EC! equ 10 ( + echo [%date% %time%] start.bat already running this boot, exiting. + echo [%date% %time%] start.bat already running this boot, exiting. >> "%START_LOG%" + echo [%date% %time%] [start] start.bat already running this boot, exiting. >> "%FULL_LOG%" + exit /b 0 +) +rem Any other non-zero means acquire_lock.ps1 ITSELF failed (parse error, +rem missing cmdlet, unwritable mount). Do NOT treat that as "held" -- that is +rem precisely what deadlocked the stack when a syntax error in the helper made +rem PowerShell exit 1 and every boot concluded the lock was taken. Fall back to +rem the plain mkdir lock so boot still proceeds with single-instance safety. +if !LOCK_EC! neq 0 ( + echo [%date% %time%] WARN acquire_lock.ps1 failed ^(exit !LOCK_EC!^), falling back to mkdir lock >> "%START_LOG%" + echo [%date% %time%] [start] WARN acquire_lock.ps1 failed ^(exit !LOCK_EC!^), falling back to mkdir lock >> "%FULL_LOG%" + mkdir "%LOCKDIR%" 2>nul + if !errorlevel! neq 0 ( + echo [%date% %time%] fallback lock held, exiting. >> "%START_LOG%" + echo [%date% %time%] [start] fallback lock held, exiting. >> "%FULL_LOG%" + exit /b 0 + ) +) + +call :log "%START_LOG%" "====== Boot ======" +call :log "%INSTALL_LOG%" "====== Boot ======" + +:: ── Run install ────────────────────────────────────────────────── +call :log "%START_LOG%" "Running install.bat..." +call "%SCRIPTS%\install.bat" +if !errorlevel! equ 3 ( + call :log "%START_LOG%" "Reboot scheduled by install.bat, stopping." + call :release_lock + exit /b 0 +) +if !errorlevel! neq 0 ( + call :log "%START_LOG%" "ERROR: install.bat failed (exit code !errorlevel!)" + call :release_lock + exit /b 1 +) +call :log "%START_LOG%" "install.bat done." + +:: ── Pip install ────────────────────────────────────────────────── +:: Install base deps first (pyyaml required for config_helper.py below). +:: numpy<2 pin: MetaTrader5 5.0.5735 was built against numpy 1.x and breaks +:: silently with numpy 2.x — reads still work but order_send fails immediately +:: with (-2, 'Unnamed arguments not allowed'). Drop the pin once MetaQuotes +:: ships a numpy-2-compatible wheel. +:: +:: Each pip command writes to a per-call temp file so we can detect whether +:: anything ACTUALLY got installed/upgraded (presence of "Successfully +:: installed" in pip's output). If so, the python processes already running +:: from the previous boot are stale → reboot to pick up the new libs. +set "PIP_TMP=%TEMP%\mt5-pip-%RANDOM%-%RANDOM%.txt" +set "PIP_CHANGED=0" + +call :log "%START_LOG%" "Installing pip packages..." +call :log "%PIP_LOG%" "Installing pip packages..." +rem MCP v2 removed mcp.server.fastmcp; keep this synchronized with requirements-api.txt. +"%PYDIR%\python.exe" -m pip install pyyaml MetaTrader5 "numpy<2" flask waitress flask-compress psutil "mcp==1.28.0" a2wsgi > "%PIP_TMP%" 2>&1 +set "PIP_EC=!errorlevel!" +type "%PIP_TMP%" >> "%PIP_LOG%" +findstr /C:"Successfully installed" "%PIP_TMP%" >nul 2>&1 && set "PIP_CHANGED=1" +del "%PIP_TMP%" 2>nul +if !PIP_EC! neq 0 ( + call :log "%START_LOG%" "ERROR: pip install (base) failed (exit code !PIP_EC!), aborting." + call :log "%PIP_LOG%" "ERROR: pip install (base) failed" + call :release_lock + exit /b 1 +) +:: Extra packages from config.yaml requirements list. +:: NOTE: no `usebackq` — with usebackq, single-quoted strings are LITERAL, +:: not commands. Without usebackq, ('cmd') executes the command. This is +:: the same pattern install.bat uses for the `ports` lookup. +for /f "delims=" %%R in ('"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" requirements 2^>nul') do ( + "%PYDIR%\python.exe" -m pip install "%%R" > "%PIP_TMP%" 2>&1 + type "%PIP_TMP%" >> "%PIP_LOG%" + findstr /C:"Successfully installed" "%PIP_TMP%" >nul 2>&1 && set "PIP_CHANGED=1" + del "%PIP_TMP%" 2>nul +) +call :log "%START_LOG%" "pip done." +call :log "%PIP_LOG%" "pip done." + +:: Routed through reboot.bat so the flag write + lock release happen in one +:: place. Previously this inlined its own flag+shutdown+rmdir sequence, which +:: was correct but meant three separate reboot implementations to keep in sync. +if "!PIP_CHANGED!"=="1" ( + call :log "%START_LOG%" "pip changed packages -> rebooting so api_runners pick up new libs" + call "%SCRIPTS%\reboot.bat" pip-changed + exit /b 0 +) + +:: ── Start Windows event log tailer (background) ──────────────── +:: Streams Warning/Error/Critical from System + Application logs into +:: %LOGDIR%\windows-events.log so OOM kills, BSODs, terminal64 crashes, +:: etc. show up alongside the API logs. Single-instance via lock file +:: inside the script. +call :log "%START_LOG%" "Starting Windows event log tailer..." +start "Win Event Tailer" /B powershell -NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File "%SCRIPTS%\event-log-tailer.ps1" + +:: ── Kill lingering MT5 terminals ──────────────────────────────── +call :log "%START_LOG%" "Killing lingering MT5 terminals..." +tasklist /fi "imagename eq terminal64.exe" 2>nul | find /i "terminal64.exe" >nul && ( + taskkill /f /im terminal64.exe >nul 2>&1 + timeout /t 2 /nobreak >nul +) + +:: ── Verify config.yaml exists ─────────────────────────────────── +if not exist "%CONFIG%\config.yaml" ( + call :log "%START_LOG%" "ERROR: config.yaml not found! Copy config/config.yaml.example and re-run." + call :release_lock + exit /b 1 +) + +:: ── Parse config.yaml terminals once ──────────────────────────── +set "TERM_LIST=%TEMP%\mt5_terminals.txt" +"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" terminals > "%TERM_LIST%" 2>"%TEMP%\mt5_parse_err.txt" +if !errorlevel! neq 0 ( + call :log "%START_LOG%" "ERROR: Failed to parse config.yaml:" + type "%TEMP%\mt5_parse_err.txt" >> "%START_LOG%" + del "%TERM_LIST%" 2>nul + call :release_lock + exit /b 1 +) + +:: ── Periodic auto-reboot scheduled task ───────────────────────── +:: MT5 terminals share a desktop with DWM, and DWM/VirtIO-GPU crashes +:: under sustained load wedge the SDK pipe (terminal64.exe stops +:: responding to GDI/IPC). Cheapest mitigation: hard-reboot every N +:: minutes to flush GPU/desktop state before it rots. +:: Configured via config.yaml reboot_interval (minutes). 0 = disabled. +:: Default: 30. /f on schtasks is idempotent -- overwrites existing task. +:: +:: The task calls reboot.bat, NOT `shutdown` directly. Calling shutdown here +:: was the root cause of the permanent-deadlock outage: it killed start.bat +:: mid-run without writing rebooting.flag and without releasing +:: %SHARED%\start.running, and that orphaned lock (living on the host mount) +:: blocked every subsequent boot. reboot.bat always does both. +:: +:: No inner quotes in /tr -- schtasks quote-escaping is fragile, and %SCRIPTS% +:: has no spaces (C:\Users\Docker\Desktop\Shared\scripts). +set "REBOOT_INTERVAL=30" +"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" reboot_interval > "%SHARED%\mt5_ri.tmp" 2>nul +for /f "usebackq delims=" %%V in ("%SHARED%\mt5_ri.tmp") do set "REBOOT_INTERVAL=%%V" +del "%SHARED%\mt5_ri.tmp" 2>nul +if "!REBOOT_INTERVAL!"=="0" ( + schtasks /delete /tn "MT5AutoReboot" /f >nul 2>&1 + call :log "%START_LOG%" "Auto-reboot disabled (reboot_interval=0)." +) else ( + schtasks /create /tn "MT5AutoReboot" /tr "cmd.exe /c %SCRIPTS%\reboot.bat scheduled" /sc minute /mo !REBOOT_INTERVAL! /ru "SYSTEM" /rl HIGHEST /f >nul 2>&1 + if !errorlevel! equ 0 ( + call :log "%START_LOG%" "MT5AutoReboot task ensured (every !REBOOT_INTERVAL! min)." + ) else ( + call :log "%START_LOG%" "WARN: failed to create MT5AutoReboot task (errorlevel !errorlevel!)." + ) +) + +:: ── Compile chartctl loader EA (zero-touch bootstrap) ──────────── +:: Compiles MT5ChartLoader in every broker base and propagates the .ex5 +:: into existing terminal instances, so the [StartUp] Expert= line in +:: mt5start.ini can auto-attach it at launch. Skipped when chartctl is +:: disabled globally in config.yaml. Non-fatal: a compile failure only +:: means chart deployments stay unavailable until fixed. +:: Tempfile read, NOT for /f ('command') — with both python.exe and the +:: script path quoted, cmd's quote-stripping mangles the subshell command +:: and it silently outputs nothing (same failure the api_token block +:: documents; also why install.bat's ports lookup falls back to 6542). +set "CHARTCTL_ON=" +"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" chartctl_enabled > "%SHARED%\mt5_cc.tmp" 2>nul +for /f "usebackq delims=" %%C in ("%SHARED%\mt5_cc.tmp") do set "CHARTCTL_ON=%%C" +del "%SHARED%\mt5_cc.tmp" 2>nul +if "!CHARTCTL_ON!"=="1" ( + call :log "%START_LOG%" "Compiling chartctl loader EA (MT5ChartLoader)..." + call "%SCRIPTS%\compile-chartctl-loader.bat" >> "%START_LOG%" 2>&1 + if !errorlevel! neq 0 ( + call :log "%START_LOG%" "WARN: chartctl loader compile failed -- chart deployments unavailable. See logs\compile-chartctl-loader.log" + ) else ( + call :log "%START_LOG%" "chartctl loader compiled and propagated." + ) +) else ( + call :log "%START_LOG%" "chartctl disabled in config.yaml -- skipping loader compile." +) + +:: ── Launch MT5 terminals ───────────────────────────────────────── +call :log "%START_LOG%" "Launching MT5 terminals..." +set TERM_COUNT=0 +for /f "usebackq delims=" %%L in ("%TERM_LIST%") do ( + call :launch_terminal %%L + if !errorlevel! neq 0 ( + call :log "%START_LOG%" "ERROR: Failed to launch terminal, aborting." + del "%TERM_LIST%" 2>nul + call :release_lock + exit /b 1 + ) + set /a TERM_COUNT+=1 +) + +if !TERM_COUNT! equ 0 ( + call :log "%START_LOG%" "ERROR: No terminals configured in config.yaml" + del "%TERM_LIST%" 2>nul + call :release_lock + exit /b 1 +) + +call :log "%START_LOG%" "Launched !TERM_COUNT! terminal(s), waiting 30s to initialize..." +timeout /t 30 /nobreak >nul + +:: ── Load API token from config.yaml (optional) ────────────────── +:: Tempfile path is more robust than `for /f`'s subshell+quoting dance — +:: any python crash, pyyaml fallback install, or stdout buffering quirk +:: showed up as "API_TOKEN empty" through the for/f path. +set "API_TOKEN=" +set "TOKEN_TMP=%TEMP%\mt5_api_token.txt" +del "%TOKEN_TMP%" 2>nul +"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" api_token > "%TOKEN_TMP%" 2>nul +if exist "%TOKEN_TMP%" set /p API_TOKEN=<"%TOKEN_TMP%" +del "%TOKEN_TMP%" 2>nul +if defined API_TOKEN ( + call :log "%START_LOG%" "API token loaded." +) else ( + call :log "%START_LOG%" "WARNING: api_token empty in config.yaml, API running without auth." +) + +:: ── Launch API processes (all background) ──────────────────────── +call :log "%START_LOG%" "Launching API processes..." +for /f "usebackq delims=" %%L in ("%TERM_LIST%") do ( + call :launch_api_bg %%L +) +del "%TERM_LIST%" 2>nul +call :release_lock +call :log "%START_LOG%" "All !TERM_COUNT! API(s) running in background." + +:: ── Foreground: status + health monitor ────────────────────────── +:status_loop +cls +echo. +echo ===================================================== +echo MT5 HTTP API RUNNING -- %DATE% %TIME% +echo ===================================================== +echo. +"%PYDIR%\python.exe" "%SCRIPTS%\check_health.py" +echo. +timeout /t 60 /nobreak >nul +goto status_loop + +:: ══════════════════════════════════════════════════════════════════ +:launch_terminal +:: %1=broker %2=account %3=instance %4=port %5=utc_offset %6=mode (live|backtest) +set "LT_BROKER=%~1" +set "LT_ACCOUNT=%~2" +set "LT_INSTANCE=%~3" +set "LT_PORT=%~4" +set "LT_MODE=%~6" +if "!LT_INSTANCE!"=="" set "LT_INSTANCE=default" +if "!LT_MODE!"=="" set "LT_MODE=live" +set "LT_BASEDIR=%BROKERS%\!LT_BROKER!\base" +set "LT_DIR=%BROKERS%\!LT_BROKER!\!LT_ACCOUNT!\!LT_INSTANCE!" + +if not exist "!LT_BASEDIR!\terminal64.exe" ( + call :log "%START_LOG%" "ERROR: No base install for !LT_BROKER! at !LT_BASEDIR!" + exit /b 1 +) + +if not exist "!LT_DIR!\terminal64.exe" ( + call :log "%START_LOG%" "Copying !LT_BROKER!\base to !LT_BROKER!\!LT_ACCOUNT!\!LT_INSTANCE!..." + xcopy "!LT_BASEDIR!\*" "!LT_DIR!\" /E /I /H /Y /Q >nul 2>&1 + if !errorlevel! neq 0 ( + call :log "%START_LOG%" "ERROR: xcopy failed for !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE!" + exit /b 1 + ) +) + +del "!LT_DIR!\Config\settings.ini" 2>nul +del "!LT_DIR!\Config\common.ini" 2>nul + +call :write_ini "!LT_DIR!" "!LT_BROKER!" "!LT_ACCOUNT!" "!LT_INSTANCE!" "!LT_MODE!" + +rem Save journal log size before launch so we only check NEW content +for /f "delims=" %%D in ('python -c "from datetime import date;print(date.today().strftime('%%Y%%m%%d'))"') do set "LT_LOGDATE=%%D" +set "LT_LOGFILE=!LT_DIR!\logs\!LT_LOGDATE!.log" +set LT_LOGSIZE=0 +if exist "!LT_LOGFILE!" ( + for %%A in ("!LT_LOGFILE!") do set LT_LOGSIZE=%%~zA +) + +if /i "!LT_MODE!"=="backtest" ( + call :log "%START_LOG%" " !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! mode=backtest -- portable dir prepared, terminal NOT launched (tester will spawn it on demand)." + exit /b 0 +) + +call :log "%START_LOG%" "Starting terminal: !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! (port !LT_PORT!) [log offset !LT_LOGSIZE!]" +powershell -Command "Start-Process '!LT_DIR!\terminal64.exe' -ArgumentList '/portable','/config:\"!LT_DIR!\mt5start.ini\"' -Verb RunAs -WindowStyle Normal" + +rem Wait for 'started for' in journal log (for /L avoids goto inside call) +set LT_STARTED=0 +for /L %%N in (1,1,120) do ( + if !LT_STARTED! equ 0 ( + python -c "import sys;f=open(sys.argv[1],'rb');f.seek(int(sys.argv[2]));d=f.read().decode('utf-16-le',errors='ignore');f.close();sys.exit(0 if 'started for' in d else 1)" "!LT_LOGFILE!" !LT_LOGSIZE! 2>nul + if !errorlevel! equ 0 ( + set LT_STARTED=1 + ) else ( + call :log "%START_LOG%" " Waiting for !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! to start (%%N)..." + timeout /t 5 /nobreak >nul + ) + ) +) +if !LT_STARTED! equ 0 ( + call :log "%START_LOG%" "ERROR: !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! failed to start after 10 minutes" + exit /b 1 +) +call :log "%START_LOG%" " !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! started." +exit /b 0 + +:: ══════════════════════════════════════════════════════════════════ +:launch_api_bg +set "LA_BROKER=%~1" +set "LA_ACCOUNT=%~2" +set "LA_INSTANCE=%~3" +set "LA_PORT=%~4" +set "LA_OFFSET=%~5" +set "LA_MODE=%~6" +if "!LA_INSTANCE!"=="" set "LA_INSTANCE=default" +if "!LA_OFFSET!"=="" set "LA_OFFSET=0" +if "!LA_MODE!"=="" set "LA_MODE=live" + +call :log "%START_LOG%" "Starting API (bg): !LA_BROKER!/!LA_ACCOUNT!/!LA_INSTANCE! on port !LA_PORT! (utc_offset=!LA_OFFSET! mode=!LA_MODE!)" +if "!LA_INSTANCE!"=="default" ( + start "MT5 API !LA_BROKER!/!LA_ACCOUNT!" cmd /c ""%SCRIPTS%\api_runner.bat" !LA_BROKER! !LA_ACCOUNT! !LA_INSTANCE! !LA_PORT! !API_TOKEN! !LA_OFFSET! !LA_MODE!" +) else ( + start "MT5 API !LA_BROKER!/!LA_ACCOUNT!/!LA_INSTANCE!" cmd /c ""%SCRIPTS%\api_runner.bat" !LA_BROKER! !LA_ACCOUNT! !LA_INSTANCE! !LA_PORT! !API_TOKEN! !LA_OFFSET! !LA_MODE!" +) +exit /b 0 + +:: ══════════════════════════════════════════════════════════════════ +:write_ini +set "WI_DIR=%~1" +set "WI_BROKER=%~2" +set "WI_ACCOUNT=%~3" +set "WI_INSTANCE=%~4" +set "WI_MODE=%~5" +if "!WI_INSTANCE!"=="" set "WI_INSTANCE=default" +if "!WI_MODE!"=="" set "WI_MODE=live" +set "WI_CFG=!WI_DIR!\mt5start.ini" +"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" write_ini "!WI_BROKER!" "!WI_ACCOUNT!" "!WI_CFG!" "!WI_INSTANCE!" "!WI_MODE!" >> "%START_LOG%" 2>&1 +if errorlevel 1 ( + call :log "%START_LOG%" "WARNING: Could not write ini for !WI_BROKER!/!WI_ACCOUNT!, using defaults" + echo [Common]> "!WI_CFG!" + echo KeepPrivate=0>> "!WI_CFG!" + echo AutoTrading=1>> "!WI_CFG!" + echo NewsEnable=0>> "!WI_CFG!" + echo [Experts]>> "!WI_CFG!" + echo AllowLiveTrading=1>> "!WI_CFG!" + echo AllowDllImport=1>> "!WI_CFG!" + echo Enabled=1>> "!WI_CFG!" + echo [Email]>> "!WI_CFG!" + echo Enable=0>> "!WI_CFG!" +) +exit /b 0 + +:: ══════════════════════════════════════════════════════════════════ +:release_lock +:: /s /q is REQUIRED: the lock dir contains acquire_lock.ps1's boot.id stamp, +:: so a bare `rmdir` fails on a non-empty directory and would leave the lock +:: behind -- reintroducing the deadlock this whole mechanism removes. +rmdir /s /q "%LOCKDIR%" 2>nul +exit /b 0 + +:: ══════════════════════════════════════════════════════════════════ +:log +echo [%date% %time%] %~2 +echo [%date% %time%] %~2 >> "%~1" +echo [%date% %time%] [start] %~2 >> "%FULL_LOG%" +exit /b 0 diff --git a/scripts/webrequest_allowlist_codec.py b/scripts/webrequest_allowlist_codec.py new file mode 100644 index 0000000..4a59c7d --- /dev/null +++ b/scripts/webrequest_allowlist_codec.py @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 +"""Codec for the MT5 ``[Experts] WebRequestUrl=`` allowlist blob (Config/common.ini). + +The WebRequest allowed-URL list (Tools -> Options -> Expert Advisors) is stored in +each terminal's ``Config/common.ini`` as ``WebRequestUrl=``. The hex is a +length-preserving encrypted blob produced by terminal64.exe. It is +NOT machine-bound: the key is fixed in the binary, so a blob generated here is +accepted verbatim by any terminal of the same build. That lets us provision the +allowlist programmatically (no RDP / no Options dialog), which is what Chart +Deployments needs so deployed EAs can call WebRequest without manual setup. + +FORMAT (fully reverse-engineered from terminal64.exe fn @0x7ff7824d4010): + ini hex string --%04X per uint16, stored LE--> ciphertext bytes (== swap16 of naive hex) + plaintext = u16le(1) + u16le(checksum) + utf16le(";".join(urls)) + where checksum = (sum of every UTF-16 code unit in the joined string) & 0xffff + cipher (decode, config-load direction) is a byte-wise CFB stream: + p[i] = ((c[i-1] + KEY[i % 16]) & 0xff) ^ c[i] with c[-1] = 0 + encode is the exact inverse (feedback taken from the ciphertext byte). + +The leading u16 is a constant 1 in every observed blob. The checksum lives at the +FRONT of the plaintext, so editing any URL character changes it and re-ciphers the +whole tail -- this is why the blob looked like a strong full-avalanche cipher in +black-box testing when it is really a single CFB pass plus a front checksum. + +Verified: round-trips byte-identically against 18 independent real broker-terminal +blobs (BlackBull, IC Markets, FP Markets, Darwinex, Ducascopy, AquaFunded, ...). +""" +from __future__ import annotations + +# 16-byte key, recovered by cryptanalysis of the CFB recurrence and confirmed +# against 50+ plaintext bytes and 18 full-blob round-trips. (The binary derives +# it at runtime via an obfuscated routine from a .rdata seed; the derived bytes +# are what matter and are reproduced here directly.) +KEY = bytes([0xe2, 0x30, 0x54, 0xb4, 0xde, 0xe5, 0xcc, 0x04, + 0x9c, 0x70, 0x8f, 0x3c, 0x6b, 0x87, 0x78, 0xf0]) + + +def _swap16(data: bytes) -> bytes: + b = bytearray(data) + for i in range(0, len(b) - 1, 2): + b[i], b[i + 1] = b[i + 1], b[i] + return bytes(b) + + +def _decode_cfb(ct: bytes) -> bytes: + out = bytearray(len(ct)) + prev = 0 + for i, c in enumerate(ct): + out[i] = ((prev + KEY[i % 16]) & 0xff) ^ c + prev = c + return bytes(out) + + +def _encode_cfb(pt: bytes) -> bytes: + out = bytearray(len(pt)) + prev = 0 + for i, p in enumerate(pt): + c = ((prev + KEY[i % 16]) & 0xff) ^ p + out[i] = c + prev = c + return bytes(out) + + +def _checksum(s: str) -> int: + # sum over UTF-16 code units (BMP chars == ord); & 0xffff + return sum(b[0] | (b[1] << 8) + for b in (s.encode("utf-16-le")[i:i + 2] + for i in range(0, len(s) * 2, 2))) & 0xffff + + +def decode_blob(hex_blob: str) -> list[str]: + """Ciphertext hex (value of ``WebRequestUrl=``) -> list of URL strings.""" + pt = _decode_cfb(_swap16(bytes.fromhex(hex_blob.strip()))) + body = pt[4:].decode("utf-16-le") + return body.split(";") if body else [] + + +def encode_urls(urls: list[str]) -> str: + """List of URL strings -> uppercase ciphertext hex for ``WebRequestUrl=``.""" + joined = ";".join(urls) + pt = (1).to_bytes(2, "little") + _checksum(joined).to_bytes(2, "little") \ + + joined.encode("utf-16-le") + return _swap16(_encode_cfb(pt)).hex().upper() + + +# --- embedded verified test vectors (real captured broker blobs) --- +_VECTORS = [ + ("13E33B7856715A56F2822DF172EBC0D0BD8DF23E89A42B2716A602C68D06506070409EEA021DA6A29" + "525874B0D86E7F7D8A8FC4898B30E0A3DCDFBBF9D166474552580CC55704642FD8D1DE13AB3CADA08D8" + "DC28AFCA0C080292FABED14A1828BA8A1B67BCD700FC69F9D094E55E2A3AAE7E17637D986B67D868511" + "562DB", + ["https://tracker.algotradingspace.com", "https://api.telegram.org"]), + ("13E33B7E56715A56F2822DF172EBC0D0BD8DF23E96B1070332C2DCA0AD263444A7774E9A324DA39FCE5" + "E783C0982D0E0CC9CF7439FBA", + ["https://aaaaaaaaaaaaaa.co"]), +] + + +def _selftest() -> None: + for blob, urls in _VECTORS: + blob = blob.replace("\n", "") + assert decode_blob(blob) == urls, decode_blob(blob) + assert encode_urls(urls) == blob, encode_urls(urls) + print("webrequest_allowlist_codec: self-test OK (%d vectors)" % len(_VECTORS)) + + +if __name__ == "__main__": + import sys + if len(sys.argv) == 1 or sys.argv[1] == "--selftest": + _selftest() + elif sys.argv[1] == "decode": + for u in decode_blob(sys.argv[2]): + print(u) + elif sys.argv[1] == "encode": + print(encode_urls(sys.argv[2:])) + else: + print("usage: webrequest_allowlist_codec.py [--selftest | decode | encode ...]") diff --git a/tests/chartctl_fake_loader.py b/tests/chartctl_fake_loader.py new file mode 100644 index 0000000..455b4cf --- /dev/null +++ b/tests/chartctl_fake_loader.py @@ -0,0 +1,111 @@ +"""A Python stand-in for the loader EA, implementing the terminal side of +Chart Control Protocol v1. Lets the full endpoint suite run on Linux with +no MT5, no Windows — the same trick conftest uses to stub the SDK. + +It reads desired.json and writes observed.json exactly as the MQL5 loader +would, so integration tests exercise the real registry merge + status +derivation against a realistic observed file. +""" +import json +import os + + +class FakeLoader: + def __init__(self, protocol_dir: str): + self.dir = protocol_dir + os.makedirs(self.dir, exist_ok=True) + os.makedirs(os.path.join(self.dir, "shots"), exist_ok=True) + self.auto_trading = True + self._next_chart_id = 133039100 + + # ── protocol files ─────────────────────────────────────────── + def _read(self, name): + try: + with open(os.path.join(self.dir, name), encoding="utf-8") as fh: + return json.load(fh) + except (OSError, ValueError): + return None + + def _write(self, name, obj): + path = os.path.join(self.dir, name) + tmp = path + ".tmp" + with open(tmp, "w", encoding="utf-8") as fh: + json.dump(obj, fh) + os.replace(tmp, path) + + # ── one reconcile pass ─────────────────────────────────────── + def reconcile(self, *, fail_ids=None, missing_ids=None): + """Materialize observed.json from desired.json. + + fail_ids: deployments that should report a terminal error. + missing_ids: enabled deployments the loader 'fails to attach' + (stay pending / degraded). + """ + fail_ids = set(fail_ids or []) + missing_ids = set(missing_ids or []) + desired = self._read("desired.json") or {"revision": 0, "deployments": []} + + charts, dep_status, errors = [], [], [] + for dep in desired.get("deployments", []): + did = dep["id"] + if not dep.get("enabled", True): + dep_status.append({"id": did, "status": "paused"}) + continue + if did in fail_ids: + errors.append({"id": did, "status": "failed", + "code": "EXPERT_NOT_ATTACHED", + "detail": "fake failure"}) + dep_status.append({"id": did, "status": "failed"}) + continue + if did in missing_ids: + dep_status.append({"id": did, "status": "pending"}) + continue + cid = self._next_chart_id + self._next_chart_id += 1 + charts.append({ + "chart_id": cid, "symbol": dep["symbol"], + "timeframe": dep["timeframe"], "expert": dep["expert"], + "expert_enabled": True, "deployment_id": did, + }) + dep_status.append({"id": did, "status": "running", "chart_id": cid}) + + observed = { + "protocol": 1, + "loader": {"name": "FakeLoader", "version": "1.0.0", + "last_loop": "now", + "applied_revision": desired.get("revision", 0)}, + "terminal": {"auto_trading": self.auto_trading}, + "charts": charts, + "deployments": dep_status, + "errors": errors, + } + self._write("observed.json", observed) + return observed + + # ── command channel ────────────────────────────────────────── + def handle_command(self): + cmd = self._read("command.json") + if not cmd: + return None + cid = cmd["command_id"] + result = {"command_id": cid} + if cmd.get("action") == "screenshot": + fname = f"{cid}.png" + with open(os.path.join(self.dir, "shots", fname), "wb") as fh: + fh.write(b"\x89PNG\r\n\x1a\n") # PNG magic, enough for a test + result.update({"status": "ok", "file": fname}) + elif cmd.get("action") == "close_chart": + if cmd.get("chart_id") == -1: # sentinel: unknown chart + result.update({"status": "error", "error_code": "CLOSE_FAILED", + "error_detail": "err=4101"}) + else: + result.update({"status": "ok"}) + else: + result.update({"status": "error", "error_code": "UNKNOWN_ACTION", + "error_detail": cmd.get("action", "")}) + self._write("command_result.json", result) + try: + os.remove(os.path.join(self.dir, "command.json")) + except OSError: + pass + return result diff --git a/tests/test_chartctl_endpoints.py b/tests/test_chartctl_endpoints.py new file mode 100644 index 0000000..260f6aa --- /dev/null +++ b/tests/test_chartctl_endpoints.py @@ -0,0 +1,305 @@ +"""End-to-end chartctl endpoint tests via Flask's test client, with the +Python FakeLoader playing the terminal side of the protocol. + +Repoints every chartctl path at a tmp dir, registers the routes on a +fresh Flask app (config gating is bypassed here — we wire handlers +directly so the suite doesn't depend on CHARTCTL_ENABLED at import). +""" +from __future__ import annotations + +import io +import json +import os + +import pytest +from flask import Flask + +from tests.chartctl_fake_loader import FakeLoader + + +@pytest.fixture +def client(monkeypatch, tmp_path): + from mt5api.chartctl import paths, registry, command + + experts = tmp_path / "experts" + sets_ = tmp_path / "sets" + proto = tmp_path / "proto" + tpls = tmp_path / "tpls" + host_e = tmp_path / "host_experts" + host_s = tmp_path / "host_sets" + for d in (experts, sets_, proto, tpls, host_e, host_s, + proto / "shots"): + d.mkdir(parents=True, exist_ok=True) + + monkeypatch.setattr(paths, "EXPERTS_DIR", str(experts)) + monkeypatch.setattr(paths, "SETS_DIR", str(sets_)) + monkeypatch.setattr(paths, "PROTOCOL_DIR", str(proto)) + monkeypatch.setattr(paths, "TEMPLATES_DIR", str(tpls)) + monkeypatch.setattr(paths, "SCREENSHOTS_DIR", str(proto / "shots")) + monkeypatch.setattr(paths, "HOST_EXPERTS_DIR", str(host_e)) + monkeypatch.setattr(paths, "HOST_SETS_DIR", str(host_s)) + monkeypatch.setattr(paths, "REGISTRY_PATH", str(tmp_path / "registry.json")) + monkeypatch.setattr(paths, "DESIRED_PATH", str(proto / "desired.json")) + monkeypatch.setattr(paths, "OBSERVED_PATH", str(proto / "observed.json")) + monkeypatch.setattr(paths, "COMMAND_PATH", str(proto / "command.json")) + monkeypatch.setattr(paths, "COMMAND_RESULT_PATH", + str(proto / "command_result.json")) + # registry caches some path constants via import — repoint those too. + monkeypatch.setattr(registry.paths, "REGISTRY_PATH", str(tmp_path / "registry.json")) + monkeypatch.setattr(registry.paths, "DESIRED_PATH", str(proto / "desired.json")) + monkeypatch.setattr(registry.paths, "OBSERVED_PATH", str(proto / "observed.json")) + monkeypatch.setattr(registry.paths, "PROTOCOL_DIR", str(proto)) + monkeypatch.setattr(registry, "_STATE", None) + # tpl_builder writes into TEMPLATES_DIR imported at module load + from mt5api.chartctl import tpl_builder + monkeypatch.setattr(tpl_builder, "TEMPLATES_DIR", str(tpls)) + # shorten command timeout so the timeout test is fast + monkeypatch.setattr(command, "CHARTCTL_COMMAND_TIMEOUT_SECONDS", 1) + + from mt5api.handlers import chartctl + app = Flask(__name__) + app.post("/experts")(chartctl.upload_expert) + app.get("/experts")(chartctl.list_experts) + app.delete("/experts/")(chartctl.delete_expert) + app.post("/sets")(chartctl.upload_set) + app.get("/sets")(chartctl.list_sets) + app.get("/sets/")(chartctl.get_set) + app.post("/deployments")(chartctl.create_deployment) + app.get("/deployments")(chartctl.list_deployments) + app.post("/deployments/reconcile")(chartctl.reconcile) + app.get("/deployments/")(chartctl.get_deployment) + app.patch("/deployments/")(chartctl.patch_deployment) + app.delete("/deployments/")(chartctl.delete_deployment) + app.get("/charts")(chartctl.charts) + app.get("/loader")(chartctl.loader_status) + app.post("/charts//screenshot")(chartctl.screenshot) + app.post("/charts//close")(chartctl.close_chart) + + c = app.test_client() + c._proto_dir = str(proto) # stash for the fake loader + return c + + +def _upload_expert(client, name="EA.ex5", content=b"MZ\x00fakeex5"): + return client.post("/experts", data={ + "expert": (io.BytesIO(content), name)}, + content_type="multipart/form-data") + + +def _upload_set(client, name="gold.set", text="Lots=0.10\nMagic=777\n"): + return client.post("/sets", data={ + "set": (io.BytesIO(text.encode("utf-16")), name)}, + content_type="multipart/form-data") + + +# ── artifacts ──────────────────────────────────────────────────────── + +def test_expert_upload_list_dedupe(client): + r = _upload_expert(client) + assert r.status_code == 201 + sha = r.get_json()["sha256"] + # re-upload identical -> skipped + r2 = _upload_expert(client) + assert r2.get_json()["skipped"] is True + lst = client.get("/experts").get_json()["experts"] + assert any(e["name"] == "EA.ex5" and e["sha256"] == sha for e in lst) + + +def test_expert_upload_conflict_on_hash_change(client): + _upload_expert(client, content=b"one") + r = _upload_expert(client, content=b"two") + assert r.status_code == 409 + assert r.get_json()["code"] == "EXISTS" + + +def test_set_upload_returns_parsed_inputs(client): + r = _upload_set(client) + assert r.status_code == 201 + inputs = r.get_json()["inputs"] + assert {"name": "Lots", "value": "0.10"} in inputs + + +def test_expert_traversal_rejected(client): + r = client.post("/experts", data={ + "expert": (io.BytesIO(b"x"), "../evil.ex5")}, + content_type="multipart/form-data") + assert r.status_code == 400 + + +# ── deployment lifecycle with fake loader ──────────────────────────── + +def test_full_deploy_verify_cycle(client): + _upload_expert(client) + _upload_set(client) + r = client.post("/deployments", json={ + "expert": "EA.ex5", "set": "gold.set", + "symbol": "XAUUSD", "timeframe": "M5"}) + assert r.status_code == 202 + dep_id = r.get_json()["id"] + + # Before the loader runs: pending, not converged. + v = client.get("/deployments").get_json() + assert v["deployments"][0]["status"] == "pending" + assert v["converged"] is False + + # A .tpl was generated. + from mt5api.chartctl import paths + assert os.path.exists(os.path.join(paths.TEMPLATES_DIR, f"{dep_id}.tpl")) + + # Loader reconciles -> running + converged. + loader = FakeLoader(client._proto_dir) + loader.reconcile() + v = client.get("/deployments").get_json() + assert v["deployments"][0]["status"] == "running" + assert v["converged"] is True + + # /charts reflects the live inventory. + charts = client.get("/charts").get_json() + assert charts["loader_alive"] is True + assert charts["charts"][0]["symbol"] == "XAUUSD" + + +def test_deploy_requires_staged_expert(client): + r = client.post("/deployments", json={ + "expert": "NOPE.ex5", "symbol": "EURUSD", "timeframe": "H1"}) + assert r.status_code == 404 + assert r.get_json()["code"] == "ARTIFACT_NOT_FOUND" + + +def test_duplicate_chart_conflict(client): + _upload_expert(client) + client.post("/deployments", json={ + "expert": "EA.ex5", "symbol": "EURUSD", "timeframe": "H1"}) + r = client.post("/deployments", json={ + "expert": "EA.ex5", "symbol": "EURUSD", "timeframe": "H1"}) + assert r.status_code == 409 + assert r.get_json()["code"] == "DUPLICATE_CHART" + + +def test_pause_then_delete(client): + _upload_expert(client) + dep_id = client.post("/deployments", json={ + "expert": "EA.ex5", "symbol": "USDJPY", "timeframe": "M30" + }).get_json()["id"] + + # pause + r = client.patch(f"/deployments/{dep_id}", json={"enabled": False}) + assert r.status_code == 200 + loader = FakeLoader(client._proto_dir) + loader.reconcile() + item = client.get(f"/deployments/{dep_id}").get_json() + assert item["status"] == "paused" + + # delete removes the tpl and the row + r = client.delete(f"/deployments/{dep_id}") + assert r.status_code == 200 + from mt5api.chartctl import paths + assert not os.path.exists(os.path.join(paths.TEMPLATES_DIR, f"{dep_id}.tpl")) + assert client.get("/deployments").get_json()["deployments"] == [] + + +def test_loader_reports_failure(client): + _upload_expert(client) + dep_id = client.post("/deployments", json={ + "expert": "EA.ex5", "symbol": "GBPUSD", "timeframe": "M15" + }).get_json()["id"] + loader = FakeLoader(client._proto_dir) + loader.reconcile(fail_ids={dep_id}) + item = client.get(f"/deployments/{dep_id}").get_json() + assert item["status"] == "failed" + assert item["error"]["code"] == "EXPERT_NOT_ATTACHED" + + +def test_patch_set_regenerates_tpl(client): + _upload_expert(client) + _upload_set(client, name="a.set", text="Lots=0.01\n") + _upload_set(client, name="b.set", text="Lots=0.99\n") + dep_id = client.post("/deployments", json={ + "expert": "EA.ex5", "set": "a.set", + "symbol": "AUDUSD", "timeframe": "H1"}).get_json()["id"] + from mt5api.chartctl import paths + tpl = os.path.join(paths.TEMPLATES_DIR, f"{dep_id}.tpl") + before = open(tpl, "rb").read() + client.patch(f"/deployments/{dep_id}", json={"set": "b.set"}) + after = open(tpl, "rb").read() + assert b"0.99" in after.decode("utf-16").encode("utf-8") or before != after + + +def test_loader_absent_hint(client): + r = client.get("/loader").get_json() + assert r["alive"] is False + assert "hint" in r + + +def test_screenshot_via_command_channel(client, monkeypatch): + _upload_expert(client) + dep_id = client.post("/deployments", json={ + "expert": "EA.ex5", "symbol": "XAUUSD", "timeframe": "M5" + }).get_json()["id"] + loader = FakeLoader(client._proto_dir) + loader.reconcile() + charts = client.get("/charts").get_json()["charts"] + chart_id = charts[0]["chart_id"] + + # Command channel is synchronous in the handler; drive the fake loader + # from a thread so it answers while the request blocks. + import threading + import time + + def answer(): + for _ in range(20): + if loader.handle_command(): + return + time.sleep(0.05) + + t = threading.Thread(target=answer) + t.start() + r = client.post(f"/charts/{chart_id}/screenshot") + t.join() + assert r.status_code == 200 + assert r.mimetype == "image/png" + + +def _run_with_fake_loader(client, loader, method, url): + """Issue a command-channel request while the fake loader answers.""" + import threading + import time + + def answer(): + for _ in range(20): + if loader.handle_command(): + return + time.sleep(0.05) + + t = threading.Thread(target=answer) + t.start() + r = getattr(client, method)(url) + t.join() + return r + + +def test_close_chart_via_command_channel(client): + _upload_expert(client) + client.post("/deployments", json={ + "expert": "EA.ex5", "symbol": "XAUUSD", "timeframe": "M5"}) + loader = FakeLoader(client._proto_dir) + loader.reconcile() + chart_id = client.get("/charts").get_json()["charts"][0]["chart_id"] + + r = _run_with_fake_loader(client, loader, "post", f"/charts/{chart_id}/close") + assert r.status_code == 200 + assert r.get_json() == {"closed": chart_id} + + +def test_close_chart_loader_failure(client): + loader = FakeLoader(client._proto_dir) + loader.reconcile() + # chart_id -1 is the fake loader's CLOSE_FAILED sentinel + r = _run_with_fake_loader(client, loader, "post", "/charts/-1/close") + assert r.status_code == 502 + assert r.get_json()["code"] == "CLOSE_FAILED" + + +def test_close_chart_bad_id(client): + r = client.post("/charts/notanint/close") + assert r.status_code == 400 diff --git a/tests/test_chartctl_units.py b/tests/test_chartctl_units.py new file mode 100644 index 0000000..b7e4dd0 --- /dev/null +++ b/tests/test_chartctl_units.py @@ -0,0 +1,206 @@ +"""Unit tests for chartctl: path safety, set parsing, tpl generation, +registry desired-state + status derivation. + +All Linux-safe: no MT5 SDK, no Windows. The registry/paths modules are +repointed at a tmp dir per test via monkeypatch, mirroring the backtest +jobs test fixture. +""" +from __future__ import annotations + +import json +import os + +import pytest + + +# ── paths.safe_name ────────────────────────────────────────────────── + +@pytest.mark.parametrize("bad", [ + "", " ", "..", ".", "a/b.ex5", "a\\b.ex5", "../evil.ex5", + "C:\\evil.ex5", "\\\\host\\share\\x.ex5", ".hidden.ex5", + "na.ex5", 'quote".ex5', "pipe|.ex5", "null\x00.ex5", +]) +def test_safe_name_rejects(bad): + from mt5api.chartctl import paths + with pytest.raises(ValueError): + paths.safe_name(bad, "expert", ".ex5") + + +def test_safe_name_accepts_and_checks_ext(): + from mt5api.chartctl import paths + assert paths.safe_name("HappyGoldScalp.ex5", "expert", ".ex5") \ + == "HappyGoldScalp.ex5" + with pytest.raises(ValueError): + paths.safe_name("HappyGoldScalp.set", "expert", ".ex5") + + +# ── setparse ───────────────────────────────────────────────────────── + +def test_parse_plain_and_optimized_set(): + from mt5api.chartctl.setparse import parse_set_text + text = ( + "; comment line\n" + "Lots=0.10\n" + "StopLoss=50||10||5||100||Y\n" + "UseTrailing=1\n" + ) + got = parse_set_text(text) + assert got[0] == {"name": "Lots", "value": "0.10"} + assert got[1]["name"] == "StopLoss" + assert got[1]["value"] == "50" + assert got[1]["optimize"] is True + assert got[1]["start"] == "10" and got[1]["stop"] == "100" + assert got[2] == {"name": "UseTrailing", "value": "1"} + + +def test_parse_utf16_bytes(): + from mt5api.chartctl.setparse import parse_set_bytes + raw = "Lots=0.01\nMagic=12345\n".encode("utf-16") + got = parse_set_bytes(raw) + assert {"name": "Lots", "value": "0.01"} in got + assert {"name": "Magic", "value": "12345"} in got + + +def test_parse_ascii_bytes_even_length(): + # Even-length ASCII input decodes "successfully" as utf-16 garbage if + # utf-16 is blind-tried first, silently yielding zero inputs — and the + # deployment would run on EA defaults. Regression for that decode bug. + from mt5api.chartctl.setparse import parse_set_bytes + raw = b"; comment\r\nLots=0.10\r\nMagic=99\r\n" + assert len(raw) % 2 == 0 + got = parse_set_bytes(raw) + assert {"name": "Lots", "value": "0.10"} in got + assert {"name": "Magic", "value": "99"} in got + + +def test_parse_bomless_utf16_bytes(): + from mt5api.chartctl.setparse import parse_set_bytes + raw = "Lots=0.01\nMagic=12345\n".encode("utf-16-le") # no BOM + got = parse_set_bytes(raw) + assert {"name": "Lots", "value": "0.01"} in got + assert {"name": "Magic", "value": "12345"} in got + + +# ── tpl_builder ────────────────────────────────────────────────────── + +def test_tpl_text_structure(): + from mt5api.chartctl import tpl_builder + text = tpl_builder.build_tpl_text( + deployment_id="dep_abc123", + expert_name="HappyGoldScalp", + expert_rel_path="Experts\\Uploaded\\HappyGoldScalp.ex5", + inputs=[{"name": "Lots", "value": "0.10"}, + {"name": "Magic", "value": "777"}], + terminal_build=4620, + ) + assert "" in text and "" in text + assert "" in text and "" in text + assert "name=HappyGoldScalp" in text + assert "path=Experts\\Uploaded\\HappyGoldScalp.ex5" in text + assert "expertmode=1" in text + assert "__chartctl_id=dep_abc123" in text # attribution input + assert "Lots=0.10" in text and "Magic=777" in text + assert "build=4620" in text # forensic stamp + assert text.endswith("\r\n") + + +def test_tpl_written_as_utf16_with_bom(tmp_path, monkeypatch): + from mt5api.chartctl import tpl_builder, paths + monkeypatch.setattr(paths, "TEMPLATES_DIR", str(tmp_path)) + monkeypatch.setattr(tpl_builder, "TEMPLATES_DIR", str(tmp_path)) + path = tpl_builder.write_tpl( + deployment_id="dep_x", expert_name="EA", + expert_rel_path="Experts\\Uploaded\\EA.ex5", inputs=[]) + data = open(path, "rb").read() + assert data[:2] == b"\xff\xfe" # UTF-16-LE BOM + assert "name=EA" in data.decode("utf-16") + + +# ── registry ───────────────────────────────────────────────────────── + +@pytest.fixture +def reg(monkeypatch, tmp_path): + from mt5api.chartctl import registry, paths + monkeypatch.setattr(paths, "REGISTRY_PATH", str(tmp_path / "registry.json")) + monkeypatch.setattr(paths, "DESIRED_PATH", str(tmp_path / "desired.json")) + monkeypatch.setattr(paths, "OBSERVED_PATH", str(tmp_path / "observed.json")) + monkeypatch.setattr(paths, "PROTOCOL_DIR", str(tmp_path)) + monkeypatch.setattr(paths, "TEMPLATES_DIR", str(tmp_path)) + monkeypatch.setattr(registry, "_STATE", None) + return registry, tmp_path + + +def test_add_bumps_revision_and_writes_desired(reg): + registry, tmp = reg + d = registry.add_deployment( + expert_file="EA.ex5", expert_name="EA", set_file=None, + symbol="XAUUSD", timeframe="M5") + assert d["id"].startswith("dep_") + assert registry.current_revision() == 1 + desired = json.loads((tmp / "desired.json").read_text()) + assert desired["revision"] == 1 + assert desired["deployments"][0]["symbol"] == "XAUUSD" + assert desired["deployments"][0]["template"].startswith("\\Files\\chartctl\\") + + +def test_duplicate_enabled_chart_rejected(reg): + registry, _ = reg + registry.add_deployment(expert_file="A.ex5", expert_name="A", + set_file=None, symbol="EURUSD", timeframe="H1") + with pytest.raises(registry.DuplicateChart): + registry.add_deployment(expert_file="B.ex5", expert_name="B", + set_file=None, symbol="EURUSD", timeframe="H1") + + +def test_disabled_does_not_conflict(reg): + registry, _ = reg + registry.add_deployment(expert_file="A.ex5", expert_name="A", + set_file=None, symbol="EURUSD", timeframe="H1", + enabled=False) + # Same slot, enabled — must be allowed since the first is paused. + registry.add_deployment(expert_file="B.ex5", expert_name="B", + set_file=None, symbol="EURUSD", timeframe="H1") + + +def test_persistence_across_reload(reg): + registry, tmp = reg + registry.add_deployment(expert_file="EA.ex5", expert_name="EA", + set_file=None, symbol="GBPUSD", timeframe="M15") + registry._STATE = None # simulate API restart + deps = registry.list_deployments() + assert len(deps) == 1 and deps[0]["symbol"] == "GBPUSD" + + +def test_remove_bumps_and_clears(reg): + registry, _ = reg + d = registry.add_deployment(expert_file="EA.ex5", expert_name="EA", + set_file=None, symbol="USDJPY", timeframe="H4") + registry.remove_deployment(d["id"]) + assert registry.list_deployments() == [] + assert registry.current_revision() == 2 + + +def test_merged_view_status_pending_without_observed(reg): + registry, _ = reg + registry.add_deployment(expert_file="EA.ex5", expert_name="EA", + set_file=None, symbol="XAUUSD", timeframe="M5") + view = registry.merged_view() + assert view["deployments"][0]["status"] == "pending" + assert view["converged"] is False + assert view["observed_stale"] is True + + +def test_merged_view_running_when_observed_matches(reg): + registry, tmp = reg + d = registry.add_deployment(expert_file="EA.ex5", expert_name="EA", + set_file=None, symbol="XAUUSD", timeframe="M5") + observed = { + "loader": {"applied_revision": registry.current_revision(), + "last_loop": "now"}, + "terminal": {"auto_trading": True}, + "charts": [{"chart_id": 1, "expert": "EA", "deployment_id": d["id"]}], + "deployments": [{"id": d["id"], "status": "running", "chart_id": 1}], + } + (tmp / "observed.json").write_text(json.dumps(observed)) + view = registry.merged_view() + assert view["deployments"][0]["status"] == "running" diff --git a/tests/test_config_generation.py b/tests/test_config_generation.py index ebe82cc..1829ac4 100644 --- a/tests/test_config_generation.py +++ b/tests/test_config_generation.py @@ -142,11 +142,10 @@ def test_absent_vms_file_keeps_every_terminal_on_the_default_container( assert "mt5-b" not in content -def test_live_terminal_ini_declares_no_startup_expert(tmp_path, monkeypatch): - """The generated INI decides what a terminal does on launch, so a `[StartUp]` - section auto-attaches an expert to every live terminal — a fleet-wide - behaviour change in a file nothing else asserts on. Adding that feature means - editing this test deliberately, not inheriting it from an unrelated commit. +def test_live_terminal_ini_startup_expert_with_chartctl(tmp_path, monkeypatch): + """Chart Deployments is enabled by default, so a live terminal's INI carries + a [StartUp] section that auto-attaches MT5ChartLoader. The loader's + GlobalVariable mutex makes repeated re-attaches idempotent. """ helper = _load_config_helper_module() config_path = _write_config( @@ -161,6 +160,33 @@ def test_live_terminal_ini_declares_no_startup_expert(tmp_path, monkeypatch): helper.main() + content = outpath.read_text(encoding="utf-8") + assert "[StartUp]" in content + assert "Expert=Advisors\\MT5ChartLoader" in content + + +def test_live_terminal_ini_declares_no_startup_when_chartctl_disabled(tmp_path, monkeypatch): + """With chartctl explicitly disabled, a live terminal's INI has no [StartUp] + section — no unexpected expert auto-attachment. + """ + helper = _load_config_helper_module() + config_path = _write_config( + tmp_path, + [{"broker": "acme", "account": "main", "port": 5001}], + ) + config_obj = yaml.safe_load(config_path.read_text(encoding="utf-8")) + config_obj["chartctl"] = {"enabled": False} + config_path.write_text(yaml.safe_dump(config_obj), encoding="utf-8") + + outpath = tmp_path / "terminal.ini" + monkeypatch.setattr(helper, "CONFIG_PATH", str(config_path)) + monkeypatch.setattr( + "sys.argv", + ["config_helper.py", "write_ini", "acme", "main", str(outpath), "default", "live"], + ) + + helper.main() + content = outpath.read_text(encoding="utf-8") assert "[StartUp]" not in content assert "Expert=" not in content diff --git a/tests/test_webrequest.py b/tests/test_webrequest.py new file mode 100644 index 0000000..a7648e8 --- /dev/null +++ b/tests/test_webrequest.py @@ -0,0 +1,338 @@ +"""Tests for the WebRequest allowlist codec + manager + endpoint. + +Runs entirely on Linux (no MT5/Windows): the codec is pure Python and the +manager is plain file I/O. The live apply (terminal restart writing common.ini) +is exercised via a stubbed restart_terminal. +""" +from __future__ import annotations + +import json +import os + +import pytest +from flask import Flask + +from mt5api.chartctl import webrequest as wr + +# A real captured broker blob and its plaintext URLs (verified byte-identical). +REAL_BLOB = ( + "13E33B7856715A56F2822DF172EBC0D0BD8DF23E89A42B2716A602C68D06506070409EEA021DA6A29" + "525874B0D86E7F7D8A8FC4898B30E0A3DCDFBBF9D166474552580CC55704642FD8D1DE13AB3CADA08D8" + "DC28AFCA0C080292FABED14A1828BA8A1B67BCD700FC69F9D094E55E2A3AAE7E17637D986B67D868511" + "562DB" +) +REAL_URLS = ["https://tracker.algotradingspace.com", "https://api.telegram.org"] + +_BOM = b"\xff\xfe" + + +def _write_utf16_ini(path: str, text: str) -> None: + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "wb") as f: + f.write(_BOM + text.replace("\n", "\r\n").encode("utf-16-le")) + + +# ── codec ──────────────────────────────────────────────────────────── +def test_codec_roundtrip_real_blob(): + assert wr.decode_blob(REAL_BLOB) == REAL_URLS + assert wr.encode_urls(REAL_URLS) == REAL_BLOB + + +def test_codec_roundtrip_arbitrary(): + urls = ["https://api.example.com", "http://nfs.faireconomy.media/x.xml"] + assert wr.decode_blob(wr.encode_urls(urls)) == urls + + +# ── url hygiene ────────────────────────────────────────────────────── +def test_clean_urls_filters_and_dedupes(): + got = wr.clean_urls([ + " https://a.com ", "https://a.com", "ftp://no.com", + "https://b.com;evil", "", 42, "HTTPS://C.com", + ]) + assert got == ["https://a.com", "HTTPS://C.com"] + + +# ── desired store ──────────────────────────────────────────────────── +def test_desired_store_roundtrip(tmp_path): + cfg = str(tmp_path / "Config") + assert wr.load_desired(cfg) is None + wr.save_desired(cfg, REAL_URLS) + assert wr.load_desired(cfg) == REAL_URLS + with open(wr.desired_path(cfg)) as f: + assert json.load(f)["urls"] == REAL_URLS + + +# ── migrate-from-current ───────────────────────────────────────────── +def test_read_current_urls_from_existing_common_ini(tmp_path): + cfg = str(tmp_path / "Config") + _write_utf16_ini( + wr.common_ini_path(cfg), + f"[Charts]\nProfileLast=Default\n[Experts]\nAllowDllImport=1\n" + f"WebRequest=1\nWebRequestUrl={REAL_BLOB}\n[Objects]\nShow=0\n", + ) + assert wr.read_current_urls(cfg) == REAL_URLS + + +def test_read_current_urls_absent(tmp_path): + assert wr.read_current_urls(str(tmp_path / "Config")) == [] + + +def test_effective_prefers_desired_then_migrates(tmp_path): + cfg = str(tmp_path / "Config") + _write_utf16_ini( + wr.common_ini_path(cfg), + f"[Experts]\nWebRequest=1\nWebRequestUrl={REAL_BLOB}\n", + ) + # no desired yet -> migrate from current + assert wr.effective_urls(cfg) == REAL_URLS + wr.save_desired(cfg, ["https://only.example.com"]) + assert wr.effective_urls(cfg) == ["https://only.example.com"] + + +# ── write_common_ini: preserve everything, upsert the two keys ─────── +def test_write_common_ini_preserves_other_sections(tmp_path): + cfg = str(tmp_path / "Config") + _write_utf16_ini( + wr.common_ini_path(cfg), + "[Charts]\nProfileLast=Default\n[Experts]\nAllowDllImport=1\n" + "WebRequest=0\nWebRequestUrl=DEADBEEF\n[Objects]\nShow=0\n", + ) + new = ["https://api.example.com"] + wr.write_common_ini(cfg, new) + lines = wr._read_lines(wr.common_ini_path(cfg)) + assert "[Charts]" in lines and "ProfileLast=Default" in lines + assert "[Objects]" in lines and "Show=0" in lines + assert "AllowDllImport=1" in lines # sibling key preserved + assert "WebRequest=1" in lines # flipped on + assert wr.read_current_urls(cfg) == new # blob replaced, decodes to new + # exactly one WebRequestUrl line + assert sum(l.strip().lower().startswith("webrequesturl=") for l in lines) == 1 + + +def test_write_common_ini_creates_file_and_section(tmp_path): + cfg = str(tmp_path / "Config") + wr.write_common_ini(cfg, REAL_URLS) + assert os.path.exists(wr.common_ini_path(cfg)) + with open(wr.common_ini_path(cfg), "rb") as f: + assert f.read(2) == _BOM # UTF-16LE BOM preserved + assert wr.read_current_urls(cfg) == REAL_URLS + + +def test_write_common_ini_is_utf16(tmp_path): + cfg = str(tmp_path / "Config") + wr.write_common_ini(cfg, REAL_URLS) + with open(wr.common_ini_path(cfg), "rb") as f: + raw = f.read() + assert b"\x00" in raw # wide chars + assert raw.decode("utf-16").count("[Experts]") == 1 + + +def test_apply_from_desired(tmp_path): + term = str(tmp_path / "term") + cfg = wr.config_dir(term) + assert wr.apply_from_desired(term) is None # no desired -> no-op + assert not os.path.exists(wr.common_ini_path(cfg)) + wr.save_desired(cfg, REAL_URLS) + assert wr.apply_from_desired(term) == len(REAL_URLS) + assert wr.read_current_urls(cfg) == REAL_URLS + + +# ── endpoint ───────────────────────────────────────────────────────── +@pytest.fixture +def client(monkeypatch, tmp_path): + from mt5api.handlers import webrequest as handler + + term = str(tmp_path / "term") + os.makedirs(wr.config_dir(term), exist_ok=True) + monkeypatch.setattr(handler, "TERMINAL_DIR", term) + + calls = {"restart": 0} + + def fake_restart(): + calls["restart"] += 1 + # emulate the real restart applying desired -> common.ini + wr.apply_from_desired(term) + return True + + monkeypatch.setattr(handler, "restart_terminal", fake_restart) + + # default: no AutoIt (bare-metal fallback path) unless a test flips it + from mt5api.chartctl import autoit_webrequest as autoit + monkeypatch.setattr(autoit, "available", lambda: False) + + app = Flask(__name__) + app.get("/webrequest")(handler.get_webrequest) + app.put("/webrequest")(handler.put_webrequest) + app.post("/webrequest/apply")(handler.apply_webrequest) + c = app.test_client() + c._term = term + c._calls = calls + c._autoit = autoit + return c + + +def test_get_empty(client): + r = client.get("/webrequest") + assert r.status_code == 200 and r.get_json() == {"urls": []} + + +def test_put_replace_restarts_and_applies(client): + r = client.put("/webrequest", json={"urls": REAL_URLS}) + assert r.status_code == 200 + body = r.get_json() + assert body["success"] is True and body["urls"] == REAL_URLS + assert body["applied_via"] == "restart" # bare-metal fallback path + assert client._calls["restart"] == 1 + # persisted + written to common.ini + assert wr.load_desired(wr.config_dir(client._term)) == REAL_URLS + assert wr.read_current_urls(wr.config_dir(client._term)) == REAL_URLS + # readable back through GET + assert client.get("/webrequest").get_json()["urls"] == REAL_URLS + + +def test_put_add_remove_migrates_from_current(client): + cfg = wr.config_dir(client._term) + _write_utf16_ini(wr.common_ini_path(cfg), + f"[Experts]\nWebRequest=1\nWebRequestUrl={REAL_BLOB}\n") + r = client.put("/webrequest", json={ + "add": ["https://new.example.com"], + "remove": ["https://api.telegram.org"], + }) + assert r.status_code == 200 + assert r.get_json()["urls"] == [ + "https://tracker.algotradingspace.com", "https://new.example.com", + ] + + +def test_put_uses_autoit_when_available(client, monkeypatch): + """On the VM (AutoIt present) the allowlist is applied via the Options + dialog, not a terminal restart.""" + seen = {} + + def fake_apply(urls, timeout=120, use_runas=False): + seen["urls"] = list(urls) + return "OK", "log" + + monkeypatch.setattr(client._autoit, "available", lambda: True) + monkeypatch.setattr(client._autoit, "apply_urls", fake_apply) + + r = client.put("/webrequest", json={"urls": REAL_URLS}) + assert r.status_code == 200 + body = r.get_json() + assert body["success"] is True and body["applied_via"] == "autoit:OK" + assert seen["urls"] == REAL_URLS + assert client._calls["restart"] == 0 # no restart on the AutoIt path + assert wr.load_desired(wr.config_dir(client._term)) == REAL_URLS + + +def test_put_autoit_failure_returns_500(client, monkeypatch): + monkeypatch.setattr(client._autoit, "available", lambda: True) + monkeypatch.setattr(client._autoit, "apply_urls", lambda urls, timeout=120, use_runas=False: ("FAIL", "boom")) + r = client.put("/webrequest", json={"urls": REAL_URLS}) + assert r.status_code == 500 + # desired still persisted so a later /apply or boot can retry + assert wr.load_desired(wr.config_dir(client._term)) == REAL_URLS + + +def test_apply_endpoint_reapplies_desired(client, monkeypatch): + seen = {} + monkeypatch.setattr(client._autoit, "available", lambda: True) + monkeypatch.setattr( + client._autoit, "apply_urls", + lambda urls, timeout=120, use_runas=False: (seen.update(urls=list(urls)) or ("OK", "")), + ) + wr.save_desired(wr.config_dir(client._term), REAL_URLS) + r = client.post("/webrequest/apply") + assert r.status_code == 200 + assert r.get_json()["urls"] == REAL_URLS + assert seen["urls"] == REAL_URLS + + +def test_apply_endpoint_noop_when_empty(client): + r = client.post("/webrequest/apply") + assert r.status_code == 200 + assert r.get_json()["urls"] == [] + + +def test_put_rejects_non_dict(client): + assert client.put("/webrequest", data="nope").status_code == 400 + + +def test_put_rejects_empty_body(client): + assert client.put("/webrequest", json={}).status_code == 400 + + +def test_put_restart_failure_returns_500(client, monkeypatch): + from mt5api.handlers import webrequest as handler + monkeypatch.setattr(handler, "restart_terminal", lambda: False) + r = client.put("/webrequest", json={"urls": REAL_URLS}) + assert r.status_code == 500 + # desired still persisted (next boot/restart will apply it) + assert wr.load_desired(wr.config_dir(client._term)) == REAL_URLS + + +# ── config_helper boot-seed (start.bat deletes common.ini every boot) ─ +def _load_config_helper(): + import importlib.util + root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + path = os.path.join(root, "scripts", "config_helper.py") + spec = importlib.util.spec_from_file_location("config_helper_under_test", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def test_config_helper_write_ini_seeds_common_ini(tmp_path, monkeypatch): + ch = _load_config_helper() + cfg_yaml = tmp_path / "config.yaml" + cfg_yaml.write_text( + "chartctl:\n enabled: true\n" + "accounts:\n testbroker:\n acct1:\n" + " login: 123\n server: TestServer\n password: pw\n" + "terminals:\n - broker: testbroker\n account: acct1\n" + " instance: default\n port: 6542\n mode: live\n" + ) + monkeypatch.setattr(ch, "CONFIG_PATH", str(cfg_yaml)) + + term = tmp_path / "term" + (term / "Config").mkdir(parents=True) + outpath = term / "mt5start.ini" + # simulate a terminal that already has a desired allowlist persisted + wr.save_desired(str(term / "Config"), REAL_URLS) + + monkeypatch.setattr( + ch.sys, "argv", + ["config_helper.py", "write_ini", "testbroker", "acct1", str(outpath), + "default", "live"], + ) + ch.main() + + # mt5start.ini written with the loader StartUp block (chartctl on) + start_ini = outpath.read_text() + assert "[StartUp]" in start_ini and "MT5ChartLoader" in start_ini + # common.ini re-emitted from the desired file, decodes back to the URLs + assert wr.read_current_urls(str(term / "Config")) == REAL_URLS + + +def test_config_helper_write_ini_no_desired_no_common_ini(tmp_path, monkeypatch): + ch = _load_config_helper() + cfg_yaml = tmp_path / "config.yaml" + cfg_yaml.write_text( + "chartctl:\n enabled: true\n" + "accounts:\n testbroker:\n acct1:\n" + " login: 123\n server: TestServer\n password: pw\n" + "terminals:\n - broker: testbroker\n account: acct1\n" + " instance: default\n port: 6542\n mode: live\n" + ) + monkeypatch.setattr(ch, "CONFIG_PATH", str(cfg_yaml)) + term = tmp_path / "term" + (term / "Config").mkdir(parents=True) + outpath = term / "mt5start.ini" + monkeypatch.setattr( + ch.sys, "argv", + ["config_helper.py", "write_ini", "testbroker", "acct1", str(outpath), + "default", "live"], + ) + ch.main() + # no desired file -> boot-seed is a no-op, common.ini left absent + assert not os.path.exists(wr.common_ini_path(str(term / "Config")))