FlaUI-MCP

An MCP (Model Context Protocol) server that enables AI agents to automate Windows desktop applications using accessibility APIs - the same way Playwright automates browsers.

Why This Exists

When Playwright's MCP server automates browsers, it provides:

browser_snapshot → Structured accessibility tree with element refs
browser_click ref="..." → Click by ref, not coordinates

FlaUI-MCP brings the same pattern to Windows desktop apps:

windows_snapshot → Accessibility tree with refs like w1e5
windows_click ref="w1e5" → Click element by ref

No screenshot parsing. No coordinate guessing. Just semantic element references.

Quick Demo

Agent: Calculate 3 × 3

1. windows_launch { "app": "calc.exe" }
   → Window handle: w1

2. windows_snapshot { "handle": "w1" }
   → - window "Calculator" [ref=w1]
       - button "Three" [ref=w1e43]
       - button "Multiply by" [ref=w1e35]
       - button "Equals" [ref=w1e38]
       - text "Display is 0" [ref=w1e15]

3. windows_batch { "actions": [
     {"action": "click", "ref": "w1e43"},
     {"action": "click", "ref": "w1e35"},
     {"action": "click", "ref": "w1e43"},
     {"action": "click", "ref": "w1e38"},
     {"action": "snapshot", "handle": "w1"}
   ]}
   → 1. click: Invoked Three
     2. click: Invoked Multiply by
     3. click: Invoked Three
     4. click: Invoked Equals
     5. snapshot: ... "Display is 9" ...

Installation

Prerequisites

Windows 10/11
.NET 8.0 Runtime

Download Release

Download the latest release from Releases and extract to a folder.

Choose the ZIP that matches your machine:

Asset	Use when
`FlaUI-MCP-win-x64-*-self-contained.zip`	64-bit Windows, no .NET runtime required
`FlaUI-MCP-win-x64-*.zip`	64-bit Windows with .NET 8 Runtime already installed
`FlaUI-MCP-win-arm64-*-self-contained.zip`	Windows on ARM64, no .NET runtime required
`FlaUI-MCP-win-arm64-*.zip`	Windows on ARM64 with .NET 8 Runtime already installed

Configure MCP Client

Add to your MCP configuration (e.g., ~/.copilot/mcp-config.json):

{
  "mcpServers": {
    "windows": {
      "type": "local",
      "command": "C:\\path\\to\\FlaUI-MCP.exe",
      "tools": ["*"]
    }
  }
}

Or using dotnet run:

{
  "mcpServers": {
    "windows": {
      "type": "local",
      "command": "dotnet",
      "args": ["run", "--project", "C:\\path\\to\\src\\FlaUI.Mcp"]
    }
  }
}

Available Tools

Tool	Description
`windows_launch`	Launch a Windows application
`windows_snapshot`	Get accessibility tree with element refs
`windows_click`	Click an element by ref
`windows_type`	Type text into an element
`windows_send_keys`	Send key presses or key chords (for example `Ctrl+A`)
`windows_fill`	Clear and fill a text field
`windows_get_text`	Get text content of an element
`windows_screenshot`	Capture window/element as PNG
`windows_list_windows`	List all open windows
`windows_focus`	Bring a window to foreground
`windows_close`	Close a window
`windows_batch`	Execute multiple actions in one call

windows_screenshot supports an optional background: true argument when a window handle is provided. This uses native background capture when available and falls back to the normal screenshot path if Windows returns a blank frame. It can also save screenshots with savePath, which must be an absolute local .png path. Existing files are not replaced unless overwrite: true is set.

Tool calls have a 30-second timeout so a blocked UI Automation provider or modal dialog returns an actionable error instead of hanging the MCP server forever.

Keeping the Screen Awake

Long automation runs generate no keyboard or mouse input, so Windows would normally turn off the display and show the lock screen mid-run — which breaks screenshots and can freeze rendering. While tools are actively being called, FlaUI-MCP holds a Windows power availability request (the same signal video players and conferencing apps send) that keeps the display on and suppresses the idle lock. The request appears in powercfg /requests (run as admin) with the reason "FlaUI-MCP is driving Windows UI automation".

The request is released after 5 minutes without a tool call, so an idle MCP server does not keep your screen on. Configure via the FLAUI_MCP_KEEP_AWAKE_SECONDS environment variable: a positive value changes the idle period, 0 disables keep-awake entirely.

Note: this covers the common idle-lock paths (display timeout, screensaver, sleep). A domain group policy that enforces a hard machine inactivity limit ("Interactive logon: Machine inactivity limit") locks based on input idle time and is not suppressed by availability requests — no application can override that policy.

Restricting Which Apps Can Be Automated

By default, FlaUI-MCP can automate any application on the desktop — including File Explorer, browsers, and terminals. If the agent driving it is ever misled (for example by prompt injection), that is a large attack surface. Set the FLAUI_MCP_ALLOWED_APPS environment variable to restrict automation to specific apps:

{
  "mcpServers": {
    "windows": {
      "type": "local",
      "command": "C:\\path\\to\\FlaUI.Mcp.exe",
      "env": {
        "FLAUI_MCP_ALLOWED_APPS": "TabularEditor3"
      }
    }
  }
}

The value is a semicolon- or comma-separated list of process names, matched case-insensitively, with or without .exe (full paths are reduced to their file name). When the variable is unset or empty, everything is allowed.

While the allowlist is active:

windows_launch refuses to start non-allowed executables.
Window handles are only ever issued for allowed processes, so every ref-based tool (snapshot, click, type, fill, get_text, screenshot by handle/ref) is automatically scoped to allowed apps. windows_list_windows lists only allowed apps' windows.
Ref-less keyboard input (windows_send_keys / windows_type without a ref) verifies the foreground window belongs to an allowed process first, and the Windows key is rejected outright (it opens system UI like the Start menu and Win+R outside any allowlist).
windows_screenshot refuses fullScreen capture and foreground-window capture of non-allowed apps, so other windows' content is not disclosed.

Scope honestly stated: matching is by process name, so this is a guard against a misdirected or prompt-injected agent driving unintended apps through this server — not a sandbox against a local attacker, and it does not restrict anything the agent can do through other tools (like a shell). Dialogs owned by the allowed process (including common file dialogs, which run in-process) keep working; apps it launches as separate processes (e.g. a browser for OAuth) are blocked unless also listed.

Tool Examples

Send a keyboard chord to a target element:

{
  "ref": "w1e5",
  "chord": "Ctrl+A"
}

Send a sequence of key presses or chords:

{
  "keys": ["Ctrl+A", "Delete", "Enter"]
}

Capture a window using opt-in background capture:

{
  "handle": "w1",
  "background": true
}

Save a screenshot to disk without replacing existing files:

{
  "handle": "w1",
  "savePath": "C:\\Temp\\capture.png"
}

Replace an existing screenshot file explicitly:

{
  "handle": "w1",
  "savePath": "C:\\Temp\\capture.png",
  "overwrite": true
}

Safety and Limitations

Keyboard input is focus-dependent. When you use windows_send_keys, the tool focuses the supplied ref first when possible, but Windows still sends keys to the active keyboard focus.
windows_screenshot background: true is only valid with a window handle. If native background capture returns a blank frame, FlaUI-MCP falls back to the normal capture path.
savePath accepts absolute local .png paths only. UNC paths, device paths, non-PNG extensions, and existing files without overwrite: true are rejected.
Desktop integration tests require an interactive Windows session because they launch real WinForms and WPF windows.
A timeout error means the MCP request returned, but a blocked Windows UI Automation provider or modal dialog may still need to be dismissed before retrying the operation.

How It Works

The Accessibility Snapshot

When you call windows_snapshot, you get a structured text tree:

- window "Calculator" [ref=w1e1]
  - group "Number pad" [ref=w1e39]
    - button "Seven" [ref=w1e47]
    - button "Eight" [ref=w1e48]
    - button "Nine" [ref=w1e49]
  - text "Display is 0" [ref=w1e15]

This comes from Windows UI Automation - the same API screen readers use. Each element has:

Role (button, text, group, textbox)
Name ("Seven", "Display is 0")
Ref (w1e47) - a handle for interaction
State ([disabled], [readonly], [checked])

Why Not Screenshots?

Approach	Pros	Cons
Accessibility Tree	Semantic, precise, fast, works at any resolution	Requires UI Automation support
Screenshot + Vision	Works with any app	Slow, expensive, imprecise, resolution-dependent

FlaUI-MCP uses accessibility because it's what screen readers use - it's designed for programmatic UI interaction.

Modal Dialogs

UI Automation pattern calls (Invoke, Toggle, Select) are synchronous cross-process calls. When a click handler opens a modal dialog (ShowDialog() in WinForms/WPF), the handler — and therefore the pattern call and the app's entire UIA provider — stays blocked until the dialog closes.

FlaUI-MCP handles this instead of hanging:

windows_click runs the pattern call on a background thread and watches the app's top-level windows via non-blocking Win32 APIs. If a modal appears, the tool returns immediately with the dialog's title.
While the call is pending, UIA-based tools targeting that app (windows_snapshot, windows_get_text, ref-based typing/clicking) fail fast with guidance instead of timing out.
Tools that don't need UIA keep working throughout: windows_screenshot, windows_send_keys / windows_type without a ref (pure keyboard input), windows_list_windows, windows_focus, and windows_close.

Typical flow: click a button → "modal dialog opened" → screenshot to see it → send keys (e.g. Enter or Tab+Enter) to dismiss it → snapshot works again.

Building from Source

# Clone
git clone https://github.com/shanselman/FlaUI-MCP.git
cd FlaUI-MCP

# Build
dotnet build src/FlaUI.Mcp

# Run
dotnet run --project src/FlaUI.Mcp

Testing

# Unit tests
dotnet test tests\FlaUI.Mcp.Tests

# Desktop integration tests; requires an interactive Windows session
dotnet test tests\FlaUI.Mcp.IntegrationTests

Architecture

┌─────────────────────────────────────────────────────────────────┐
│  AI Agent (GitHub Copilot, Claude, etc.)                        │
│  - Calls MCP tools: windows_snapshot, windows_click, etc.       │
└─────────────────────────────────────────────────────────────────┘
                              │ MCP Protocol (JSON-RPC over stdio)
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  FlaUI-MCP Server (.NET 8)                                      │
│  - Implements MCP tool handlers                                 │
│  - Builds agent-friendly accessibility snapshots                │
│  - Maps element refs ↔ AutomationElements                       │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  FlaUI Library (github.com/FlaUI/FlaUI)                         │
│  - UIA3Automation for modern apps (WPF, UWP, Win32)            │
│  - Control patterns: Invoke, Value, Toggle, Selection           │
│  - Tree walking and element discovery                           │
└─────────────────────────────────────────────────────────────────┘

Supported Applications

Works with any Windows application that supports UI Automation:

✅ Win32 apps (Notepad, Explorer, etc.)
✅ WPF applications
✅ WinForms applications
✅ UWP/Store apps (Calculator, Settings, etc.)
⚠️ Electron apps (partial - depends on accessibility implementation)
❌ Games (typically no UI Automation support)

Contributing

Contributions welcome! Please see CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE for details.

Acknowledgments

FlaUI - The excellent .NET UI Automation library this project is built on
Playwright - Inspiration for the snapshot/ref interaction model
Model Context Protocol - The protocol that makes this possible

Name		Name	Last commit message	Last commit date
.github		.github
src/FlaUI.Mcp		src/FlaUI.Mcp
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
FlaUI.Mcp.slnx		FlaUI.Mcp.slnx
GitVersion.yml		GitVersion.yml
LICENSE		LICENSE
README.md		README.md

Navigation Menu

Uh oh!

Repository files navigation

FlaUI-MCP

Why This Exists

Quick Demo

Installation

Prerequisites

Download Release

Configure MCP Client

Available Tools

Keeping the Screen Awake

Restricting Which Apps Can Be Automated

Tool Examples

Safety and Limitations

How It Works

The Accessibility Snapshot

Why Not Screenshots?

Modal Dialogs

Building from Source

Testing

Architecture

Supported Applications

Contributing

License

Acknowledgments

About

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

FlaUI-MCP

Why This Exists

Quick Demo

Installation

Prerequisites

Download Release

Configure MCP Client

Available Tools

Keeping the Screen Awake

Restricting Which Apps Can Be Automated

Tool Examples

Safety and Limitations

How It Works

The Accessibility Snapshot

Why Not Screenshots?

Modal Dialogs

Building from Source

Testing

Architecture

Supported Applications

Contributing

License

Acknowledgments

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages