Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added
- Blocked-provider and modal-detected guidance messages now recommend capturing the dialog via its own window handle from `windows_list_windows` (which works while the provider is blocked and under an app allowlist) instead of `fullScreen: true` (refused while an allowlist is active), and mention waiting for the dialog to appear and take focus before sending keys.
- Optional app allowlist via the `FLAUI_MCP_ALLOWED_APPS` environment variable (semicolon- or comma-separated process names). When set, the server refuses to launch, list, snapshot, screenshot or send input to any other application: window handles are only issued for allowed processes (scoping every ref-based tool), ref-less keyboard input requires an allowed foreground window and rejects the Windows key, and `fullScreen` screenshots are disabled. Unset means everything is allowed, as before.
- The server now keeps the display awake while tools are actively being called, so Windows does not turn off the screen or show the lock screen in the middle of a long automation run. Implemented with a Windows power availability request (`PowerCreateRequest`/`PowerSetRequest` with `PowerRequestDisplayRequired` + `PowerRequestSystemRequired`) — the same mechanism video players and conferencing apps use, visible in `powercfg /requests`. The request is released after 5 minutes without a tool call; configure the idle period (or disable with `0`) via the `FLAUI_MCP_KEEP_AWAKE_SECONDS` environment variable.

### Fixed
Expand Down
48 changes: 48 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,54 @@ sleep). A domain group policy that enforces a hard machine inactivity limit
and is not suppressed by availability requests — no application can override
that policy.

### Restricting Which Apps Can Be Automated

By default, FlaUI-MCP can automate **any** application on the desktop — including
File Explorer, browsers, and terminals. If the agent driving it is ever misled
(for example by prompt injection), that is a large attack surface. Set the
`FLAUI_MCP_ALLOWED_APPS` environment variable to restrict automation to specific
apps:

```json
{
"mcpServers": {
"windows": {
"type": "local",
"command": "C:\\path\\to\\FlaUI.Mcp.exe",
"env": {
"FLAUI_MCP_ALLOWED_APPS": "TabularEditor3"
}
}
}
}
```

The value is a semicolon- or comma-separated list of process names, matched
case-insensitively, with or without `.exe` (full paths are reduced to their
file name). When the variable is unset or empty, everything is allowed.

While the allowlist is active:

- `windows_launch` refuses to start non-allowed executables.
- Window handles are only ever issued for allowed processes, so every ref-based
tool (snapshot, click, type, fill, get_text, screenshot by handle/ref) is
automatically scoped to allowed apps. `windows_list_windows` lists only
allowed apps' windows.
- Ref-less keyboard input (`windows_send_keys` / `windows_type` without a ref)
verifies the **foreground window** belongs to an allowed process first, and
the Windows key is rejected outright (it opens system UI like the Start menu
and Win+R outside any allowlist).
- `windows_screenshot` refuses `fullScreen` capture and foreground-window
capture of non-allowed apps, so other windows' content is not disclosed.

Scope honestly stated: matching is by process name, so this is a guard against
a misdirected or prompt-injected agent driving unintended apps *through this
server* — not a sandbox against a local attacker, and it does not restrict
anything the agent can do through other tools (like a shell). Dialogs owned by
the allowed process (including common file dialogs, which run in-process) keep
working; apps it launches as separate processes (e.g. a browser for OAuth) are
blocked unless also listed.

### Tool Examples

Send a keyboard chord to a target element:
Expand Down
5 changes: 3 additions & 2 deletions src/FlaUI.Mcp/Core/PendingInvokeTracker.cs
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,9 @@ public static string DescribeBlocked(PendingInvokeInfo info)
: "";
var elapsed = (int)(DateTime.UtcNow - info.StartedUtc).TotalSeconds;
return $"UI Automation for this app is blocked by a pending '{info.Description}' call " +
$"started {elapsed}s ago{modalPart}. UIA-based tools will hang until it completes. " +
"Interact with the dialog using windows_screenshot (fullScreen: true) to see it, " +
$"started {elapsed}s ago{modalPart}. Ref-based tools on this app will fail until it completes. " +
"To interact with the dialog: find its window handle via windows_list_windows (it is a separate " +
"window of the same process), see it with windows_screenshot using that handle, use " +
"windows_send_keys (without ref) for keyboard input, or dismiss it; then retry.";
}
}
148 changes: 148 additions & 0 deletions src/FlaUI.Mcp/Core/ProcessPolicy.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
namespace PlaywrightWindows.Mcp.Core;

/// <summary>
/// Optional allowlist restricting which applications the MCP tools may automate.
///
/// Configured via the FLAUI_MCP_ALLOWED_APPS environment variable: a semicolon-
/// or comma-separated list of process names, matched case-insensitively and with
/// or without an ".exe" suffix (full paths are reduced to their file name).
/// When the variable is unset or empty, everything is allowed (the historical
/// behavior). When set, the server refuses to launch, register windows for,
/// screenshot, or send input to any process not on the list.
///
/// Matching is by process name, so this is a guard against a misdirected or
/// prompt-injected agent driving unintended apps through this server - not a
/// sandbox against a local attacker who can rename executables.
/// </summary>
public sealed class ProcessPolicy
{
public const string EnvironmentVariable = "FLAUI_MCP_ALLOWED_APPS";

private readonly HashSet<string> _allowedNames;

/// <summary>A policy that allows every application (no allowlist configured).</summary>
public static ProcessPolicy AllowAll { get; } = new(Array.Empty<string>());

public ProcessPolicy(IEnumerable<string> allowedApps)
{
_allowedNames = allowedApps
.Select(Normalize)
.Where(name => name.Length > 0)
.ToHashSet(StringComparer.OrdinalIgnoreCase);
}

public static ProcessPolicy FromEnvironment()
{
var raw = Environment.GetEnvironmentVariable(EnvironmentVariable);
if (string.IsNullOrWhiteSpace(raw))
{
return AllowAll;
}

return new ProcessPolicy(raw.Split(
new[] { ';', ',' },
StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries));
}

/// <summary>True when an allowlist is configured; false means allow everything.</summary>
public bool IsRestricted => _allowedNames.Count > 0;

public bool IsNameAllowed(string? processName)
{
if (!IsRestricted)
{
return true;
}
if (string.IsNullOrWhiteSpace(processName))
{
return false;
}
return _allowedNames.Contains(Normalize(processName));
}

/// <summary>
/// Check a process by id. When restricted, an unknown or exited process is denied.
/// </summary>
public bool IsProcessAllowed(int processId)
{
if (!IsRestricted)
{
return true;
}
return IsNameAllowed(TryGetProcessName(processId));
}

/// <summary>
/// Check the executable path or name a launch request points at.
/// </summary>
public bool IsExecutableAllowed(string appPath)
{
if (!IsRestricted)
{
return true;
}
return IsNameAllowed(appPath);
}

/// <summary>
/// Verify the current foreground window's process against the allowlist.
/// Returns null when allowed (or unrestricted), otherwise an error message.
/// Used to gate ref-less keyboard input, which goes to whatever has focus.
/// </summary>
public string? CheckForegroundWindowAllowed()
{
if (!IsRestricted)
{
return null;
}

var processId = Win32Desktop.GetForegroundWindowProcessId();
if (IsProcessAllowed(processId))
{
return null;
}

var name = TryGetProcessName(processId) ?? "unknown";
return DescribeDenied($"The foreground window's process '{name}'") +
" Focus an allowed window first, or target an element ref directly.";
}

public string DescribeDenied(string subject)
{
return $"{subject} is not in the FlaUI-MCP app allowlist. " +
$"Only these apps can be automated: {string.Join(", ", _allowedNames.OrderBy(n => n, StringComparer.OrdinalIgnoreCase))} " +
$"(configured via the {EnvironmentVariable} environment variable).";
}

public static string? TryGetProcessName(int processId)
{
if (processId <= 0)
{
return null;
}
try
{
return System.Diagnostics.Process.GetProcessById(processId).ProcessName;
}
catch
{
return null;
}
}

private static string Normalize(string name)
{
var trimmed = name.Trim().Trim('"');
try
{
trimmed = Path.GetFileName(trimmed);
}
catch { /* keep the raw value if the entry is not path-like */ }

if (trimmed.EndsWith(".exe", StringComparison.OrdinalIgnoreCase))
{
trimmed = trimmed[..^4];
}
return trimmed;
}
}
50 changes: 44 additions & 6 deletions src/FlaUI.Mcp/Core/SessionManager.cs
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,24 @@ public class SessionManager : IDisposable
private readonly Dictionary<string, nint> _windowHwnds = new();
private readonly Dictionary<string, int> _windowPids = new();
private readonly Dictionary<nint, string> _hwndToHandle = new();
private readonly ProcessPolicy _processPolicy;
private int _windowCounter = 0;

public SessionManager()
public SessionManager(ProcessPolicy? processPolicy = null)
{
_automation = new UIA3Automation();
_processPolicy = processPolicy ?? ProcessPolicy.AllowAll;
}

public UIA3Automation Automation => _automation;

public (string handle, Window window) LaunchApp(string appPath, string[]? args = null)
{
if (!_processPolicy.IsExecutableAllowed(appPath))
{
throw new Exception(_processPolicy.DescribeDenied($"'{appPath}'"));
}

// Use Process.Start for more reliable launching
var psi = new System.Diagnostics.ProcessStartInfo
{
Expand Down Expand Up @@ -130,6 +137,8 @@ public string RegisterWindow(Window window)
}
catch { /* best effort */ }

EnsureProcessAllowed(pid);

if (hwnd != 0 && _hwndToHandle.TryGetValue(hwnd, out var existing))
{
_windows[existing] = window;
Expand All @@ -156,6 +165,8 @@ public string RegisterWindow(Window window)
/// </summary>
public string RegisterNativeWindow(nint hwnd, int processId)
{
EnsureProcessAllowed(processId);

if (_hwndToHandle.TryGetValue(hwnd, out var existing))
{
_windowPids[existing] = processId;
Expand Down Expand Up @@ -223,15 +234,16 @@ public nint GetWindowHwnd(string handle)
continue;
}

var handle = RegisterNativeWindow(info.Hwnd, info.ProcessId);
var processName = ProcessPolicy.TryGetProcessName(info.ProcessId);

string? processName = null;
try
// When an allowlist is active, windows of other apps are not listed
// at all - no handle is registered, so they stay unreachable.
if (!_processPolicy.IsNameAllowed(processName))
{
processName = System.Diagnostics.Process.GetProcessById(info.ProcessId).ProcessName;
continue;
}
catch { }

var handle = RegisterNativeWindow(info.Hwnd, info.ProcessId);
result.Add((handle, info.Title, processName));
}
return result;
Expand Down Expand Up @@ -281,6 +293,32 @@ public void CloseWindow(string handle)
_windowPids.Remove(handle);
}

/// <summary>
/// Throw when an app allowlist is active and the process is not on it (or
/// cannot be identified). Every window-handle registration funnels through
/// this, so refs and handles can only ever point at allowed apps.
/// </summary>
private void EnsureProcessAllowed(int processId)
{
if (!_processPolicy.IsRestricted)
{
return;
}

if (processId == 0)
{
throw new Exception(
"Cannot verify this window's owning process against the app allowlist " +
$"({ProcessPolicy.EnvironmentVariable}), so it is not controllable.");
}

if (!_processPolicy.IsProcessAllowed(processId))
{
var name = ProcessPolicy.TryGetProcessName(processId) ?? $"pid {processId}";
throw new Exception(_processPolicy.DescribeDenied($"Process '{name}'"));
}
}

public void Dispose()
{
foreach (var app in _applications.Values)
Expand Down
18 changes: 18 additions & 0 deletions src/FlaUI.Mcp/Core/Win32Desktop.cs
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ public static class Win32Desktop
[DllImport("user32.dll")]
private static extern bool GetWindowRect(nint hWnd, out RECT lpRect);

[DllImport("user32.dll")]
private static extern nint GetForegroundWindow();

[DllImport("dwmapi.dll")]
private static extern int DwmGetWindowAttribute(nint hWnd, int dwAttribute, out int pvAttribute, int cbAttribute);

Expand Down Expand Up @@ -140,6 +143,21 @@ public static void CloseWindow(nint hwnd)
PostMessage(hwnd, WM_CLOSE, 0, 0);
}

/// <summary>
/// Get the process id owning the current foreground window, or 0 if there
/// is no foreground window.
/// </summary>
public static int GetForegroundWindowProcessId()
{
var hwnd = GetForegroundWindow();
if (hwnd == 0)
{
return 0;
}
GetWindowThreadProcessId(hwnd, out var pid);
return (int)pid;
}

/// <summary>
/// Get a window's bounding rectangle in screen coordinates, or null on failure.
/// </summary>
Expand Down
15 changes: 10 additions & 5 deletions src/FlaUI.Mcp/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,13 @@

DpiUtility.EnablePerMonitorV2();

// Optional app allowlist: when FLAUI_MCP_ALLOWED_APPS is set (semicolon- or
// comma-separated process names, e.g. "TabularEditor3;notepad"), only those
// apps can be launched, listed, snapshotted, screenshotted or receive input.
var processPolicy = ProcessPolicy.FromEnvironment();

// Create shared services
var sessionManager = new SessionManager();
var sessionManager = new SessionManager(processPolicy);
var elementRegistry = new ElementRegistry();
var invokeTracker = new PendingInvokeTracker();

Expand All @@ -30,15 +35,15 @@
toolRegistry.RegisterTool(new LaunchTool(sessionManager));
toolRegistry.RegisterTool(new SnapshotTool(sessionManager, elementRegistry, invokeTracker));
toolRegistry.RegisterTool(new ClickTool(elementRegistry, invokeTracker));
toolRegistry.RegisterTool(new TypeTool(elementRegistry, invokeTracker));
toolRegistry.RegisterTool(new TypeTool(elementRegistry, invokeTracker, processPolicy));
toolRegistry.RegisterTool(new FillTool(elementRegistry, invokeTracker));
toolRegistry.RegisterTool(new GetTextTool(elementRegistry, invokeTracker));
toolRegistry.RegisterTool(new SendKeysTool(elementRegistry, invokeTracker));
toolRegistry.RegisterTool(new ScreenshotTool(sessionManager, elementRegistry, invokeTracker));
toolRegistry.RegisterTool(new SendKeysTool(elementRegistry, invokeTracker, processPolicy));
toolRegistry.RegisterTool(new ScreenshotTool(sessionManager, elementRegistry, invokeTracker, processPolicy));
toolRegistry.RegisterTool(new ListWindowsTool(sessionManager));
toolRegistry.RegisterTool(new FocusWindowTool(sessionManager));
toolRegistry.RegisterTool(new CloseWindowTool(sessionManager));
toolRegistry.RegisterTool(new BatchTool(sessionManager, elementRegistry, invokeTracker));
toolRegistry.RegisterTool(new BatchTool(sessionManager, elementRegistry, invokeTracker, processPolicy));

// Create and run MCP server
var server = new McpServer(toolRegistry);
Expand Down
Loading
Loading