Wayland MCP Server

Model Context Protocol server for Wayland desktop automation

Features • Installation • Usage • API • Security

Overview

Wayland MCP Server enables AI assistants to interact with your Wayland desktop through the Model Context Protocol. It provides screenshot capture with VLM analysis, mouse control, keyboard input, and action chaining capabilities.

Why This Project?

Existing Wayland screenshot and automation tools often have reliability issues. This project provides a robust, MCP-native solution specifically designed for AI-driven desktop automation on modern Linux systems.

Quick Example

# AI Assistant: "Take a screenshot and tell me what's on screen"
→ Captures screen, analyzes with VLM, responds with description

# AI Assistant: "Click the OK button"  
→ Identifies button location from screenshot, moves mouse, clicks

# AI Assistant: "Fill out this form with test data"
→ Chains clicks and keyboard input to complete form automatically

Features

Visual Analysis

Screenshot capture with precision ruler overlays
VLM-powered image analysis via OpenRouter or Google Gemini
Multiple vision model support (Claude, GPT-4V, Gemini, Qwen)
Side-by-side image comparison and diff detection

Mouse Automation

Absolute and relative cursor positioning
Click operations (left, right, middle button)
Drag and drop with coordinate precision
Bidirectional scrolling (vertical/horizontal)

Keyboard Control

Text input simulation
Individual key press events
Complex key combinations

Action Sequences

Chain multiple operations together
Flexible syntax: chain:action1;action2;action3
Example: chain:click:100,200;type:hello;press:Enter

Installation

Prerequisites

Python 3.8 or higher
Wayland compositor (GNOME, KDE Plasma, Hyprland, Sway, etc.)
grim and slurp for screenshots (usually pre-installed)

Quick Install

uvx wayland-mcp

From Source

git clone https://github.com/kurojs/wayland-mcp.git
cd wayland-mcp
pip install -e .

Input Control Setup

For mouse and keyboard automation, run the setup script:

sudo ./setup.sh

What it does:

Installs evemu-tools package
Configures setuid for evemu-event
Adds user to input group
Creates udev rules for device access

After setup, log out and back in for group changes to take effect.

Usage

MCP Configuration

The server supports two VLM providers:

Option 1: OpenRouter (multiple models via proxy)

{
  "mcpServers": {
    "wayland": {
      "command": "uvx",
      "args": ["wayland-mcp"],
      "env": {
        "OPENROUTER_API_KEY": "sk-or-v1-...",
        "VLM_PROVIDER": "openrouter",
        "VLM_MODEL": "qwen/qwen2.5-vl-72b-instruct:free",
        "XDG_RUNTIME_DIR": "/run/user/1000",
        "WAYLAND_DISPLAY": "wayland-0"
      }
    }
  }
}

Option 2: Google Gemini Direct (native API, faster)

{
  "mcpServers": {
    "wayland": {
      "command": "uvx",
      "args": ["wayland-mcp"],
      "env": {
        "GEMINI_API_KEY": "AIza...",
        "VLM_PROVIDER": "gemini",
        "VLM_MODEL": "gemini-2.5-flash",
        "XDG_RUNTIME_DIR": "/run/user/1000",
        "WAYLAND_DISPLAY": "wayland-0"
      }
    }
  }
}

Example for Claude Desktop (~/.config/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "wayland": {
      "command": "uvx",
      "args": ["wayland-mcp"],
      "env": {
        "GEMINI_API_KEY": "AIza...",
        "VLM_PROVIDER": "gemini",
        "VLM_MODEL": "gemini-2.5-flash",
        "XDG_RUNTIME_DIR": "/run/user/1000",
        "WAYLAND_DISPLAY": "wayland-0"
      }
    }
  }
}

Note: See CONFIG_EXAMPLES.md for more configuration examples including Cursor, OpenRouter models, and VLM provider options.

Environment Variables

Variable	Description	Default	Required
VLM Provider Options
`VLM_PROVIDER`	Vision provider: `openrouter` or `gemini`	`openrouter`	No
`OPENROUTER_API_KEY`	OpenRouter API key	-	For OpenRouter
`GEMINI_API_KEY`	Google Gemini API key	-	For Gemini
`VLM_MODEL`	Model identifier	`qwen/qwen2.5-vl-72b-instruct:free` (OpenRouter) or `gemini-2.5-flash` (Gemini)	No
Wayland Environment
`XDG_RUNTIME_DIR`	Wayland runtime directory	`/run/user/1000`	Yes
`WAYLAND_DISPLAY`	Display identifier	`wayland-0`	Yes
Optional
`WAYLAND_MCP_PORT`	Server listen port	`4999`	No

Getting API Keys:

OpenRouter: openrouter.ai → Keys section
Google Gemini: Google AI Studio

Desktop Environment Compatibility

Desktop	Status	Notes
GNOME	✅ Tested	Wayland by default on modern versions
KDE Plasma	✅ Tested	Enable Wayland session at login
Hyprland	✅ Tested	Native Wayland compositor
Sway	✅ Should work	i3-compatible Wayland compositor
Others	⚠️ Untested	Any wlroots-based compositor should work

Example Commands

Through an MCP client, you can request actions like:

"Take a screenshot and analyze what's on the screen"
"Move the mouse to coordinates (100, 200) and click"
"Type 'hello world' and press Enter"
"Click at (50, 50), then drag to (200, 200)"

Available Tools

The server exposes the following MCP tools:

Screen Capture

capture_screenshot - Take a screenshot with optional ruler overlays
capture_and_analyze - Capture and analyze using VLM in one step

Vision Analysis

analyze_screenshot - Analyze an existing screenshot with custom prompt
compare_images - Compare two screenshots to detect differences

Mouse Control

move_mouse - Move cursor to coordinates (absolute or relative)
click_mouse - Perform left click at current position
drag_mouse - Drag between two coordinate points
scroll_mouse - Vertical scroll (positive=up, negative=down)

Action Execution

execute_action - Execute single action or chain multiple actions

Action Chain Syntax

Combine multiple actions with semicolons:

chain:action1;action2;action3

Supported Actions:

type:text - Type a text string
press:key - Press a specific key
click: or click:x,y - Click at position or current location
move_to:x,y - Move to absolute coordinates
move_to:rel:x,y - Move relative to current position
drag:x1,y1:x2,y2 - Drag from point to point
scroll:amount - Scroll vertically (typical values: 15-120)
scroll:horizontal:amount - Scroll horizontally

Example Chains:

chain:move_to:100,200;click:;type:hello;press:Enter
chain:click:50,50;drag:50,50:200,200
chain:scroll:120;move_to:rel:0,-50;click:

Security

⚠️ IMPORTANT SECURITY CONSIDERATIONS

This server grants extensive control over your desktop environment:

Full mouse and keyboard control
Screen capture capabilities
Ability to execute arbitrary input sequences

Best Practices

Only use with trusted AI models and MCP clients
Review action chains before execution in sensitive contexts
Consider running in a sandboxed or test environment
Be aware that the AI can perform any action you could perform manually

Permission Model

The setup script requires sudo access to:

Install system packages (evemu-tools)
Modify file permissions
Configure udev rules

After setup, the server runs with your user privileges but can control input devices through configured permissions.

Architecture

                    ┌─────────────────────────────────┐
                    │      MCP Client Layer           │
                    │   (Claude, Cursor, VS Code)     │
                    └───────────────┬─────────────────┘
                                    │
                            MCP Protocol (stdio/HTTP)
                                    │
                    ┌───────────────▼─────────────────┐
                    │    Wayland MCP Server           │
                    │    ┌─────────────────────┐      │
                    │    │  Core Components    │      │
                    │    ├─────────────────────┤      │
                    │    │ • FastMCP Handler   │      │
                    │    │ • Action Processor  │      │
                    │    │ • Chain Parser      │      │
                    │    └─────────────────────┘      │
                    └────┬────────────────┬────────────┘
                         │                │
         ┌───────────────┴────┐      ┌────┴──────────────┐
         │                    │      │                   │
    ┌────▼─────┐      ┌──────▼───┐  │  ┌──────────────┐ │
    │  Vision  │      │  Input   │  │  │   Screen     │ │
    │          │      │ Control  │  │  │   Capture    │ │
    ├──────────┤      ├──────────┤  │  ├──────────────┤ │
    │ • VLM    │      │ • evemu  │  │  │ • grim       │ │
    │ • Compare│      │ • Mouse  │  │  │ • slurp      │ │
    │          │      │ • Keyboard│  │  │ • PIL        │ │
    └──────────┘      └──────────┘  │  └──────────────┘ │
                                    │                    │
                                    └────────────────────┘
                                      Wayland Compositor

Troubleshooting

Input control not working

Ensure you ran sudo ./setup.sh
Log out and back in after setup
Verify you're in the input group: groups | grep input

Screenshots failing

Check if grim is installed: which grim
Verify WAYLAND_DISPLAY matches your session: echo $WAYLAND_DISPLAY

VLM analysis not working

Confirm OPENROUTER_API_KEY is set correctly
Check API key permissions on OpenRouter dashboard
Test model availability: some models have usage limits

Server won't start

Check Python version: python3 --version (needs 3.8+)
Verify all dependencies: pip install -e .
Look for port conflicts if using custom WAYLAND_MCP_PORT

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Project Structure

wayland-mcp/
├── wayland_mcp/          # Main package
│   ├── server_mcp.py     # MCP server implementation
│   ├── screen_utils.py   # Screenshot & VLM analysis
│   ├── mouse_utils.py    # Mouse control functions
│   ├── keyboard_utils.py # Keyboard input handling
│   ├── chain_processor.py# Action chain parser
│   └── ...
├── README.md             # This file
├── CONFIG_EXAMPLES.md    # Configuration examples
├── CONTRIBUTING.md       # Contribution guidelines
├── setup.sh              # Permission setup script
└── pyproject.toml        # Package metadata

License

GPL-3.0 License - See LICENSE for details.

Acknowledgments

Built on the Model Context Protocol
Uses FastMCP for server implementation
Inspired by the need for reliable Wayland automation tools

Made for the Wayland desktop environment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Wayland MCP Server

Overview

Why This Project?

Quick Example

Features

Installation

Prerequisites

Quick Install

From Source

Input Control Setup

Usage

MCP Configuration

Environment Variables

Desktop Environment Compatibility

Example Commands

Available Tools

Screen Capture

Vision Analysis

Mouse Control

Action Execution

Action Chain Syntax

Security

Best Practices

Permission Model

Architecture

Troubleshooting

Contributing

Project Structure

License

Acknowledgments

About

Uh oh!

Contributors

Uh oh!

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 34 Commits
wayland_mcp		wayland_mcp
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONFIG_EXAMPLES.md		CONFIG_EXAMPLES.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml
setup.sh		setup.sh

Folders and files

Latest commit

History

Repository files navigation

Wayland MCP Server

Overview

Why This Project?

Quick Example

Features

Installation

Prerequisites

Quick Install

From Source

Input Control Setup

Usage

MCP Configuration

Environment Variables

Desktop Environment Compatibility

Example Commands

Available Tools

Screen Capture

Vision Analysis

Mouse Control

Action Execution

Action Chain Syntax

Security

Best Practices

Permission Model

Architecture

Troubleshooting

Contributing

Project Structure

License

Acknowledgments

About

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Contributors

Uh oh!

Languages