Skip to content

Commit f419e63

Browse files
authored
Merge pull request #265 from microsoft/vyokky/dev
Mobile Agent
2 parents 0ea43c7 + 61f5ad0 commit f419e63

52 files changed

Lines changed: 12496 additions & 155 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

README.md

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,7 @@ UFO³ introduces **Galaxy**, a revolutionary multi-device orchestration framewor
141141
| **Task Model** | Sequential ReAct Loop | DAG-based Constellation Workflows |
142142
| **Scope** | Single device, multi-app | Multi-device, cross-platform |
143143
| **Coordination** | HostAgent + AppAgents | ConstellationAgent + TaskOrchestrator |
144-
| **Device Support** | Windows Desktop | Windows, Linux, macOS, Android, Web |
144+
| **Device Support** | Windows Desktop | Windows, Linux, Android (more coming) |
145145
| **Task Planning** | Application-level | Device-level with dependencies |
146146
| **Execution** | Sequential | Parallel DAG execution |
147147
| **Device Agent Role** | Standalone | Can serve as Galaxy device agent |
@@ -280,23 +280,26 @@ pip install -r requirements.txt
280280
copy config\galaxy\agent.yaml.template config\galaxy\agent.yaml
281281
# Edit and add your API keys
282282
283-
# 3. Start device agents (with platform flags)
284-
# Windows:
285-
python -m ufo.server.app --port 5000
286-
python -m ufo.client.client --ws --ws-server ws://localhost:5000/ws --client-id windows_device_1 --platform windows
283+
# 3. Configure devices
284+
# Edit config\galaxy\devices.yaml to register your devices
287285
288-
# Linux:
289-
python -m ufo.server.app --port 5001
290-
python -m ufo.client.client --ws --ws-server ws://localhost:5001/ws --client-id linux_device_1 --platform linux
286+
# 4. Start device agents (with platform flags)
287+
# Windows: Start server + client
288+
# Linux: Start server + MCP servers + client
289+
# Mobile (Android): Start server + MCP servers + client
290+
# See platform-specific guides for detailed setup
291291
292-
# 4. Launch Galaxy
292+
# 5. Launch Galaxy
293293
python -m galaxy --interactive
294294
```
295295

296296
**📖 Complete Guide:**
297297
- [Galaxy README](./galaxy/README.md) – Architecture & concepts
298298
- [Online Quick Start](https://microsoft.github.io/UFO/getting_started/quick_start_galaxy/) – Step-by-step tutorial
299-
- [Configuration](https://microsoft.github.io/UFO/configuration/system/galaxy_devices/) – Device setup
299+
- [Windows Device Setup](https://microsoft.github.io/UFO/getting_started/quick_start_ufo2/)
300+
- [Linux Device Setup](https://microsoft.github.io/UFO/getting_started/quick_start_linux/)
301+
- [Mobile Device Setup](https://microsoft.github.io/UFO/getting_started/quick_start_mobile/) – Android agent setup
302+
- [Configuration](https://microsoft.github.io/UFO/configuration/system/galaxy_devices/) – Device pool configuration
300303

301304
</td>
302305
<td width="50%" valign="top">

README_ZH.md

Lines changed: 15 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -136,7 +136,7 @@ UFO³ 引入了 **Galaxy**,这是一个革命性的多设备编排框架,可
136136
| **任务模型** | 顺序 ReAct 循环 | 基于 DAG 的星座工作流 |
137137
| **范围** | 单设备,多应用 | 多设备,跨平台 |
138138
| **协调** | HostAgent + AppAgents | ConstellationAgent + TaskOrchestrator |
139-
| **设备支持** | Windows 桌面 | Windows、Linux、macOS、Android、Web |
139+
| **设备支持** | Windows 桌面 | Windows、Linux、Android(更多平台即将推出) |
140140
| **任务规划** | 应用程序级别 | 设备级别,带依赖关系 |
141141
| **执行** | 顺序 | 并行 DAG 执行 |
142142
| **设备智能体角色** | 独立 | 可作为 Galaxy 设备智能体 |
@@ -268,30 +268,33 @@ UFO² 扮演双重角色:**独立 Windows 自动化**和 Windows 平台的 **G
268268
**用于跨设备编排**
269269

270270
```powershell
271-
# 1. 安装
271+
# 1. 安装依赖
272272
pip install -r requirements.txt
273273
274274
# 2. 配置 ConstellationAgent
275275
copy config\galaxy\agent.yaml.template config\galaxy\agent.yaml
276-
# 编辑并添加您的 API 密钥
276+
# 编辑配置文件,添加 API Key
277277
278-
# 3. 启动设备智能体(带平台标志)
279-
# Windows:
280-
python -m ufo.server.app --port 5000
281-
python -m ufo.client.client --ws --ws-server ws://localhost:5000/ws --client-id windows_device_1 --platform windows
278+
# 3. 配置设备
279+
# 编辑 config\galaxy\devices.yaml 注册您的设备
282280
283-
# Linux:
284-
python -m ufo.server.app --port 5001
285-
python -m ufo.client.client --ws --ws-server ws://localhost:5001/ws --client-id linux_device_1 --platform linux
281+
# 4. 启动设备智能体(带平台标志)
282+
# Windows: 启动服务器 + 客户端
283+
# Linux: 启动服务器 + MCP 服务器 + 客户端
284+
# Mobile (Android): 启动服务器 + MCP 服务器 + 客户端
285+
# 请参阅特定平台指南了解详细设置
286286
287-
# 4. 启动 Galaxy
287+
# 5. 启动 Galaxy
288288
python -m galaxy --interactive
289289
```
290290

291291
**📖 完整指南:**
292292
- [Galaxy 中文文档](./galaxy/README_ZH.md) – 架构和概念
293293
- [在线快速入门](https://microsoft.github.io/UFO/getting_started/quick_start_galaxy/) – 分步教程
294-
- [配置](https://microsoft.github.io/UFO/configuration/system/galaxy_devices/) – 设备设置
294+
- [Windows 设备设置](https://microsoft.github.io/UFO/getting_started/quick_start_ufo2/)
295+
- [Linux 设备设置](https://microsoft.github.io/UFO/getting_started/quick_start_linux/)
296+
- [Mobile 设备设置](https://microsoft.github.io/UFO/getting_started/quick_start_mobile/) – Android 智能体设置
297+
- [配置](https://microsoft.github.io/UFO/configuration/system/galaxy_devices/) – 设备池配置
295298

296299
</td>
297300
<td width="50%" valign="top">

aip/messages.py

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -474,3 +474,82 @@ def validate_server_message(msg: ServerMessage) -> bool:
474474
return False
475475

476476
return True
477+
478+
479+
# ============================================================================
480+
# Binary Transfer Message Types (New Feature)
481+
# ============================================================================
482+
483+
484+
class BinaryMetadata(BaseModel):
485+
"""
486+
Metadata for binary data transfer.
487+
488+
This metadata is sent as a text frame before the actual binary data,
489+
allowing receivers to prepare for and validate incoming binary transfers.
490+
"""
491+
492+
type: Literal["binary_data"] = "binary_data"
493+
filename: Optional[str] = None
494+
mime_type: Optional[str] = None
495+
size: int = Field(..., description="Size of binary data in bytes")
496+
checksum: Optional[str] = Field(
497+
None, description="MD5 or SHA256 checksum for validation"
498+
)
499+
session_id: Optional[str] = None
500+
description: Optional[str] = None
501+
timestamp: Optional[str] = None
502+
# Allow additional custom fields
503+
model_config = ConfigDict(extra="allow")
504+
505+
506+
class FileTransferStart(BaseModel):
507+
"""
508+
Message to initiate a chunked file transfer.
509+
510+
Sent before sending file chunks to inform the receiver about
511+
the file details and transfer parameters.
512+
"""
513+
514+
type: Literal["file_transfer_start"] = "file_transfer_start"
515+
filename: str = Field(..., description="Name of file being transferred")
516+
size: int = Field(..., description="Total file size in bytes")
517+
chunk_size: int = Field(..., description="Size of each chunk in bytes")
518+
total_chunks: int = Field(..., description="Total number of chunks")
519+
mime_type: Optional[str] = Field(None, description="MIME type of file")
520+
session_id: Optional[str] = None
521+
description: Optional[str] = None
522+
# Allow additional custom fields
523+
model_config = ConfigDict(extra="allow")
524+
525+
526+
class FileTransferComplete(BaseModel):
527+
"""
528+
Message to signal completion of a chunked file transfer.
529+
530+
Sent after all file chunks have been transmitted, includes
531+
checksum for validation.
532+
"""
533+
534+
type: Literal["file_transfer_complete"] = "file_transfer_complete"
535+
filename: str = Field(..., description="Name of transferred file")
536+
total_chunks: int = Field(..., description="Total chunks sent")
537+
checksum: Optional[str] = Field(None, description="MD5 checksum of complete file")
538+
session_id: Optional[str] = None
539+
# Allow additional custom fields
540+
model_config = ConfigDict(extra="allow")
541+
542+
543+
class ChunkMetadata(BaseModel):
544+
"""
545+
Metadata for a single file chunk.
546+
547+
Sent with each chunk during chunked file transfer to track
548+
chunk sequence and validate chunk integrity.
549+
"""
550+
551+
chunk_num: int = Field(..., description="Chunk sequence number (0-indexed)")
552+
chunk_size: int = Field(..., description="Size of this chunk in bytes")
553+
checksum: Optional[str] = Field(None, description="Checksum of this chunk")
554+
# Allow additional custom fields
555+
model_config = ConfigDict(extra="allow")

0 commit comments

Comments
 (0)