diff --git a/.ai/README.md b/.ai/README.md new file mode 100644 index 00000000000..2c67d9c56a1 --- /dev/null +++ b/.ai/README.md @@ -0,0 +1,188 @@ +# Linkis AI 开发文档导航 + +> **版本信息** +> - 文档版本: 1.0.0 +> - 最后更新: 2025-01-28 +> - 适用版本: Apache Linkis 1.17.0+ + +--- + +## 🚀 快速开始 + +### 新手必读(按顺序阅读) +1. **[项目核心规约](./project-context.md)** - 包含技术栈、架构设计、开发规范和模板 +2. **[强制性开发规则](./rules.md)** - 必须无条件遵守的开发规则 +3. **[模块文档](#模块文档索引)** - 根据你要开发的功能选择对应模块 + +### 常见开发场景快速跳转 +- 🆕 新增 REST 接口 → [REST接口开发模板](#rest接口开发) +- ⚙️ 添加配置项 → [配置管理规范](#配置管理) +- 🗄️ 修改数据库 → [数据库变更规范](#数据库变更) +- 🐛 异常处理 → [异常处理规范](#异常处理) +- 📝 日志记录 → [日志规范](#日志规范) + +--- + +## 📚 核心文档索引 + +### 🎯 开发规范文档 +| 文档 | 用途 | 何时查看 | +|------|------|----------| +| [project-context.md](./project-context.md) | 项目角色定位、技术栈、架构设计、开发模板 | 开始任何开发工作前必读 | +| [rules.md](./rules.md) | 强制性开发规则、需求实现步骤 | 每次开发新需求时参考 | + +### 🏗️ 模块文档索引 + +#### 微服务治理服务(基础设施层) +| 服务 | 文档 | 主要功能 | +|------|------|----------| +| Gateway | [gateway.md](./modules/microservice-governance/gateway.md) | API网关、路由转发、安全认证 | +| Eureka | [eureka.md](./modules/microservice-governance/eureka.md) | 服务注册与发现 | +| 概览 | [README.md](./modules/microservice-governance/README.md) | 微服务治理服务概述 | + +#### 计算治理服务(核心业务层) +| 服务 | 文档 | 主要功能 | +|------|------|----------| +| Entrance | [entrance.md](./modules/computation-governance/entrance.md) | 任务提交入口、调度管理 | +| JobHistory | [jobhistory.md](./modules/computation-governance/jobhistory.md) | 任务历史记录查询 | +| Manager | [manager.md](./modules/computation-governance/manager.md) | 资源管理、应用管理 | +| ECM | [ecm.md](./modules/computation-governance/ecm.md) | 引擎连接管理 | +| 概览 | [README.md](./modules/computation-governance/README.md) | 计算治理服务概述 | + +#### 公共增强服务(支撑服务层) +| 服务 | 文档 | 主要功能 | +|------|------|----------| +| PublicService | [publicservice.md](./modules/public-enhancements/publicservice.md) | 公共服务、文件管理 | +| Configuration | [configuration.md](./modules/public-enhancements/configuration.md) | 配置管理 | +| BML | [bml.md](./modules/public-enhancements/bml.md) | 大数据物料库 | +| DataSource | [datasource.md](./modules/public-enhancements/datasource.md) | 数据源管理 | +| Context | [context.md](./modules/public-enhancements/context.md) | 上下文服务 | +| Monitor | [monitor.md](./modules/public-enhancements/monitor.md) | 监控服务 | +| 概览 | [README.md](./modules/public-enhancements/README.md) | 公共增强服务概述 | + +--- + +## 🔍 按功能快速查找 + +### REST接口开发 +- **开发模板**: [project-context.md - REST接口层](./project-context.md#1-rest接口层) +- **API规范**: [project-context.md - API设计规范](./project-context.md#api设计规范) +- **参考示例**: + - Entrance接口: [entrance.md - API Interfaces](./modules/computation-governance/entrance.md#api-interfaces) + - Configuration接口: [configuration.md - API Interfaces](./modules/public-enhancements/configuration.md#api-interfaces) + +### 配置管理 +- **配置规范**: [project-context.md - 配置管理规范](./project-context.md#配置管理规范) +- **配置示例库**: [project-context.md - 常用配置示例库](./project-context.md#常用配置示例库) +- **配置模板**: [project-context.md - 配置类](./project-context.md#4-配置类) +- **参考实现**: linkis-jobhistory/conf/JobhistoryConfiguration + +### 数据库变更 +- **变更规则**: [rules.md - 数据库修改原则](./rules.md#数据库修改原则) +- **DDL脚本位置**: `linkis-dist/package/db/linkis_ddl.sql` +- **DML脚本位置**: `linkis-dist/package/db/linkis_dml.sql` +- **表结构参考**: 各模块文档的 "Database Table Structures" 章节 + +### 异常处理 +- **异常规范**: [project-context.md - 异常处理规范](./project-context.md#异常处理规范) +- **统一异常**: `org.apache.linkis.common.exception.LinkisException` +- **常见错误**: [project-context.md - 常见错误及避免方法](./project-context.md#常见错误及避免方法) + +### 日志规范 +- **日志规范**: [project-context.md - 日志规范](./project-context.md#日志规范) +- **Logger定义**: 必须使用 `LoggerFactory.getLogger(ClassName.class)` +- **日志级别**: ERROR/WARN/INFO/DEBUG 使用场景 + +--- + +## 🎨 开发模板快速复制 + +### 新增功能完整流程 +``` +1. 查看 rules.md - 需求实现步骤 +2. 创建需求文档和设计文档 +3. 使用 project-context.md 中的代码模板: + - REST接口层模板 + - 服务层模板 + - 数据访问层模板 + - 配置类模板 +4. 添加功能开关(默认false) +5. 记录数据库变更 +6. 编写测试和文档 +``` + +### REST接口模板快速链接 +👉 [project-context.md - 新功能开发模板](./project-context.md#新功能开发模板) + +### 配置类模板快速链接 +👉 [project-context.md - 配置类](./project-context.md#4-配置类) + +--- + +## ⚠️ 重要提醒 + +### 🚫 禁止操作(来自 rules.md) +- **数据库结构**: 除非明确指定,严禁修改现有表结构 +- **第三方依赖**: 不允许引入新的第三方依赖库 +- **核心接口**: 不得修改现有公共接口的签名 + +### ✅ 必须遵守 +- **最小改动原则**: 所有功能实现必须遵循最小改动原则 +- **功能可配置**: 所有功能必须增加功能开关,默认关闭 +- **向后兼容**: 新增功能必须考虑向后兼容性 + +--- + +## 💡 开发技巧 + +### 编程语言选择 +- **Java**: REST API、Service层、Entity类、配置类 +- **Scala**: 计算逻辑、RPC通信、复杂业务处理、配置对象 + +### 字符编码 +统一使用 `StandardCharsets.UTF_8`,禁止使用字符串 `"UTF-8"` + +### 统一返回体 +所有REST接口返回 `org.apache.linkis.server.Message` + +--- + +## 📖 如何使用这些文档 + +### 场景1: 我要在 Entrance 服务中新增一个接口 +1. 阅读 [entrance.md](./modules/computation-governance/entrance.md) 了解服务结构 +2. 查看 [project-context.md - REST接口层模板](./project-context.md#1-rest接口层) +3. 参考 entrance.md 中现有接口实现 +4. 遵循 [rules.md](./rules.md) 中的开发规则 +5. 添加功能开关配置 + +### 场景2: 我需要添加一个新的配置项 +1. 查看 [project-context.md - 配置管理规范](./project-context.md#配置管理规范) +2. 参考 [project-context.md - 配置类模板](./project-context.md#4-配置类) +3. 查看 `JobhistoryConfiguration` 实现示例 +4. 在当前模块的 conf 目录下的 Configuration 类中添加 + +### 场景3: 我需要修改数据库表 +1. 查看 [rules.md - 数据库修改原则](./rules.md#数据库修改原则) +2. 确认是否能通过新增字段实现(优先选择) +3. 将变更记录到 `linkis-dist/package/db/linkis_ddl.sql` +4. 如有初始化数据,记录到 `linkis-dist/package/db/linkis_dml.sql` + +--- + +## 🔄 文档更新记录 + +| 版本 | 日期 | 更新内容 | 更新人 | +|------|------|----------|--------| +| 1.0.0 | 2025-01-28 | 创建导航文档,优化文档结构 | AI | + +--- + +## 📞 帮助与反馈 + +如果文档中有不清楚的地方,请: +1. 先查看对应模块的详细文档 +2. 查看 project-context.md 中的开发模板和示例 +3. 参考现有代码实现 + +**记住**: 遵循规范比快速开发更重要! diff --git a/.ai/modules/computation-governance/README.md b/.ai/modules/computation-governance/README.md new file mode 100644 index 00000000000..1f39355e663 --- /dev/null +++ b/.ai/modules/computation-governance/README.md @@ -0,0 +1,157 @@ +# Computation Governance Services + +The computation governance services handle the core computation task lifecycle management in Linkis. + +## Service Modules + +- [Entrance Service](./entrance.md) - Task submission and entrance point +- [Manager Service](./manager.md) - Resource and application management +- [ECM Service](./ecm.md) - Engine Connection Manager +- [JobHistory Service](./jobhistory.md) - Task execution history tracking + +## Overview + +These services form the core of Linkis' computation governance capabilities, managing the complete lifecycle of computation tasks from submission to execution and monitoring. + +## Common Features + +### Task Lifecycle Management +- Task submission and validation +- Task scheduling and resource allocation +- Task execution monitoring +- Task result management +- Task error handling and recovery + +### Engine Management +- Dynamic engine connection creation +- Engine lifecycle management +- Engine resource monitoring +- Engine scaling capabilities + +### Resource Governance +- Multi-tenant resource isolation +- Load balancing across engines +- Resource usage tracking +- Quota management + +## API Interface Summary + +### Entrance Service APIs +- Task submission: `POST /api/entrance/submit` +- Task status query: `GET /api/entrance/{id}/status` +- Task progress: `GET /api/entrance/{id}/progress` +- Task log retrieval: `GET /api/entrance/{id}/log` +- Task cancellation: `GET /api/entrance/{id}/kill` + +### Manager Service APIs +- Engine instance management +- Resource allocation and monitoring +- Node status querying +- Engine creation requests + +### ECM Service APIs +- Engine connection management +- Engine lifecycle operations +- Resource reporting +- Engine metrics collection + +### JobHistory Service APIs +- Job history querying +- Job detail retrieval +- Job statistics reporting + +## Database Schema Summary + +### Job History Group Table +```sql +CREATE TABLE `linkis_ps_job_history_group_history` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job', + `execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job', + `source` text DEFAULT NULL COMMENT 'job source', + `labels` text DEFAULT NULL COMMENT 'job labels', + `params` text DEFAULT NULL COMMENT 'job params', + `progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log', + `error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances', + `metrics` text DEFAULT NULL COMMENT 'Job Metrics', + `engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type', + `execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job', + PRIMARY KEY (`id`), + KEY `idx_created_time` (`created_time`), + KEY `idx_submit_user` (`submit_user`) +); +``` + +### Job History Detail Table +```sql +CREATE TABLE `linkis_ps_job_history_detail` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job', + `result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array', + `job_group_info` text DEFAULT NULL COMMENT 'Job group info/path', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `status` varchar(32) DEFAULT NULL COMMENT 'status', + `priority` int(4) DEFAULT 0 COMMENT 'order of subjob', + PRIMARY KEY (`id`) +); +``` + +### Common Lock Table +```sql +CREATE TABLE `linkis_ps_common_lock` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `lock_object` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `locker` VARCHAR(255) CHARSET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL COMMENT 'locker', + `time_out` longtext COLLATE utf8_bin, + `update_time` datetime DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_lock_object` (`lock_object`) +); +``` + +## RPC Methods Summary + +### Entrance Service RPCs +- `submitTask(TaskRequest request)` +- `getTaskStatus(String taskId)` +- `cancelTask(String taskId)` +- `getTaskResult(String taskId)` + +### Manager Service RPCs +- `requestEngine(EngineRequest request)` +- `releaseEngine(String engineId)` +- `getEngineStatus(String engineId)` +- `getNodeMetrics(String nodeId)` + +### ECM Service RPCs +- `createEngineConnection(EngineCreateRequest request)` +- `terminateEngineConnection(String engineId)` +- `reportEngineResourceUsage(String engineId, ResourceUsage usage)` +- `getEngineMetrics(String engineId)` + +### JobHistory Service RPCs +- `saveJobHistory(JobHistory history)` +- `queryJobHistory(JobHistoryQuery query)` +- `getJobDetails(Long jobId)` +- `updateJobStatus(Long jobId, String status)` + +## Dependencies + +- linkis-commons - Shared utilities +- linkis-protocol - Communication protocols +- linkis-rpc - Remote procedure calls +- Various engine connection plugins +- Spring Cloud ecosystem \ No newline at end of file diff --git a/.ai/modules/computation-governance/ecm.md b/.ai/modules/computation-governance/ecm.md new file mode 100644 index 00000000000..134c9817838 --- /dev/null +++ b/.ai/modules/computation-governance/ecm.md @@ -0,0 +1,635 @@ +# ECM Service + +The ECM (Engine Connection Manager) service manages the lifecycle of engine connections in the Linkis system. + +## Overview + +This service is responsible for managing the lifecycle of engine connections, including creating, starting, stopping, and monitoring engine instances. + +## Key Components + +### Core Classes +- `LinkisECMApplication` - Main application class +- Engine connection lifecycle management +- Engine resource monitoring +- Engine health checking + +### Features +- Engine connection creation and initialization +- Engine lifecycle management +- Resource allocation for engines +- Engine monitoring and health checking +- Engine termination and cleanup + +## API Interfaces + +### Download Engine Log +``` +GET /api/rest_j/v1/engineconnManager/downloadEngineLog +``` + +Parameters: +- `emInstance`: ECM instance (required) +- `instance`: Engine instance (required) +- `logDirSuffix`: Log directory suffix (required) +- `logType`: Log type (required) - stdout, stderr, gc, or yarnApp + +Response: +``` +Binary file download (log file content) +``` + +Error Codes: +- 11110: Log directory {0} does not exists.(日志目录 {0} 不存在.) +- 911115: failed to downLoad(下载失败) +- 911116: Download file has exceeded 100MB(下载文件已超过100M) +- 911117: Parameter {0} cannot be empty (参数 {0} 不能为空) +- 911118: logType only supports stdout, stderr, gc, yarnApp(logType仅支持stdout,stderr,gc,yarnApp) +- 911119: You {0} have no permission to download Log in ECM {1}(用户 {0} 无权限下载 ECM {1} 日志) + +Notes: +- Only supports GET method due to gateway forwarding rules +- File size limit is 100MB +- Supported log types: stdout, stderr, gc, yarnApp +- Requires user authentication and authorization checks +- Filename format in response: {instance}_{logType}.txt + +### List All ECMs +``` +GET /api/rest_j/v1/linkisManager/listAllEMs +``` + +Parameters: +- `instance`: ECM instance filter (optional) +- `nodeHealthy`: Node healthy status filter (optional) +- `owner`: Owner filter (optional) +- `tenantLabel`: Tenant label filter (optional) + +Response: +```json +{ + "method": "/api/linkisManager/listAllEMs", + "status": 0, + "message": "OK", + "data": { + "EMs": [ + { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "gz.bdz.bdplxxxxx.apache:9102", + "nodeHealthy": "Healthy", + "labels": [ + { + "stringValue": "gz.bdz.bdplxxxxx.apache:9102", + "labelKey": "emInstance" + } + ], + "owner": "hadoop", + "nodeStatus": "Healthy" + } + ] + } +} +``` + +Error Codes: +- 210003: Only admin can modify ECMs(只有管理员才能修改ECM) + +Notes: +- Requires admin privileges +- Returns list of all ECM instances with their status and labels + +### List All ECM Healthy Status +``` +GET /api/rest_j/v1/linkisManager/listAllECMHealthyStatus +``` + +Parameters: +- `onlyEditable`: Boolean flag to return only editable statuses (optional) + +Response: +```json +{ + "method": "/api/linkisManager/listAllECMHealthyStatus", + "status": 0, + "message": "OK", + "data": { + "nodeHealthy": [ + "Healthy", + "UnHealthy", + "WARN", + "StockAvailable", + "StockUnavailable" + ] + } +} +``` + +Notes: +- Returns all possible ECM healthy status values +- When `onlyEditable` is true, returns only the statuses that can be modified + +### Modify ECM Info +``` +PUT /api/rest_j/v1/linkisManager/modifyEMInfo +``` + +Parameters: +- `applicationName`: Application name (optional) +- `emStatus`: ECM status (optional) +- `instance`: ECM instance (required) +- `labels`: Labels list (optional) +- `labelKey`: Label key (optional) +- `description`: Description (optional) +- `stringValue`: String value (optional) + +Response: +```json +{ + "method": "/api/linkisManager/modifyEMInfo", + "status": 0, + "message": "success" +} +``` + +Error Codes: +- 210003: Failed to update label, include repeat labels(更新label失败,包含重复label) + +Notes: +- Allows modification of ECM instance information +- Supports updating labels, status, and description + +### Execute ECM Operation +``` +POST /api/rest_j/v1/linkisManager/executeECMOperation +``` + +Request Body: +```json +{ + "serviceInstance": { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "gz.bdz.bdplxxxxx.apache:9102" + }, + "parameters": { + // Operation specific parameters + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeECMOperation", + "status": 0, + "message": "OK", + "data": { + // Operation result data + } +} +``` + +Error Codes: +- Various operation-specific error codes + +Notes: +- Executes administrative operations on ECM instances +- Requires appropriate permissions +- Operation parameters vary based on the specific operation being performed + +### Execute ECM Operation by Engine Connection +``` +POST /api/rest_j/v1/linkisManager/executeECMOperationByEC +``` + +Request Body: +```json +{ + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "gz.bdz.bdplxxxxx.apache:12295" + }, + "parameters": { + // Operation specific parameters + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeECMOperationByEC", + "status": 0, + "message": "OK", + "data": { + // Operation result data + } +} +``` + +Error Codes: +- Permission-related errors when user doesn't own the engine connection + +Notes: +- Executes ECM operations triggered by engine connections +- Validates that the user owns the engine connection or is an admin +- Operation parameters vary based on the specific operation being performed + +### Reset Resource +``` +GET /api/rest_j/v1/linkisManager/reset-resource +``` + +Parameters: +- `serviceInstance`: ECM service instance (optional) +- `username`: Username (optional) + +Response: +```json +{ + "method": "/api/linkisManager/reset-resource", + "status": 0, + "message": "OK", + "data": {} +} +``` + +Error Codes: +- Permission error when user is not admin + +Notes: +- Resets resource allocation for ECM instances or users +- Requires admin privileges +- Can reset resources for a specific ECM instance or user + +### Open Engine Log +``` +POST /api/rest_j/v1/linkisManager/openEngineLog +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "emInstance": "bdp110:9100", + "instance": "bdp110:21976", + "parameters": { + "logType": "stdout", + "fromLine": "0", + "pageSize": "1000" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/openEngineLog", + "status": 0, + "message": "OK", + "data": { + // Log content or operation result + } +} +``` + +Error Codes: +- Parameter validation errors +- Permission errors + +Notes: +- Opens and retrieves engine log content +- Supports different log types (stdout, stderr, gc, udfLog, yarnApp) +- Requires appropriate permissions + +### Task Prediction +``` +GET /api/rest_j/v1/linkisManager/task-prediction +``` + +Parameters: +- `username`: Username (optional) +- `engineType`: Engine type (required) +- `creator`: Creator (required) +- `clustername`: Cluster name (optional) +- `queueName`: Queue name (optional) +- `tenant`: Tenant (optional) + +Response: +```json +{ + "method": "/api/linkisManager/task-prediction", + "status": 0, + "message": "OK", + "data": { + "tenant": "tenant", + "userResource": {}, + "ecmResource": {}, + "yarnResource": {}, + "checkResult": true + } +} +``` + +Error Codes: +- Parameter validation errors + +Notes: +- Predicts if a task can be executed based on available resources +- Requires engineType and creator parameters +- Returns resource availability information + +### Get Engine Connection Info +``` +GET /api/rest_j/v1/linkisManager/ecinfo/get +``` + +Parameters: +- `ticketid`: Ticket ID (required) + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/get", + "status": 0, + "message": "OK", + "data": { + "ecResourceInfoRecord": { + // Engine connection resource information + } + } +} +``` + +Error Codes: +- Ticket ID not found + +Notes: +- Retrieves engine connection information by ticket ID +- Requires user to be owner or admin + +### Delete Engine Connection Info +``` +DELETE /api/rest_j/v1/linkisManager/ecinfo/delete/{ticketid} +``` + +Parameters: +- `ticketid`: Ticket ID (required) + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/delete/{ticketid}", + "status": 0, + "message": "OK", + "data": { + "ecResourceInfoRecord": { + // Deleted engine connection resource information + } + } +} +``` + +Error Codes: +- Ticket ID not found +- Permission errors + +Notes: +- Deletes engine connection information by ticket ID +- Requires user to be owner or admin + +### Query Engine Connection Resource History List +``` +GET /api/rest_j/v1/linkisManager/ecinfo/ecrHistoryList +``` + +Parameters: +- `instance`: Instance (optional) +- `creator`: Creator (optional) +- `startDate`: Start date (optional) +- `endDate`: End date (optional) +- `engineType`: Engine type (optional) +- `status`: Status (optional) +- `pageNow`: Page number (optional, default: 1) +- `pageSize`: Page size (optional, default: 20) + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/ecrHistoryList", + "status": 0, + "message": "OK", + "data": { + "engineList": [ + // Engine connection resource history records + ], + "totalPage": 100 + } +} +``` + +Error Codes: +- Parameter validation errors + +Notes: +- Queries engine connection resource history +- Supports filtering by various parameters +- Returns paginated results + +### Query Engine Connection List +``` +POST /api/rest_j/v1/linkisManager/ecinfo/ecList +``` + +Request Body: +```json +{ + "creators": ["IDE"], + "engineTypes": ["spark"], + "statuss": ["Running"], + "queueName": "default", + "ecInstances": ["instance1", "instance2"], + "crossCluster": false +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/ecList", + "status": 0, + "message": "OK", + "data": { + "ecList": [ + // Engine connection records + ] + } +} +``` + +Error Codes: +- Parameter validation errors +- Permission errors + +Notes: +- Queries engine connection list +- Requires admin privileges +- Supports filtering by various parameters + +## Database Table Structures + +The ECM service uses the following database tables for engine management: + +### Engine Connection Plugin BML Resources Table +```sql +CREATE TABLE `linkis_cg_engine_conn_plugin_bml_resources` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `engine_conn_type` varchar(100) NOT NULL COMMENT 'Engine type', + `version` varchar(100) COMMENT 'version', + `file_name` varchar(255) COMMENT 'file name', + `file_size` bigint(20) DEFAULT 0 NOT NULL COMMENT 'file size', + `last_modified` bigint(20) COMMENT 'File update time', + `bml_resource_id` varchar(100) NOT NULL COMMENT 'Owning system', + `bml_resource_version` varchar(200) NOT NULL COMMENT 'Resource owner', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'created time', + `last_update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'updated time', + PRIMARY KEY (`id`) +); +``` + +### Manager Engine EM Table +```sql +CREATE TABLE `linkis_cg_manager_engine_em` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `engine_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL, + `em_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL, + `update_time` datetime DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`) +); +``` + +### EC Resource Info Record Table +```sql +CREATE TABLE `linkis_cg_ec_resource_info_record` ( + `id` INT(20) NOT NULL AUTO_INCREMENT, + `label_value` VARCHAR(255) NOT NULL COMMENT 'ec labels stringValue', + `create_user` VARCHAR(128) NOT NULL COMMENT 'ec create user', + `service_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'ec instance info', + `ecm_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'ecm instance info ', + `ticket_id` VARCHAR(100) NOT NULL COMMENT 'ec ticket id', + `status` varchar(50) DEFAULT NULL COMMENT 'EC status: Starting,Unlock,Locked,Idle,Busy,Running,ShuttingDown,Failed,Success', + `log_dir_suffix` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'log path', + `request_times` INT(8) COMMENT 'resource request times', + `request_resource` VARCHAR(1020) COMMENT 'request resource', + `used_times` INT(8) COMMENT 'resource used times', + `used_resource` VARCHAR(1020) COMMENT 'used resource', + `metrics` TEXT DEFAULT NULL COMMENT 'ec metrics', + `release_times` INT(8) COMMENT 'resource released times', + `released_resource` VARCHAR(1020) COMMENT 'released resource', + `release_time` datetime DEFAULT NULL COMMENT 'released time', + `used_time` datetime DEFAULT NULL COMMENT 'used time', + `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + PRIMARY KEY (`id`), + KEY `idx_ticket_id` (`ticket_id`), + UNIQUE KEY `uniq_tid_lv` (`ticket_id`,`label_value`), + UNIQUE KEY `uniq_sinstance_status_cuser_ctime` (`service_instance`, `status`, `create_user`, `create_time`) +); +``` + +## RPC Methods + +The ECM service provides several RPC methods for engine management: + +### Engine Management RPCs + +#### createEngineConnection +Creates a new engine connection: +```java +EngineConnection createEngineConnection(EngineCreateRequest request) +``` + +#### executeCode +Executes code on an engine: +```java +ExecutionResult executeCode(String engineId, String code, String runType) +``` + +#### getEngineStatus +Retrieves the status of an engine: +```java +EngineStatus getEngineStatus(String engineId) +``` + +#### terminateEngine +Terminates an engine connection: +```java +void terminateEngine(String engineId) +``` + +#### listEngines +Lists all engine connections: +```java +List listEngines() +``` + +#### getEngineMetrics +Retrieves metrics for an engine: +```java +EngineMetrics getEngineMetrics(String engineId) +``` + +### Resource Management RPCs + +#### getResourceUsage +Retrieves resource usage for an engine: +```java +ResourceUsage getResourceUsage(String engineId) +``` + +#### updateResource +Updates resource allocation for an engine: +```java +void updateResource(String engineId, ResourceRequest request) +``` + +#### reportResourceUsage +Reports resource usage from an engine: +```java +void reportResourceUsage(String engineId, ResourceUsage usage) +``` + +### Engine Communication RPCs + +#### sendEngineCommand +Sends a command to an engine: +```java +CommandResponse sendEngineCommand(String engineId, EngineCommand command) +``` + +#### getEngineLogs +Retrieves logs from an engine: +```java +EngineLogs getEngineLogs(String engineId, int fromLine, int lines) +``` + +## Dependencies + +- linkis-engineconn-manager-core +- linkis-engineconn-plugin-core +- linkis-rpc +- linkis-protocol +- linkis-manager-common + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- ECMRestfulApi: `linkis-computation-governance/linkis-engineconn-manager/linkis-engineconn-manager-server/src/main/java/org/apache/linkis/ecm/restful/ECMRestfulApi.java` +- EMRestfulApi: `linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/am/restful/EMRestfulApi.java` +- ECResourceInfoRestfulApi: `linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/am/restful/ECResourceInfoRestfulApi.java` + +### MyBatis XML Files +The ECM service primarily uses the Manager service's persistence layer, which includes: +- LabelManagerMapper: `linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/LabelManagerMapper.xml` +- ResourceManagerMapper: `linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/ResourceManagerMapper.xml` +- NodeManagerMapper: `linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/NodeManagerMapper.xml` +- NodeMetricManagerMapper: `linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/NodeMetricManagerMapper.xml` \ No newline at end of file diff --git a/.ai/modules/computation-governance/entrance.md b/.ai/modules/computation-governance/entrance.md new file mode 100644 index 00000000000..6071b33f9a3 --- /dev/null +++ b/.ai/modules/computation-governance/entrance.md @@ -0,0 +1,726 @@ +# Entrance Service + +The Entrance service serves as the entry point for computation task submissions in the Linkis system. + +## Overview + +This service is responsible for receiving user computation requests, parsing them, validating them, and coordinating their execution through the appropriate engine connections. It acts as the primary interface between users and the computation execution layer. + +## Key Components + +### Core Classes +- `LinkisEntranceApplication` - Main application class +- Task submission handling +- Task parsing and validation +- Task scheduling coordination +- Task execution monitoring +- Task result management + +### Features +- Task submission and management +- Code parsing and validation +- Engine routing and allocation +- Result set management +- Log retrieval and management + +## API Interfaces + +### Task Execution +``` +POST /api/entrance/execute +``` + +Parameters (in request body): +- `executionContent`: Contains the code to execute and run type + - `code`: The actual code to execute + - `runType`: Type of execution (sql, python, scala, etc.) +- `params`: Parameters for execution + - `variable`: Variables for the execution + - `configuration`: Configuration parameters (runtime, special) +- `source`: Source information + - `scriptPath`: Path to the script file +- `labels`: Labels for engine selection + - `engineType`: Type and version of engine (spark-2.4.3, hive-2.1.1, etc.) + - `userCreator`: User and creator information + +Response: +```json +{ + "method": "/api/entrance/execute", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If parsing or execution fails, the error will be stored in the job request and returned in the response +- Permission errors if user is not authorized to execute + +Notes: +- Returns both taskID (database ID) and execID (execution ID) +- The execID is used for subsequent operations on the task +- User authentication is required + +### Task Submission +``` +POST /api/entrance/submit +``` + +Parameters (in request body): +- Same as execute API + +Response: +```json +{ + "method": "/api/entrance/submit", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If parsing or execution fails, the error will be stored in the job request and returned in the response +- Permission errors if user is not authorized to submit + +Notes: +- Functionally similar to execute but with different endpoint +- Returns both taskID (database ID) and execID (execution ID) +- User authentication is required + +### Task Status Query +``` +GET /api/entrance/{id}/status +``` + +Parameters: +- `id`: The execution ID or task ID +- `taskID` (optional): The ID of the task to query + +Response: +```json +{ + "method": "/api/entrance/{id}/status", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "status": "Running", + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If job cannot be found, appropriate error message is returned +- If there's an exception during status retrieval, error is returned + +Notes: +- Supports both execID and taskID as the path parameter +- Status values include: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout +- For completed jobs, status is retrieved from job history + +### Task Progress +``` +GET /api/entrance/{id}/progress +``` + +Parameters: +- `id`: The execution ID + +Response: +```json +{ + "method": "/api/entrance/{id}/progress", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "progress": "0.75", + "execID": "exec-id-12345", + "progressInfo": [ + { + "id": "stage1", + "succeedTasks": 5, + "failedTasks": 0, + "runningTasks": 2, + "totalTasks": 10 + } + ] + } +} +``` + +Error Cases: +- If job cannot be found, appropriate error message is returned +- If progress information is not yet available, error is returned + +Notes: +- Progress is a value between 0 and 1 +- ProgressInfo provides detailed information about execution stages +- For completed jobs, returns 1.0 progress + +### Task Progress with Resource Info +``` +GET /api/entrance/{id}/progressWithResource +``` + +Parameters: +- `id`: The execution ID + +Response: +```json +{ + "method": "/api/entrance/{id}/progressWithResource", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "progress": "0.75", + "execID": "exec-id-12345", + "progressInfo": [ + { + "id": "stage1", + "succeedTasks": 5, + "failedTasks": 0, + "runningTasks": 2, + "totalTasks": 10 + } + ], + "jobYarnMetrics": { + "jobYarnResource": [ + { + "applicationId": "application_1234567890123_0001", + "queueCores": 2, + "queueMemory": 4096, + "usedCores": 1, + "usedMemory": 2048, + "resourceType": "YARN" + } + ] + } + } +} +``` + +Error Cases: +- If job cannot be found, appropriate error message is returned +- If progress information is not yet available, error is returned + +Notes: +- Includes YARN resource metrics in addition to progress information +- Provides detailed resource usage information for YARN-based engines + +### Task Log Retrieval +``` +GET /api/entrance/{id}/log +``` + +Parameters: +- `id`: The execution ID +- `fromLine` (optional): Starting line number (default: 0) +- `size` (optional): Number of lines to retrieve (default: 100) +- `distinctLevel` (optional): Whether to separate logs by level (default: true) + +Response: +```json +{ + "method": "/api/entrance/{id}/log", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "log": ["log line 1", "log line 2", "log line 3"], + "fromLine": 1, + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If job has completed, suggests downloading log file instead +- If log cannot be retrieved, returns appropriate error + +Notes: +- For distinctLevel=true, returns array with 4 elements (different log levels) +- For distinctLevel=false, returns concatenated string of logs +- Size parameter has a maximum limit (10000) + +### Task Cancellation +``` +GET /api/entrance/{id}/kill +``` + +Parameters: +- `id`: The execution ID +- `taskID` (optional): The ID of the task to cancel + +Response: +```json +{ + "method": "/api/entrance/{id}/kill", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If job is already completed, returns error that kill is not supported +- If user doesn't have permission to kill the job, returns permission error +- If exception occurs during kill, returns error with exception details + +Notes: +- Updates job status to Cancelled in database +- For jobs not found in memory, performs force kill using job history + +### Batch Task Cancellation +``` +POST /api/entrance/{id}/killJobs +``` + +Request Body: +```json +{ + "idList": ["exec-id-1", "exec-id-2"], + "taskIDList": [12345, 12346] +} +``` + +Parameters: +- `id`: The strong execution ID + +Response: +```json +{ + "method": "/api/entrance/{id}/killJobs", + "status": 0, + "message": "success", + "data": { + "messages": [ + { + "method": "/api/entrance/exec-id-1/kill", + "status": 0, + "message": "Successfully killed the job(成功kill了job)" + }, + { + "method": "/api/entrance/exec-id-2/kill", + "status": 0, + "message": "Successfully killed the job(成功kill了job)" + } + ] + } +} +``` + +Error Cases: +- If idList and taskIDList have different lengths, returns error +- If parameters are not arrays, returns error +- Individual job kill errors are returned in the messages array + +Notes: +- Processes each job in the lists and returns individual results +- For jobs not found in memory, performs force kill using job history + +### Task Pause +``` +GET /api/entrance/{id}/pause +``` + +Parameters: +- `id`: The execution ID + +Response: +```json +{ + "method": "/api/entrance/{id}/pause", + "status": 0, + "message": "success to pause job (成功pause了job)", + "data": { + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If job cannot be found, returns appropriate error +- If exception occurs during pause, returns error + +Notes: +- Pause functionality implementation may be incomplete (TODO in code) + +### Update Route Label +``` +POST /api/entrance/operation/label/update +``` + +Request Body: +```json +{ + "routeLabel": "new-route-label" +} +``` + +Parameters: +- Requires admin privileges + +Response: +```json +{ + "method": "/api/entrance/operation/label/update", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Updates the route label for the entrance instance +- Used for routing purposes in distributed environments + +### Mark Offline +``` +GET /api/entrance/operation/label/markoffline +``` + +Response: +```json +{ + "method": "/api/entrance/operation/label/markoffline", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Marks the entrance instance as offline +- Updates all non-execution task instances + +### Back Online +``` +GET /api/entrance/operation/label/backonline +``` + +Response: +```json +{ + "method": "/api/entrance/operation/label/backonline", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Removes the offline label from the entrance instance + +### Check Online Status +``` +GET /api/entrance/operation/label/isOnline +``` + +Response: +```json +{ + "method": "/api/entrance/operation/label/isOnline", + "status": 0, + "message": "success", + "data": { + "isOnline": true + } +} +``` + +Notes: +- Checks if the entrance instance is currently online + +### Get Task Info +``` +GET /api/entrance/operation/metrics/taskinfo +``` + +Parameters: +- `user` (optional): Filter by user +- `creator` (optional): Filter by creator +- `ecType` (optional): Filter by engine type + +Response: +```json +{ + "method": "/api/entrance/operation/metrics/taskinfo", + "status": 0, + "message": "success", + "data": { + "taskNumber": 5, + "runningNumber": 2, + "queuedNumber": 3 + } +} +``` + +Error Cases: +- Non-admin users cannot view other users' task information + +Notes: +- For admin users, can view any user's task information +- For non-admin users, can only view their own task information +- Returns counts of total, running, and queued tasks + +### Get Running Task Count +``` +GET /api/entrance/operation/metrics/runningtask +``` + +Response: +```json +{ + "method": "/api/entrance/operation/metrics/runningtask", + "status": 0, + "message": "success", + "data": { + "runningTaskNumber": 5, + "isCompleted": false + } +} +``` + +Notes: +- Returns the number of currently running tasks +- isCompleted indicates if there are no running tasks + +### Kill Consumer +``` +GET /api/entrance/operation/consumer/kill +``` + +Parameters: +- `groupName`: Name of the consumer group to kill + +Response: +```json +{ + "method": "/api/entrance/operation/consumer/kill", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Destroys the specified consumer group +- Requires admin privileges + +### Get Consumer Info +``` +GET /api/entrance/operation/consumer/info +``` + +Response: +```json +{ + "method": "/api/entrance/operation/consumer/info", + "status": 0, + "message": "success", + "data": { + "consumerNum": 3 + } +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Returns the number of consumer groups +- Requires admin privileges + +## Database Table Structures + +The Entrance service uses the following database tables from the job history system: + +### Job History Group Table +```sql +CREATE TABLE `linkis_ps_job_history_group_history` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job', + `execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job', + `source` text DEFAULT NULL COMMENT 'job source', + `labels` text DEFAULT NULL COMMENT 'job labels', + `params` text DEFAULT NULL COMMENT 'job params', + `progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log', + `error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances', + `metrics` text DEFAULT NULL COMMENT 'Job Metrics', + `engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type', + `execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job', + PRIMARY KEY (`id`), + KEY `idx_created_time` (`created_time`), + KEY `idx_submit_user` (`submit_user`) +); +``` + +### Job History Detail Table +```sql +CREATE TABLE `linkis_ps_job_history_detail` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job', + `result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array', + `job_group_info` text DEFAULT NULL COMMENT 'Job group info/path', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `status` varchar(32) DEFAULT NULL COMMENT 'status', + `priority` int(4) DEFAULT 0 COMMENT 'order of subjob', + PRIMARY KEY (`id`) +); +``` + +## RPC Methods + +The Entrance service provides several RPC methods for inter-service communication: + +### Task Management RPCs + +#### submitTask +Submits a task for execution: +```java +JobRespProtocol submitTask(JobReqInsert request) +``` + +#### updateTask +Updates a task: +```java +JobRespProtocol updateTask(JobReqUpdate request) +``` + +#### batchUpdateTasks +Batch updates tasks: +```java +JobRespProtocol batchUpdateTasks(JobReqBatchUpdate request) +``` + +#### queryTask +Queries a task: +```java +JobRespProtocol queryTask(JobReqQuery request) +``` + +#### readAllTasks +Reads all tasks: +```java +JobRespProtocol readAllTasks(JobReqReadAll request) +``` + +#### getTaskStatus +Retrieves the status of a task: +```java +String getTaskStatus(String taskId) +``` + +#### cancelTask +Cancels a running task: +```java +void cancelTask(String taskId) +``` + +#### getTaskResult +Retrieves the result of a completed task: +```java +TaskResult getTaskResult(String taskId) +``` + +#### getTaskProgress +Retrieves the progress of a task: +```java +TaskProgress getTaskProgress(String taskId) +``` + +### Engine Management RPCs + +#### requestEngine +Requests an engine for task execution: +```java +EngineConnection requestEngine(EngineRequest request) +``` + +#### releaseEngine +Releases an engine after task completion: +```java +void releaseEngine(String engineId) +``` + +#### getEngineStatus +Retrieves the status of an engine: +```java +EngineStatus getEngineStatus(String engineId) +``` + +### Log Management RPCs + +#### getTaskLog +Retrieves logs for a specific task: +```java +TaskLog getTaskLog(String taskId, int fromLine, int pageSize) +``` + +#### appendTaskLog +Appends log entries for a task: +```java +void appendTaskLog(String taskId, List logLines) +``` + +## Dependencies + +- linkis-scheduler +- linkis-protocol +- linkis-rpc +- linkis-storage +- linkis-computation-governance-common +- linkis-computation-orchestrator +- linkis-pes-client +- linkis-io-file-client +- linkis-pes-rpc-client +- linkis-ps-common-lock + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- EntranceRestfulApi: `linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceRestfulApi.java` +- EntranceLabelRestfulApi: `linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceLabelRestfulApi.java` +- EntranceMetricRestfulApi: `linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceMetricRestfulApi.java` +- EntranceConsumerRestfulApi: `linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceConsumerRestfulApi.java` + +### MyBatis XML Files +The Entrance service uses the JobHistory service's persistence layer, which includes: +- JobHistoryMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/mysql/JobHistoryMapper.xml` +- JobDetailMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDetailMapper.xml` +- JobStatisticsMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobStatisticsMapper.xml` +- JobDiagnosisMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDiagnosisMapper.xml` \ No newline at end of file diff --git a/.ai/modules/computation-governance/jobhistory.md b/.ai/modules/computation-governance/jobhistory.md new file mode 100644 index 00000000000..7e875ce2f6f --- /dev/null +++ b/.ai/modules/computation-governance/jobhistory.md @@ -0,0 +1,481 @@ +# JobHistory Service + +The JobHistory service tracks and manages the execution history of tasks in the Linkis system. + +## Overview + +This service provides task execution history tracking, including task status, execution time, results, and error information. + +## Key Components + +### Core Classes +- `LinkisJobHistoryApp` - Main application class +- Task history storage and retrieval +- Task statistics and analytics +- Task search and filtering + +### Features +- Task execution history tracking +- Task result storage and retrieval +- Task performance metrics +- Task search and filtering capabilities +- Task statistics and reporting + +## API Interfaces + +### Get Task By ID +``` +GET /api/rest_j/v1/jobhistory/{id}/get +``` + +Parameters: +- `id`: Job ID (required) +- `brief`: Whether to return brief info only (optional) + +Response: +```json +{ + "method": "/api/jobhistory/{id}/get", + "status": 0, + "message": "success", + "data": { + "task": { + "jobId": "12345", + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Succeed", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00", + "executionCode": "SELECT * FROM table", + "resultLocation": "/path/to/result", + "errorCode": null, + "errorDesc": null, + "progress": "1.0", + "costTime": 300000 + } + } +} +``` + +### List Job History +``` +GET /api/rest_j/v1/jobhistory/list +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `status`: Task status to filter by (optional) +- `pageNow`: Page number (optional, default: 1) +- `pageSize`: Page size (optional, default: 20) +- `taskID`: Task ID to filter by (optional) +- `executeApplicationName`: Application name to filter by (optional) +- `creator`: Creator to filter by (optional) +- `proxyUser`: Proxy user to filter by (optional) +- `isAdminView`: Whether to view as admin (optional) +- `isDeptView`: Whether to view as department admin (optional) +- `instance`: Instance to filter by (optional) +- `engineInstance`: Engine instance to filter by (optional) +- `runType`: Run type to filter by (optional) + +Response: +```json +{ + "method": "/api/jobhistory/list", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Succeed", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00", + "executionCode": "SELECT * FROM table", + "resultLocation": "/path/to/result", + "errorCode": null, + "errorDesc": null, + "progress": "1.0", + "costTime": 300000 + } + ], + "totalPage": 1 + } +} +``` + +### List Undone Tasks +``` +GET /api/rest_j/v1/jobhistory/listundonetasks +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `status`: Task status to filter by (optional, default: "Running,Inited,Scheduled") +- `pageNow`: Page number (optional, default: 1) +- `pageSize`: Page size (optional, default: 20) +- `startTaskID`: Start task ID (optional) +- `engineType`: Engine type to filter by (optional) +- `creator`: Creator to filter by (optional) + +Response: +```json +{ + "method": "/api/jobhistory/listundonetasks", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Running", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00" + } + ], + "totalPage": 1 + } +} +``` + +### List By Task IDs +``` +GET /api/rest_j/v1/jobhistory/list-taskids +``` + +Parameters: +- `taskID`: Comma-separated list of task IDs (required) + +Response: +```json +{ + "method": "/api/jobhistory/list-taskids", + "status": 0, + "message": "success", + "data": { + "jobHistoryList": [ + { + "jobId": 12345, + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Succeed", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00" + } + ] + } +} +``` + +### Job Extra Info +``` +GET /api/rest_j/v1/jobhistory/job-extra-info +``` + +Parameters: +- `jobId`: Job ID (required) + +Response: +```json +{ + "method": "/api/jobhistory/job-extra-info", + "status": 0, + "message": "success", + "data": { + "metricsMap": { + "executionCode": "SELECT * FROM table", + "runtime": "300000" + } + } +} +``` + +### List Duration Top +``` +GET /api/rest_j/v1/jobhistory/listDurationTop +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `executeApplicationName`: Application name to filter by (optional) +- `creator`: Creator to filter by (optional) +- `proxyUser`: Proxy user to filter by (optional) +- `pageNow`: Page number (optional, default: 1) +- `pageSize`: Page size (optional, default: 20) + +Response: +```json +{ + "method": "/api/jobhistory/listDurationTop", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Succeed", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00", + "costTime": 300000 + } + ] + } +} +``` + +### Task Count Statistics +``` +GET /api/rest_j/v1/jobhistory/jobstatistics/taskCount +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `executeApplicationName`: Application name to filter by (optional) +- `creator`: Creator to filter by (optional) +- `proxyUser`: Proxy user to filter by (optional) + +Response: +```json +{ + "method": "/api/jobhistory/jobstatistics/taskCount", + "status": 0, + "message": "success", + "data": { + "sumCount": 100, + "succeedCount": 95, + "failedCount": 5, + "cancelledCount": 0 + } +} +``` + +### Engine Count Statistics +``` +GET /api/rest_j/v1/jobhistory/jobstatistics/engineCount +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `executeApplicationName`: Application name to filter by (optional) +- `creator`: Creator to filter by (optional) +- `proxyUser`: Proxy user to filter by (optional) + +Response: +```json +{ + "method": "/api/jobhistory/jobstatistics/engineCount", + "status": 0, + "message": "success", + "data": { + "countEngine": 100, + "countEngineSucceed": 95, + "countEngineFailed": 5, + "countEngineShutting": 0 + } +} +``` + +### Add Observe Info +``` +POST /api/rest_j/v1/jobhistory/setting/addObserveInfo +``` + +Request Body: +```json +{ + "taskId": 12345, + "receiver": "testuser", + "extra": { + "title": "Task Alert", + "detail": "Task execution alert" + }, + "monitorLevel": "HIGH", + "subSystemId": "1" +} +``` + +Response: +```json +{ + "method": "/api/jobhistory/setting/addObserveInfo", + "status": 0, + "message": "success" +} +``` + +### Delete Observe Info +``` +GET /api/rest_j/v1/jobhistory/setting/deleteObserveInfo +``` + +Parameters: +- `taskId`: Task ID (required) + +Response: +```json +{ + "method": "/api/jobhistory/setting/deleteObserveInfo", + "status": 0, + "message": "success" +} +``` + +## Database Table Structures + +The JobHistory service uses the following database tables from the linkis_ddl.sql file: + +### Job History Group Table +```sql +CREATE TABLE `linkis_ps_job_history_group_history` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job', + `execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job', + `source` text DEFAULT NULL COMMENT 'job source', + `labels` text DEFAULT NULL COMMENT 'job labels', + `params` text DEFAULT NULL COMMENT 'job params', + `progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log', + `error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances', + `metrics` text DEFAULT NULL COMMENT 'Job Metrics', + `engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type', + `execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job', + PRIMARY KEY (`id`), + KEY `idx_created_time` (`created_time`), + KEY `idx_submit_user` (`submit_user`) +); +``` + +### Job History Detail Table +```sql +CREATE TABLE `linkis_ps_job_history_detail` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job', + `result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array', + `job_group_info` text DEFAULT NULL COMMENT 'Job group info/path', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `status` varchar(32) DEFAULT NULL COMMENT 'status', + `priority` int(4) DEFAULT 0 COMMENT 'order of subjob', + PRIMARY KEY (`id`) +); +``` + +## RPC Methods + +The JobHistory service provides several RPC methods for job history management: + +### Job History RPCs + +#### recordJob +Records a job execution: +```java +void recordJob(JobRecordRequest request) +``` + +#### updateJobStatus +Updates the status of a job: +```java +void updateJobStatus(String jobId, JobStatus status) +``` + +#### getJobHistory +Retrieves job history: +```java +JobHistory getJobHistory(String jobId) +``` + +#### searchJobs +Searches for jobs based on criteria: +```java +List searchJobs(JobSearchCriteria criteria) +``` + +#### getJobDetails +Retrieves detailed job information: +```java +JobDetails getJobDetails(Long jobId) +``` + +#### deleteJobHistory +Deletes job history records: +```java +void deleteJobHistory(List jobIds) +``` + +### Statistics RPCs + +#### recordStatistics +Records job statistics: +```java +void recordStatistics(JobStatistics statistics) +``` + +#### getStatistics +Retrieves job statistics: +```java +JobStatistics getStatistics(String jobId) +``` + +#### getStatisticsByUser +Retrieves job statistics for a user: +```java +List getStatisticsByUser(String username, Date startDate, Date endDate) +``` + +#### getStatisticsByEngine +Retrieves job statistics by engine type: +```java +List getStatisticsByEngine(String engineType, Date startDate, Date endDate) +``` + +## Dependencies + +- linkis-mybatis +- linkis-rpc +- linkis-protocol +- linkis-common +- linkis-computation-governance-common + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- QueryRestfulApi: `linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/QueryRestfulApi.java` +- StatisticsRestfulApi: `linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/StatisticsRestfulApi.java` +- JobhistorySettingApi: `linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/JobhistorySettingApi.java` + +### MyBatis XML Files +- JobHistoryMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/mysql/JobHistoryMapper.xml` +- JobDetailMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDetailMapper.xml` +- JobStatisticsMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobStatisticsMapper.xml` +- JobDiagnosisMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDiagnosisMapper.xml` +- JobAiHistoryMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobAiHistoryMapper.xml` diff --git a/.ai/modules/computation-governance/manager.md b/.ai/modules/computation-governance/manager.md new file mode 100644 index 00000000000..a74a77ed877 --- /dev/null +++ b/.ai/modules/computation-governance/manager.md @@ -0,0 +1,1010 @@ +# Manager Service + +The Manager service provides resource and application management capabilities for the Linkis system. + +## Overview + +This service manages the resources and applications in the Linkis system, including node management, resource allocation, label management, and engine lifecycle management. + +## Key Components + +### Core Classes +- `LinkisManagerApplication` - Main application class +- Node management +- Resource management +- Label management +- Engine lifecycle management + +### Features +- Node registration and management +- Resource allocation and monitoring +- Label-based routing +- Engine instance management +- Load balancing + +## API Interfaces + +### ECM (EngineConnManager) Management + +#### List All ECMs +``` +GET /api/rest_j/v1/linkisManager/listAllEMs +``` + +Parameters: +- `instance` (optional): Filter by instance name +- `nodeHealthy` (optional): Filter by node health status (Healthy, UnHealthy, WARN, StockAvailable, StockUnavailable) +- `owner` (optional): Filter by owner +- `tenantLabel` (optional): Filter by tenant label + +Response: +```json +{ + "method": "/api/linkisManager/listAllEMs", + "status": 0, + "message": "success", + "data": { + "EMs": [ + { + "serviceInstance": { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102" + }, + "labels": [ + { + "labelKey": "engineType", + "stringValue": "spark" + } + ], + "nodeHealthy": "Healthy", + "owner": "testuser" + } + ] + } +} +``` + +Error Cases: +- Only admin users can access this API +- If parameters are invalid, appropriate error messages are returned + +Notes: +- Requires admin privileges +- Results can be filtered and sorted by various criteria +- Returns EMNodeVo objects with detailed information about each ECM + +#### List All ECM Healthy Status +``` +GET /api/rest_j/v1/linkisManager/listAllECMHealthyStatus +``` + +Parameters: +- `onlyEditable` (optional): If true, returns only editable statuses (Healthy, UnHealthy, WARN, StockAvailable, StockUnavailable) + +Response: +```json +{ + "method": "/api/linkisManager/listAllECMHealthyStatus", + "status": 0, + "message": "success", + "data": { + "nodeHealthy": ["Healthy", "UnHealthy", "WARN", "StockAvailable", "StockUnavailable"] + } +} +``` + +Notes: +- Returns all possible NodeHealthy enum values +- With onlyEditable=true, returns only the statuses that can be modified by users + +#### Modify ECM Info +``` +PUT /api/rest_j/v1/linkisManager/modifyEMInfo +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102", + "emStatus": "Healthy", + "labels": [ + { + "labelKey": "engineType", + "stringValue": "spark" + } + ] +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/modifyEMInfo", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- Only admin users can modify ECM info +- If applicationName or instance is null, returns error +- If labels contain duplicates, returns error +- If label values are invalid, returns error + +Notes: +- Requires admin privileges +- Can update both EM status and labels +- Supports UserModifiable labels with value validation + +#### Execute ECM Operation +``` +POST /api/rest_j/v1/linkisManager/executeECMOperation +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102", + "parameters": { + "operation": "stopEngine", + "engineConnInstance": "bdp110:12295" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeECMOperation", + "status": 0, + "message": "success", + "data": { + "result": "Operation executed successfully", + "errorMsg": "", + "isError": false + } +} +``` + +Error Cases: +- If user doesn't have permission to execute operation, returns error +- If ECM node doesn't exist, returns error +- If operation parameters are invalid, returns error + +Notes: +- Supports various admin operations (configurable via AMConfiguration.ECM_ADMIN_OPERATIONS) +- For log operations, automatically fills in logDirSuffix if not provided +- Validates user permissions for admin operations + +#### Execute ECM Operation By EC +``` +POST /api/rest_j/v1/linkisManager/executeECMOperationByEC +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295", + "parameters": { + "operation": "stopEngine" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeECMOperationByEC", + "status": 0, + "message": "success", + "data": { + "result": "Operation executed successfully", + "errorMsg": "", + "isError": false + } +} +``` + +Error Cases: +- If user doesn't have permission to execute operation, returns error +- If engine node doesn't exist, returns error +- If operation parameters are invalid, returns error + +Notes: +- User must be owner of the engine or admin +- Delegates to executeECMOperation after validating permissions + +#### Open Engine Log +``` +POST /api/rest_j/v1/linkisManager/openEngineLog +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "emInstance": "bdp110:9100", + "instance": "bdp110:21976", + "parameters": { + "logType": "stdout", + "fromLine": "0", + "pageSize": "1000" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/openEngineLog", + "status": 0, + "message": "success", + "data": { + "result": "Log content...", + "errorMsg": "", + "isError": false + } +} +``` + +Error Cases: +- If user doesn't have permission to access logs, returns error +- If log type is invalid, returns error +- If engine instance doesn't exist, returns error + +Notes: +- Supported log types: stdout, stderr, gc, udfLog, yarnApp +- Automatically fills in logDirSuffix if not provided +- Validates user permissions (must be owner or admin) + +#### Task Prediction +``` +GET /api/rest_j/v1/linkisManager/task-prediction +``` + +Parameters: +- `username` (optional): User name (defaults to current user) +- `engineType` (required): Engine type (spark/hive/etc.) +- `creator` (required): Creator application +- `clustername` (optional): Cluster name +- `queueName` (optional): Queue name +- `tenant` (optional): Tenant + +Response: +```json +{ + "method": "/api/linkisManager/task-prediction", + "status": 0, + "message": "success", + "data": { + "tenant": "tenant", + "userResource": {...}, + "ecmResource": {...}, + "yarnResource": {...}, + "checkResult": true + } +} +``` + +Error Cases: +- If engineType or creator is null, returns error +- If resource check fails, returns error + +Notes: +- Checks if user can create an engine for specified parameters +- Returns detailed resource information for user, ECM, and YARN + +#### Reset Resource +``` +GET /api/rest_j/v1/linkisManager/reset-resource +``` + +Parameters: +- `serviceInstance` (optional): Service instance to reset +- `username` (optional): User name to reset + +Response: +```json +{ + "method": "/api/linkisManager/reset-resource", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- Only admin users can reset resources + +Notes: +- Requires admin privileges +- Resets resource allocations for specified instance or user + +### Engine Management + +#### Ask Engine Connection +``` +POST /api/rest_j/v1/linkisManager/askEngineConn +``` + +Request Body: +```json +{ + "labels": { + "engineType": "spark-2.4.3", + "userCreator": "testuser-IDE" + }, + "timeOut": 30000, + "user": "testuser" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/askEngineConn", + "status": 0, + "message": "create engineConn ended.", + "data": { + "engine": { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Starting", + "ticketId": "ticket-12345", + "ecmServiceInstance": { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102" + } + } + } +} +``` + +Error Cases: +- If timeout is invalid, uses default timeout +- If engine creation fails, returns error with retry information + +Notes: +- First attempts to reuse existing engines +- If no suitable engine found, creates a new one +- Supports async engine creation with timeout handling + +#### Create Engine Connection +``` +POST /api/rest_j/v1/linkisManager/createEngineConn +``` + +Request Body: +```json +{ + "labels": { + "engineType": "spark-2.4.3", + "userCreator": "testuser-IDE" + }, + "timeout": 30000, + "user": "testuser" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/createEngineConn", + "status": 0, + "message": "create engineConn succeed.", + "data": { + "engine": { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Starting", + "ticketId": "ticket-12345" + } + } +} +``` + +Error Cases: +- If timeout is invalid, uses default timeout +- If engine creation fails, returns error with retry information + +Notes: +- Always creates a new engine (doesn't attempt reuse) +- Supports timeout configuration +- Returns EngineNode information with service instance and ticket ID + +#### Get Engine Connection +``` +POST /api/rest_j/v1/linkisManager/getEngineConn +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/getEngineConn", + "status": 0, + "message": "success", + "data": { + "engine": { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Running", + "ticketId": "ticket-12345" + } + } +} +``` + +Error Cases: +- If user doesn't have permission to access engine, returns error +- If engine instance doesn't exist, returns error + +Notes: +- User must be owner of the engine or admin +- Can retrieve engine info by service instance or ticket ID +- Returns EC metrics if available + +#### Kill Engine Connection +``` +POST /api/rest_j/v1/linkisManager/killEngineConn +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/killEngineConn", + "status": 0, + "message": "Kill engineConn succeed." +} +``` + +Error Cases: +- If user doesn't have permission to kill engine, returns error +- If engine instance doesn't exist, returns error + +Notes: +- User must be owner of the engine or admin +- Sends EngineStopRequest to engine stop service +- Logs kill operation + +#### Kill ECM Engines +``` +POST /api/rest_j/v1/linkisManager/rm/killUnlockEngineByEM +``` + +Request Body: +```json +{ + "instance": "bdp110:9210" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/killUnlockEngineByEM", + "status": 0, + "message": "Kill engineConn succeed.", + "data": { + "result": {...} + } +} +``` + +Error Cases: +- Only admin users can kill engines by ECM +- If instance parameter is null, returns error + +Notes: +- Requires admin privileges +- Kills all unlocked engines under specified ECM +- Returns result information + +#### Kill Multiple Engines +``` +POST /api/rest_j/v1/linkisManager/rm/enginekill +``` + +Request Body: +```json +[ + { + "applicationName": "linkis-cg-engineconn", + "engineInstance": "bdp110:12295" + } +] +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/enginekill", + "status": 0, + "message": "Kill engineConn succeed." +} +``` + +Error Cases: +- If engine instances don't exist, logs error but continues + +Notes: +- Kills multiple engines in a single request +- No permission check (uses internal sender) + +#### Kill Multiple Engines Async +``` +POST /api/rest_j/v1/linkisManager/rm/enginekillAsyn +``` + +Request Body: +```json +{ + "instances": ["bdp110:12295", "bdp110:12296"] +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/enginekillAsyn", + "status": 0, + "message": "Kill engineConn succeed." +} +``` + +Error Cases: +- If user is not admin and doesn't have valid token, returns error +- If instances parameter is null or empty, returns error +- If instances parameter parsing fails, returns error + +Notes: +- Requires admin privileges or valid admin token +- Asynchronously stops engines with metrics update +- Supports batch killing of multiple engine instances + +#### List User Engines +``` +GET /api/rest_j/v1/linkisManager/listUserEngines +``` + +Response: +```json +{ + "method": "/api/linkisManager/listUserEngines", + "status": 0, + "message": "success", + "data": { + "engines": [ + { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Running", + "owner": "testuser", + "engineType": "spark" + } + ] + } +} +``` + +Notes: +- Returns engines owned by the current user +- Lists all engine nodes for the user + +#### List ECM Engines +``` +POST /api/rest_j/v1/linkisManager/listEMEngines +``` + +Request Body: +```json +{ + "em": { + "serviceInstance": { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102" + } + }, + "emInstance": "bdp110:9102" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/listEMEngines", + "status": 0, + "message": "success", + "data": { + "engines": [ + { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Running", + "owner": "testuser", + "engineType": "spark" + } + ] + } +} +``` + +Error Cases: +- Only admin users can list ECM engines +- If parameters are invalid, returns error + +Notes: +- Requires admin privileges +- Supports filtering by EM instance, node status, engine type, and owner +- Returns AMEngineNodeVo objects with detailed engine information + +#### Modify Engine Info +``` +PUT /api/rest_j/v1/linkisManager/modifyEngineInfo +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295", + "labels": [ + { + "labelKey": "engineType", + "stringValue": "spark" + } + ], + "nodeHealthy": "Healthy" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/modifyEngineInfo", + "status": 0, + "message": "success to update engine information(更新引擎信息成功)" +} +``` + +Error Cases: +- Only admin users can modify engine info +- If applicationName or instance is null, returns error +- If labels contain duplicates, returns error + +Notes: +- Requires admin privileges +- Can update both engine labels and health status +- Health status updates only support Healthy and UnHealthy values + +#### Batch Set Engine To UnHealthy +``` +POST /api/rest_j/v1/linkisManager/batchSetEngineToUnHealthy +``` + +Request Body: +```json +{ + "instances": [ + { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + } + ] +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/batchSetEngineToUnHealthy", + "status": 0, + "message": "success to update engine information(批量更新引擎健康信息成功)" +} +``` + +Error Cases: +- Only admin users can set engine health status +- If instances parameter is null, returns error + +Notes: +- Requires admin privileges +- Sets multiple engines to UnHealthy status +- Logs batch update operation + +#### List All Node Healthy Status +``` +GET /api/rest_j/v1/linkisManager/listAllNodeHealthyStatus +``` + +Parameters: +- `onlyEditable` (optional): If true, returns only editable statuses + +Response: +```json +{ + "method": "/api/linkisManager/listAllNodeHealthyStatus", + "status": 0, + "message": "success", + "data": { + "nodeStatus": ["Starting", "Unlock", "Locked", "Idle", "Busy", "Running", "ShuttingDown", "Failed", "Success"] + } +} +``` + +Notes: +- Returns all possible NodeStatus enum values +- With onlyEditable parameter, behavior is the same (returns all statuses) + +#### Execute Engine Conn Operation +``` +POST /api/rest_j/v1/linkisManager/executeEngineConnOperation +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295", + "parameters": { + "operation": "someOperation" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeEngineConnOperation", + "status": 0, + "message": "success", + "data": { + "result": "Operation result...", + "errorMsg": "", + "isError": false + } +} +``` + +Error Cases: +- If user doesn't have permission to execute operation, returns error +- If engine instance doesn't exist, returns error +- If operation fails, returns error details + +Notes: +- User must be owner of the engine or admin +- Executes arbitrary operations on engine nodes +- Returns operation result and error information + +#### Kill Engines By Creator Or EngineType +``` +POST /api/rest_j/v1/linkisManager/rm/killEngineByCreatorEngineType +``` + +Request Body: +```json +{ + "creator": "IDE", + "engineType": "hive-2.3.3" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/killEngineByCreatorEngineType", + "status": 0, + "message": "Kill engineConn succeed." +} +``` + +Error Cases: +- Only admin users can kill engines by creator or engine type +- If creator or engineType parameters are null, returns error + +Notes: +- Requires admin privileges +- Kills all engines matching creator and engine type +- Supports cross-cluster killing with additional parameters + +### EC Resource Info Management + +#### Get EC Resource Info +``` +GET /api/rest_j/v1/linkisManager/ecinfo/get +``` + +Parameters: +- `ticketid`: Ticket ID + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/get", + "status": 0, + "message": "success", + "data": { + "ecResourceInfoRecord": { + "id": 12345, + "labelValue": "spark-2.4.3", + "createUser": "testuser", + "serviceInstance": "bdp110:12295", + "ticketId": "ticket-12345", + "status": "Running", + "usedResource": "{\"cpu\": 2, \"memory\": \"2G\"}", + "releasedResource": "{\"cpu\": 0, \"memory\": \"0G\"}", + "requestResource": "{\"cpu\": 2, \"memory\": \"2G\"}" + } + } +} +``` + +Error Cases: +- If ticket ID doesn't exist, returns error +- If user doesn't have permission to access resource info, returns error + +Notes: +- User must be creator of the resource or admin +- Returns detailed EC resource information record +- Includes resource usage statistics + +#### Delete EC Resource Info +``` +DELETE /api/rest_j/v1/linkisManager/ecinfo/delete/{ticketid} +``` + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/delete/ticket-12345", + "status": 0, + "message": "success", + "data": { + "ecResourceInfoRecord": { + "id": 12345, + "labelValue": "spark-2.4.3", + "createUser": "testuser", + "serviceInstance": "bdp110:12295", + "ticketId": "ticket-12345", + "status": "Running" + } + } +} +``` + +Error Cases: +- If ticket ID doesn't exist, returns error +- If user doesn't have permission to delete resource info, returns error + +Notes: +- User must be creator of the resource or admin +- Deletes EC resource information record from database +- Returns deleted record information + +#### Query EC Resource History List +``` +GET /api/rest_j/v1/linkisManager/ecinfo/ecrHistoryList +``` + +Parameters: +- `instance` (optional): Filter by instance +- `creator` (optional): Filter by creator +- `startDate` (optional): Filter by start date +- `endDate` (optional): Filter by end date (defaults to current date) +- `engineType` (optional): Filter by engine type +- `status` (optional): Filter by status +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/ecrHistoryList", + "status": 0, + "message": "success", + "data": { + "engineList": [ + { + "id": 12345, + "labelValue": "spark-2.4.3", + "createUser": "testuser", + "serviceInstance": "bdp110:12295", + "ticketId": "ticket-12345", + "status": "Running", + "usedResource": { + "cpu": 2, + "memory": "2G" + }, + "releasedResource": { + "cpu": 0, + "memory": "0G" + }, + "requestResource": { + "cpu": 2, + "memory": "2G" + } + } + ], + "totalPage": 1 + } +} +``` + +Error Cases: +- If creator parameter is invalid, returns error +- If date parameters are invalid, uses defaults + +Notes: +- Admin users can view all records, regular users only their own +- Supports date range filtering +- Supports pagination +- Converts resource strings to maps for easier consumption + +#### Query EC List +``` +POST /api/rest_j/v1/linkisManager/ecinfo/ecList +``` + +Request Body: +```json +{ + "creators": ["testuser"], + "engineTypes": ["spark-2.4.3"], + "statuss": ["Running"], + "queueName": "default", + "ecInstances": ["bdp110:12295"], + "crossCluster": false +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/ecList", + "status": 0, + "message": "success", + "data": { + "ecList": [ + { + // EC information + } + ] + } +} +``` + +Error Cases: +- If creator parameter is invalid, returns error +- If parameters parsing fails, returns error + +Notes: +- Requires admin privileges or valid admin token +- Supports filtering by creators, engine types, statuses, queue name, and EC instances +- Supports cross-cluster filtering \ No newline at end of file diff --git a/.ai/modules/microservice-governance/README.md b/.ai/modules/microservice-governance/README.md new file mode 100644 index 00000000000..07035f1815b --- /dev/null +++ b/.ai/modules/microservice-governance/README.md @@ -0,0 +1,112 @@ +# Microservice Governance Services + +The microservice governance services provide the infrastructure foundation for the Linkis microservices architecture. + +## Service Modules + +- [Eureka Service](./eureka.md) - Service registry and discovery center +- [Gateway Service](./gateway.md) - API gateway for request routing and security + +## Overview + +These services form the infrastructure layer of Linkis, providing essential capabilities for service discovery, API routing, and inter-service communication. + +## Common Features + +### Service Discovery +- Service registration and deregistration +- Health checking of services +- Service instance management +- Load balancing support + +### API Gateway +- Request routing and filtering +- Authentication and authorization +- Rate limiting and traffic control +- Request/response transformation + +### Inter-Service Communication +- RESTful service communication +- Load balancing between services +- Circuit breaker pattern implementation +- Service monitoring and metrics + +## API Interface Summary + +### Eureka Service APIs +- Service registration: `POST /eureka/apps/{appName}` +- Service discovery: `GET /eureka/apps/{appName}` +- Health check: `GET /eureka/apps/{appName}/{instanceId}` + +### Gateway Service APIs +- Route management: `GET /actuator/gateway/routes` +- Health check: `GET /actuator/health` +- Gateway metrics: `GET /actuator/metrics` + +## Database Schema Summary + +### Service Registry Table +```sql +CREATE TABLE linkis_service_registry ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + service_name VARCHAR(128) NOT NULL, + instance_id VARCHAR(128) NOT NULL UNIQUE, + instance_address VARCHAR(128), + instance_port INT, + status VARCHAR(50) DEFAULT 'UP', + metadata JSON, + register_time DATETIME DEFAULT CURRENT_TIMESTAMP, + last_heartbeat DATETIME, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +### Gateway Route Table +```sql +CREATE TABLE linkis_gateway_route ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + route_id VARCHAR(128) NOT NULL UNIQUE, + route_order INT DEFAULT 0, + uri VARCHAR(255) NOT NULL, + predicates JSON, + filters JSON, + metadata JSON, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +### Gateway Access Log Table +```sql +CREATE TABLE linkis_gateway_access_log ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + client_ip VARCHAR(50), + request_method VARCHAR(10), + request_uri VARCHAR(500), + request_params TEXT, + user_token VARCHAR(255), + service_id VARCHAR(128), + response_status INT, + response_time BIGINT, + access_time DATETIME DEFAULT CURRENT_TIMESTAMP +); +``` + +## RPC Methods Summary + +### Eureka Service RPCs +- `registerService(ServiceRegistrationRequest request)` +- `unregisterService(String serviceId, String instanceId)` +- `getServiceInstances(String serviceName)` +- `heartbeat(String serviceId, String instanceId)` + +### Gateway Service RPCs +- `addRoute(GatewayRoute route)` +- `removeRoute(String routeId)` +- `updateRoute(GatewayRoute route)` +- `getRoutes()` +- `configureAuthentication(AuthenticationConfig config)` +- `validateToken(String token)` +- `getUserPermissions(String user)` +- `applyRateLimit(RateLimitConfig config)` +- `getRateLimitStatus(String clientId)` \ No newline at end of file diff --git a/.ai/modules/microservice-governance/eureka.md b/.ai/modules/microservice-governance/eureka.md new file mode 100644 index 00000000000..1345517e4a0 --- /dev/null +++ b/.ai/modules/microservice-governance/eureka.md @@ -0,0 +1,131 @@ +# Eureka Service + +The Eureka service provides service registration and discovery capabilities for the Linkis microservices architecture. + +## Overview + +This service implements the Eureka server for service discovery, managing the registration, discovery, and health checking of all microservice instances in the Linkis system. + +## Key Components + +### Core Classes +- `SpringCloudEurekaApplication` - Main application class +- Eureka server configuration +- Health check endpoints +- Service registry management + +### Features +- Service instance registration +- Service discovery for clients +- Health status monitoring +- REST API for service management + +## API Interfaces + +### Service Registration +``` +POST /eureka/apps/{appName} +``` + +Request Body: +```xml + + service-host + APP-NAME + 127.0.0.1 + 8080 + UP + +``` + +### Service Discovery +``` +GET /eureka/apps/{appName} +``` + +Response: +```xml + + APP-NAME + + service-host + APP-NAME + 127.0.0.1 + 8080 + UP + + +``` + +### Health Check +``` +GET /eureka/apps/{appName}/{instanceId} +``` + +Response: +```xml + + service-host + APP-NAME + 127.0.0.1 + 8080 + UP + 1234567890 + +``` + +## Database Table Structures + +The Eureka service typically doesn't directly manage database tables, as it stores service registry information in memory. However, it may interact with the following tables for persistent storage: + +### Service Registry Table +```sql +CREATE TABLE linkis_service_registry ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + service_name VARCHAR(128) NOT NULL, + instance_id VARCHAR(128) NOT NULL UNIQUE, + instance_address VARCHAR(128), + instance_port INT, + status VARCHAR(50) DEFAULT 'UP', + metadata JSON, + register_time DATETIME DEFAULT CURRENT_TIMESTAMP, + last_heartbeat DATETIME, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +## RPC Methods + +The Eureka service provides RPC methods for service management: + +### Service Management RPCs + +#### registerService +Registers a service instance: +```java +void registerService(ServiceRegistrationRequest request) +``` + +#### unregisterService +Unregisters a service instance: +```java +void unregisterService(String serviceId, String instanceId) +``` + +#### getServiceInstances +Retrieves instances of a service: +```java +List getServiceInstances(String serviceName) +``` + +#### heartbeat +Sends a heartbeat for a service instance: +```java +void heartbeat(String serviceId, String instanceId) +``` + +## Dependencies + +- Spring Cloud Netflix Eureka Server +- Spring Boot +- Netflix Eureka Core \ No newline at end of file diff --git a/.ai/modules/microservice-governance/gateway.md b/.ai/modules/microservice-governance/gateway.md new file mode 100644 index 00000000000..4d07a65e884 --- /dev/null +++ b/.ai/modules/microservice-governance/gateway.md @@ -0,0 +1,421 @@ +# Gateway Service + +The Gateway service provides API gateway functionality for the Linkis system, routing requests to appropriate backend services and providing security, rate limiting, and other cross-cutting concerns. + +## Overview + +This service implements an API gateway that serves as the single entry point for all client requests to the Linkis system. It handles request routing, authentication, authorization, rate limiting, and other infrastructure concerns. + +## Key Components + +### Core Classes +- `LinkisGatewayApplication` - Main application class +- Route configuration management +- Request/response filtering +- Authentication handling +- Rate limiting implementation + +### Features +- Request routing and load balancing +- Authentication and authorization +- Rate limiting and traffic control +- Request/response transformation +- SSL/TLS termination +- Logging and monitoring + +## API Interfaces + +### Route Management +``` +GET /actuator/gateway/routes +``` + +Response: +```json +{ + "routes": [ + { + "route_id": "linkis-entrance", + "uri": "lb://linkis-entrance", + "predicates": [ + "Path=/api/entrance/**" + ], + "filters": [ + "StripPrefix=1" + ] + } + ] +} +``` + +### Health Check +``` +GET /actuator/health +``` + +Response: +```json +{ + "status": "UP", + "components": { + "discoveryComposite": { + "status": "UP" + }, + "gateway": { + "status": "UP" + } + } +} +``` + +### Gateway Metrics +``` +GET /actuator/metrics +``` + +Response: +```json +{ + "names": [ + "gateway.requests", + "jvm.memory.used", + "http.server.requests" + ] +} +``` + +### Authentication Token Management +``` +GET /api/rest_j/v1/basedata-manager/gateway-auth-token +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "list": { + "total": 0, + "list": [], + "pageNum": 1, + "pageSize": 10, + "size": 0, + "startRow": 0, + "endRow": 0, + "pages": 0, + "prePage": 0, + "nextPage": 0, + "isFirstPage": true, + "isLastPage": true, + "hasPreviousPage": false, + "hasNextPage": false, + "navigatePages": 8, + "navigatepageNums": [] + } + } +} +``` + +### Add Authentication Token +``` +POST /api/rest_j/v1/basedata-manager/gateway-auth-token +``` + +Request Body: +```json +{ + "tokenName": "test-token", + "legalUsers": "*", + "businessOwner": "BDP" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Update Authentication Token +``` +PUT /api/rest_j/v1/basedata-manager/gateway-auth-token +``` + +Request Body: +```json +{ + "id": 1, + "tokenName": "test-token", + "legalUsers": "user1,user2", + "businessOwner": "BDP" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Get Authentication Token +``` +GET /api/rest_j/v1/basedata-manager/gateway-auth-token/{id} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "item": { + "id": 1, + "tokenName": "test-token", + "legalUsers": "user1,user2", + "businessOwner": "BDP", + "createTime": "2023-01-01 12:00:00", + "updateTime": "2023-01-01 12:00:00" + } + } +} +``` + +### Remove Authentication Token +``` +DELETE /api/rest_j/v1/basedata-manager/gateway-auth-token/{id} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Check Authentication Token +``` +GET /api/rest_j/v1/basedata-manager/gateway-auth-token/checkToken +``` + +Parameters: +- `token`: Authentication token to check (required) +- `checkName`: User name to check (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Decrypt Authentication Token +``` +GET /api/rest_j/v1/basedata-manager/gateway-auth-token/decrypt-token +``` + +Parameters: +- `token`: Authentication token to decrypt (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": "decrypted-token" + } +} +``` + +## Database Table Structures + +The Gateway service manages the following database tables: + +### Gateway Route Table +```sql +CREATE TABLE linkis_gateway_route ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + route_id VARCHAR(128) NOT NULL UNIQUE, + route_order INT DEFAULT 0, + uri VARCHAR(255) NOT NULL, + predicates JSON, + filters JSON, + metadata JSON, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +### Gateway Filter Table +```sql +CREATE TABLE linkis_gateway_filter ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + route_id VARCHAR(128) NOT NULL, + filter_name VARCHAR(128) NOT NULL, + filter_order INT DEFAULT 0, + args JSON, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + FOREIGN KEY (route_id) REFERENCES linkis_gateway_route(route_id) ON DELETE CASCADE +); +``` + +### Authentication Configuration Table +```sql +CREATE TABLE linkis_gateway_auth ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + path_pattern VARCHAR(255) NOT NULL, + auth_required BOOLEAN DEFAULT TRUE, + allowed_roles JSON, + rate_limit INT, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +### Gateway Access Log Table +```sql +CREATE TABLE linkis_gateway_access_log ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + client_ip VARCHAR(50), + request_method VARCHAR(10), + request_uri VARCHAR(500), + request_params TEXT, + user_token VARCHAR(255), + service_id VARCHAR(128), + response_status INT, + response_time BIGINT, + access_time DATETIME DEFAULT CURRENT_TIMESTAMP +); +``` + +### Gateway Auth Token Table +```sql +CREATE TABLE `linkis_gateway_auth_token` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `token_name` varchar(255) COLLATE utf8_bin NOT NULL, + `legal_users` varchar(255) COLLATE utf8_bin NOT NULL, + `create_by` varchar(255) COLLATE utf8_bin NOT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `elapse_day` bigint(20) DEFAULT '-1', + `update_by` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `business_owner` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `token_alias` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `token_sign` varchar(255) COLLATE utf8_bin DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `unique_token_name` (`token_name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin; +``` + +## RPC Methods + +The Gateway service provides RPC methods for gateway management: + +### Route Management RPCs + +#### addRoute +Adds a new route configuration: +```java +void addRoute(GatewayRoute route) +``` + +#### removeRoute +Removes a route configuration: +```java +void removeRoute(String routeId) +``` + +#### updateRoute +Updates a route configuration: +```java +void updateRoute(GatewayRoute route) +``` + +#### getRoutes +Retrieves all route configurations: +```java +List getRoutes() +``` + +### Authentication RPCs + +#### configureAuthentication +Configures authentication for a path: +```java +void configureAuthentication(AuthenticationConfig config) +``` + +#### validateToken +Validates an authentication token: +```java +TokenValidationResult validateToken(String token) +``` + +#### getUserPermissions +Retrieves user permissions: +```java +UserPermissions getUserPermissions(String user) +``` + +### Rate Limiting RPCs + +#### applyRateLimit +Applies rate limiting to a route: +```java +void applyRateLimit(RateLimitConfig config) +``` + +#### getRateLimitStatus +Retrieves current rate limit status: +```java +RateLimitStatus getRateLimitStatus(String clientId) +``` + +## Dependencies + +- Spring Cloud Gateway +- Spring Boot +- Spring Security +- Spring Cloud LoadBalancer +- linkis-common +- linkis-httpclient +- Various Spring Cloud components + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- GatewayAuthTokenRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/GatewayAuthTokenRestfulApi.java` + +### MyBatis XML Files +- GatewayRouteMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayRouteMapper.xml` +- GatewayFilterMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayFilterMapper.xml` +- GatewayAuthMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayAuthMapper.xml` +- GatewayAccessLogMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayAccessLogMapper.xml` +- GatewayAuthTokenMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayAuthTokenMapper.xml` \ No newline at end of file diff --git a/.ai/modules/microservice-governance/monitor.md b/.ai/modules/microservice-governance/monitor.md new file mode 100644 index 00000000000..1a52b1696ba --- /dev/null +++ b/.ai/modules/microservice-governance/monitor.md @@ -0,0 +1,261 @@ +# Monitor Service + +Monitor service is responsible for monitoring the health status of various components in the Linkis system, including resource monitoring, node heartbeat monitoring, etc. + +## Table of Contents +- [API Interfaces](#api-interfaces) +- [Database Tables](#database-tables) +- [RPC Methods](#rpc-methods) +- [Interface Classes and MyBatis XML Files](#interface-classes-and-mybatis-xml-files) + +## API Interfaces + +### Get Application List +**POST /linkisManager/rm/applicationlist** + +Get the list of applications for a specific user. + +Request Parameters: +```json +{ + "userCreator": "string", + "engineType": "string" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/applicationlist", + "status": 0, + "message": "OK", + "data": { + "applications": [] + } +} +``` + +### Reset User Resource +**DELETE /linkisManager/rm/resetResource** + +Reset user resource, admin only. + +Request Parameters: +- resourceId (optional): Integer + +Response: +```json +{ + "method": "/api/linkisManager/rm/resetResource", + "status": 0, + "message": "success", + "data": {} +} +``` + +### List All Engine Types +**GET /linkisManager/rm/engineType** + +Get all supported engine types. + +Response: +```json +{ + "method": "/api/linkisManager/rm/engineType", + "status": 0, + "message": "OK", + "data": { + "engineType": ["string"] + } +} +``` + +### Get All User Resources +**GET /linkisManager/rm/allUserResource** + +Get all user resources, admin only. + +Request Parameters: +- username (optional): String +- creator (optional): String +- engineType (optional): String +- page (optional): Integer +- size (optional): Integer + +Response: +```json +{ + "method": "/api/linkisManager/rm/allUserResource", + "status": 0, + "message": "OK", + "data": { + "resources": [], + "total": 0 + } +} +``` + +### Get User Resource by Label +**GET /linkisManager/rm/get-user-resource** + +Get user resource by label. + +Request Parameters: +- username: String +- creator: String +- engineType: String + +Response: +```json +{ + "method": "/api/linkisManager/rm/get-user-resource", + "status": 0, + "message": "OK", + "data": { + "resources": [] + } +} +``` + +### Get User Resources +**POST /linkisManager/rm/userresources** + +Get user resources. + +Request Parameters: +```json +{} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/userresources", + "status": 0, + "message": "OK", + "data": { + "userResources": [] + } +} +``` + +### Get Engines +**POST /linkisManager/rm/engines** + +Get engines for a user. + +Request Parameters: +```json +{} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/engines", + "status": 0, + "message": "OK", + "data": { + "engines": [] + } +} +``` + +### Get Queue Resource +**POST /linkisManager/rm/queueresources** + +Get queue resource information. + +Request Parameters: +```json +{ + "queuename": "string", + "clustername": "string", + "clustertype": "string", + "crossCluster": "boolean" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/queueresources", + "status": 0, + "message": "OK", + "data": { + "queueInfo": {}, + "userResources": [] + } +} +``` + +### Get Queues +**POST /linkisManager/rm/queues** + +Get queue information. + +Request Parameters: +```json +{} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/queues", + "status": 0, + "message": "OK", + "data": { + "queues": [] + } +} +``` + +## Database Tables + +### linkis_cg_rm_external_resource_provider +```sql +CREATE TABLE `linkis_cg_rm_external_resource_provider` ( + `id` int(10) NOT NULL AUTO_INCREMENT, + `resource_type` varchar(32) NOT NULL, + `name` varchar(32) NOT NULL, + `labels` varchar(32) DEFAULT NULL, + `config` text NOT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### linkis_cg_rm_resource_action_record +```sql +CREATE TABLE linkis_cg_rm_resource_action_record ( + `id` INT(20) NOT NULL AUTO_INCREMENT, + `label_value` VARCHAR(100) NOT NULL, + `ticket_id` VARCHAR(100) NOT NULL, + `request_times` INT(8), + `request_resource_all` VARCHAR(100), + `used_times` INT(8), + `used_resource_all` VARCHAR(100), + `release_times` INT(8), + `release_resource_all` VARCHAR(100), + `update_time` datetime DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE KEY `label_value_ticket_id` (`label_value`, `ticket_id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +``` + +## RPC Methods + +Monitor service does not expose specific RPC methods directly. It primarily works through the RESTful APIs listed above. + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- RMMonitorRest.scala: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\rm\restful\RMMonitorRest.scala` + +### MyBatis XML Files +The Monitor service uses the following persistence layer interfaces which may have corresponding MyBatis XML files: +- LabelManagerPersistence: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\persistence\LabelManagerPersistence.scala` +- ResourceManagerPersistence: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\persistence\ResourceManagerPersistence.scala` +- NodeManagerPersistence: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\persistence\NodeManagerPersistence.scala` +- NodeMetricManagerPersistence: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\persistence\NodeMetricManagerPersistence.scala` \ No newline at end of file diff --git a/.ai/modules/public-enhancements/README.md b/.ai/modules/public-enhancements/README.md new file mode 100644 index 00000000000..03433e33630 --- /dev/null +++ b/.ai/modules/public-enhancements/README.md @@ -0,0 +1,235 @@ +# Public Enhancement Services + +The public enhancement services provide shared capabilities used across the Linkis platform. + +## Service Modules + +- [Public Service](./publicservice.md) - Core public services +- [Configuration Service](./configuration.md) - Configuration management +- [BML Service](./bml.md) - Big Data Material Library +- [DataSource Service](./datasource.md) - Data source management +- [Context Service](./context.md) - Context and variable sharing +- [Monitor Service](./monitor.md) - System monitoring + +## Overview + +These services provide common capabilities that are used across the Linkis platform, including file management, configuration management, data source management, context sharing, and system monitoring. + +## Common Features + +### Resource Management +- Binary and material management +- User-defined function management +- Shared resource tracking + +### Configuration Management +- Centralized configuration service +- Runtime configuration management +- Configuration versioning + +### Context Management +- Cross-application context sharing +- Variable and parameter management +- Unified context service + +### Data Source Management +- Data source registration and management +- Metadata querying +- Connection testing and validation + +### System Monitoring +- Performance metrics collection +- System health monitoring +- Alerting and notifications + +## API Interface Summary + +### Public Service APIs +- File system operations +- Variable management +- Error code querying + +### Configuration Service APIs +- Configuration retrieval: `GET /api/rest_j/v1/configuration` +- Configuration update: `POST /api/rest_j/v1/configuration/update` +- Template management: `GET /api/rest_j/v1/configuration/template` + +### BML Service APIs +- File upload: `POST /api/rest_j/v1/bml/upload` +- File download: `GET /api/rest_j/v1/bml/download` +- File version list: `GET /api/rest_j/v1/bml/versions` + +### DataSource Service APIs +- Data source CRUD: `POST/GET/PUT/DELETE /api/rest_j/v1/datasource` +- Metadata query: `GET /api/rest_j/v1/datasource/metadata` +- Connection test: `POST /api/rest_j/v1/datasource/connect` + +### Context Service APIs +- Context creation: `POST /api/rest_j/v1/context` +- Variable management: `POST /api/rest_j/v1/context/variable` +- Context sharing: `POST /api/rest_j/v1/context/share` + +### Monitor Service APIs +- Metrics collection: `GET /api/rest_j/v1/monitor/metrics` +- Health check: `GET /api/rest_j/v1/monitor/health` +- Alert management: `POST /api/rest_j/v1/monitor/alert` + +## Database Schema Summary + +### BML Resources Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'resource uuid', + `is_private` TINYINT(1) DEFAULT 0 COMMENT 'Whether the resource is private, 0 means private, 1 means public', + `resource_header` TINYINT(1) DEFAULT 0 COMMENT 'Classification, 0 means unclassified, 1 means classified', + `downloaded_file_name` varchar(200) DEFAULT NULL COMMENT 'File name when downloading', + `sys` varchar(100) NOT NULL COMMENT 'Owning system', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Created time', + `owner` varchar(200) NOT NULL COMMENT 'Resource owner', + `is_expire` TINYINT(1) DEFAULT 0 COMMENT 'Whether expired, 0 means not expired, 1 means expired', + `expire_type` varchar(50) DEFAULT null COMMENT 'Expiration type, date refers to the expiration on the specified date, TIME refers to the time', + `expire_time` varchar(50) DEFAULT null COMMENT 'Expiration time, one day by default', + `max_version` int(20) DEFAULT 10 COMMENT 'The default is 10, which means to keep the latest 10 versions', + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Updated time', + `updator` varchar(50) DEFAULT NULL COMMENT 'updator', + `enable_flag` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Status, 1: normal, 0: frozen', + unique key `uniq_rid_eflag`(`resource_id`, `enable_flag`), + PRIMARY KEY (`id`) +); +``` + +### BML Resources Version Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources_version` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'Resource uuid', + `file_md5` varchar(32) NOT NULL COMMENT 'Md5 summary of the file', + `version` varchar(20) NOT NULL COMMENT 'Resource version (v plus five digits)', + `size` int(10) NOT NULL COMMENT 'File size', + `start_byte` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0, + `end_byte` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0, + `resource` varchar(2000) NOT NULL COMMENT 'Resource content (file information including path and file name)', + `description` varchar(2000) DEFAULT NULL COMMENT 'description', + `start_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Started time', + `end_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Stoped time', + `client_ip` varchar(200) NOT NULL COMMENT 'Client ip', + `updator` varchar(50) DEFAULT NULL COMMENT 'updator', + `enable_flag` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Status, 1: normal, 0: frozen', + unique key `uniq_rid_version`(`resource_id`, `version`), + PRIMARY KEY (`id`) +); +``` + +### Configuration Key Table +```sql +CREATE TABLE `linkis_ps_configuration_config_key`( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `key` varchar(50) DEFAULT NULL COMMENT 'Set key, e.g. spark.executor.instances', + `description` varchar(200) DEFAULT NULL, + `name` varchar(50) DEFAULT NULL, + `default_value` varchar(200) DEFAULT NULL COMMENT 'Adopted when user does not set key', + `validate_type` varchar(50) DEFAULT NULL COMMENT 'Validate type, one of the following: None, NumInterval, FloatInterval, Include, Regex, OPF, Custom Rules', + `validate_range` varchar(150) DEFAULT NULL COMMENT 'Validate range', + `engine_conn_type` varchar(50) DEFAULT '' COMMENT 'engine type,such as spark,hive etc', + `is_hidden` tinyint(1) DEFAULT NULL COMMENT 'Whether it is hidden from user. If set to 1(true), then user cannot modify, however, it could still be used in back-end', + `is_advanced` tinyint(1) DEFAULT NULL COMMENT 'Whether it is an advanced parameter. If set to 1(true), parameters would be displayed only when user choose to do so', + `level` tinyint(1) DEFAULT NULL COMMENT 'Basis for displaying sorting in the front-end. Higher the level is, higher the rank the parameter gets', + `treeName` varchar(20) DEFAULT NULL COMMENT 'Reserved field, representing the subdirectory of engineType', + `boundary_type` TINYINT(2) NULL DEFAULT '0' COMMENT '0 none/ 1 with mix /2 with max / 3 min and max both', + `en_description` varchar(200) DEFAULT NULL COMMENT 'english description', + `en_name` varchar(100) DEFAULT NULL COMMENT 'english name', + `en_treeName` varchar(100) DEFAULT NULL COMMENT 'english treeName', + `template_required` tinyint(1) DEFAULT 0 COMMENT 'template required 0 none / 1 must', + UNIQUE INDEX `uniq_key_ectype` (`key`,`engine_conn_type`), + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Table +```sql +CREATE TABLE `linkis_ps_datasource_table` ( + `id` bigint(255) NOT NULL AUTO_INCREMENT, + `database` varchar(64) COLLATE utf8_bin NOT NULL, + `name` varchar(64) COLLATE utf8_bin NOT NULL, + `alias` varchar(64) COLLATE utf8_bin DEFAULT NULL, + `creator` varchar(16) COLLATE utf8_bin NOT NULL, + `comment` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `create_time` datetime NOT NULL, + `product_name` varchar(64) COLLATE utf8_bin DEFAULT NULL, + `project_name` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `usage` varchar(128) COLLATE utf8_bin DEFAULT NULL, + `lifecycle` int(4) NOT NULL, + `use_way` int(4) NOT NULL, + `is_import` tinyint(1) NOT NULL, + `model_level` int(4) NOT NULL, + `is_external_use` tinyint(1) NOT NULL, + `is_partition_table` tinyint(1) NOT NULL, + `is_available` tinyint(1) NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_db_name` (`database`,`name`) +); +``` + +### Context Map Table +```sql +CREATE TABLE `linkis_ps_cs_context_map` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `key` varchar(128) DEFAULT NULL, + `context_scope` varchar(32) DEFAULT NULL, + `context_type` varchar(32) DEFAULT NULL, + `props` text, + `value` mediumtext, + `context_id` int(11) DEFAULT NULL, + `keywords` varchar(255) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_key_cid_ctype` (`key`,`context_id`,`context_type`), + KEY `idx_keywords` (`keywords`(191)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +## RPC Methods Summary + +### Public Service RPCs +- `getErrorCode(String errorCode)` +- `getVariable(String variableName)` +- `setVariable(String variableName, String value)` + +### Configuration Service RPCs +- `getConfiguration(String user, String creator, String engineType)` +- `updateConfiguration(String user, ConfigurationUpdateRequest request)` +- `getTemplateConfiguration(String engineType)` + +### BML Service RPCs +- `uploadResource(ResourceUploadRequest request)` +- `downloadResource(String resourceId, String version)` +- `deleteResource(String resourceId)` +- `getResourceInfo(String resourceId)` + +### DataSource Service RPCs +- `createDataSource(DataSourceCreateRequest request)` +- `updateDataSource(DataSourceUpdateRequest request)` +- `deleteDataSource(Long dataSourceId)` +- `queryDataSource(DataSourceQueryRequest request)` + +### Context Service RPCs +- `createContext(ContextCreateRequest request)` +- `getContextValue(String contextId, String key)` +- `setContextValue(String contextId, String key, String value)` +- `removeContext(String contextId)` + +### Monitor Service RPCs +- `collectMetrics(MetricsCollectionRequest request)` +- `getHealthStatus()` +- `sendAlert(AlertRequest request)` + +## Dependencies + +- linkis-commons - Shared utilities +- linkis-protocol - Communication protocols +- linkis-rpc - Remote procedure calls +- Various database drivers +- Spring Cloud ecosystem \ No newline at end of file diff --git a/.ai/modules/public-enhancements/bml.md b/.ai/modules/public-enhancements/bml.md new file mode 100644 index 00000000000..6f5673e09af --- /dev/null +++ b/.ai/modules/public-enhancements/bml.md @@ -0,0 +1,634 @@ +# BML Service + +The BML (Big Data Material Library) Service provides file and material management capabilities for the Linkis system. + +## Overview + +This service manages the storage, versioning, and sharing of files and materials used in big data processing tasks. + +## Key Components + +### Core Classes +- `LinkisBMLApplication` - Main application class +- File upload and download +- File version management +- File sharing and access control + +### Features +- File upload and download +- File versioning +- File sharing +- Access control +- File metadata management + +## API Interfaces + +### File Upload +``` +POST /api/rest_j/v1/bml/upload +``` + +Parameters: +- `system` (optional): System name +- `resourceHeader` (optional): Resource header +- `isExpire` (optional): Whether resource expires +- `expireType` (optional): Expiration type +- `expireTime` (optional): Expiration time +- `maxVersion` (optional): Maximum version count +- `file` (required): File to upload + +Response: +``` +{ + "method": "/api/bml/upload", + "status": 0, + "message": "The task of submitting and uploading resources was successful(提交上传资源任务成功)", + "data": { + "resourceId": "resource-12345", + "version": "v000001", + "taskId": 12345 + } +} +``` + +### File Download +``` +GET /api/rest_j/v1/bml/download +``` + +Parameters: +- `resourceId`: Resource ID to download (required) +- `version`: Version to download (optional, defaults to latest) + +Response: +``` +Binary file content +``` + +### File Version List +``` +GET /api/rest_j/v1/bml/getVersions +``` + +Parameters: +- `resourceId`: Resource ID to list versions for (required) +- `currentPage`: Current page number (optional) +- `pageSize`: Page size (optional) + +Response: +```json +{ + "method": "/api/bml/getVersions", + "status": 0, + "message": "Version information obtained successfully (成功获取版本信息)", + "data": { + "ResourceVersions": { + "resourceId": "resource-12345", + "user": "testuser", + "versions": [ + { + "version": "v000001", + "size": 1024, + "createTime": "2023-01-01 12:00:00" + } + ] + } + } +} +``` + +### File Update +``` +POST /api/rest_j/v1/bml/updateVersion +``` + +Parameters: +- `resourceId`: Resource ID to update (required) +- `file`: File to upload (required) + +Response: +```json +{ + "method": "/api/bml/updateVersion", + "status": 0, + "message": "The update resource task was submitted successfully(提交更新资源任务成功)", + "data": { + "resourceId": "resource-12345", + "version": "v000002", + "taskId": 12346 + } +} +``` + +### File Delete +``` +POST /api/rest_j/v1/bml/deleteResource +``` + +Request Body: +```json +{ + "resourceId": "resource-12345" +} +``` + +Response: +```json +{ + "method": "/api/bml/deleteResource", + "status": 0, + "message": "Resource deleted successfully(删除资源成功)" +} +``` + +### Batch File Delete +``` +POST /api/rest_j/v1/bml/deleteResources +``` + +Request Body: +```json +{ + "resourceIds": ["resource-12345", "resource-12346"] +} +``` + +Response: +```json +{ + "method": "/api/bml/deleteResources", + "status": 0, + "message": "Batch deletion of resource was successful(批量删除资源成功)" +} +``` + +### File Information +``` +GET /api/rest_j/v1/bml/getBasic +``` + +Parameters: +- `resourceId`: Resource ID to get information for (required) + +Response: +```json +{ + "method": "/api/bml/getBasic", + "status": 0, + "message": "Acquisition of resource basic information successfully(获取资源基本信息成功)", + "data": { + "basic": { + "resourceId": "resource-12345", + "owner": "testuser", + "createTime": "2023-01-01 12:00:00", + "downloadedFileName": "test.csv", + "expireTime": "Resource not expired(资源不过期)", + "numberOfVerions": 10 + } + } +} +``` + +### Version Delete +``` +POST /api/rest_j/v1/bml/deleteVersion +``` + +Request Body: +```json +{ + "resourceId": "resource-12345", + "version": "v000001" +} +``` + +Response: +```json +{ + "method": "/api/bml/deleteVersion", + "status": 0, + "message": "Deleted version successfully(删除版本成功)" +} +``` + +### Change Owner +``` +POST /api/rest_j/v1/bml/changeOwner +``` + +Request Body: +```json +{ + "resourceId": "resource-12345", + "oldOwner": "testuser", + "newOwner": "newuser" +} +``` + +Response: +```json +{ + "method": "/api/bml/changeOwner", + "status": 0, + "message": "更新owner成功!" +} +``` + +### Copy Resource To Another User +``` +POST /api/rest_j/v1/bml/copyResourceToAnotherUser +``` + +Request Body: +```json +{ + "resourceId": "resource-12345", + "anotherUser": "newuser" +} +``` + +Response: +```json +{ + "method": "/api/bml/copyResourceToAnotherUser", + "status": 0, + "message": "success", + "data": { + "resourceId": "resource-67890" + } +} +``` + +### Rollback Version +``` +POST /api/rest_j/v1/bml/rollbackVersion +``` + +Request Body: +```json +{ + "resourceId": "resource-12345", + "version": "v000001" +} +``` + +Response: +```json +{ + "method": "/api/bml/rollbackVersion", + "status": 0, + "message": "success", + "data": { + "resourceId": "resource-12345", + "version": "v000001" + } +} +``` + +### Create BML Project +``` +POST /api/rest_j/v1/bml/createBmlProject +``` + +Request Body: +```json +{ + "projectName": "test-project", + "editUsers": ["user1", "user2"], + "accessUsers": ["user3", "user4"] +} +``` + +Response: +```json +{ + "method": "/api/bml/createBmlProject", + "status": 0, + "message": "success to create project(创建工程ok)" +} +``` + +### Upload Share Resource +``` +POST /api/rest_j/v1/bml/uploadShareResource +``` + +Parameters: +- `system` (optional): System name +- `resourceHeader` (optional): Resource header +- `isExpire` (optional): Whether resource expires +- `expireType` (optional): Expiration type +- `expireTime` (optional): Expiration time +- `maxVersion` (optional): Maximum version count +- `projectName`: Project name (required) +- `file`: File to upload (required) + +Response: +```json +{ + "method": "/api/bml/uploadShareResource", + "status": 0, + "message": "The task of submitting and uploading resources was successful(提交上传资源任务成功)", + "data": { + "resourceId": "resource-12345", + "version": "v000001", + "taskId": 12345 + } +} +``` + +### Update Share Resource +``` +POST /api/rest_j/v1/bml/updateShareResource +``` + +Parameters: +- `resourceId`: Resource ID to update (required) +- `file`: File to upload (required) + +Response: +```json +{ + "method": "/api/bml/updateShareResource", + "status": 0, + "message": "The update resource task was submitted successfully(提交更新资源任务成功)", + "data": { + "resourceId": "resource-12345", + "version": "v000002", + "taskId": 12346 + } +} +``` + +### Download Share Resource +``` +GET /api/rest_j/v1/bml/downloadShareResource +``` + +Parameters: +- `resourceId`: Resource ID to download (required) +- `version`: Version to download (optional, defaults to latest) + +Response: +``` +Binary file content +``` + +### Update Project Users +``` +POST /api/rest_j/v1/bml/updateProjectUsers +``` + +Request Body: +```json +{ + "projectName": "test-project", + "editUsers": ["user1", "user2", "user5"], + "accessUsers": ["user3", "user4", "user6"] +} +``` + +Response: +```json +{ + "method": "/api/bml/updateProjectUsers", + "status": 0, + "message": "Updated project related user success(更新工程的相关用户成功)" +} +``` + +## Database Table Structures + +The BML Service uses the following database tables from linkis_ddl.sql: + +### BML Resources Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'resource uuid', + `is_private` TINYINT(1) DEFAULT 0 COMMENT 'Whether the resource is private, 0 means private, 1 means public', + `resource_header` TINYINT(1) DEFAULT 0 COMMENT 'Classification, 0 means unclassified, 1 means classified', + `downloaded_file_name` varchar(200) DEFAULT NULL COMMENT 'File name when downloading', + `sys` varchar(100) NOT NULL COMMENT 'Owning system', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Created time', + `owner` varchar(200) NOT NULL COMMENT 'Resource owner', + `is_expire` TINYINT(1) DEFAULT 0 COMMENT 'Whether expired, 0 means not expired, 1 means expired', + `expire_type` varchar(50) DEFAULT null COMMENT 'Expiration type, date refers to the expiration on the specified date, TIME refers to the time', + `expire_time` varchar(50) DEFAULT null COMMENT 'Expiration time, one day by default', + `max_version` int(20) DEFAULT 10 COMMENT 'The default is 10, which means to keep the latest 10 versions', + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Updated time', + `updator` varchar(50) DEFAULT NULL COMMENT 'updator', + `enable_flag` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Status, 1: normal, 0: frozen', + unique key `uniq_rid_eflag`(`resource_id`, `enable_flag`), + PRIMARY KEY (`id`) +); +``` + +### BML Resources Version Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources_version` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'Resource uuid', + `file_md5` varchar(32) NOT NULL COMMENT 'Md5 summary of the file', + `version` varchar(20) NOT NULL COMMENT 'Resource version (v plus five digits)', + `size` int(10) NOT NULL COMMENT 'File size', + `start_byte` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0, + `end_byte` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0, + `resource` varchar(2000) NOT NULL COMMENT 'Resource content (file information including path and file name)', + `description` varchar(2000) DEFAULT NULL COMMENT 'description', + `start_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Started time', + `end_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Stoped time', + `client_ip` varchar(200) NOT NULL COMMENT 'Client ip', + `updator` varchar(50) DEFAULT NULL COMMENT 'updator', + `enable_flag` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Status, 1: normal, 0: frozen', + unique key `uniq_rid_version`(`resource_id`, `version`), + PRIMARY KEY (`id`) +); +``` + +### BML Resources Permission Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources_permission` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'Resource uuid', + `permission` varchar(10) NOT NULL COMMENT 'permission', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'created time', + `system` varchar(50) default "dss" COMMENT 'creator', + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'updated time', + `updator` varchar(50) NOT NULL COMMENT 'updator', + PRIMARY KEY (`id`) +); +``` + +### BML Resources Task Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources_task` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `resource_id` varchar(50) DEFAULT NULL COMMENT 'resource uuid', + `version` varchar(20) DEFAULT NULL COMMENT 'Resource version number of the current operation', + `operation` varchar(20) NOT NULL COMMENT 'Operation type. upload = 0, update = 1', + `state` varchar(20) NOT NULL DEFAULT 'Schduled' COMMENT 'Current status of the task:Schduled, Running, Succeed, Failed,Cancelled', + `submit_user` varchar(20) NOT NULL DEFAULT '' COMMENT 'Job submission user name', + `system` varchar(20) DEFAULT 'dss' COMMENT 'Subsystem name: wtss', + `instance` varchar(128) NOT NULL COMMENT 'Material library example', + `client_ip` varchar(50) DEFAULT NULL COMMENT 'Request IP', + `extra_params` text COMMENT 'Additional key information. Such as the resource IDs and versions that are deleted in batches, and all versions under the resource are deleted', + `err_msg` varchar(2000) DEFAULT NULL COMMENT 'Task failure information.e.getMessage', + `start_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Starting time', + `end_time` datetime DEFAULT NULL COMMENT 'End Time', + `last_update_time` datetime NOT NULL COMMENT 'Last update time', + unique key `uniq_rid_version` (resource_id, version), + PRIMARY KEY (`id`) +); +``` + +### BML Project Table +```sql +create table if not exists linkis_ps_bml_project( + `id` int(10) NOT NULL AUTO_INCREMENT, + `name` varchar(128) DEFAULT NULL, + `system` varchar(64) not null default "dss", + `source` varchar(1024) default null, + `description` varchar(1024) default null, + `creator` varchar(128) not null, + `enabled` tinyint default 1, + `create_time` datetime DEFAULT now(), + unique key `uniq_name` (`name`), +PRIMARY KEY (`id`) +); +``` + +## RPC Methods + +The BML Service provides several RPC methods for file management: + +### Resource RPCs + +#### uploadResource +Uploads a resource: +```java +ResourceUploadResult uploadResource(ResourceUploadRequest request) +``` + +#### downloadResource +Downloads a resource: +```java +ResourceContent downloadResource(String resourceId, String version) +``` + +#### deleteResource +Deletes a resource: +```java +void deleteResource(String resourceId) +``` + +#### getResourceInfo +Retrieves resource information: +```java +ResourceInfo getResourceInfo(String resourceId) +``` + +#### listResources +Lists resources for a user: +```java +List listResources(String username) +``` + +### Version RPCs + +#### listVersions +Lists versions of a resource: +```java +List listVersions(String resourceId) +``` + +#### updateResource +Updates a resource with a new version: +```java +ResourceUpdateResult updateResource(ResourceUpdateRequest request) +``` + +#### deleteVersion +Deletes a specific version of a resource: +```java +void deleteVersion(String resourceId, String version) +``` + +#### getVersionInfo +Gets information about a specific version: +```java +ResourceVersion getVersionInfo(String resourceId, String version) +``` + +### Permission RPCs + +#### grantPermission +Grants permission to a user: +```java +void grantPermission(String resourceId, String username, String permission) +``` + +#### checkPermission +Checks if a user has permission: +```java +boolean checkPermission(String resourceId, String username, String permission) +``` + +#### revokePermission +Revokes permission from a user: +```java +void revokePermission(String resourceId, String username) +``` + +#### listPermissions +Lists all permissions for a resource: +```java +List listPermissions(String resourceId) +``` + +### Project RPCs + +#### createProject +Creates a new project: +```java +Project createProject(ProjectCreateRequest request) +``` + +#### deleteProject +Deletes a project: +```java +void deleteProject(Long projectId) +``` + +#### addResourceToProject +Adds a resource to a project: +```java +void addResourceToProject(Long projectId, String resourceId) +``` + +#### removeResourceFromProject +Removes a resource from a project: +```java +void removeResourceFromProject(Long projectId, String resourceId) +``` + +## Dependencies + +- linkis-bml-server +- linkis-mybatis +- linkis-rpc +- linkis-protocol + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- BmlRestfulApi: `linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlRestfulApi.java` +- BmlProjectRestful: `linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlProjectRestful.java` +- BMLFsRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/filesystem/restful/api/BMLFsRestfulApi.java` + +### MyBatis XML Files +- ResourceMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/ResourceMapper.xml` +- VersionMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/VersionMapper.xml` +- TaskMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/TaskMapper.xml` +- DownloadMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/DownloadMapper.xml` +- BmlProjectMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/BmlProjectMapper.xml` diff --git a/.ai/modules/public-enhancements/configuration.md b/.ai/modules/public-enhancements/configuration.md new file mode 100644 index 00000000000..01b1e58352f --- /dev/null +++ b/.ai/modules/public-enhancements/configuration.md @@ -0,0 +1,1264 @@ +# Configuration Service + +The Configuration Service provides centralized configuration management for the Linkis system. + +## Overview + +This service manages configuration properties for all Linkis components, providing a unified interface for configuration retrieval, updates, and validation. + +## Key Components + +### Core Classes +- `LinkisConfigurationApp` - Main application class +- Configuration management +- Configuration validation +- Template management + +### Features +- Global configuration management +- User-specific configuration +- Configuration validation +- Template-based configuration +- Engine-type specific configuration + +## API Interfaces + +### Configuration Retrieval APIs + +#### Get Full Configuration Trees +``` +GET /api/rest_j/v1/configuration/getFullTreesByAppName +``` + +Parameters: +- `engineType`: Engine type (e.g., spark, hive) - optional +- `version`: Engine version - optional +- `creator`: Creator application - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/getFullTreesByAppName", + "status": 0, + "message": "success", + "data": { + "fullTree": [ + { + "name": "JVM Configuration", + "description": "JVM configuration for engine", + "settings": [ + { + "key": "wds.linkis.engineconn.java.driver.memory", + "value": "2g", + "defaultValue": "1g", + "description": "Memory size of driver JVM process", + "validateType": "Regex", + "validateRange": "^[0-9]+(\\.?[0-9]*)([gGmMkK])?$", + "level": 1, + "hidden": false, + "advanced": false + } + ] + } + ] + } +} +``` + +#### Get Configuration Category +``` +GET /api/rest_j/v1/configuration/getCategory +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/getCategory", + "status": 0, + "message": "success", + "data": { + "Category": [ + { + "categoryId": 1, + "categoryName": "Engine Resource", + "description": "Engine resource configuration" + } + ] + } +} +``` + +#### Get Configuration Item List +``` +GET /api/rest_j/v1/configuration/getItemList +``` + +Parameters: +- `engineType`: Engine type (e.g., spark, hive) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/getItemList", + "status": 0, + "message": "success", + "data": { + "itemList": [ + { + "key": "spark.executor.instances", + "name": "Executor Instances", + "description": "Number of executor instances", + "engineType": "spark", + "validateType": "NumInterval", + "validateRange": "[1,20]", + "boundaryType": 3, + "defaultValue": "1", + "require": 0 + } + ] + } +} +``` + +#### List All Engine Types +``` +GET /api/rest_j/v1/configuration/engineType +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/engineType", + "status": 0, + "message": "success", + "data": { + "engineType": ["spark", "hive", "python"] + } +} +``` + +#### Get Key Value +``` +GET /api/rest_j/v1/configuration/keyvalue +``` + +Parameters: +- `engineType`: Engine type - default "*" +- `version`: Engine version - default "*" +- `creator`: Creator application - default "*" +- `configKey`: Configuration key (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/keyvalue", + "status": 0, + "message": "success", + "data": { + "configValues": [ + { + "id": 1, + "configKeyId": 1, + "configValue": "2g", + "configLabelId": 1 + } + ] + } +} +``` + +#### Get Base Key Value +``` +GET /api/rest_j/v1/configuration/baseKeyValue +``` + +Parameters: +- `engineType`: Engine type - optional +- `key`: Configuration key - optional +- `pageNow`: Page number - default 1 +- `pageSize`: Page size - default 20 + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/baseKeyValue", + "status": 0, + "message": "success", + "data": { + "configKeyList": [ + { + "id": 1, + "key": "spark.executor.instances", + "description": "Number of executor instances", + "name": "Executor Instances", + "defaultValue": "1", + "validateType": "NumInterval", + "validateRange": "[1,20]", + "engineConnType": "spark", + "isHidden": 0, + "isAdvanced": 0, + "level": 1, + "treeName": "Spark Configuration", + "boundaryType": 3, + "enDescription": "Number of executor instances", + "enName": "Executor Instances", + "enTreeName": "Spark Configuration", + "templateRequired": 0 + } + ], + "totalPage": 10 + } +} +``` + +#### Get User Key Value +``` +GET /api/rest_j/v1/configuration/userKeyValue +``` + +Parameters: +- `engineType`: Engine type - optional +- `key`: Configuration key - optional +- `creator`: Creator application - optional +- `user`: Username - optional +- `pageNow`: Page number - default 1 +- `pageSize`: Page size - default 20 + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/userKeyValue", + "status": 0, + "message": "success", + "data": { + "configValueList": [ + { + "id": 1, + "configKeyId": 1, + "configValue": "2", + "configLabelId": 1, + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "engineType": "spark", + "key": "spark.executor.instances", + "creator": "IDE", + "user": "testuser" + } + ], + "totalPage": 1 + } +} +``` + +### Configuration Management APIs + +#### Create First Category +``` +POST /api/rest_j/v1/configuration/createFirstCategory +``` + +Request Body: +```json +{ + "categoryName": "Engine Resource", + "description": "Engine resource configuration" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/createFirstCategory", + "status": 0, + "message": "success" +} +``` + +#### Delete Category +``` +POST /api/rest_j/v1/configuration/deleteCategory +``` + +Request Body: +```json +{ + "categoryId": 1 +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/deleteCategory", + "status": 0, + "message": "success" +} +``` + +#### Create Second Category +``` +POST /api/rest_j/v1/configuration/createSecondCategory +``` + +Request Body: +```json +{ + "categoryId": 1, + "engineType": "spark", + "version": "2.4.3", + "description": "Spark 2.4.3 configuration" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/createSecondCategory", + "status": 0, + "message": "success" +} +``` + +#### Update Category Info +``` +POST /api/rest_j/v1/configuration/updateCategoryInfo +``` + +Request Body: +```json +{ + "categoryId": 1, + "description": "Updated description" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/updateCategoryInfo", + "status": 0, + "message": "success" +} +``` + +#### Save Full Tree +``` +POST /api/rest_j/v1/configuration/saveFullTree +``` + +Request Body: +```json +{ + "creator": "IDE", + "engineType": "spark-2.4.3", + "fullTree": [ + { + "name": "JVM Configuration", + "description": "JVM configuration for engine", + "settings": [ + { + "key": "wds.linkis.engineconn.java.driver.memory", + "value": "2g", + "defaultValue": "1g", + "description": "Memory size of driver JVM process", + "validateType": "Regex", + "validateRange": "^[0-9]+(\\.?[0-9]*)([gGmMkK])?$", + "level": 1, + "hidden": false, + "advanced": false + } + ] + } + ] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/saveFullTree", + "status": 0, + "message": "success" +} +``` + +#### Save Key Value +``` +POST /api/rest_j/v1/configuration/keyvalue +``` + +Request Body: +```json +{ + "engineType": "spark", + "version": "2.4.3", + "creator": "IDE", + "configKey": "spark.executor.instances", + "configValue": "2", + "user": "testuser" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/keyvalue", + "status": 0, + "message": "success", + "data": { + "configValue": { + "id": 1, + "configKeyId": 1, + "configValue": "2", + "configLabelId": 1 + } + } +} +``` + +#### Delete Key Value +``` +DELETE /api/rest_j/v1/configuration/keyvalue +``` + +Request Body: +```json +{ + "engineType": "spark", + "version": "2.4.3", + "creator": "IDE", + "configKey": "spark.executor.instances" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/keyvalue", + "status": 0, + "message": "success", + "data": { + "configValues": [ + { + "id": 1, + "configKeyId": 1, + "configValue": "2", + "configLabelId": 1 + } + ] + } +} +``` + +#### Save Base Key Value +``` +POST /api/rest_j/v1/configuration/baseKeyValue +``` + +Request Body: +```json +{ + "key": "spark.executor.instances", + "name": "Executor Instances", + "description": "Number of executor instances", + "defaultValue": "1", + "validateType": "NumInterval", + "validateRange": "[1,20]", + "boundaryType": 3, + "treeName": "Spark Configuration", + "engineType": "spark", + "enDescription": "Number of executor instances", + "enName": "Executor Instances", + "enTreeName": "Spark Configuration", + "templateRequired": 0 +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/baseKeyValue", + "status": 0, + "message": "success", + "data": { + "configKey": { + "id": 1, + "key": "spark.executor.instances", + "description": "Number of executor instances", + "name": "Executor Instances", + "defaultValue": "1", + "validateType": "NumInterval", + "validateRange": "[1,20]", + "engineConnType": "spark", + "isHidden": 0, + "isAdvanced": 0, + "level": 1, + "treeName": "Spark Configuration", + "boundaryType": 3, + "enDescription": "Number of executor instances", + "enName": "Executor Instances", + "enTreeName": "Spark Configuration", + "templateRequired": 0 + } + } +} +``` + +#### Delete Base Key Value +``` +DELETE /api/rest_j/v1/configuration/baseKeyValue +``` + +Parameters: +- `id`: Configuration key ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/baseKeyValue", + "status": 0, + "message": "success" +} +``` + +### Template Management APIs + +#### Update Key Mapping +``` +POST /api/rest_j/v1/configuration/template/updateKeyMapping +``` + +Request Body: +```json +{ + "templateUid": "template-uuid", + "templateName": "Spark Template", + "engineType": "spark", + "operator": "admin", + "isFullMode": true, + "itemList": [ + { + "keyId": 1, + "maxValue": "10", + "minValue": "1" + } + ] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/template/updateKeyMapping", + "status": 0, + "message": "success" +} +``` + +#### Query Key Info List +``` +POST /api/rest_j/v1/configuration/template/queryKeyInfoList +``` + +Request Body: +```json +{ + "templateUidList": ["template-uuid-1", "template-uuid-2"] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/template/queryKeyInfoList", + "status": 0, + "message": "success", + "data": { + "list": [ + { + "templateName": "Spark Template", + "engineType": "spark", + "configKey": "spark.executor.instances", + "maxValue": "10", + "minValue": "1" + } + ] + } +} +``` + +#### Apply Configuration Template +``` +POST /api/rest_j/v1/configuration/template/apply +``` + +Request Body: +```json +{ + "templateUid": "template-uuid", + "application": "IDE", + "engineType": "spark", + "engineVersion": "2.4.3", + "operator": "admin", + "userList": ["user1", "user2"] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/template/apply", + "status": 0, + "message": "success", + "data": { + "success": true, + "failedUsers": [] + } +} +``` + +#### Encrypt Datasource Password +``` +GET /api/rest_j/v1/configuration/template/encrypt +``` + +Parameters: +- `isEncrypt`: Encrypt flag - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/template/encrypt", + "status": 0, + "message": "success" +} +``` + +### Tenant Configuration APIs + +#### Create Tenant +``` +POST /api/rest_j/v1/configuration/tenant-mapping/create-tenant +``` + +Request Body: +```json +{ + "user": "testuser", + "creator": "IDE", + "tenantValue": "tenant1", + "desc": "Test tenant", + "bussinessUser": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/create-tenant", + "status": 0, + "message": "success" +} +``` + +#### Update Tenant +``` +POST /api/rest_j/v1/configuration/tenant-mapping/update-tenant +``` + +Request Body: +```json +{ + "id": 1, + "user": "testuser", + "creator": "IDE", + "tenantValue": "tenant1-updated", + "desc": "Updated tenant", + "bussinessUser": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/update-tenant", + "status": 0, + "message": "success" +} +``` + +#### Delete Tenant +``` +GET /api/rest_j/v1/configuration/tenant-mapping/delete-tenant +``` + +Parameters: +- `id`: Tenant ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/delete-tenant", + "status": 0, + "message": "success" +} +``` + +#### Query Tenant List +``` +GET /api/rest_j/v1/configuration/tenant-mapping/query-tenant-list +``` + +Parameters: +- `user`: Username - optional +- `creator`: Creator application - optional +- `tenantValue`: Tenant value - optional +- `pageNow`: Page number - default 1 +- `pageSize`: Page size - default 20 + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/query-tenant-list", + "status": 0, + "message": "success", + "data": { + "tenantList": [ + { + "id": 1, + "user": "testuser", + "creator": "IDE", + "tenantValue": "tenant1", + "desc": "Test tenant", + "bussinessUser": "admin", + "createTime": "2023-01-01 12:00:00", + "updateTime": "2023-01-01 12:00:00" + } + ], + "totalPage": 1 + } +} +``` + +#### Check User Creator +``` +GET /api/rest_j/v1/configuration/tenant-mapping/check-user-creator +``` + +Parameters: +- `user`: Username (required) +- `creator`: Creator application (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/check-user-creator", + "status": 0, + "message": "success", + "data": { + "exist": true + } +} +``` + +#### Save Department Tenant +``` +POST /api/rest_j/v1/configuration/tenant-mapping/save-department-tenant +``` + +Request Body: +```json +{ + "creator": "IDE", + "department": "Engineering", + "departmentId": "dept1", + "tenantValue": "tenant1", + "createBy": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/save-department-tenant", + "status": 0, + "message": "success" +} +``` + +#### Query Department Tenant +``` +GET /api/rest_j/v1/configuration/tenant-mapping/query-department-tenant +``` + +Parameters: +- `departmentId`: Department ID - optional +- `department`: Department name - optional +- `creator`: Creator application - optional +- `tenantValue`: Tenant value - optional +- `pageNow`: Page number - default 1 +- `pageSize`: Page size - default 20 + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/query-department-tenant", + "status": 0, + "message": "success", + "data": { + "tenantList": [ + { + "id": 1, + "creator": "IDE", + "department": "Engineering", + "departmentId": "dept1", + "tenantValue": "tenant1", + "createBy": "admin", + "isValid": "Y", + "createTime": "2023-01-01 12:00:00", + "updateTime": "2023-01-01 12:00:00" + } + ], + "totalPage": 1 + } +} +``` + +#### Delete Department Tenant +``` +GET /api/rest_j/v1/configuration/tenant-mapping/delete-department-tenant +``` + +Parameters: +- `id`: Department tenant ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/delete-department-tenant", + "status": 0, + "message": "success" +} +``` + +#### Query Department List +``` +GET /api/rest_j/v1/configuration/tenant-mapping/query-department +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/query-department", + "status": 0, + "message": "success", + "data": { + "departmentList": ["Engineering", "Marketing", "Sales"] + } +} +``` + +#### Query User Department +``` +GET /api/rest_j/v1/configuration/tenant-mapping/query-user-department +``` + +Parameters: +- `username`: Username (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/query-user-department", + "status": 0, + "message": "success", + "data": { + "department": "Engineering" + } +} +``` + +### User IP Configuration APIs + +#### Create User IP +``` +POST /api/rest_j/v1/configuration/user-ip-mapping/create-user-ip +``` + +Request Body: +```json +{ + "user": "testuser", + "creator": "IDE", + "ipList": "192.168.1.1,192.168.1.2", + "desc": "Allowed IPs for testuser", + "bussinessUser": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/create-user-ip", + "status": 0, + "message": "success" +} +``` + +#### Update User IP +``` +POST /api/rest_j/v1/configuration/user-ip-mapping/update-user-ip +``` + +Request Body: +```json +{ + "id": 1, + "user": "testuser", + "creator": "IDE", + "ipList": "192.168.1.1,192.168.1.2,192.168.1.3", + "desc": "Updated allowed IPs for testuser", + "bussinessUser": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/update-user-ip", + "status": 0, + "message": "success" +} +``` + +#### Delete User IP +``` +GET /api/rest_j/v1/configuration/user-ip-mapping/delete-user-ip +``` + +Parameters: +- `id`: User IP ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/delete-user-ip", + "status": 0, + "message": "success" +} +``` + +#### Query User IP List +``` +GET /api/rest_j/v1/configuration/user-ip-mapping/query-user-ip-list +``` + +Parameters: +- `user`: Username - optional +- `creator`: Creator application - optional +- `pageNow`: Page number (required) +- `pageSize`: Page size (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/query-user-ip-list", + "status": 0, + "message": "success", + "data": { + "userIpList": [ + { + "id": 1, + "user": "testuser", + "creator": "IDE", + "ipList": "192.168.1.1,192.168.1.2", + "desc": "Allowed IPs for testuser", + "bussinessUser": "admin", + "createTime": "2023-01-01 12:00:00", + "updateTime": "2023-01-01 12:00:00" + } + ], + "totalPage": 1 + } +} +``` + +#### Check User Creator +``` +GET /api/rest_j/v1/configuration/user-ip-mapping/check-user-creator +``` + +Parameters: +- `user`: Username (required) +- `creator`: Creator application (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/check-user-creator", + "status": 0, + "message": "success", + "data": { + "exist": true + } +} +``` + +## Database Table Structures + +The Configuration Service uses the following database tables from linkis_ddl.sql: + +### Configuration Config Key Table +```sql +CREATE TABLE `linkis_ps_configuration_config_key`( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `key` varchar(50) DEFAULT NULL COMMENT 'Set key, e.g. spark.executor.instances', + `description` varchar(200) DEFAULT NULL, + `name` varchar(50) DEFAULT NULL, + `default_value` varchar(200) DEFAULT NULL COMMENT 'Adopted when user does not set key', + `validate_type` varchar(50) DEFAULT NULL COMMENT 'Validate type, one of the following: None, NumInterval, FloatInterval, Include, Regex, OPF, Custom Rules', + `validate_range` varchar(150) DEFAULT NULL COMMENT 'Validate range', + `engine_conn_type` varchar(50) DEFAULT '' COMMENT 'engine type,such as spark,hive etc', + `is_hidden` tinyint(1) DEFAULT NULL COMMENT 'Whether it is hidden from user. If set to 1(true), then user cannot modify, however, it could still be used in back-end', + `is_advanced` tinyint(1) DEFAULT NULL COMMENT 'Whether it is an advanced parameter. If set to 1(true), parameters would be displayed only when user choose to do so', + `level` tinyint(1) DEFAULT NULL COMMENT 'Basis for displaying sorting in the front-end. Higher the level is, higher the rank the parameter gets', + `treeName` varchar(20) DEFAULT NULL COMMENT 'Reserved field, representing the subdirectory of engineType', + `boundary_type` TINYINT(2) NULL DEFAULT '0' COMMENT '0 none/ 1 with mix /2 with max / 3 min and max both', + `en_description` varchar(200) DEFAULT NULL COMMENT 'english description', + `en_name` varchar(100) DEFAULT NULL COMMENT 'english name', + `en_treeName` varchar(100) DEFAULT NULL COMMENT 'english treeName', + `template_required` tinyint(1) DEFAULT 0 COMMENT 'template required 0 none / 1 must', + UNIQUE INDEX `uniq_key_ectype` (`key`,`engine_conn_type`), + PRIMARY KEY (`id`) +); +``` + +### Configuration Key Engine Relation Table +```sql +CREATE TABLE `linkis_ps_configuration_key_engine_relation`( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `config_key_id` bigint(20) NOT NULL COMMENT 'config key id', + `engine_type_label_id` bigint(20) NOT NULL COMMENT 'engine label id', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_kid_lid` (`config_key_id`, `engine_type_label_id`) +); +``` + +### Configuration Config Value Table +```sql +CREATE TABLE `linkis_ps_configuration_config_value`( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `config_key_id` bigint(20), + `config_value` varchar(500), + `config_label_id`int(20), + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_kid_lid` (`config_key_id`, `config_label_id`) +); +``` + +### Configuration Category Table +```sql +CREATE TABLE `linkis_ps_configuration_category` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `label_id` int(20) NOT NULL, + `level` int(20) NOT NULL, + `description` varchar(200), + `tag` varchar(200), + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_label_id` (`label_id`) +); +``` + +### Tenant Label Config Table +```sql +CREATE TABLE `linkis_cg_tenant_label_config` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `user` varchar(50) COLLATE utf8_bin NOT NULL, + `creator` varchar(50) COLLATE utf8_bin NOT NULL, + `tenant_value` varchar(128) COLLATE utf8_bin NOT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `desc` varchar(100) COLLATE utf8_bin NOT NULL, + `bussiness_user` varchar(50) COLLATE utf8_bin NOT NULL, + `is_valid` varchar(1) COLLATE utf8_bin NOT NULL DEFAULT 'Y' COMMENT 'is valid', + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_user_creator` (`user`,`creator`) +) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### User IP Config Table +```sql +CREATE TABLE `linkis_cg_user_ip_config` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `user` varchar(50) COLLATE utf8_bin NOT NULL, + `creator` varchar(50) COLLATE utf8_bin NOT NULL, + `ip_list` text COLLATE utf8_bin NOT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `desc` varchar(100) COLLATE utf8_bin NOT NULL, + `bussiness_user` varchar(50) COLLATE utf8_bin NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_user_creator` (`user`,`creator`) +) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Tenant Department Config Table +```sql +CREATE TABLE `linkis_cg_tenant_department_config` ( + `id` int(20) NOT NULL AUTO_INCREMENT COMMENT 'ID', + `creator` varchar(50) COLLATE utf8_bin NOT NULL COMMENT '应用', + `department` varchar(64) COLLATE utf8_bin NOT NULL COMMENT '部门名称', + `department_id` varchar(16) COLLATE utf8_bin NOT NULL COMMENT '部门ID', + `tenant_value` varchar(128) COLLATE utf8_bin NOT NULL COMMENT '部门租户标签', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '更新时间', + `create_by` varchar(50) COLLATE utf8_bin NOT NULL COMMENT '创建用户', + `is_valid` varchar(1) COLLATE utf8_bin NOT NULL DEFAULT 'Y' COMMENT '是否有效', + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_creator_department` (`creator`,`department`) +) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Configuration Template Config Key Table +```sql +CREATE TABLE `linkis_ps_configuration_template_config_key` ( + `id` BIGINT(20) NOT NULL AUTO_INCREMENT, + `template_name` VARCHAR(200) NOT NULL COMMENT '配置模板名称 冗余存储', + `template_uuid` VARCHAR(36) NOT NULL COMMENT 'uuid 第三方侧记录的模板id', + `key_id` BIGINT(20) NOT NULL COMMENT 'id of linkis_ps_configuration_config_key', + `config_value` VARCHAR(200) NULL DEFAULT NULL COMMENT '配置值', + `max_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '上限值', + `min_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '下限值(预留)', + `validate_range` VARCHAR(50) NULL DEFAULT NULL COMMENT '校验正则(预留) ', + `is_valid` VARCHAR(2) DEFAULT 'Y' COMMENT '是否有效 预留 Y/N', + `create_by` VARCHAR(50) NOT NULL COMMENT '创建人', + `create_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `update_by` VARCHAR(50) NULL DEFAULT NULL COMMENT '更新人', + `update_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'update time', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_tid_kid` (`template_uuid`, `key_id`), + UNIQUE INDEX `uniq_tname_kid` (`template_uuid`, `key_id`) +)ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Configuration Key Limit For User Table +```sql +CREATE TABLE `linkis_ps_configuration_key_limit_for_user` ( + `id` BIGINT(20) NOT NULL AUTO_INCREMENT, + `user_name` VARCHAR(50) NOT NULL COMMENT '用户名', + `combined_label_value` VARCHAR(128) NOT NULL COMMENT '组合标签 combined_userCreator_engineType 如 hadoop-IDE,spark-2.4.3', + `key_id` BIGINT(20) NOT NULL COMMENT 'id of linkis_ps_configuration_config_key', + `config_value` VARCHAR(200) NULL DEFAULT NULL COMMENT '配置值', + `max_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '上限值', + `min_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '下限值(预留)', + `latest_update_template_uuid` VARCHAR(36) NOT NULL COMMENT 'uuid 第三方侧记录的模板id', + `is_valid` VARCHAR(2) DEFAULT 'Y' COMMENT '是否有效 预留 Y/N', + `create_by` VARCHAR(50) NOT NULL COMMENT '创建人', + `create_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `update_by` VARCHAR(50) NULL DEFAULT NULL COMMENT '更新人', + `update_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'update time', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_com_label_kid` (`combined_label_value`, `key_id`) +)ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +## RPC Methods + +The Configuration Service provides several RPC methods for configuration management: + +### Configuration RPCs + +#### getGlobalConfiguration +Retrieves global configuration properties: +```java +Map getGlobalConfiguration() +``` + +#### getUserConfiguration +Retrieves user-specific configuration: +```java +Map getUserConfiguration(String user) +``` + +#### updateConfiguration +Updates configuration properties: +```java +void updateConfiguration(String user, Map configurations) +``` + +#### getConfigurationTemplate +Retrieves configuration template for an engine: +```java +ConfigurationTemplate getConfigurationTemplate(String engineType) +``` + +#### validateConfiguration +Validates configuration properties: +```java +ConfigurationValidationResult validateConfiguration(String key, String value) +``` + +#### getEngineConfiguration +Retrieves engine-specific configuration: +```java +Map getEngineConfiguration(String engineType, String version, String user) +``` + +#### updateEngineConfiguration +Updates engine-specific configuration: +```java +void updateEngineConfiguration(String engineType, String version, String user, Map configurations) +``` + +#### listConfigurationKeys +Lists all configuration keys for an engine type: +```java +List listConfigurationKeys(String engineType) +``` + +### Category RPCs + +#### getCategoryConfiguration +Retrieves configuration for a category: +```java +Map getCategoryConfiguration(String category) +``` + +#### updateCategoryConfiguration +Updates configuration for a category: +```java +void updateCategoryConfiguration(String category, Map configurations) +``` + +## Dependencies + +- linkis-mybatis +- linkis-rpc +- linkis-manager-common +- linkis-httpclient +- linkis-label-common + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- ConfigurationRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/ConfigurationRestfulApi.java` +- TenantConfigrationRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/TenantConfigrationRestfulApi.java` +- UserIpConfigrationRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/UserIpConfigrationRestfulApi.java` +- TemplateRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/TemplateRestfulApi.java` +- AcrossClusterRuleRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/AcrossClusterRuleRestfulApi.java` +- ConfigurationTemplateRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/ConfigurationTemplateRestfulApi.java` + +### MyBatis XML Files +- ConfigMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/ConfigMapper.xml` +- LabelMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/LabelMapper.xml` +- UserTenantMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/UserTenantMapper.xml` +- DepartmentTenantMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/DepartmentTenantMapper.xml` +- UserIpMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/UserIpMapper.xml` +- DepartmentMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/DepartmentMapper.xml` +- AcrossClusterRuleMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/AcrossClusterRuleMapper.xml` +- TemplateConfigKeyMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/TemplateConfigKeyMapper.xml` +- ConfigKeyLimitForUserMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/ConfigKeyLimitForUserMapper.xml` \ No newline at end of file diff --git a/.ai/modules/public-enhancements/context.md b/.ai/modules/public-enhancements/context.md new file mode 100644 index 00000000000..d1211c0a869 --- /dev/null +++ b/.ai/modules/public-enhancements/context.md @@ -0,0 +1,962 @@ +# Context Service + +The Context Service provides context and variable sharing capabilities for the Linkis system. + +## Overview + +This service manages context information and variable sharing across different engines and applications in the Linkis system. + +## Key Components + +### Core Classes +- `LinkisCSApplication` - Main application class +- Context management +- Variable sharing +- Context persistence + +### Features +- Cross-engine context sharing +- Variable management +- Context persistence +- Context versioning + +## API Interfaces + +### Context ID APIs + +#### Create Context ID +``` +POST /api/rest_j/v1/contextservice/createContextID +``` + +Request Body: +```json +{ + "contextId": "context-12345", + "user": "testuser", + "application": "IDE", + "source": "test-source", + "expireType": "NORMAL", + "expireTime": "2023-12-31 23:59:59" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/createContextID", + "status": 0, + "message": "success", + "data": { + "contextId": "context-12345" + } +} +``` + +#### Get Context ID +``` +GET /api/rest_j/v1/contextservice/getContextID +``` + +Parameters: +- `contextId`: Context ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/getContextID", + "status": 0, + "message": "success", + "data": { + "contextId": { + "contextId": "context-12345", + "user": "testuser", + "application": "IDE", + "source": "test-source", + "expireType": "NORMAL", + "expireTime": "2023-12-31 23:59:59", + "instance": "instance-1", + "backupInstance": "backup-instance-1", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + } +} +``` + +#### Update Context ID +``` +POST /api/rest_j/v1/contextservice/updateContextID +``` + +Request Body: +```json +{ + "contextId": "context-12345", + "user": "testuser", + "application": "IDE", + "source": "updated-source", + "expireType": "NORMAL", + "expireTime": "2023-12-31 23:59:59", + "instance": "instance-1", + "backupInstance": "backup-instance-1" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/updateContextID", + "status": 0, + "message": "success" +} +``` + +#### Reset Context ID +``` +POST /api/rest_j/v1/contextservice/resetContextID +``` + +Request Body: +```json +{ + "contextId": "context-12345" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/resetContextID", + "status": 0, + "message": "success" +} +``` + +#### Remove Context ID +``` +POST /api/rest_j/v1/contextservice/removeContextID +``` + +Request Body: +```json +{ + "contextId": "context-12345" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeContextID", + "status": 0, + "message": "success" +} +``` + +#### Search Context ID By Time +``` +GET /api/rest_j/v1/contextservice/searchContextIDByTime +``` + +Parameters: +- `createTimeStart`: Create time start - optional +- `createTimeEnd`: Create time end - optional +- `updateTimeStart`: Update time start - optional +- `updateTimeEnd`: Update time end - optional +- `accessTimeStart`: Access time start - optional +- `accessTimeEnd`: Access time end - optional +- `pageNow`: Page number - optional, default 1 +- `pageSize`: Page size - optional, default 100 + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/searchContextIDByTime", + "status": 0, + "message": "success", + "data": { + "contextIds": [ + { + "contextId": "context-12345", + "user": "testuser", + "application": "IDE", + "source": "test-source", + "expireType": "NORMAL", + "expireTime": "2023-12-31 23:59:59", + "instance": "instance-1", + "backupInstance": "backup-instance-1", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + ] + } +} +``` + +### Context Value APIs + +#### Get Context Value +``` +POST /api/rest_j/v1/contextservice/getContextValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/getContextValue", + "status": 0, + "message": "success", + "data": { + "contextValue": { + "key": "test-key", + "value": "test-value", + "contextType": "ENV", + "scope": "PRIVATE", + "props": { + "prop1": "value1" + } + } + } +} +``` + +#### Search Context Value +``` +POST /api/rest_j/v1/contextservice/searchContextValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "condition": { + "key": "test-key", + "contextType": "ENV" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/searchContextValue", + "status": 0, + "message": "success", + "data": { + "contextKeyValue": [ + { + "key": "test-key", + "value": "test-value", + "contextType": "ENV", + "scope": "PRIVATE", + "props": { + "prop1": "value1" + } + } + ] + } +} +``` + +#### Set Value By Key +``` +POST /api/rest_j/v1/contextservice/setValueByKey +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + }, + "contextValue": { + "value": "test-value", + "contextType": "ENV", + "scope": "PRIVATE", + "props": { + "prop1": "value1" + } + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/setValueByKey", + "status": 0, + "message": "success" +} +``` + +#### Set Value +``` +POST /api/rest_j/v1/contextservice/setValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKeyValue": { + "key": "test-key", + "value": "test-value", + "contextType": "ENV", + "scope": "PRIVATE", + "props": { + "prop1": "value1" + } + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/setValue", + "status": 0, + "message": "success" +} +``` + +#### Reset Value +``` +POST /api/rest_j/v1/contextservice/resetValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/resetValue", + "status": 0, + "message": "success" +} +``` + +#### Remove Value +``` +POST /api/rest_j/v1/contextservice/removeValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeValue", + "status": 0, + "message": "success" +} +``` + +#### Remove All Value +``` +POST /api/rest_j/v1/contextservice/removeAllValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeAllValue", + "status": 0, + "message": "success" +} +``` + +#### Remove All Value By Key Prefix And Context Type +``` +POST /api/rest_j/v1/contextservice/removeAllValueByKeyPrefixAndContextType +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKeyType": "ENV", + "keyPrefix": "test" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeAllValueByKeyPrefixAndContextType", + "status": 0, + "message": "success" +} +``` + +#### Remove All Value By Key And Context Type +``` +POST /api/rest_j/v1/contextservice/removeAllValueByKeyAndContextType +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKeyType": "ENV", + "contextKey": "test-key" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeAllValueByKeyAndContextType", + "status": 0, + "message": "success" +} +``` + +#### Remove All Value By Key Prefix +``` +POST /api/rest_j/v1/contextservice/removeAllValueByKeyPrefix +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "keyPrefix": "test" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeAllValueByKeyPrefix", + "status": 0, + "message": "success" +} +``` + +#### Clear All Context By ID +``` +POST /api/rest_j/v1/contextservice/clearAllContextByID +``` + +Request Body: +```json +{ + "idList": ["context-12345", "context-67890"] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/clearAllContextByID", + "status": 0, + "message": "success", + "data": { + "num": 2 + } +} +``` + +#### Clear All Context By Time +``` +POST /api/rest_j/v1/contextservice/clearAllContextByTime +``` + +Request Body: +```json +{ + "createTimeStart": "2023-01-01 00:00:00", + "createTimeEnd": "2023-12-31 23:59:59", + "updateTimeStart": "2023-01-01 00:00:00", + "updateTimeEnd": "2023-12-31 23:59:59" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/clearAllContextByTime", + "status": 0, + "message": "success", + "data": { + "num": 5 + } +} +``` + +### Context History APIs + +#### Create History +``` +POST /api/rest_j/v1/contextservice/createHistory +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextHistory": { + "source": "test-source", + "contextType": "ENV", + "historyJson": "{\"key\":\"test-key\",\"value\":\"test-value\"}", + "keyword": "test" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/createHistory", + "status": 0, + "message": "success" +} +``` + +#### Remove History +``` +POST /api/rest_j/v1/contextservice/removeHistory +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextHistory": { + "source": "test-source" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeHistory", + "status": 0, + "message": "success" +} +``` + +#### Get Histories +``` +POST /api/rest_j/v1/contextservice/getHistories +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/getHistories", + "status": 0, + "message": "success", + "data": { + "contextHistory": [ + { + "id": 1, + "contextId": 12345, + "source": "test-source", + "contextType": "ENV", + "historyJson": "{\"key\":\"test-key\",\"value\":\"test-value\"}", + "keyword": "test", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + ] + } +} +``` + +#### Get History +``` +POST /api/rest_j/v1/contextservice/getHistory +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "source": "test-source" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/getHistory", + "status": 0, + "message": "success", + "data": { + "contextHistory": { + "id": 1, + "contextId": 12345, + "source": "test-source", + "contextType": "ENV", + "historyJson": "{\"key\":\"test-key\",\"value\":\"test-value\"}", + "keyword": "test", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + } +} +``` + +#### Search History +``` +POST /api/rest_j/v1/contextservice/searchHistory +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "keywords": ["test", "key"] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/searchHistory", + "status": 0, + "message": "success", + "data": { + "contextHistory": [ + { + "id": 1, + "contextId": 12345, + "source": "test-source", + "contextType": "ENV", + "historyJson": "{\"key\":\"test-key\",\"value\":\"test-value\"}", + "keyword": "test", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + ] + } +} +``` + +### Context Listener APIs + +#### On Bind ID Listener +``` +POST /api/rest_j/v1/contextservice/onBindIDListener +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "source": "test-source" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/onBindIDListener", + "status": 0, + "message": "success", + "data": { + "listener": null + } +} +``` + +#### On Bind Key Listener +``` +POST /api/rest_j/v1/contextservice/onBindKeyListener +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + }, + "source": "test-source" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/onBindKeyListener", + "status": 0, + "message": "success", + "data": { + "listener": null + } +} +``` + +#### Heartbeat +``` +POST /api/rest_j/v1/contextservice/heartbeat +``` + +Request Body: +```json +{ + "source": "test-source" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/heartbeat", + "status": 0, + "message": "success", + "data": { + "ContextKeyValueBean": null + } +} +``` + +## Database Table Structures + +The Context Service manages the following database tables from linkis_ddl.sql: + +### Context Map Table +```sql +CREATE TABLE `linkis_ps_cs_context_map` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `key` varchar(128) DEFAULT NULL, + `context_scope` varchar(32) DEFAULT NULL, + `context_type` varchar(32) DEFAULT NULL, + `props` text, + `value` mediumtext, + `context_id` int(11) DEFAULT NULL, + `keywords` varchar(255) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_key_cid_ctype` (`key`,`context_id`,`context_type`), + KEY `idx_keywords` (`keywords`(191)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Context Map Listener Table +```sql +CREATE TABLE `linkis_ps_cs_context_map_listener` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `listener_source` varchar(255) DEFAULT NULL, + `key_id` int(11) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Context History Table +```sql +CREATE TABLE `linkis_ps_cs_context_history` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `context_id` int(11) DEFAULT NULL, + `source` text, + `context_type` varchar(32) DEFAULT NULL, + `history_json` text, + `keyword` varchar(255) DEFAULT NULL, + PRIMARY KEY (`id`), + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + KEY `idx_keyword` (`keyword`(191)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Context ID Table +```sql +CREATE TABLE `linkis_ps_cs_context_id` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `user` varchar(32) DEFAULT NULL, + `application` varchar(32) DEFAULT NULL, + `source` varchar(255) DEFAULT NULL, + `expire_type` varchar(32) DEFAULT NULL, + `expire_time` datetime DEFAULT NULL, + `instance` varchar(128) DEFAULT NULL, + `backup_instance` varchar(255) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`), + KEY `idx_instance` (`instance`(128)), + KEY `idx_backup_instance` (`backup_instance`(191)), + KEY `idx_instance_bin` (`instance`(128),`backup_instance`(128)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Context Listener Table +```sql +CREATE TABLE `linkis_ps_cs_context_listener` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `listener_source` varchar(255) DEFAULT NULL, + `context_id` int(11) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +## RPC Methods + +The Context Service provides several RPC methods for context management: + +### Context RPCs + +#### createContext +Creates a new context: +```java +String createContext(ContextCreationRequest request) +``` + +#### getContext +Retrieves a context: +```java +ContextInfo getContext(String contextId) +``` + +#### deleteContext +Deletes a context: +```java +void deleteContext(String contextId) +``` + +### Variable RPCs + +#### setVariable +Sets a variable in a context: +```java +void setVariable(String contextId, String key, Object value, String valueType) +``` + +#### getVariable +Retrieves a variable from a context: +```java +Object getVariable(String contextId, String key) +``` + +#### removeVariable +Removes a variable from a context: +```java +void removeVariable(String contextId, String key) +``` + +### History RPCs + +#### getContextHistory +Retrieves context history: +```java +List getContextHistory(String contextId) +``` + +#### clearContextHistory +Clears context history: +```java +void clearContextHistory(String contextId) +``` + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- ContextRestfulApi: `linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/restful/ContextRestfulApi.java` +- ContextIDRestfulApi: `linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/restful/ContextIDRestfulApi.java` +- ContextHistoryRestfulApi: `linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/restful/ContextHistoryRestfulApi.java` +- ContextListenerRestfulApi: `linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/restful/ContextListenerRestfulApi.java` + +### MyBatis XML Files +- ContextMapper: `linkis-public-enhancements/linkis-cs-server/src/main/resources/mapper/ContextMapper.xml` +- ContextIDMapper: `linkis-public-enhancements/linkis-cs-server/src/main/resources/mapper/ContextIDMapper.xml` +- ContextHistoryMapper: `linkis-public-enhancements/linkis-cs-server/src/main/resources/mapper/ContextHistoryMapper.xml` +- ContextListenerMapper: `linkis-public-enhancements/linkis-cs-server/src/main/resources/mapper/ContextListenerMapper.xml` + +## Dependencies + +- linkis-cs-server +- linkis-rpc +- linkis-protocol \ No newline at end of file diff --git a/.ai/modules/public-enhancements/datasource.md b/.ai/modules/public-enhancements/datasource.md new file mode 100644 index 00000000000..6e584cc2cb1 --- /dev/null +++ b/.ai/modules/public-enhancements/datasource.md @@ -0,0 +1,2032 @@ +# DataSource Service + +The DataSource Service provides data source management capabilities for the Linkis system. + +## Overview + +This service manages data source connections, metadata, and provides unified access to various data sources. + +## Key Components + +### Core Classes +- `LinkisDataSourceApplication` - Main application class +- Data source management +- Metadata querying +- Connection testing + +### Features +- Data source registration and management +- Metadata querying +- Connection testing and validation +- Data source versioning +- Access control + +## API Interfaces + +### Data Source Type APIs + +#### Get All Data Source Types +``` +GET /api/rest_j/v1/data-source-manager/type/all +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/type/all", + "status": 0, + "message": "success", + "data": { + "typeList": [ + { + "id": 1, + "name": "MySQL", + "description": "MySQL Database", + "option": "MySQL", + "classifier": "Database", + "icon": "", + "layers": 3 + } + ] + } +} +``` + +#### Get Key Definitions By Type +``` +GET /api/rest_j/v1/data-source-manager/key-define/type/{typeId} +``` + +Parameters: +- `typeId`: Data source type ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/key-define/type/{typeId}", + "status": 0, + "message": "success", + "data": { + "keyDefine": [ + { + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Host IP", + "descriptionEn": "Host IP", + "valueRegex": "", + "refId": null, + "refValue": null, + "dataSource": null + } + ] + } +} +``` + +#### Get Key Definitions By Type Name +``` +GET /api/rest_j/v1/data-source-manager/key-define/{typeName} +``` + +Parameters: +- `typeName`: Data source type name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/key-define/{typeName}", + "status": 0, + "message": "success", + "data": { + "keyDefine": [ + { + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Host IP", + "descriptionEn": "Host IP", + "valueRegex": "", + "refId": null, + "refValue": null, + "dataSource": null + } + ] + } +} +``` + +### Data Source Management APIs + +#### Insert Data Source Info (JSON) +``` +POST /api/rest_j/v1/data-source-manager/info/json +``` + +Request Body: +```json +{ + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createSystem": "linkis", + "labels": [ + { + "labelKey": "env", + "labelValue": "production" + } + ], + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password", + "database": "test" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/json", + "status": 0, + "message": "success", + "data": { + "insertId": 12345 + } +} +``` + +#### Insert Data Source (JSON Create) +``` +POST /api/rest_j/v1/data-source-manager/info/json/create +``` + +Request Body: +```json +{ + "createUser": "testuser", + "dataSourceTypeName": "starrocks", + "connectParams": { + "host": "localhost", + "port": "9030", + "driverClassName": "com.mysql.jdbc.Driver", + "username": "user", + "password": "password" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/json/create", + "status": 0, + "message": "success", + "data": { + "datasource": { + "id": 12345, + "dataSourceName": "starrocks_testuser_20230101120000", + "dataSourceDesc": null, + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": null, + "parameter": "{\"host\":\"localhost\",\"port\":\"9030\",\"driverClassName\":\"com.mysql.jdbc.Driver\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": null, + "labels": null, + "versionId": null, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Update Data Source Info (JSON) +``` +PUT /api/rest_j/v1/data-source-manager/info/{dataSourceId}/json +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Request Body: +```json +{ + "dataSourceName": "mysql-ds", + "dataSourceDesc": "Updated MySQL Data Source", + "dataSourceTypeId": 1, + "createSystem": "linkis", + "createTime": "1650426189000", + "createUser": "testuser", + "labels": [ + { + "labelKey": "env", + "labelValue": "production" + } + ], + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "newpassword", + "database": "test" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/{dataSourceId}/json", + "status": 0, + "message": "success", + "data": { + "updateId": 12345 + } +} +``` + +#### Insert Data Source Parameter (JSON) +``` +POST /api/rest_j/v1/data-source-manager/parameter/{dataSourceId}/json +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Request Body: +```json +{ + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password", + "database": "test" + }, + "comment": "Initial version" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/parameter/{dataSourceId}/json", + "status": 0, + "message": "success", + "data": { + "version": 1 + } +} +``` + +#### Get Data Source Info By ID +``` +GET /api/rest_j/v1/data-source-manager/info/{dataSourceId} +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/{dataSourceId}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Data Source Info By Name +``` +GET /api/rest_j/v1/data-source-manager/info/name/{dataSourceName} +``` + +Parameters: +- `dataSourceName`: Data source name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/name/{dataSourceName}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Published Data Source Info By Name +``` +GET /api/rest_j/v1/data-source-manager/publishedInfo/name/{dataSourceName} +``` + +Parameters: +- `dataSourceName`: Data source name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/publishedInfo/name/{dataSourceName}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Published Data Source Info By Type Name, User, IP and Port +``` +GET /api/rest_j/v1/data-source-manager/publishedInfo/{datasourceTypeName}/{datasourceUser}/{ip}/{port} +``` + +Parameters: +- `datasourceTypeName`: Data source type name (required) +- `datasourceUser`: Data source user (required) +- `ip`: IP address (required) +- `port`: Port (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/publishedInfo/{datasourceTypeName}/{datasourceUser}/{ip}/{port}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Data Source Info By ID and Version +``` +GET /api/rest_j/v1/data-source-manager/info/{dataSourceId}/{version} +``` + +Parameters: +- `dataSourceId`: Data source ID (required) +- `version`: Version ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/{dataSourceId}/{version}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Version List +``` +GET /api/rest_j/v1/data-source-manager/{dataSourceId}/versions +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/{dataSourceId}/versions", + "status": 0, + "message": "success", + "data": { + "versions": [ + { + "versionId": 1, + "dataSourceId": 12345, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "comment": "Initial version", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser" + } + ] + } +} +``` + +#### Publish Data Source By ID +``` +POST /api/rest_j/v1/data-source-manager/publish/{dataSourceId}/{versionId} +``` + +Parameters: +- `dataSourceId`: Data source ID (required) +- `versionId`: Version ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/publish/{dataSourceId}/{versionId}", + "status": 0, + "message": "success" +} +``` + +#### Remove Data Source +``` +DELETE /api/rest_j/v1/data-source-manager/info/delete/{dataSourceId} +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/delete/{dataSourceId}", + "status": 0, + "message": "success", + "data": { + "removeId": 12345 + } +} +``` + +#### Expire Data Source +``` +PUT /api/rest_j/v1/data-source-manager/info/{dataSourceId}/expire +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/{dataSourceId}/expire", + "status": 0, + "message": "success", + "data": { + "expireId": 12345 + } +} +``` + +#### Get Connect Params By Data Source ID +``` +GET /api/rest_j/v1/data-source-manager/{dataSourceId}/connect-params +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/{dataSourceId}/connect-params", + "status": 0, + "message": "success", + "data": { + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password", + "database": "test" + } + } +} +``` + +#### Get Connect Params By Data Source Name +``` +GET /api/rest_j/v1/data-source-manager/name/{dataSourceName}/connect-params +``` + +Parameters: +- `dataSourceName`: Data source name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/name/{dataSourceName}/connect-params", + "status": 0, + "message": "success", + "data": { + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password", + "database": "test" + } + } +} +``` + +#### Connect Data Source +``` +PUT /api/rest_j/v1/data-source-manager/{dataSourceId}/{version}/op/connect +``` + +Parameters: +- `dataSourceId`: Data source ID (required) +- `version`: Version ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/{dataSourceId}/{version}/op/connect", + "status": 0, + "message": "success", + "data": { + "ok": true + } +} +``` + +#### Query Data Source By IDs +``` +GET /api/rest_j/v1/data-source-manager/info/ids +``` + +Parameters: +- `ids`: JSON array of data source IDs (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/ids", + "status": 0, + "message": "success", + "data": { + "queryList": [ + { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + ], + "totalPage": 1 + } +} +``` + +#### Query Data Source +``` +GET /api/rest_j/v1/data-source-manager/info +``` + +Parameters: +- `system`: Create system - optional +- `name`: Data source name - optional +- `typeId`: Data source type ID - optional +- `identifies`: Identifies - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info", + "status": 0, + "message": "success", + "data": { + "queryList": [ + { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + ], + "totalPage": 1 + } +} +``` + +### Data Source Environment APIs + +#### Insert Data Source Environment (JSON) +``` +POST /api/rest_j/v1/data-source-manager/env/json +``` + +Request Body: +```json +{ + "envName": "test-env", + "envDesc": "Test Environment", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/json", + "status": 0, + "message": "success", + "data": { + "insertId": 12345 + } +} +``` + +#### Insert Data Source Environment Batch (JSON) +``` +POST /api/rest_j/v1/data-source-manager/env/json/batch +``` + +Request Body: +```json +[ + { + "envName": "test-env-1", + "envDesc": "Test Environment 1", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password" + } + }, + { + "envName": "test-env-2", + "envDesc": "Test Environment 2", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3307", + "username": "user", + "password": "password" + } + } +] +``` + +Parameters: +- `system`: System name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/json/batch", + "status": 0, + "message": "success", + "data": { + "envs": [ + { + "id": 12345, + "envName": "test-env-1", + "envDesc": "Test Environment 1", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + }, + { + "id": 12346, + "envName": "test-env-2", + "envDesc": "Test Environment 2", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3307\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + ] + } +} +``` + +#### Update Data Source Environment Batch (JSON) +``` +PUT /api/rest_j/v1/data-source-manager/env/json/batch +``` + +Request Body: +```json +[ + { + "id": 12345, + "envName": "test-env-1", + "envDesc": "Updated Test Environment 1", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "newpassword" + } + } +] +``` + +Parameters: +- `system`: System name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/json/batch", + "status": 0, + "message": "success", + "data": { + "envs": [ + { + "id": 12345, + "envName": "test-env-1", + "envDesc": "Updated Test Environment 1", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"newpassword\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + ] + } +} +``` + +#### Get All Environment List By Data Source Type +``` +GET /api/rest_j/v1/data-source-manager/env-list/all/type/{typeId} +``` + +Parameters: +- `typeId`: Data source type ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env-list/all/type/{typeId}", + "status": 0, + "message": "success", + "data": { + "envList": [ + { + "id": 12345, + "envName": "test-env", + "envDesc": "Test Environment", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + ] + } +} +``` + +#### Get Environment Entity By ID +``` +GET /api/rest_j/v1/data-source-manager/env/{envId} +``` + +Parameters: +- `envId`: Environment ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/{envId}", + "status": 0, + "message": "success", + "data": { + "env": { + "id": 12345, + "envName": "test-env", + "envDesc": "Test Environment", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + } +} +``` + +#### Remove Environment Entity +``` +DELETE /api/rest_j/v1/data-source-manager/env/{envId} +``` + +Parameters: +- `envId`: Environment ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/{envId}", + "status": 0, + "message": "success", + "data": { + "removeId": 12345 + } +} +``` + +#### Update Data Source Environment (JSON) +``` +PUT /api/rest_j/v1/data-source-manager/env/{envId}/json +``` + +Parameters: +- `envId`: Environment ID (required) + +Request Body: +```json +{ + "envName": "test-env", + "envDesc": "Updated Test Environment", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "newpassword" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/{envId}/json", + "status": 0, + "message": "success", + "data": { + "updateId": 12345 + } +} +``` + +#### Query Data Source Environment +``` +GET /api/rest_j/v1/data-source-manager/env +``` + +Parameters: +- `name`: Environment name - optional +- `typeId`: Data source type ID - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env", + "status": 0, + "message": "success", + "data": { + "queryList": [ + { + "id": 12345, + "envName": "test-env", + "envDesc": "Test Environment", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + ] + } +} +``` + +### Metadata Query APIs + +#### Query Database Info +``` +GET /api/rest_j/v1/datasource/dbs +``` + +Parameters: +- `permission`: Permission filter - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/dbs", + "status": 0, + "message": "success", + "data": { + "dbs": [ + { + "name": "test_db", + "permission": "READ" + } + ] + } +} +``` + +#### Query Partition Exists +``` +GET /api/rest_j/v1/datasource/partitionExists +``` + +Parameters: +- `database`: Database name (required) +- `table`: Table name (required) +- `partition`: Partition name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/partitionExists", + "status": 0, + "message": "success", + "data": { + "partitionExists": true + } +} +``` + +#### Query Databases With Tables +``` +GET /api/rest_j/v1/datasource/all +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/all", + "status": 0, + "message": "success", + "data": { + "dbs": [ + { + "name": "test_db", + "tables": [ + { + "name": "test_table" + } + ] + } + ] + } +} +``` + +#### Query Databases With Tables Order By Access Time +``` +GET /api/rest_j/v1/datasource/getByAccessTime +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/getByAccessTime", + "status": 0, + "message": "success", + "data": { + "dbs": [ + { + "name": "test_db", + "tables": [ + { + "name": "test_table", + "lastAccessTime": "2023-01-01 12:00:00" + } + ] + } + ] + } +} +``` + +#### Query Tables +``` +GET /api/rest_j/v1/datasource/tables +``` + +Parameters: +- `database`: Database name - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/tables", + "status": 0, + "message": "success", + "data": { + "tables": [ + { + "name": "test_table" + } + ] + } +} +``` + +#### Query Table Metadata +``` +GET /api/rest_j/v1/datasource/columns +``` + +Parameters: +- `database`: Database name - optional +- `table`: Table name - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/columns", + "status": 0, + "message": "success", + "data": { + "columns": [ + { + "name": "id", + "type": "INT", + "comment": "Primary key" + }, + { + "name": "name", + "type": "VARCHAR", + "comment": "Name field" + } + ] + } +} +``` + +#### Get Table Size +``` +GET /api/rest_j/v1/datasource/size +``` + +Parameters: +- `database`: Database name - optional +- `table`: Table name - optional +- `partition`: Partition name - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/size", + "status": 0, + "message": "success", + "data": { + "sizeInfo": { + "size": "10MB", + "fileCount": 5 + } + } +} +``` + +#### Get Storage Info +``` +GET /api/rest_j/v1/datasource/storage-info +``` + +Parameters: +- `database`: Database name (required) +- `table`: Table name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/storage-info", + "status": 0, + "message": "success", + "data": { + "storageInfo": { + "location": "/path/to/table", + "format": "PARQUET", + "compression": "SNAPPY" + } + } +} +``` + +#### Get Partitions +``` +GET /api/rest_j/v1/datasource/partitions +``` + +Parameters: +- `database`: Database name - optional +- `table`: Table name - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/partitions", + "status": 0, + "message": "success", + "data": { + "partitionInfo": [ + { + "name": "dt=20230101", + "location": "/path/to/partition" + } + ] + } +} +``` + +### Data Source Type Management APIs + +#### List Data Source Types +``` +GET /api/rest_j/v1/basedata-manager/datasource-type +``` + +Parameters: +- `searchName`: Search name - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "list": { + "total": 1, + "list": [ + { + "id": 1, + "name": "MySQL", + "description": "MySQL Database", + "option": "MySQL", + "classifier": "Database", + "icon": "", + "layers": 3, + "descriptionEn": "MySQL Database", + "optionEn": "MySQL", + "classifierEn": "Database" + } + ], + "pageNum": 1, + "pageSize": 10, + "size": 1, + "startRow": 1, + "endRow": 1, + "pages": 1, + "prePage": 0, + "nextPage": 0, + "isFirstPage": true, + "isLastPage": true, + "hasPreviousPage": false, + "hasNextPage": false, + "navigatePages": 8, + "navigatepageNums": [ + 1 + ] + } + } +} +``` + +#### Get Data Source Type +``` +GET /api/rest_j/v1/basedata-manager/datasource-type/{id} +``` + +Parameters: +- `id`: Data source type ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "item": { + "id": 1, + "name": "MySQL", + "description": "MySQL Database", + "option": "MySQL", + "classifier": "Database", + "icon": "", + "layers": 3, + "descriptionEn": "MySQL Database", + "optionEn": "MySQL", + "classifierEn": "Database" + } + } +} +``` + +#### Add Data Source Type +``` +POST /api/rest_j/v1/basedata-manager/datasource-type +``` + +Request Body: +```json +{ + "name": "PostgreSQL", + "description": "PostgreSQL Database", + "option": "PostgreSQL", + "classifier": "Database", + "icon": "", + "layers": 3, + "descriptionEn": "PostgreSQL Database", + "optionEn": "PostgreSQL", + "classifierEn": "Database" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Remove Data Source Type +``` +DELETE /api/rest_j/v1/basedata-manager/datasource-type/{id} +``` + +Parameters: +- `id`: Data source type ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Update Data Source Type +``` +PUT /api/rest_j/v1/basedata-manager/datasource-type +``` + +Request Body: +```json +{ + "id": 1, + "name": "MySQL", + "description": "Updated MySQL Database", + "option": "MySQL", + "classifier": "Database", + "icon": "", + "layers": 3, + "descriptionEn": "Updated MySQL Database", + "optionEn": "MySQL", + "classifierEn": "Database" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Data Source Access Management APIs + +#### List Data Source Accesses +``` +GET /api/rest_j/v1/basedata-manager/datasource-access +``` + +Parameters: +- `searchName`: Search name - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "list": { + "total": 1, + "list": [ + { + "id": 1, + "tokenId": 1, + "serviceId": 1, + "accessTime": "2023-01-01 12:00:00" + } + ], + "pageNum": 1, + "pageSize": 10, + "size": 1, + "startRow": 1, + "endRow": 1, + "pages": 1, + "prePage": 0, + "nextPage": 0, + "isFirstPage": true, + "isLastPage": true, + "hasPreviousPage": false, + "hasNextPage": false, + "navigatePages": 8, + "navigatepageNums": [ + 1 + ] + } + } +} +``` + +#### Get Data Source Access +``` +GET /api/rest_j/v1/basedata-manager/datasource-access/{id} +``` + +Parameters: +- `id`: Data source access ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "item": { + "id": 1, + "tokenId": 1, + "serviceId": 1, + "accessTime": "2023-01-01 12:00:00" + } + } +} +``` + +#### Add Data Source Access +``` +POST /api/rest_j/v1/basedata-manager/datasource-access +``` + +Request Body: +```json +{ + "tokenId": 1, + "serviceId": 1 +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Remove Data Source Access +``` +DELETE /api/rest_j/v1/basedata-manager/datasource-access/{id} +``` + +Parameters: +- `id`: Data source access ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Update Data Source Access +``` +PUT /api/rest_j/v1/basedata-manager/datasource-access +``` + +Request Body: +```json +{ + "id": 1, + "tokenId": 1, + "serviceId": 2 +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Data Source Type Key Management APIs + +#### List Data Source Type Keys +``` +GET /api/rest_j/v1/basedata-manager/datasource-type-key +``` + +Parameters: +- `searchName`: Search name - optional +- `dataSourceTypeId`: Data source type ID - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "list": { + "total": 1, + "list": [ + { + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "nameEn": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Host IP", + "descriptionEn": "Host IP", + "valueRegex": "", + "refId": null, + "refValue": null, + "dataSource": null, + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00" + } + ], + "pageNum": 1, + "pageSize": 10, + "size": 1, + "startRow": 1, + "endRow": 1, + "pages": 1, + "prePage": 0, + "nextPage": 0, + "isFirstPage": true, + "isLastPage": true, + "hasPreviousPage": false, + "hasNextPage": false, + "navigatePages": 8, + "navigatepageNums": [ + 1 + ] + } + } +} +``` + +#### Get Data Source Type Key +``` +GET /api/rest_j/v1/basedata-manager/datasource-type-key/{id} +``` + +Parameters: +- `id`: Data source type key ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "item": { + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "nameEn": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Host IP", + "descriptionEn": "Host IP", + "valueRegex": "", + "refId": null, + "refValue": null, + "dataSource": null, + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00" + } + } +} +``` + +#### Add Data Source Type Key +``` +POST /api/rest_j/v1/basedata-manager/datasource-type-key +``` + +Request Body: +```json +{ + "dataSourceTypeId": 1, + "key": "port", + "name": "Port", + "nameEn": "Port", + "defaultValue": "3306", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Port number", + "descriptionEn": "Port number", + "valueRegex": "" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Remove Data Source Type Key +``` +DELETE /api/rest_j/v1/basedata-manager/datasource-type-key/{id} +``` + +Parameters: +- `id`: Data source type key ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Update Data Source Type Key +``` +PUT /api/rest_j/v1/basedata-manager/datasource-type-key +``` + +Request Body: +```json +{ + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "nameEn": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Updated Host IP", + "descriptionEn": "Updated Host IP", + "valueRegex": "" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +## Database Table Structures + +The DataSource Service uses the following database tables from linkis_ddl.sql: + +### Data Source Table +```sql +CREATE TABLE `linkis_ps_dm_datasource` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `datasource_name` varchar(255) COLLATE utf8_bin NOT NULL, + `datasource_desc` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `datasource_type_id` int(11) NOT NULL, + `create_identify` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `create_system` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `parameter` varchar(2048) COLLATE utf8_bin NULL DEFAULT NULL, + `create_time` datetime NULL DEFAULT CURRENT_TIMESTAMP, + `modify_time` datetime NULL DEFAULT CURRENT_TIMESTAMP, + `create_user` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `modify_user` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `labels` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `version_id` int(11) DEFAULT NULL COMMENT 'current version id', + `expire` tinyint(1) DEFAULT 0, + `published_version_id` int(11) DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_datasource_name` (`datasource_name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Environment Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_env` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `env_name` varchar(32) COLLATE utf8_bin NOT NULL, + `env_desc` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `datasource_type_id` int(11) NOT NULL, + `parameter` varchar(2048) COLLATE utf8_bin DEFAULT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_user` varchar(255) COLLATE utf8_bin NULL DEFAULT NULL, + `modify_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `modify_user` varchar(255) COLLATE utf8_bin NULL DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_env_name` (`env_name`), + UNIQUE INDEX `uniq_name_dtid` (`env_name`, `datasource_type_id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Type Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_type` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `name` varchar(32) COLLATE utf8_bin NOT NULL, + `description` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `option` varchar(32) COLLATE utf8_bin DEFAULT NULL, + `classifier` varchar(32) COLLATE utf8_bin NOT NULL, + `icon` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `layers` int(3) NOT NULL, + `description_en` varchar(255) DEFAULT NULL COMMENT 'english description', + `option_en` varchar(32) DEFAULT NULL COMMENT 'english option', + `classifier_en` varchar(32) DEFAULT NULL COMMENT 'english classifier', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_name` (`name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Type Key Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_type_key` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `data_source_type_id` int(11) NOT NULL, + `key` varchar(32) COLLATE utf8_bin NOT NULL, + `name` varchar(32) COLLATE utf8_bin NOT NULL, + `name_en` varchar(32) COLLATE utf8_bin NULL DEFAULT NULL, + `default_value` varchar(50) COLLATE utf8_bin NULL DEFAULT NULL, + `value_type` varchar(50) COLLATE utf8_bin NOT NULL, + `scope` varchar(50) COLLATE utf8_bin NULL DEFAULT NULL, + `require` tinyint(1) NULL DEFAULT 0, + `description` varchar(200) COLLATE utf8_bin NULL DEFAULT NULL, + `description_en` varchar(200) COLLATE utf8_bin NULL DEFAULT NULL, + `value_regex` varchar(200) COLLATE utf8_bin NULL DEFAULT NULL, + `ref_id` bigint(20) NULL DEFAULT NULL, + `ref_value` varchar(50) COLLATE utf8_bin NULL DEFAULT NULL, + `data_source` varchar(200) COLLATE utf8_bin NULL DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_dstid_key` (`data_source_type_id`, `key`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Version Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_version` +( + `version_id` int(11) NOT NULL AUTO_INCREMENT, + `datasource_id` int(11) NOT NULL, + `parameter` varchar(2048) COLLATE utf8_bin NULL DEFAULT NULL, + `comment` varchar(255) COLLATE utf8_bin NULL DEFAULT NULL, + `create_time` datetime(0) NULL DEFAULT CURRENT_TIMESTAMP, + `create_user` varchar(255) COLLATE utf8_bin NULL DEFAULT NULL, + PRIMARY KEY `uniq_vid_did` (`version_id`, `datasource_id`) USING BTREE +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Access Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_access` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `token_id` int(11) NOT NULL, + `service_id` int(11) NOT NULL, + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +## RPC Methods + +The DataSource Service provides several RPC methods for data source management: + +### Data Source RPCs + +#### createDataSource +Creates a new data source: +```java +Long createDataSource(DataSourceCreationRequest request) +``` + +#### getDataSource +Retrieves a data source: +```java +DataSourceInfo getDataSource(Long dataSourceId) +``` + +#### updateDataSource +Updates a data source: +```java +void updateDataSource(DataSourceUpdateRequest request) +``` + +#### deleteDataSource +Deletes a data source: +```java +void deleteDataSource(Long dataSourceId) +``` + +#### listDataSources +Lists data sources with filtering: +```java +List listDataSources(DataSourceQueryRequest request) +``` + +### Metadata RPCs + +#### getMetadata +Retrieves metadata for a data source: +```java +DataSourceMetadata getMetadata(DataSourceMetadataRequest request) +``` + +#### testConnection +Tests connection to a data source: +```java +ConnectionTestResult testConnection(Long dataSourceId) +``` + +#### getTableSchema +Retrieves table schema information: +```java +TableSchema getTableSchema(Long dataSourceId, String database, String table) +``` + +#### getDatabaseList +Retrieves list of databases: +```java +List getDatabaseList(Long dataSourceId) +``` + +#### getTableList +Retrieves list of tables in a database: +```java +List getTableList(Long dataSourceId, String database) +``` + +### Environment RPCs + +#### createEnvironment +Creates a new environment: +```java +Long createEnvironment(EnvironmentCreationRequest request) +``` + +#### getEnvironment +Retrieves an environment: +```java +EnvironmentInfo getEnvironment(Long environmentId) +``` + +#### updateEnvironment +Updates an environment: +```java +void updateEnvironment(EnvironmentUpdateRequest request) +``` + +#### deleteEnvironment +Deletes an environment: +```java +void deleteEnvironment(Long environmentId) +``` + +### Access RPCs + +#### grantAccess +Grants access to a data source: +```java +void grantAccess(DataSourceAccessRequest request) +``` + +#### revokeAccess +Revokes access from a data source: +```java +void revokeAccess(DataSourceAccessRequest request) +``` + +#### checkAccess +Checks if a user has access to a data source: +```java +boolean checkAccess(String user, Long dataSourceId) +``` + +## Dependencies + +- linkis-datasource-manager +- linkis-metadata +- linkis-rpc +- linkis-protocol + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- DataSourceCoreRestfulApi: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceCoreRestfulApi.java` +- DataSourceAdminRestfulApi: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceAdminRestfulApi.java` +- DataSourceRestfulApi: `linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/java/org/apache/linkis/metadata/restful/api/DataSourceRestfulApi.java` +- DatasourceTypeRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/DatasourceTypeRestfulApi.java` +- DatasourceAccessRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/DatasourceAccessRestfulApi.java` +- DatasourceTypeKeyRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/DatasourceTypeKeyRestfulApi.java` + +### MyBatis XML Files +- DataSouceMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSouceMapper.xml` +- DataSourceEnvMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSourceEnvMapper.xml` +- DataSourceParamKeyMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSourceParamKeyMapper.xml` +- DataSourceTypeMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSourceTypeMapper.xml` +- DataSourceVersionMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSourceVersionMapper.xml` +- DataSourceAccessMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/DataSourceAccessMapper.xml` +- DatasourceTypeMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/DatasourceTypeMapper.xml` +- DatasourceTypeKeyMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/DatasourceTypeKeyMapper.xml` \ No newline at end of file diff --git a/.ai/modules/public-enhancements/jobhistory.md b/.ai/modules/public-enhancements/jobhistory.md new file mode 100644 index 00000000000..508c55ba925 --- /dev/null +++ b/.ai/modules/public-enhancements/jobhistory.md @@ -0,0 +1,579 @@ +# JobHistory Service + +The JobHistory service manages job execution history and provides querying capabilities for completed tasks in the Linkis system. + +## Overview + +This service tracks and stores information about job executions, including task status, execution results, logs, and performance metrics. It provides APIs for querying job history, statistics, and diagnostics. + +## Key Components + +### Core Classes +- `LinkisJobHistoryApplication` - Main application class +- Job history persistence and querying +- Task statistics and metrics collection +- Job diagnosis and failure analysis + +### Features +- Job execution history tracking +- Task result and log storage +- Performance statistics and metrics +- Job failure diagnosis +- Historical data querying and filtering + +## API Interfaces + +### Query Task by ID +``` +GET /api/rest_j/v1/jobhistory/{id}/get +``` + +Parameters: +- `id` (required): Task ID +- `brief` (optional): If true, only returns brief info + +Response: +```json +{ + "method": "/api/jobhistory/{id}/get", + "status": 0, + "message": "success", + "data": { + "task": { + "jobId": 12345, + "status": "Succeed", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "1.0", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql", + "params": { + "configuration": { + "runtime": { + "spark.executor.instances": "2" + } + } + } + } + } +} +``` + +### List Tasks +``` +GET /api/rest_j/v1/jobhistory/list +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `status` (optional): Task status filter +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) +- `taskID` (optional): Specific task ID +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter +- `isAdminView` (optional): Admin view flag +- `isDeptView` (optional): Department view flag +- `instance` (optional): Instance filter +- `engineInstance` (optional): Engine instance filter +- `runType` (optional): Run type filter + +Response: +```json +{ + "method": "/api/jobhistory/list", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "status": "Succeed", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "1.0", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql" + } + ], + "totalPage": 100 + } +} +``` + +### List Undone Tasks +``` +GET /api/rest_j/v1/jobhistory/listundonetasks +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `status` (optional): Task status filter (default: "Running,Inited,Scheduled") +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) +- `startTaskID` (optional): Start task ID +- `engineType` (optional): Engine type filter +- `creator` (optional): Creator filter + +Response: +```json +{ + "method": "/api/jobhistory/listundonetasks", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "status": "Running", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "0.5", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql" + } + ], + "totalPage": 10 + } +} +``` + +### List Undone Task Count +``` +GET /api/rest_j/v1/jobhistory/listundone +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) +- `startTaskID` (optional): Start task ID +- `engineType` (optional): Engine type filter +- `creator` (optional): Creator filter + +Response: +```json +{ + "method": "/api/jobhistory/listundone", + "status": 0, + "message": "success", + "data": { + "totalPage": 10 + } +} +``` + +### List Tasks by Task IDs +``` +GET /api/rest_j/v1/jobhistory/list-taskids +``` + +Parameters: +- `taskID` (required): Comma-separated list of task IDs (max 30) + +Response: +```json +{ + "method": "/api/jobhistory/list-taskids", + "status": 0, + "message": "success", + "data": { + "jobHistoryList": [ + { + "jobId": 12345, + "status": "Succeed", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "1.0", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql" + } + ] + } +} +``` + +### Get Job Extra Info +``` +GET /api/rest_j/v1/jobhistory/job-extra-info +``` + +Parameters: +- `jobId` (required): Job ID + +Response: +```json +{ + "method": "/api/jobhistory/job-extra-info", + "status": 0, + "message": "success", + "data": { + "metricsMap": { + "executionCode": "SELECT * FROM table", + "runtime": "300s", + // Additional metrics data + } + } +} +``` + +### Download Job List +``` +GET /api/rest_j/v1/jobhistory/download-job-list +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `status` (optional): Task status filter +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) +- `taskID` (optional): Specific task ID +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter +- `isAdminView` (optional): Admin view flag +- `isDeptView` (optional): Department view flag +- `instance` (optional): Instance filter +- `engineInstance` (optional): Engine instance filter + +Response: +``` +Excel file download +``` + +### List Duration Top Tasks +``` +GET /api/rest_j/v1/jobhistory/listDurationTop +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) + +Response: +```json +{ + "method": "/api/jobhistory/listDurationTop", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "status": "Succeed", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "1.0", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql" + } + ] + } +} +``` + +### Query Failed Task Diagnosis +``` +GET /api/rest_j/v1/jobhistory/diagnosis-query +``` + +Parameters: +- `taskID` (required): Task ID + +Response: +```json +{ + "method": "/api/jobhistory/diagnosis-query", + "status": 0, + "message": "success", + "data": { + "diagnosisMsg": "Diagnosis message content" + } +} +``` + +### Get Governance Station Admin Info +``` +GET /api/rest_j/v1/jobhistory/governanceStationAdmin +``` + +Response: +```json +{ + "method": "/api/jobhistory/governanceStationAdmin", + "status": 0, + "message": "success", + "data": { + "admin": true, + "historyAdmin": true, + "deptAdmin": false, + "canResultSet": true, + "errorMsgTip": "Error message tip" + } +} +``` + +### Task Count Statistics +``` +GET /api/rest_j/v1/jobhistory/jobstatistics/taskCount +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter + +Response: +```json +{ + "method": "/api/jobhistory/jobstatistics/taskCount", + "status": 0, + "message": "success", + "data": { + "sumCount": 100, + "succeedCount": 80, + "failedCount": 15, + "cancelledCount": 5 + } +} +``` + +### Engine Count Statistics +``` +GET /api/rest_j/v1/jobhistory/jobstatistics/engineCount +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter + +Response: +```json +{ + "method": "/api/jobhistory/jobstatistics/engineCount", + "status": 0, + "message": "success", + "data": { + "countEngine": 50, + "countEngineSucceed": 40, + "countEngineFailed": 8, + "countEngineShutting": 2 + } +} +``` + +### Add Observe Info +``` +POST /api/rest_j/v1/jobhistory/setting/addObserveInfo +``` + +Request Body: +```json +{ + "taskId": 12345, + "receiver": "user@example.com", + "extra": { + "title": "Job Alert", + "detail": "Job execution details" + }, + "monitorLevel": "HIGH", + "subSystemId": "subsystem1" +} +``` + +Response: +```json +{ + "method": "/api/jobhistory/setting/addObserveInfo", + "status": 0, + "message": "success" +} +``` + +### Delete Observe Info +``` +GET /api/rest_j/v1/jobhistory/setting/deleteObserveInfo +``` + +Parameters: +- `taskId` (required): Task ID + +Response: +```json +{ + "method": "/api/jobhistory/setting/deleteObserveInfo", + "status": 0, + "message": "success" +} +``` + +## Database Table Structures + +The JobHistory service uses the following database tables for job execution history management: + +### Job History Group History Table +```sql +CREATE TABLE `linkis_ps_job_history_group_history` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job', + `execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job', + `source` text DEFAULT NULL COMMENT 'job source', + `labels` text DEFAULT NULL COMMENT 'job labels', + `params` text DEFAULT NULL COMMENT 'job params', + `progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log', + `error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances', + `metrics` text DEFAULT NULL COMMENT 'Job Metrics', + `engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type', + `execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job', + PRIMARY KEY (`id`), + KEY `idx_created_time` (`created_time`), + KEY `idx_submit_user` (`submit_user`) +); +``` + +### Job History Detail Table +```sql +CREATE TABLE `linkis_ps_job_history_detail` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job', + `result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array', + `job_group_info` text DEFAULT NULL COMMENT 'Job group info/path', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `status` varchar(32) DEFAULT NULL COMMENT 'status', + `priority` int(4) DEFAULT 0 COMMENT 'order of subjob', + PRIMARY KEY (`id`) +); +``` + +## RPC Methods + +The JobHistory service provides several RPC methods for job history management: + +### Job History Query RPCs + +#### getJobHistoryById +Retrieves job history by ID: +```java +JobHistory getJobHistoryById(Long jobId) +``` + +#### searchJobHistory +Searches job history with filters: +```java +List searchJobHistory(JobHistorySearchRequest request) +``` + +#### updateJobHistory +Updates job history information: +```java +void updateJobHistory(JobHistory jobHistory) +``` + +#### deleteJobHistory +Deletes job history: +```java +void deleteJobHistory(Long jobId) +``` + +### Job Statistics RPCs + +#### taskExecutionStatistics +Retrieves task execution statistics: +```java +JobStatistics taskExecutionStatistics(StatisticsRequest request) +``` + +#### engineExecutionStatistics +Retrieves engine execution statistics: +```java +JobStatistics engineExecutionStatistics(StatisticsRequest request) +``` + +### Job Diagnosis RPCs + +#### diagnoseJob +Performs job diagnosis: +```java +JobDiagnosis diagnoseJob(Long jobId) +``` + +#### getDiagnosisInfo +Retrieves diagnosis information: +```java +JobDiagnosis getDiagnosisInfo(Long jobId) +``` + +## Dependencies + +- linkis-jobhistory-server +- linkis-rpc +- linkis-protocol +- linkis-commons +- Database drivers (MySQL, etc.) \ No newline at end of file diff --git a/.ai/modules/public-enhancements/publicservice.md b/.ai/modules/public-enhancements/publicservice.md new file mode 100644 index 00000000000..6606d8a296d --- /dev/null +++ b/.ai/modules/public-enhancements/publicservice.md @@ -0,0 +1,151 @@ +# Public Service + +The Public Service provides core public services for the Linkis system. + +## Overview + +This service provides common public services including file system operations, variable management, and other shared functionalities. + +## Key Components + +### Core Classes +- `LinkisPublicServiceApp` - Main application class +- File system operations +- Variable management +- Shared service utilities + +### Features +- File system operations (upload, download, list) +- Variable management +- Shared service utilities +- Common REST APIs + +## API Interfaces + +### File System Operations +``` +POST /api/rest_j/v1/filesystem/upload +``` + +Request: +``` +multipart/form-data with file content +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/filesystem/upload", + "status": 0, + "message": "success", + "data": { + "path": "/path/to/uploaded/file" + } +} +``` + +### Variable Management +``` +POST /api/rest_j/v1/variable/add +``` + +Request Body: +```json +{ + "key": "variableKey", + "value": "variableValue", + "user": "testuser" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/variable/add", + "status": 0, + "message": "success", + "data": {} +} +``` + +## Database Table Structures + +The Public Service manages the following database tables: + +### File System Metadata Table +```sql +CREATE TABLE linkis_filesystem_meta ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + user_name VARCHAR(32) NOT NULL, + path VARCHAR(500) NOT NULL, + file_type VARCHAR(50), + file_size BIGINT, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + UNIQUE KEY uk_user_path (user_name, path) +); +``` + +### Variable Table +```sql +CREATE TABLE linkis_variable ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + user_name VARCHAR(32) NOT NULL, + key_name VARCHAR(128) NOT NULL, + value TEXT, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + UNIQUE KEY uk_user_key (user_name, key_name) +); +``` + +## RPC Methods + +The Public Service provides several RPC methods for common operations: + +### File System RPCs + +#### uploadFile +Uploads a file: +```java +String uploadFile(FileUploadRequest request) +``` + +#### downloadFile +Downloads a file: +```java +FileContent downloadFile(String path, String user) +``` + +#### listFiles +Lists files in a directory: +```java +List listFiles(String path, String user) +``` + +### Variable RPCs + +#### setVariable +Sets a variable: +```java +void setVariable(String key, String value, String user) +``` + +#### getVariable +Retrieves a variable: +```java +String getVariable(String key, String user) +``` + +#### deleteVariable +Deletes a variable: +```java +void deleteVariable(String key, String user) +``` + +## Dependencies + +- linkis-filesystem +- linkis-variable +- linkis-rpc +- linkis-protocol \ No newline at end of file diff --git a/.ai/project-context.md b/.ai/project-context.md new file mode 100644 index 00000000000..fcddd703cb9 --- /dev/null +++ b/.ai/project-context.md @@ -0,0 +1,1027 @@ +# Apache Linkis AI IDE 开发规约 + +> **文档版本信息** +> - 版本: 1.0.0 +> - 最后更新: 2025-01-28 +> - 适用版本: Apache Linkis 1.17.0+ + +## 角色定位 +你是Apache Linkis项目的资深后端开发专家,熟练掌握: +- **核心技术栈**:Spring Boot 2.7 + Spring Cloud 2021.0.8 + MyBatis-Plus 3.5.7 +- **编程语言**:Java 8 + Scala 2.12(混合开发模式) +- **数据库**:MySQL 8.0 + Hive(通过JDBC) +- **微服务架构**:Eureka服务发现 + Gateway网关 + Feign远程调用 +- **大数据引擎**:Spark、Hive、Flink、Python、Shell等多引擎支持 + +--- + +# 项目核心信息 + +## 基础配置 +- **项目根目录**:linkis +- **基础包名**:org.apache.linkis +- **版本信息**:Apache Linkis 1.x +- **构建工具**:Maven 3.5+ +- **JDK版本**:1.8 +- **字符编码**:统一使用StandardCharsets.UTF_8 + +## 关键组件 +- **统一返回体**:`org.apache.linkis.server.Message` +- **统一异常**:`org.apache.linkis.common.exception.LinkisException` +- **配置管理**:`org.apache.linkis.common.conf.CommonVars` +- **数据库脚本**: + - DDL:`linkis-dist/package/db/linkis_ddl.sql` + - DML:`linkis-dist/package/db/linkis_dml.sql` + +--- + +# 系统架构设计 + +## 三层架构模式 +Linkis采用微服务架构,按功能职责划分为三大服务类别: + +### 1. 微服务治理服务(基础设施层) +负责微服务的基础设施支撑,包括服务发现、网关路由、配置管理等。 +- Spring Cloud Gateway:API网关服务 +- Eureka:服务注册与发现中心 +- Open Feign:声明式HTTP客户端 + +### 2. 计算治理服务(核心业务层) +负责计算任务的生命周期管理,从任务提交到执行完成的全流程控制。 +- Entrance:任务提交入口服务 +- JobHistory:任务历史记录服务 +- LinkisManager:资源管理服务 +- EngineConnManager:引擎连接管理服务 +- EngineConn:引擎连接器 + +### 3. 公共增强服务(支撑服务层) +提供跨服务的公共能力,如文件管理、数据源管理、配置管理等。 +- PublicService:公共服务 +- BML:大数据物料库 +- DataSource:数据源管理 +- Configuration:配置管理 +- ContextServer:上下文服务 +- Monitor:监控服务 + +## 服务交互模式 +``` +上层应用 -> Gateway -> Entrance -> Manager -> ECM -> EngineConn -> 底层引擎 + ↓ + 公共增强服务(BML、DataSource、Configuration等) +``` + +## 各服务模块说明 +### 微服务治理服务 +Spring Cloud Gateway +功能:API网关服务,负责请求路由转发、负载均衡、安全认证等 +主类入口:org.apache.linkis.gateway.springcloud.LinkisGatewayApplication +模块路径:linkis-spring-cloud-services/linkis-service-gateway/linkis-spring-cloud-gateway + +Eureka +功能:服务注册与发现中心,管理微服务实例的注册、发现和健康检查 +主类入口:org.apache.linkis.eureka.SpringCloudEurekaApplication +模块路径:linkis-spring-cloud-services/linkis-service-discovery/linkis-eureka + +Open Feign +功能:声明式HTTP客户端,简化微服务间的远程调用 +主类入口:集成在各个微服务模块中,无独立启动类 +模块路径:集成在linkis-commons/linkis-rpc等公共模块中 + +### 计算治理服务 +Entrance +功能:任务提交入口服务,负责任务调度、状态管控、任务信息推送等核心功能 +主类入口:org.apache.linkis.entrance.LinkisEntranceApplication +模块路径:linkis-computation-governance/linkis-entrance + +JobHistory +功能:任务历史记录服务,提供任务执行历史的查询、统计和管理功能 +主类入口:org.apache.linkis.jobhistory.LinkisJobHistoryApp +模块路径:linkis-public-enhancements/linkis-jobhistory + +LinkisManager +功能:计算治理层的管理服务,包含AppManager、ResourceManager、LabelManager等管理控制服务 +主类入口:org.apache.linkis.manager.LinkisManagerApplication +模块路径:linkis-computation-governance/linkis-manager/linkis-application-manager + +EngineConnManager +功能:引擎连接器管理服务,负责控制EngineConn的生命周期(启动、停止) +主类入口:org.apache.linkis.ecm.server.LinkisECMApplication +模块路径:linkis-computation-governance/linkis-engineconn-manager/linkis-engineconn-manager-server + +EngineConn +功能:引擎连接器,负责接收任务并提交到Spark、Hive、Flink等底层引擎执行 +主类入口:org.apache.linkis.engineconn.LinkisEngineConnApplication +模块路径:linkis-computation-governance/linkis-engineconn + +### 公共增强服务 +PublicService +功能:公共服务模块,提供统一配置管理、微服务管理等基础服务能力 +主类入口:org.apache.linkis.filesystem.LinkisPublicServiceApp +模块路径:linkis-public-enhancements/linkis-pes-publicservice + +BML +功能:大数据物料库服务(BigData Material Library),提供文件上传、下载、版本管理等功能 +主类入口:org.apache.linkis.bml.LinkisBMLApplication +模块路径:linkis-public-enhancements/linkis-bml-server + +DataSource +功能:数据源管理服务,提供统一的数据源连接、管理和元数据服务 +主类入口:org.apache.linkis.metadata.LinkisDataSourceApplication(数据源服务) +模块路径:linkis-public-enhancements/linkis-datasource + +Configuration +功能:配置管理服务,提供系统级、用户级、引擎级等多层次的配置管理 +主类入口:org.apache.linkis.configuration.LinkisConfigurationApp +模块路径:linkis-public-enhancements/linkis-configuration + +ContextServer +功能:上下文服务,支持跨引擎的资源共享、变量传递和会话管理 +主类入口:org.apache.linkis.cs.server.LinkisCSApplication +模块路径:linkis-public-enhancements/linkis-cs-server + +Monitor +功能:监控服务,提供系统性能监控、告警和运维管理功能,包括任务监控、资源监控、用户模式监控等 +主类入口:org.apache.linkis.monitor.LinksMonitorApplication +模块路径:linkis-extensions/linkis-et-monitor + +--- + +# 开发规范与约束 + +## 代码边界约束 + +### 🚫 禁止操作 +- **数据库结构**:除非明确指定,严禁修改现有表结构 +- **第三方依赖**:不允许引入新的第三方依赖库 +- **核心接口**:不得修改现有公共接口的签名 + +### ✅ 允许操作 +- **新增功能**:在不破坏现有逻辑的前提下扩展功能 +- **新增配置**:在现有配置文件中新增配置项 +- **新增表字段**:在现有表基础上新增字段 + +## 技术规范 + +### 编程语言使用 +- **Java**:主要用于REST API、Service层、Entity类、配置类 +- **Scala**:主要用于计算逻辑、RPC通信、复杂业务处理 + +### 日志规范 +```java +// 必须使用统一的Logger +private static final Logger logger = LoggerFactory.getLogger(ClassName.class); + +// 日志级别使用: +// ERROR: 系统错误、业务异常 +// WARN: 警告信息、降级处理 +// INFO: 关键业务节点、状态变更 +// DEBUG: 详细调试信息 + +logger.info("User {} starts processing task {}", username, taskId); +logger.error("Failed to process task {} for user {}", taskId, username, e); +``` + +### 配置管理规范 +- 所有配置统一使用`org.apache.linkis.common.conf.CommonVars` +- 参考示例:`org.apache.linkis.jobhistory.conf.JobhistoryConfiguration` +- 所有新需求必须添加配置开关,默认设置false +- 配置存放位置:当前模块的conf目录,一般为xxxConfiguration类 + +### 字符编码规范 +```java +// 统一使用StandardCharsets.UTF_8,禁止使用字符串"UTF-8" +import java.nio.charset.StandardCharsets; + +String content = new String(bytes, StandardCharsets.UTF_8); +Files.write(path, content.getBytes(StandardCharsets.UTF_8)); +``` + +### API设计规范 +```java +@Api(tags = "module operation") +@RestController +@RequestMapping(path = "/api/rest_j/v1/module") +public class ModuleRestfulApi { + + @ApiOperation(value = "operation", notes = "description", response = Message.class) + @RequestMapping(path = "/operation", method = RequestMethod.POST) + public Message operation(HttpServletRequest req, @RequestBody JsonNode jsonNode) { + String username = ModuleUserUtils.getOperationUser(req, "operation"); + // 业务逻辑处理 + return Message.ok("success").data("result", data); + } +} +``` + +### 异常处理规范 +```java +// 统一使用LinkisException及其子类 +try { + // 业务逻辑 +} catch (Exception e) { + logger.error("Operation failed", e); + throw new YourModuleException("Error message", e); +} +``` + +--- + +# 开发模板与示例 + +## 新功能开发模板 + +### 1. REST接口层 +```java +@Api(tags = "功能模块操作") +@RestController +@RequestMapping(path = "/api/rest_j/v1/module") +public class ModuleRestfulApi { + + @Autowired + private ModuleService moduleService; + + @ApiOperation(value = "功能操作", response = Message.class) + @RequestMapping(path = "/action", method = RequestMethod.POST) + public Message action(HttpServletRequest req, @RequestBody JsonNode jsonNode) { + String username = ModuleUserUtils.getOperationUser(req, "action"); + + // 参数解析和验证 + String param = jsonNode.get("param").asText(); + if (StringUtils.isBlank(param)) { + return Message.error("参数不能为空"); + } + + try { + Object result = moduleService.performAction(param, username); + return Message.ok("操作成功").data("result", result); + } catch (Exception e) { + logger.error("操作失败", e); + return Message.error("操作失败:" + e.getMessage()); + } + } +} +``` + +### 2. 服务层 +```java +@Service +public class ModuleServiceImpl implements ModuleService { + + private static final Logger logger = LoggerFactory.getLogger(ModuleServiceImpl.class); + + @Autowired + private ModuleMapper moduleMapper; + + @Override + @Transactional(rollbackFor = Exception.class) + public Object performAction(String param, String username) { + logger.info("User {} starts action with param: {}", username, param); + + // 业务逻辑处理 + ModuleEntity entity = new ModuleEntity(); + entity.setParam(param); + entity.setCreateUser(username); + + moduleMapper.insert(entity); + + logger.info("User {} completed action successfully", username); + return entity.getId(); + } +} +``` + +### 3. 数据访问层 +```java +@Mapper +public interface ModuleMapper { + + @Insert("INSERT INTO linkis_module_table (param, create_user, create_time) " + + "VALUES (#{param}, #{createUser}, NOW())") + @Options(useGeneratedKeys = true, keyProperty = "id") + void insert(ModuleEntity entity); + + @Select("SELECT * FROM linkis_module_table WHERE id = #{id}") + ModuleEntity selectById(@Param("id") Long id); +} +``` + +### 4. 配置类 +```scala +object ModuleConfiguration { + val MODULE_FEATURE_ENABLE = CommonVars("linkis.module.feature.enable", false) + val MODULE_TIMEOUT = CommonVars("linkis.module.timeout", 30000L) + val MODULE_BATCH_SIZE = CommonVars("linkis.module.batch.size", 1000) +} +``` + +--- + +# 常用配置示例库 + +## 配置定义示例 + +### 功能开关配置 +```scala +object FeatureConfiguration { + // 布尔型开关 - 用于控制功能是否启用 + val FEATURE_ENABLE = CommonVars("linkis.feature.enable", false) + + // 数值型配置 - 批处理大小 + val BATCH_SIZE = CommonVars("linkis.feature.batch.size", 1000) + + // 长整型配置 - 超时时间(毫秒) + val TIMEOUT = CommonVars("linkis.feature.timeout", 30000L) + + // 字符串配置 - 运行模式 + val MODE = CommonVars("linkis.feature.mode", "default") + + // 浮点型配置 - 阈值 + val THRESHOLD = CommonVars("linkis.feature.threshold", 0.8) + + // 列表型配置 - 逗号分隔 + val ALLOWED_TYPES = CommonVars("linkis.feature.allowed.types", "spark,hive,python") +} +``` + +### 性能相关配置 +```scala +object PerformanceConfiguration { + // 线程池大小 + val THREAD_POOL_SIZE = CommonVars("linkis.performance.thread.pool.size", 10) + + // 队列容量 + val QUEUE_CAPACITY = CommonVars("linkis.performance.queue.capacity", 1000) + + // 连接池配置 + val MAX_CONNECTIONS = CommonVars("linkis.performance.max.connections", 50) + val MIN_IDLE = CommonVars("linkis.performance.min.idle", 5) + + // 缓存配置 + val CACHE_ENABLE = CommonVars("linkis.performance.cache.enable", true) + val CACHE_SIZE = CommonVars("linkis.performance.cache.size", 10000) + val CACHE_EXPIRE_SECONDS = CommonVars("linkis.performance.cache.expire.seconds", 3600L) +} +``` + +### 重试和容错配置 +```scala +object ResilienceConfiguration { + // 重试次数 + val MAX_RETRY_TIMES = CommonVars("linkis.resilience.max.retry.times", 3) + + // 重试间隔(毫秒) + val RETRY_INTERVAL = CommonVars("linkis.resilience.retry.interval", 1000L) + + // 熔断开关 + val CIRCUIT_BREAKER_ENABLE = CommonVars("linkis.resilience.circuit.breaker.enable", false) + + // 失败率阈值 + val FAILURE_RATE_THRESHOLD = CommonVars("linkis.resilience.failure.rate.threshold", 0.5) +} +``` + +## 配置使用示例 + +### 在Java代码中使用配置 +```java +@Service +public class FeatureServiceImpl implements FeatureService { + + private static final Logger logger = LoggerFactory.getLogger(FeatureServiceImpl.class); + + @Override + public void executeFeature() { + // 检查功能开关 + if (!FeatureConfiguration.FEATURE_ENABLE.getValue()) { + logger.info("Feature is disabled, skipping execution"); + return; // 功能关闭时不执行 + } + + // 使用配置参数 + int batchSize = FeatureConfiguration.BATCH_SIZE.getValue(); + long timeout = FeatureConfiguration.TIMEOUT.getValue(); + String mode = FeatureConfiguration.MODE.getValue(); + + logger.info("Executing feature with batchSize={}, timeout={}, mode={}", + batchSize, timeout, mode); + + // 业务逻辑... + } +} +``` + +### 在Scala代码中使用配置 +```scala +class FeatureExecutor { + + def execute(): Unit = { + // 检查功能开关 + if (!FeatureConfiguration.FEATURE_ENABLE.getValue) { + logger.info("Feature is disabled") + return + } + + // 获取配置值 + val batchSize = FeatureConfiguration.BATCH_SIZE.getValue + val timeout = FeatureConfiguration.TIMEOUT.getValue + val allowedTypes = FeatureConfiguration.ALLOWED_TYPES.getValue.split(",").toList + + // 使用配置执行业务逻辑 + processBatch(batchSize, timeout, allowedTypes) + } +} +``` + +### 带降级逻辑的配置使用 +```java +public class SmartFeatureService { + + public void processWithFallback(List dataList) { + // 检查功能开关 + if (!FeatureConfiguration.FEATURE_ENABLE.getValue()) { + // 降级到旧逻辑 + processLegacy(dataList); + return; + } + + try { + // 新功能逻辑 + int batchSize = FeatureConfiguration.BATCH_SIZE.getValue(); + processInBatches(dataList, batchSize); + } catch (Exception e) { + logger.error("New feature failed, falling back to legacy", e); + // 异常时降级 + processLegacy(dataList); + } + } + + private void processLegacy(List dataList) { + // 原有的稳定逻辑 + } +} +``` + +### 配置验证和边界检查 +```java +public class ConfigValidator { + + public static void validateAndExecute() { + // 获取配置 + int batchSize = FeatureConfiguration.BATCH_SIZE.getValue(); + + // 验证配置合法性 + if (batchSize <= 0 || batchSize > 10000) { + logger.error("Invalid batch size: {}, using default 1000", batchSize); + batchSize = 1000; + } + + // 使用验证后的配置 + processBatch(batchSize); + } +} +``` + +## 配置文件示例 + +### linkis.properties 配置示例 +```properties +# 功能开关配置 +linkis.feature.enable=false +linkis.feature.batch.size=1000 +linkis.feature.timeout=30000 +linkis.feature.mode=default + +# 性能配置 +linkis.performance.thread.pool.size=10 +linkis.performance.queue.capacity=1000 +linkis.performance.cache.enable=true + +# 重试配置 +linkis.resilience.max.retry.times=3 +linkis.resilience.retry.interval=1000 +``` + +## 配置最佳实践 + +### ✅ 推荐做法 +1. **所有新功能必须有开关**,默认值设为 `false` +2. **配置命名规范**:`linkis.[模块].[功能].[属性]` +3. **提供合理的默认值**,确保不配置时系统能正常运行 +4. **添加配置注释**,说明配置的作用和取值范围 +5. **配置集中管理**,放在对应模块的 Configuration 类中 + +### ❌ 避免做法 +1. 不要硬编码配置值 +2. 不要在多处重复定义相同配置 +3. 不要使用不合理的默认值(如 0、空字符串) +4. 不要忘记在 linkis.properties 中添加配置说明 + +--- + +# 常见错误及避免方法 + +## ❌ 错误1:字符编码使用不规范 + +### 错误示例 +```java +// ❌ 错误:使用字符串 "UTF-8" +String content = new String(bytes, "UTF-8"); +FileWriter writer = new FileWriter(file, "UTF-8"); +response.setCharacterEncoding("UTF-8"); + +// 问题: +// 1. 字符串容易拼写错误 +// 2. 编译器无法检查 +// 3. 不符合项目规范 +``` + +### 正确示例 +```java +// ✅ 正确:使用 StandardCharsets.UTF_8 +import java.nio.charset.StandardCharsets; + +String content = new String(bytes, StandardCharsets.UTF_8); +Files.write(path, content.getBytes(StandardCharsets.UTF_8)); +response.setCharacterEncoding(StandardCharsets.UTF_8.name()); + +// 优点: +// 1. 编译时检查 +// 2. 不会拼写错误 +// 3. 符合项目规范 +``` + +--- + +## ❌ 错误2:新功能未添加开关 + +### 错误示例 +```java +// ❌ 错误:新功能直接生效,无法回退 +@Service +public class NewFeatureService { + + public void executeNewFeature() { + // 直接实现新逻辑 + // 如果出现问题,只能通过代码回退或重新部署 + newAlgorithm(); + } +} +``` + +### 正确示例 +```java +// ✅ 正确:添加功能开关,支持热切换 +@Service +public class SmartFeatureService { + + private static final Logger logger = LoggerFactory.getLogger(SmartFeatureService.class); + + public void executeFeature() { + // 检查功能开关 + if (!NewFeatureConfiguration.ENABLE.getValue()) { + logger.info("New feature is disabled, using legacy implementation"); + executeLegacyFeature(); // 降级到旧逻辑 + return; + } + + try { + logger.info("New feature is enabled"); + executeNewFeature(); // 执行新逻辑 + } catch (Exception e) { + logger.error("New feature failed, falling back to legacy", e); + executeLegacyFeature(); // 异常时降级 + } + } + + private void executeNewFeature() { + // 新功能实现 + } + + private void executeLegacyFeature() { + // 原有稳定实现 + } +} + +// 配置类 +object NewFeatureConfiguration { + val ENABLE = CommonVars("linkis.new.feature.enable", false) +} +``` + +--- + +## ❌ 错误3:修改现有表结构未记录 + +### 错误示例 +```sql +-- ❌ 错误:直接在数据库执行 ALTER TABLE +-- 问题: +-- 1. 其他环境无法同步 +-- 2. 没有变更记录 +-- 3. 无法回滚 + +ALTER TABLE linkis_ps_job_history_group_history +ADD COLUMN new_field VARCHAR(50) COMMENT 'new field'; +``` + +### 正确示例 +```sql +-- ✅ 正确:在 linkis-dist/package/db/linkis_ddl.sql 中添加变更 + +-- Step 1: 在 linkis_ddl.sql 文件末尾添加变更记录 +-- ================================================================ +-- 版本: 1.17.0 +-- 需求: 添加任务扩展字段支持 +-- 日期: 2025-01-28 +-- ================================================================ + +ALTER TABLE linkis_ps_job_history_group_history +ADD COLUMN new_field VARCHAR(50) COMMENT 'new field for extended info'; + +-- 如果有索引变更 +CREATE INDEX idx_new_field ON linkis_ps_job_history_group_history(new_field); + +-- Step 2: 如果需要初始化数据,在 linkis_dml.sql 中添加 +-- 在 linkis-dist/package/db/linkis_dml.sql 添加: +UPDATE linkis_ps_job_history_group_history +SET new_field = 'default_value' +WHERE new_field IS NULL; +``` + +--- + +## ❌ 错误4:异常处理不规范 + +### 错误示例 +```java +// ❌ 错误示例1:吞掉异常 +try { + processData(); +} catch (Exception e) { + // 什么都不做,异常被吞掉 +} + +// ❌ 错误示例2:打印后继续抛出原始异常 +try { + processData(); +} catch (Exception e) { + e.printStackTrace(); // 不要使用 printStackTrace + throw e; // 直接抛出原始异常 +} + +// ❌ 错误示例3:捕获过于宽泛 +try { + processData(); +} catch (Throwable t) { // 不要捕获 Throwable + logger.error("Error", t); +} +``` + +### 正确示例 +```java +// ✅ 正确示例1:记录日志并抛出业务异常 +try { + processData(); +} catch (IOException e) { + logger.error("Failed to process data", e); + throw new DataProcessException("Failed to process data", e); +} + +// ✅ 正确示例2:捕获具体异常,提供有意义的错误信息 +try { + String result = processData(param); + return result; +} catch (IllegalArgumentException e) { + logger.error("Invalid parameter: {}", param, e); + throw new ValidationException("Invalid parameter: " + param, e); +} catch (IOException e) { + logger.error("IO error while processing data", e); + throw new DataAccessException("IO error while processing data", e); +} + +// ✅ 正确示例3:在Service层统一处理异常 +@Service +public class DataServiceImpl implements DataService { + + @Override + public Result processData(String param) { + try { + // 业务逻辑 + String data = fetchData(param); + return Result.success(data); + } catch (DataNotFoundException e) { + logger.warn("Data not found for param: {}", param); + return Result.error("Data not found"); + } catch (Exception e) { + logger.error("Unexpected error while processing data", e); + throw new ServiceException("Failed to process data", e); + } + } +} +``` + +--- + +## ❌ 错误5:日志记录不规范 + +### 错误示例 +```java +// ❌ 错误示例1:使用 System.out +System.out.println("Processing data: " + data); + +// ❌ 错误示例2:日志级别使用不当 +logger.error("User {} logged in", username); // 登录不是错误 + +// ❌ 错误示例3:字符串拼接 +logger.info("Processing user: " + username + ", id: " + userId); + +// ❌ 错误示例4:敏感信息直接打印 +logger.info("User password: {}", password); +``` + +### 正确示例 +```java +// ✅ 正确示例1:使用 Logger +private static final Logger logger = LoggerFactory.getLogger(ClassName.class); + +// ✅ 正确示例2:使用正确的日志级别 +logger.info("User {} logged in successfully", username); // INFO +logger.warn("Login attempt from unknown IP: {}", ip); // WARN +logger.error("Failed to authenticate user {}", username, exception); // ERROR + +// ✅ 正确示例3:使用占位符 +logger.info("Processing user: {}, id: {}, type: {}", username, userId, userType); + +// ✅ 正确示例4:脱敏处理敏感信息 +logger.info("User {} password updated", username); // 不打印密码 +logger.debug("Token: {}***", token.substring(0, 4)); // 只打印前几位 + +// ✅ 正确示例5:关键业务节点记录完整上下文 +logger.info("Task submitted: taskId={}, user={}, engineType={}, code={}", + taskId, username, engineType, codePreview); +logger.error("Task execution failed: taskId={}, user={}, error={}", + taskId, username, e.getMessage(), e); +``` + +--- + +## ❌ 错误6:REST接口返回值不规范 + +### 错误示例 +```java +// ❌ 错误:直接返回业务对象或String +@RequestMapping(path = "/getData", method = RequestMethod.GET) +public UserData getData() { + return userData; // 不符合统一返回体规范 +} + +@RequestMapping(path = "/save", method = RequestMethod.POST) +public String save(@RequestBody Data data) { + return "success"; // 不符合规范 +} +``` + +### 正确示例 +```java +// ✅ 正确:使用统一返回体 Message +import org.apache.linkis.server.Message; + +@RequestMapping(path = "/getData", method = RequestMethod.GET) +public Message getData(HttpServletRequest req) { + try { + String username = ModuleUserUtils.getOperationUser(req, "getData"); + UserData data = userService.getData(username); + return Message.ok("Query successful").data("userData", data); + } catch (Exception e) { + logger.error("Failed to get user data", e); + return Message.error("Failed to get user data: " + e.getMessage()); + } +} + +@RequestMapping(path = "/save", method = RequestMethod.POST) +public Message save(HttpServletRequest req, @RequestBody JsonNode jsonNode) { + try { + String username = ModuleUserUtils.getOperationUser(req, "save"); + + // 参数验证 + String name = jsonNode.get("name").asText(); + if (StringUtils.isBlank(name)) { + return Message.error("Name cannot be empty"); + } + + Long id = dataService.save(name, username); + return Message.ok("Save successful").data("id", id); + } catch (Exception e) { + logger.error("Failed to save data", e); + return Message.error("Failed to save data: " + e.getMessage()); + } +} +``` + +--- + +## ❌ 错误7:MyBatis SQL注入风险 + +### 错误示例 +```xml + + +``` + +### 正确示例 +```xml + + + + + +``` + +```java +// 在Service层验证动态字段 +public List selectWithOrder(String orderBy) { + // 白名单验证 + List allowedFields = Arrays.asList("id", "name", "create_time"); + if (!allowedFields.contains(orderBy)) { + throw new IllegalArgumentException("Invalid order field: " + orderBy); + } + return userMapper.selectWithOrder(orderBy); +} +``` + +--- + +## ❌ 错误8:事务使用不当 + +### 错误示例 +```java +// ❌ 错误示例1:没有添加事务注解 +@Service +public class OrderService { + public void createOrder(Order order) { + orderMapper.insert(order); // 插入订单 + stockMapper.decrease(order.getProductId()); // 减库存 + // 如果减库存失败,订单已经插入,数据不一致 + } +} + +// ❌ 错误示例2:捕获异常后未抛出,事务不会回滚 +@Transactional +public void processOrder(Order order) { + try { + orderMapper.insert(order); + stockMapper.decrease(order.getProductId()); + } catch (Exception e) { + logger.error("Error", e); + // 异常被吞掉,事务不会回滚 + } +} +``` + +### 正确示例 +```java +// ✅ 正确示例1:添加事务注解,指定回滚异常 +@Service +public class OrderService { + + @Transactional(rollbackFor = Exception.class) + public void createOrder(Order order) { + orderMapper.insert(order); + stockMapper.decrease(order.getProductId()); + // 任何异常都会回滚 + } +} + +// ✅ 正确示例2:如果需要捕获异常,重新抛出 +@Transactional(rollbackFor = Exception.class) +public void processOrder(Order order) { + try { + orderMapper.insert(order); + stockMapper.decrease(order.getProductId()); + } catch (StockNotEnoughException e) { + logger.warn("Stock not enough for product: {}", order.getProductId()); + throw e; // 重新抛出,触发回滚 + } catch (Exception e) { + logger.error("Unexpected error while processing order", e); + throw new OrderProcessException("Failed to process order", e); + } +} + +// ✅ 正确示例3:部分操作不需要事务 +@Service +public class OrderService { + + @Transactional(rollbackFor = Exception.class) + public Long createOrder(Order order) { + // 数据库操作在事务中 + orderMapper.insert(order); + stockMapper.decrease(order.getProductId()); + + Long orderId = order.getId(); + + // 发送通知不在事务中(避免外部调用导致事务超时) + sendNotificationAsync(orderId); + + return orderId; + } + + private void sendNotificationAsync(Long orderId) { + // 异步发送,不阻塞事务 + executor.submit(() -> notificationService.send(orderId)); + } +} +``` + +--- + +## 🎯 错误排查清单 + +开发完成后,请检查以下项目: + +- [ ] 字符编码统一使用 `StandardCharsets.UTF_8` +- [ ] 新功能已添加开关配置(默认false) +- [ ] 数据库变更已记录到 DDL/DML 文件 +- [ ] 异常处理规范,使用 LinkisException 及其子类 +- [ ] 日志使用 Logger,不使用 System.out +- [ ] REST接口使用统一返回体 Message +- [ ] SQL 使用参数化查询,避免注入 +- [ ] 事务注解正确使用,异常能正确回滚 +- [ ] 敏感信息已脱敏处理 +- [ ] 代码遵循最小改动原则 + +--- + +# 需求开发流程 + +## 需求分析模板 + +### 【背景说明】 +描述业务场景、现有问题或痛点、期望解决的目标 + +### 【验收标准】 +- 功能验收点(具体、可测量) +- 性能要求(响应时间、并发数等) +- 安全要求(权限控制、数据保护) +- 兼容性要求(向后兼容) + +## 开发交付清单 + +### 变更清单 +- 新增/修改的文件路径列表 +- 数据库变更脚本(DDL/DML) +- 配置文件变更 + +### 测试验证 +- 单元测试代码 +- 集成测试用例 +- 手动测试命令(curl等) + +### 质量检查 +- [ ] 代码符合项目规范 +- [ ] 异常处理完整 +- [ ] 日志记录充分 +- [ ] 单元测试覆盖 +- [ ] 配置开关完整 +- [ ] 向后兼容性检查 + +--- + +# AI IDE开发提示 + +## 开发技巧 +1. **优先查看现有代码**:在新增功能前,先查看相似功能的实现方式 +2. **遵循现有模式**:保持与现有代码风格一致 +3. **充分测试**:编写充分的单元测试和集成测试 +4. **考虑边界情况**:处理各种异常和边界条件 + +## 常见问题及解决方案 + +### 1. 字符编码问题 +**问题**:HTTP传输过程中出现中文乱码 +**解决**:统一使用`StandardCharsets.UTF_8` + +### 2. 配置热更新问题 +**问题**:配置修改后需要重启服务 +**解决**:使用`CommonVars`并配合`@RefreshScope`注解 + +### 3. 性能优化问题 +**问题**:大批量数据处理性能差 +**解决**:采用分页处理,单次处理不超过5000条 + +--- + +**📝 重要提示** +1. 严格遵循现有架构设计,不得随意修改核心组件 +2. 新增功能必须考虑向后兼容性 +3. 关键业务逻辑必须有完整的异常处理和日志记录 +4. 所有配置项必须有合理的默认值 +5. 代码提交前必须通过本地测试验证 \ No newline at end of file diff --git a/.ai/rules.md b/.ai/rules.md new file mode 100644 index 00000000000..d77bf964353 --- /dev/null +++ b/.ai/rules.md @@ -0,0 +1,401 @@ +# AI Development Rules + +> **文档版本信息** +> - 版本: 1.0.0 +> - 最后更新: 2025-01-28 +> - 适用版本: Apache Linkis 1.17.0+ + +> ⚠️ **CRITICAL**: 这些是强制性规则,AI必须无条件遵守。违反规则的代码将被拒绝合并。 + +## 📋 目录 +- [需求实现步骤](#需求实现步骤) +- [最小改动原则](#最小改动原则) +- [功能可配置原则](#功能可配置原则) +- [数据库修改原则](#数据库修改原则) +- [配置管理规则](#配置管理规则) +- [代码边界约束](#代码边界约束) + +### 需求实现步骤 + +#### 步骤1:确定当前版本号 +- 查看pom.xml文件中的``配置 +- 如配置为`1.17.0-wds`,则提取版本号为`1.17.0` +- 后文用`${current_version}`代替 + +#### 步骤2:环境准备检查 +**⚠️ 开始开发前,请确认以下环境准备工作已完成** + +**AI操作:** 提示用户确认以下条件是否满足: + +``` +请在开始开发前,手动确认以下条件: + +1. ✅ 当前在正确的基础分支上(dev-${current_version}-webank) + 验证命令: git branch --show-current + +2. ✅ 工作目录干净(无未提交修改) + 验证命令: git status + 预期输出: "working tree clean" 或 "nothing to commit" + +3. ✅ 本地分支已与远程同步 + 验证命令: git status + 预期输出: "Your branch is up to date" + +如果以上条件未满足,请先处理后再继续。需要我协助处理吗? +``` + +**用户确认后,AI才继续执行后续步骤。** + +#### 步骤3:创建新的需求修改分支 +- 在确认的基础分支上创建新分支 +- 分支命名规则:`feature/${current_version}-<需求简述>` + +#### 步骤4:创建文档目录 +- 创建目录:`docs/${current_version}/requirements`和`docs/${current_version}/design` +- 如果目录已存在则跳过 + +#### 步骤5:创建需求文档 +- 按项目标准需求文档格式创建markdown文档 +- 存放路径:`docs/${current_version}/requirements/<需求名称>.md` + +#### 步骤6:创建设计文档 +- 按项目标准设计文档格式创建markdown文档 +- 存放路径:`docs/${current_version}/design/<需求名称>-design.md` + +#### 步骤7:代码开发 +- 按需求和设计文档进行开发 +- 必须遵循本文件中的所有原则(最小改动、功能可配置等) + +### 最小改动原则 +- 所有功能实现必须遵循最小改动原则,修改内容尽量不影响现有功能。 + +### 功能可配置原则 +- 所有功能必须增加功能开关,在开关关闭时功能相当于回退到上一个版本。开关配置遵循配置管理规则 + +### 数据库修改原则 +- 在能不改动现有表结构和表数据的情况下尽量不改动 +- 对于必须改动表结构和数据的情况下,将改动存档。具体路径如下 + - DDL:`linkis-dist/package/db/linkis_ddl.sql` + - DML:`linkis-dist/package/db/linkis_dml.sql` + +### 配置管理规则 +- 所有配置统一使用`org.apache.linkis.common.conf.CommonVars` +- 参考示例:`org.apache.linkis.jobhistory.conf.JobhistoryConfiguration` +- 配置存放位置:当前模块的conf目录,一般为xxxConfiguration类 + +### 代码边界约束 + +#### 🚫 禁止操作 +- **数据库结构**:除非明确指定,严禁修改现有表结构 +- **第三方依赖**:不允许引入新的第三方依赖库 +- **核心接口**:不得修改现有公共接口的签名 + +#### ✅ 允许操作 +- **新增功能**:在不破坏现有逻辑的前提下扩展功能 +- **新增配置**:在现有配置文件中新增配置项 +- **新增表字段**:在现有表基础上新增字段 + +### 其它规则 +- 所有功能只用实现后端接口功能,无需考虑前端设计和开发 + +--- + +# 开发决策流程图 + +## 🤔 决策1:是否需要修改数据库? + +``` +┌─────────────────────┐ +│ 需求分析开始 │ +└──────────┬──────────┘ + │ + ▼ + ┌──────────────┐ + │ 需要修改数据库? │ + └──┬───────┬───┘ + │ │ + 是 │ │ 否 + │ │ + ▼ ▼ + ┌─────────┐ ┌──────────────┐ + │ 能否通过 │ │ 直接开发代码 │ + │新增字段实现?│ │ (添加功能开关) │ + └─┬───┬──┘ └──────────────┘ + │ │ + 是 │ │ 否 + │ │ + ▼ ▼ + 优先 修改表结构 + 新增 (需评审) + 字段 + │ │ + └─┬─┘ + │ + ▼ + ┌──────────────────┐ + │ 记录DDL/DML变更 │ + │ + 版本信息 │ + │ + 需求说明 │ + └──────────────────┘ +``` + +**决策规则:** +1. **优先选择**: 新增字段(向后兼容) +2. **谨慎选择**: 修改字段类型、删除字段(需评审) +3. **必须记录**: 所有DDL/DML变更到规定文件 +4. **必须标注**: 版本号、需求描述、日期 + +--- + +## 🔧 决策2:如何选择开发语言? + +``` +┌─────────────────────┐ +│ 功能模块分析 │ +└──────────┬──────────┘ + │ + ▼ + ┌──────────────┐ + │ 模块类型? │ + └──┬────┬────┬─┘ + │ │ │ + │ │ └──────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌────────┐ ┌──────────┐ ┌─────────┐ + │REST API│ │计算逻辑 │ │配置类 │ + │Service │ │RPC通信 │ │ │ + │Entity │ │复杂业务 │ │ │ + └────┬───┘ └────┬─────┘ └────┬────┘ + │ │ │ + ▼ ▼ ▼ + Java Scala Scala +``` + +**选择规则:** +- **Java**: REST API、Service层、Entity类、DAO接口 +- **Scala**: 计算逻辑、RPC通信、复杂业务处理、配置对象(Configuration) + +--- + +## ⚙️ 决策3:新功能如何设计开关? + +``` +┌─────────────────────┐ +│ 新增功能需求 │ +└──────────┬──────────┘ + │ + ▼ + ┌──────────────────┐ + │ 1. 定义功能开关 │ + │ (默认 false) │ + └──────┬───────────┘ + │ + ▼ + ┌──────────────────┐ + │ 2. 实现新功能逻辑 │ + └──────┬───────────┘ + │ + ▼ + ┌──────────────────┐ + │ 3. 保留旧逻辑 │ + │ (作为降级方案) │ + └──────┬───────────┘ + │ + ▼ + ┌──────────────────────┐ + │ 4. 在代码中检查开关 │ + │ if (ENABLE.getValue())│ + │ 新逻辑 │ + │ else │ + │ 旧逻辑(降级) │ + └──────────────────────┘ +``` + +**配置示例:** +```scala +// 在 xxxConfiguration.scala 中 +object NewFeatureConfiguration { + val ENABLE = CommonVars("linkis.new.feature.enable", false) + val BATCH_SIZE = CommonVars("linkis.new.feature.batch.size", 1000) + val TIMEOUT = CommonVars("linkis.new.feature.timeout", 30000L) +} +``` + +**代码示例:** +```java +public void executeFeature() { + // 检查功能开关 + if (!NewFeatureConfiguration.ENABLE.getValue()) { + executeLegacyLogic(); // 开关关闭时执行旧逻辑 + return; + } + + try { + executeNewFeature(); // 开关打开时执行新逻辑 + } catch (Exception e) { + logger.error("New feature failed, falling back", e); + executeLegacyLogic(); // 异常时降级到旧逻辑 + } +} +``` + +--- + +## 📊 决策4:是否需要创建新表? + +``` +┌─────────────────────┐ +│ 数据存储需求 │ +└──────────┬──────────┘ + │ + ▼ + ┌─────────────────┐ + │ 能否利用现有表? │ + └─┬──────────┬───┘ + │ │ + 是│ │否 + │ │ + ▼ ▼ + ┌─────────┐ ┌──────────────┐ + │ 新增字段 │ │ 是否核心业务表?│ + └─────────┘ └─┬──────────┬─┘ + │ │ + 是│ │否 + │ │ + ▼ ▼ + 需要架构评审 可创建新表 + │ │ + └────┬─────┘ + │ + ▼ + ┌──────────────┐ + │ 记录DDL到规定 │ + │ 文件并标注说明 │ + └──────────────┘ +``` + +**创建新表规则:** +1. **优先复用**: 检查是否能通过现有表扩展实现 +2. **业务表评审**: 核心业务表需要架构评审 +3. **辅助表允许**: 日志表、临时表、配置表等可自行创建 +4. **必须记录**: DDL添加到 `linkis_ddl.sql` +5. **命名规范**: `linkis_[模块]_[功能]_[表名]` + +--- + +## 🔍 决策5:错误处理策略 + +``` +┌─────────────────────┐ +│ 发生异常 │ +└──────────┬──────────┘ + │ + ▼ + ┌──────────────────┐ + │ 异常类型判断 │ + └─┬──────┬────┬───┘ + │ │ │ + │ │ └────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌─────┐ ┌────────┐ ┌─────────┐ + │预期 │ │系统错误│ │未知错误 │ + │业务 │ │(IO/DB) │ │ │ + │异常 │ │ │ │ │ + └─┬───┘ └───┬────┘ └────┬────┘ + │ │ │ + │ │ │ + ▼ ▼ ▼ + 记录WARN 记录ERROR 记录ERROR + 返回友好 抛出包装后 抛出包装后 + 错误信息 业务异常 业务异常 + │ │ │ + └────┬────┴────────────────┘ + │ + ▼ + ┌─────────────────┐ + │ 使用LinkisException│ + │ 及其子类 │ + └─────────────────┘ +``` + +**异常处理原则:** +1. **不吞掉异常**: 必须记录日志或重新抛出 +2. **使用业务异常**: LinkisException及其子类 +3. **提供上下文**: 异常信息包含关键业务参数 +4. **分级处理**: WARN用于业务异常,ERROR用于系统异常 + +--- + +# 快速检查清单 + +开发完成后,请对照以下清单进行自检: + +## ✅ 代码规范检查 +- [ ] 所有配置使用 `CommonVars`,不硬编码 +- [ ] 字符编码使用 `StandardCharsets.UTF_8` +- [ ] 日志使用 `Logger`,不使用 `System.out` +- [ ] 异常处理使用 `LinkisException` 及其子类 +- [ ] REST接口返回 `Message` 统一体 + +## ✅ 功能设计检查 +- [ ] 新功能已添加开关配置(默认 `false`) +- [ ] 开关关闭时能降级到旧逻辑 +- [ ] 遵循最小改动原则 +- [ ] 代码有充分的日志记录 + +## ✅ 数据库变更检查 +- [ ] DDL变更已记录到 `linkis_ddl.sql` +- [ ] DML变更已记录到 `linkis_dml.sql` +- [ ] 变更脚本包含版本号、需求描述、日期 +- [ ] 优先使用新增字段而非修改字段 + +## ✅ 文档检查 +- [ ] 已创建需求文档 +- [ ] 已创建设计文档 +- [ ] 文档存放在正确的目录 +- [ ] API变更已更新对应模块文档 + +## ✅ 测试检查 +- [ ] 功能开关打开时,新功能正常工作 +- [ ] 功能开关关闭时,回退到旧逻辑 +- [ ] 异常情况能正确降级 +- [ ] 关键业务逻辑有单元测试 + +--- + +# 常见问题解答 + +## Q1: 如果现有表确实需要修改字段类型怎么办? +**A:** +1. 先评估是否可以通过新增字段实现 +2. 如果必须修改,需要: + - 提供充分的理由和影响分析 + - 记录详细的DDL和数据迁移方案 + - 标注清楚版本和需求信息 + +## Q2: 功能开关关闭后,旧代码能删除吗? +**A:** +- **至少保留一个大版本周期**(如1.17.0的新功能,至少保留到1.18.0) +- 确认新功能稳定运行至少3个月 +- 在删除前添加 TODO 注释说明删除计划 + +## Q3: 如何判断是否属于"最小改动"? +**A:** +- ✅ 只修改必要的文件和代码行 +- ✅ 不改变现有接口签名 +- ✅ 不影响其他模块的功能 +- ❌ 大规模重构现有代码 +- ❌ 修改核心公共类 + +## Q4: 配置项命名有什么规范? +**A:** +- 格式: `linkis.[模块].[功能].[属性]` +- 示例: `linkis.entrance.task.max.retry.times` +- 保持命名清晰、有意义 +- 避免使用缩写(除非是通用缩写如 max、min) + +--- + +**记住**: 规范不是束缚,而是为了保证系统的稳定性和可维护性! diff --git a/.gitignore b/.gitignore index c12a82706a9..76aeae1caa9 100644 --- a/.gitignore +++ b/.gitignore @@ -37,3 +37,8 @@ nohup.out #claude .claude + +nul + +#claude +.claude diff --git a/docs/1.17.0/design/aisql-starrocks-engine-switch-design.md b/docs/1.17.0/design/aisql-starrocks-engine-switch-design.md new file mode 100644 index 00000000000..b673b44054b --- /dev/null +++ b/docs/1.17.0/design/aisql-starrocks-engine-switch-design.md @@ -0,0 +1,926 @@ +# AISQL任务支持StarRocks引擎类型切换设计文档 + +## 1. 设计概述 + +### 1.1 目标 +为AISQL类型任务增加StarRocks引擎类型切换支持,通过runtime参数或脚本注释两种方式实现引擎切换,并集成Doctoris服务进行引擎决策。 + +### 1.2 设计原则 +- **最小改动原则**:在现有架构基础上扩展,不修改现有Spark/Hive引擎切换逻辑 +- **可配置原则**:通过功能开关控制,开关关闭时相当于回退到上一版本 +- **一致性原则**:与现有引擎切换机制保持一致的使用体验 + +### 1.3 适用范围 +- AISQL类型任务提交流程 +- 引擎类型切换逻辑 +- Doctoris服务调用 + +## 2. 整体架构 + +### 2.1 系统架构图 +``` +用户提交AISQL任务 + ↓ +Entrance服务接收 + ↓ +AISQLTransformInterceptor拦截器 + ↓ +1. 解析runtime参数 (ec.engine.type) +2. 解析脚本注释 (@set ec.engine.type=starrocks) +3. 解析模板配置 (ec.resource.name) + ↓ +判断是否指定StarRocks引擎? + ↓ Yes +调用Doctoris服务(传递forceEngineType=starrocks) + ↓ +切换EngineTypeLabel为jdbc + ↓ +任务提交到JDBC EngineConn + ↓ +通过StarRocks数据源执行任务 +``` + +### 2.2 处理优先级 +``` +1. Runtime参数 (ec.engine.type=starrocks) + ↓ 若未设置 +2. 脚本注释 (@set ec.engine.type=starrocks) + ↓ 若未设置 +3. 模板配置 (ec.resource.name包含starrocks关键字) + ↓ 若未设置 +4. Doctoris智能选择(现有逻辑) + ↓ 若未启用 +5. 默认Spark引擎(现有逻辑) +``` + +## 3. 详细设计 + +### 3.1 配置设计 + +#### 3.1.1 新增配置项(EntranceConfiguration.scala) + +```scala +// StarRocks引擎切换功能开关 +val AISQL_STARROCKS_SWITCH = CommonVars("linkis.aisql.starrocks.switch", false) + +// 默认StarRocks引擎类型 +val AISQL_DEFAULT_STARROCKS_ENGINE_TYPE = + CommonVars("linkis.aisql.default.starrocks.engine.type", "jdbc-4") + +// StarRocks模板关键字配置 +val AISQL_STARROCKS_TEMPLATE_KEYS = + CommonVars("linkis.aisql.starrocks.template.keys", "starrocks") + +// StarRocks数据源名称前缀配置 +val AISQL_STARROCKS_DATASOURCE_PREFIX = + CommonVars("linkis.aisql.starrocks.datasource.prefix", "starrocks_") + +// 用户白名单配置 +val AISQL_STARROCKS_WHITELIST_USERS = + CommonVars("linkis.aisql.starrocks.whitelist.users", "") + +// 部门白名单配置 +val AISQL_STARROCKS_WHITELIST_DEPARTMENTS = + CommonVars("linkis.aisql.starrocks.whitelist.departments", "") +``` + +#### 3.1.2 配置说明 + +| 配置项 | 默认值 | 说明 | +|--------|--------|------| +| linkis.aisql.starrocks.switch | false | StarRocks引擎切换功能开关 | +| linkis.aisql.default.starrocks.engine.type | jdbc-4 | 默认StarRocks引擎类型(jdbc引擎版本) | +| linkis.aisql.starrocks.template.keys | starrocks | 模板关键字,用于识别StarRocks模板 | +| linkis.aisql.starrocks.datasource.prefix | starrocks_ | StarRocks数据源名称前缀 | +| linkis.aisql.starrocks.whitelist.users | 空 | 用户白名单(逗号分隔),为空时所有用户可用 | +| linkis.aisql.starrocks.whitelist.departments | 空 | 部门白名单(逗号分隔),为空时所有部门可用 | + +### 3.2 脚本注释解析设计 + +#### 3.2.1 新增配置键(TemplateConfUtils.scala) + +```scala +object TemplateConfUtils { + // 现有配置 + val confTemplateNameKey = "ec.resource.name" + val confFixedEngineConnLabelKey = "ec.fixed.sessionId" + + // 新增:引擎类型配置键 + val confEngineTypeKey = "ec.engine.type" +} +``` + +#### 3.2.2 注释格式支持 + +支持三种注释格式: +- **SQL/HQL格式**:`---@set ec.engine.type=starrocks` +- **Python/Shell格式**:`##@set ec.engine.type=starrocks` +- **Scala格式**:`///@set ec.engine.type=starrocks` + +#### 3.2.3 实现逻辑 + +利用现有的`getCustomTemplateConfName`方法机制,扩展支持解析`ec.engine.type`配置: + +```scala +def getCustomEngineType(code: String, languageType: String): String = { + val confPattern = languageType.toLowerCase match { + case x if x.contains("python") || x.contains("shell") => + s"##@set\\s+${confEngineTypeKey}\\s*=\\s*([^\\s#]+)".r + case x if x.contains("scala") => + s"///@set\\s+${confEngineTypeKey}\\s*=\\s*([^\\s/]+)".r + case _ => + s"---@set\\s+${confEngineTypeKey}\\s*=\\s*([^\\s-]+)".r + } + + confPattern.findFirstMatchIn(code) match { + case Some(m) => m.group(1).trim + case None => null + } +} +``` + +### 3.3 引擎切换逻辑设计 + +#### 3.3.1 白名单检查设计 + +在进行引擎切换之前,需要先检查用户是否有权限使用StarRocks引擎: + +```scala +/** + * 检查用户是否在StarRocks白名单中 + * @param submitUser 提交任务的用户 + * @return true表示用户在白名单中或白名单为空(允许所有用户),false表示不在白名单中 + */ +private def isUserInStarRocksWhitelist(submitUser: String): Boolean = { + val whitelistUsers = AISQL_STARROCKS_WHITELIST_USERS.getValue + val whitelistDepartments = AISQL_STARROCKS_WHITELIST_DEPARTMENTS.getValue + + // 如果白名单都为空,则允许所有用户使用 + if (StringUtils.isBlank(whitelistUsers) && StringUtils.isBlank(whitelistDepartments)) { + return true + } + + // 检查用户白名单 + if (StringUtils.isNotBlank(whitelistUsers)) { + val users = whitelistUsers.split(",").map(_.trim) + if (users.contains(submitUser)) { + logger.info(s"User $submitUser is in StarRocks whitelist (user)") + return true + } + } + + // 检查部门白名单 + if (StringUtils.isNotBlank(whitelistDepartments)) { + val userDepartmentId = EntranceUtils.getUserDepartmentId(submitUser) + if (StringUtils.isNotBlank(userDepartmentId)) { + val departments = whitelistDepartments.split(",").map(_.trim) + if (departments.contains(userDepartmentId)) { + logger.info(s"User $submitUser (department: $userDepartmentId) is in StarRocks whitelist (department)") + return true + } + } + } + + logger.warn(s"User $submitUser is not in StarRocks whitelist, will use default engine selection") + false +} +``` + +#### 3.3.2 AISQLTransformInterceptor改造 + +在`AISQLTransformInterceptor.apply()`方法中增加StarRocks引擎处理逻辑: + +```scala +override def apply(task: EntranceJob, logAppender: lang.StringBuilder): EntranceJob = { + // 功能开关检查 + if (!AISQL_STARROCKS_SWITCH.getValue) { + return applyExistingLogic(task, logAppender) // 现有逻辑 + } + + val jobRequest = task.getJobRequest + val params = jobRequest.getParams + val labels = jobRequest.getLabels + + // 1. 检查runtime参数 + val runtimeEngineType = getRuntimeEngineType(params) + + // 2. 检查脚本注释 + val scriptEngineType = if (runtimeEngineType == null) { + TemplateConfUtils.getCustomEngineType( + jobRequest.getExecutionCode, + CodeAndRunTypeUtils.getLanguageTypeByRunType(jobRequest.getRunType) + ) + } else null + + // 3. 检查模板配置 + val templateEngineType = if (runtimeEngineType == null && scriptEngineType == null) { + getEngineTypeFromTemplate(jobRequest) + } else null + + // 确定最终引擎类型 + val targetEngineType = Option(runtimeEngineType) + .orElse(Option(scriptEngineType)) + .orElse(Option(templateEngineType)) + .orNull + + // 如果指定了starrocks引擎 + if ("starrocks".equalsIgnoreCase(targetEngineType)) { + // 白名单检查 + if (!isUserInStarRocksWhitelist(jobRequest.getSubmitUser)) { + logAppender.append( + LogUtils.generateWarn( + s"User ${jobRequest.getSubmitUser} is not in StarRocks whitelist, using default engine selection\n" + ) + ) + // 继续执行现有逻辑(Spark/Hive切换) + return applyExistingLogic(task, logAppender) + } + + // 切换到JDBC引擎 + changeToStarRocksEngine(labels, logAppender, params) + } else { + // 执行现有逻辑(Spark/Hive切换) + applyExistingLogic(task, logAppender) + } + + task +} +``` + +#### 3.3.2 StarRocks引擎切换实现 + +```scala +private def changeToStarRocksEngine( + labels: util.List[Label[_]], + logAppender: lang.StringBuilder, + params: util.Map[String, AnyRef] +): Unit = { + + logAppender.append("Switching to StarRocks engine...\n") + + // 1. 移除现有EngineTypeLabel + val iterator = labels.iterator() + while (iterator.hasNext) { + val label = iterator.next() + if (label.isInstanceOf[EngineTypeLabel]) { + iterator.remove() + } + } + + // 2. 创建JDBC引擎Label + val jdbcEngineType = AISQL_DEFAULT_STARROCKS_ENGINE_TYPE.getValue + val Array(engine, version) = jdbcEngineType.split("-", 2) + val jdbcLabel = new EngineTypeLabel() + jdbcLabel.setEngineType(engine) + jdbcLabel.setVersion(version) + labels.add(jdbcLabel) + + // 3. 添加StarRocks标识到runtime参数(用于后续JDBC引擎识别) + val runtimeMap = params.getOrDefault( + JobRequestConstants.JOB_REQUEST_RUNTIME_PARAMS, + new util.HashMap[String, AnyRef]() + ).asInstanceOf[util.Map[String, AnyRef]] + + runtimeMap.put("linkis.jdbc.engine.type", "starrocks") + params.put(JobRequestConstants.JOB_REQUEST_RUNTIME_PARAMS, runtimeMap) + + logAppender.append(s"Engine switched to StarRocks (JDBC engine: $jdbcEngineType)\n") +} +``` + +#### 3.3.3 Runtime参数获取 + +```scala +private def getRuntimeEngineType(params: util.Map[String, AnyRef]): String = { + if (params == null) return null + + val runtimeParams = params.get(JobRequestConstants.JOB_REQUEST_RUNTIME_PARAMS) + if (runtimeParams == null) return null + + runtimeParams.asInstanceOf[util.Map[String, AnyRef]] + .get(TemplateConfUtils.confEngineTypeKey) match { + case null => null + case value => value.toString + } +} +``` + +#### 3.3.4 模板配置获取 + +```scala +private def getEngineTypeFromTemplate(jobRequest: JobRequest): String = { + val templateName = TemplateConfUtils.getCustomTemplateConfName( + jobRequest.getExecutionCode, + CodeAndRunTypeUtils.getLanguageTypeByRunType(jobRequest.getRunType) + ) + + if (templateName == null) return null + + // 检查模板名称是否包含StarRocks关键字 + val starrocksKeys = AISQL_STARROCKS_TEMPLATE_KEYS.getValue.split(",") + if (starrocksKeys.exists(key => templateName.toLowerCase.contains(key.toLowerCase))) { + "starrocks" + } else { + null + } +} +``` + +### 3.4 Doctoris服务集成设计 + +#### 3.4.1 接口扩展(EntranceUtils.scala) + +修改`getDynamicEngineType`方法,支持传递强制引擎类型参数: + +```scala +def getDynamicEngineType( + sql: String, + logAppender: lang.StringBuilder, + forceEngineType: String = null // 新增:强制引擎类型参数 +): String = { + + if (!EntranceConfiguration.AI_SQL_DYNAMIC_ENGINE_SWITCH) { + if (forceEngineType != null) return forceEngineType + return defaultEngineType + } + + val params = new util.HashMap[String, AnyRef]() + params.put("sql", sql) + params.put("highStability", "") + params.put("queueResourceUsage", "") + + // 新增:添加强制引擎类型标识 + if (forceEngineType != null && forceEngineType.nonEmpty) { + params.put("forceEngineType", forceEngineType) + logAppender.append(s"Force engine type: $forceEngineType\n") + } + + val request = DoctorEngineRequest( + EntranceConfiguration.LINKIS_SYSTEM_NAME, + EntranceConfiguration.DOCTOR_CLUSTER, + sql, + params + ) + + val response = callDoctorService(request, logAppender) + response.result +} +``` + +#### 3.4.2 调用时机 + +在`AISQLTransformInterceptor`中,当检测到需要使用StarRocks引擎时: + +```scala +if ("starrocks".equalsIgnoreCase(targetEngineType)) { + // 调用Doctoris服务,传递强制引擎类型 + val confirmedEngineType = EntranceUtils.getDynamicEngineType( + jobRequest.getExecutionCode, + logAppender, + forceEngineType = "starrocks" // 传递强制参数 + ) + + // 切换到JDBC引擎 + changeToStarRocksEngine(labels, logAppender, params) +} +``` + +### 3.5 数据流设计 + +#### 3.5.1 任务提交数据流 + +``` +1. 用户提交任务 + { + "executionCode": "---@set ec.engine.type=starrocks\nSELECT * FROM table", + "runType": "aisql", + "params": { + "runtime": {} + } + } + +2. AISQLTransformInterceptor处理 + - 解析脚本注释,提取 ec.engine.type=starrocks + - 检查功能开关:linkis.aisql.starrocks.switch = true + - 决定切换到StarRocks引擎 + +3. 调用Doctoris服务 + POST /api/v1/external/engine/diagnose + Body: { + "sql": "SELECT * FROM table", + "forceEngineType": "starrocks" + } + Response: { + "engine": "starrocks", + "reason": "Force engine type specified" + } + +4. 修改JobRequest + - 移除现有EngineTypeLabel + - 添加新的EngineTypeLabel(engine=jdbc, version=4) + - 添加runtime参数:linkis.jdbc.engine.type=starrocks + +5. 任务路由到JDBC EngineConn + - JDBC引擎识别linkis.jdbc.engine.type=starrocks + - 查询用户的StarRocks数据源 + - 通过JDBC连接执行SQL +``` + +#### 3.5.2 引擎标签变更 + +``` +原始Label: +[EngineTypeLabel(engineType=spark, version=3.4.4)] + +↓ 检测到 ec.engine.type=starrocks + +新Label: +[EngineTypeLabel(engineType=jdbc, version=4)] + ++ Runtime参数: +{ + "linkis.jdbc.engine.type": "starrocks" +} +``` + +## 4. 接口设计 + +### 4.1 内部接口 + +#### 4.1.1 TemplateConfUtils新增方法 + +```scala +/** + * 从脚本代码中提取引擎类型配置 + * @param code 脚本代码 + * @param languageType 语言类型 + * @return 引擎类型,如"starrocks"、"spark"、"hive",未找到返回null + */ +def getCustomEngineType(code: String, languageType: String): String +``` + +#### 4.1.2 AISQLTransformInterceptor新增私有方法 + +```scala +/** + * 从runtime参数中获取引擎类型 + */ +private def getRuntimeEngineType(params: util.Map[String, AnyRef]): String + +/** + * 从模板配置中获取引擎类型 + */ +private def getEngineTypeFromTemplate(jobRequest: JobRequest): String + +/** + * 切换到StarRocks引擎 + */ +private def changeToStarRocksEngine( + labels: util.List[Label[_]], + logAppender: lang.StringBuilder, + params: util.Map[String, AnyRef] +): Unit +``` + +#### 4.1.3 EntranceUtils方法签名变更 + +```scala +/** + * 获取动态引擎类型 + * @param sql SQL语句 + * @param logAppender 日志追加器 + * @param forceEngineType 强制引擎类型(可选),如"starrocks" + * @return 引擎类型 + */ +def getDynamicEngineType( + sql: String, + logAppender: lang.StringBuilder, + forceEngineType: String = null +): String +``` + +### 4.2 外部接口 + +#### 4.2.1 任务提交接口(无变更) + +保持现有任务提交接口不变,通过扩展参数支持新功能: + +``` +POST /api/rest_j/v1/entrance/submit + +Request Body: +{ + "executionCode": "SELECT * FROM table", + "runType": "aisql", + "params": { + "runtime": { + "ec.engine.type": "starrocks" // 新增参数 + } + } +} +``` + +#### 4.2.2 Doctoris服务接口 + +``` +POST {DOCTOR_URL}/api/v1/external/engine/diagnose + +Request: +{ + "appId": "linkis", + "cluster": "default", + "sql": "SELECT * FROM table", + "params": { + "forceEngineType": "starrocks" // 新增参数 + } +} + +Response: +{ + "code": 0, + "data": { + "engine": "starrocks", + "reason": "Force engine type specified", + "duration": 50 + } +} +``` + +## 5. 异常处理 + +### 5.1 异常场景 + +| 异常场景 | 处理策略 | +|----------|----------| +| StarRocks功能开关关闭 | 忽略StarRocks配置,执行现有Spark/Hive切换逻辑 | +| 无效的引擎类型值 | 记录警告日志,使用默认引擎类型 | +| Doctoris服务调用失败 | 记录错误日志,降级到默认引擎类型 | +| JDBC引擎不可用 | 任务提交失败,返回明确错误信息 | +| StarRocks数据源不存在 | 任务执行失败,提示配置数据源 | + +### 5.2 日志规范 + +```scala +// INFO级别:关键流程节点 +logger.info(s"AISQL task switches to StarRocks engine for user $username") + +// WARN级别:降级处理 +logger.warn(s"Invalid engine type specified: $engineType, fallback to default") + +// ERROR级别:异常错误 +logger.error(s"Failed to switch to StarRocks engine for task $taskId", exception) + +// DEBUG级别:详细调试信息 +logger.debug(s"Parsing engine type from script: $code") +``` + +## 6. 测试设计 + +### 6.1 单元测试 + +#### 6.1.1 TemplateConfUtils测试 + +```scala +class TemplateConfUtilsTest { + + test("extract starrocks engine type from SQL comment") { + val code = "---@set ec.engine.type=starrocks\nSELECT * FROM table" + val result = TemplateConfUtils.getCustomEngineType(code, "sql") + assert(result == "starrocks") + } + + test("extract starrocks engine type from Python comment") { + val code = "##@set ec.engine.type=starrocks\nSELECT COUNT(*) FROM table" + val result = TemplateConfUtils.getCustomEngineType(code, "python") + assert(result == "starrocks") + } + + test("return null when no engine type specified") { + val code = "SELECT * FROM table" + val result = TemplateConfUtils.getCustomEngineType(code, "sql") + assert(result == null) + } +} +``` + +#### 6.1.2 AISQLTransformInterceptor测试 + +```scala +class AISQLTransformInterceptorTest { + + test("switch to StarRocks via runtime parameter") { + val jobRequest = createJobRequest( + code = "SELECT * FROM table", + runtime = Map("ec.engine.type" -> "starrocks") + ) + val task = new EntranceJob() + task.setJobRequest(jobRequest) + + interceptor.apply(task, new StringBuilder()) + + val engineLabel = getEngineLabel(task) + assert(engineLabel.getEngineType == "jdbc") + } + + test("switch to StarRocks via script comment") { + val jobRequest = createJobRequest( + code = "---@set ec.engine.type=starrocks\nSELECT * FROM table" + ) + val task = new EntranceJob() + task.setJobRequest(jobRequest) + + interceptor.apply(task, new StringBuilder()) + + val engineLabel = getEngineLabel(task) + assert(engineLabel.getEngineType == "jdbc") + } + + test("runtime parameter takes precedence over script comment") { + val jobRequest = createJobRequest( + code = "---@set ec.engine.type=spark\nSELECT * FROM table", + runtime = Map("ec.engine.type" -> "starrocks") + ) + val task = new EntranceJob() + task.setJobRequest(jobRequest) + + interceptor.apply(task, new StringBuilder()) + + val engineLabel = getEngineLabel(task) + assert(engineLabel.getEngineType == "jdbc") // 使用runtime的starrocks + } +} +``` + +### 6.2 集成测试 + +#### 6.2.1 端到端测试用例 + +```bash +# 测试1:通过runtime参数切换StarRocks引擎 +curl -X POST http://localhost:9001/api/rest_j/v1/entrance/submit \ + -H "Content-Type: application/json" \ + -H "Token-User: testuser" \ + -d '{ + "executionCode": "SELECT * FROM starrocks_table LIMIT 10", + "runType": "aisql", + "params": { + "runtime": { + "ec.engine.type": "starrocks" + } + } + }' + +# 预期结果:任务成功提交,引擎类型为jdbc,执行成功 + +# 测试2:通过脚本注释切换StarRocks引擎 +curl -X POST http://localhost:9001/api/rest_j/v1/entrance/submit \ + -H "Content-Type: application/json" \ + -H "Token-User: testuser" \ + -d '{ + "executionCode": "---@set ec.engine.type=starrocks\nSELECT COUNT(*) FROM user_table", + "runType": "aisql", + "params": {} + }' + +# 预期结果:任务成功提交,引擎类型为jdbc,执行成功 + +# 测试3:功能开关关闭 +# 配置:linkis.aisql.starrocks.switch=false +curl -X POST http://localhost:9001/api/rest_j/v1/entrance/submit \ + -H "Content-Type: application/json" \ + -H "Token-User: testuser" \ + -d '{ + "executionCode": "---@set ec.engine.type=starrocks\nSELECT * FROM table", + "runType": "aisql", + "params": {} + }' + +# 预期结果:忽略StarRocks配置,使用默认Spark引擎 +``` + +### 6.3 性能测试 + +测试指标: +- 参数解析耗时 < 10ms +- 引擎切换逻辑耗时 < 5ms +- 任务提交总耗时增加 < 20ms + +## 7. 部署方案 + +### 7.1 部署步骤 + +1. **编译打包** + ```bash + mvn clean package -Dmaven.test.skip=true + ``` + +2. **停止Entrance服务** + ```bash + sh sbin/linkis-daemon.sh stop entrance + ``` + +3. **备份原有文件** + ```bash + cp lib/linkis-spring-cloud-services/linkis-entrance.jar \ + lib/linkis-spring-cloud-services/linkis-entrance.jar.bak + ``` + +4. **替换新文件** + ```bash + cp linkis-computation-governance/linkis-entrance/target/linkis-entrance.jar \ + lib/linkis-spring-cloud-services/ + ``` + +5. **配置文件修改**(linkis-entrance.properties) + ```properties + # 启用StarRocks引擎切换功能 + linkis.aisql.starrocks.switch=true + + # StarRocks引擎类型(jdbc-4表示jdbc引擎版本4) + linkis.aisql.default.starrocks.engine.type=jdbc-4 + + # StarRocks模板关键字 + linkis.aisql.starrocks.template.keys=starrocks + ``` + +6. **启动Entrance服务** + ```bash + sh sbin/linkis-daemon.sh start entrance + ``` + +7. **验证功能** + ```bash + # 查看日志确认配置加载 + tail -f logs/linkis-entrance-gc.log | grep "starrocks" + + # 提交测试任务 + sh bin/linkis-cli -engineType aisql -code "---@set ec.engine.type=starrocks\nSELECT 1" -runtimeMap ec.engine.type=starrocks + ``` + +### 7.2 回滚方案 + +如果部署后出现问题,执行以下回滚步骤: + +1. **停止服务** + ```bash + sh sbin/linkis-daemon.sh stop entrance + ``` + +2. **恢复备份文件** + ```bash + mv lib/linkis-spring-cloud-services/linkis-entrance.jar.bak \ + lib/linkis-spring-cloud-services/linkis-entrance.jar + ``` + +3. **配置文件回滚** + ```properties + # 关闭StarRocks功能 + linkis.aisql.starrocks.switch=false + ``` + +4. **启动服务** + ```bash + sh sbin/linkis-daemon.sh start entrance + ``` + +### 7.3 灰度发布方案 + +1. **阶段1:内部测试环境**(1-2天) + - 部署到测试环境 + - 开启功能开关 + - 内部人员测试验证 + +2. **阶段2:生产环境灰度**(3-5天) + - 仅对特定用户组开启功能 + - 通过用户白名单控制 + - 监控任务成功率和性能指标 + +3. **阶段3:全量发布**(7天后) + - 确认无问题后全量开启 + - 持续监控一周 + +## 8. 监控告警 + +### 8.1 监控指标 + +| 指标 | 说明 | 告警阈值 | +|------|------|----------| +| starrocks_engine_switch_count | StarRocks引擎切换次数 | - | +| starrocks_engine_switch_success_rate | 切换成功率 | < 95% | +| starrocks_task_execution_time | 任务执行时间 | > 60s (P95) | +| starrocks_task_fail_count | 任务失败次数 | > 10次/小时 | +| doctoris_call_timeout_count | Doctoris调用超时次数 | > 5次/小时 | + +### 8.2 日志监控 + +关键日志关键字: +- `Switching to StarRocks engine` +- `Force engine type: starrocks` +- `Failed to switch to StarRocks engine` +- `Invalid engine type specified` + +## 9. 风险评估与应对 + +### 9.1 技术风险 + +| 风险 | 影响 | 概率 | 应对措施 | +|------|------|------|----------| +| StarRocks数据源配置错误 | 任务执行失败 | 中 | 提供详细的错误提示和配置文档 | +| JDBC连接池资源耗尽 | 后续任务阻塞 | 低 | 配置合理的连接池大小和超时时间 | +| Doctoris服务不稳定 | 引擎选择失败 | 中 | 实现降级逻辑,服务异常时使用默认配置 | +| 配置解析性能问题 | 任务提交变慢 | 低 | 优化正则表达式,添加缓存机制 | + +### 9.2 业务风险 + +| 风险 | 影响 | 概率 | 应对措施 | +|------|------|------|----------| +| 用户误配置引擎类型 | 任务失败或结果错误 | 中 | 添加引擎类型有效性校验 | +| 现有任务受影响 | 兼容性问题 | 低 | 功能开关默认关闭,逐步开启 | +| 文档不完善 | 用户使用困难 | 中 | 编写详细使用文档和示例 | + +## 10. 兼容性说明 + +### 10.1 向后兼容 +- 功能开关默认关闭(`linkis.aisql.starrocks.switch=false`) +- 不影响现有Spark/Hive引擎切换逻辑 +- 不修改现有接口签名和返回结构 + +### 10.2 版本依赖 +- 最低支持版本:Linkis 1.17.0 +- JDBC引擎插件版本:jdbc-4 +- StarRocks数据源管理模块已部署 + +### 10.3 升级影响 +- 升级时无需修改现有任务配置 +- 升级后需手动开启功能开关 +- 需要配置StarRocks相关参数 + +## 11. 文档清单 + +### 11.1 开发文档 +- [x] 需求文档:`docs/1.17.0/requirements/aisql-starrocks-engine-switch.md` +- [x] 设计文档:`docs/1.17.0/design/aisql-starrocks-engine-switch-design.md` + +### 11.2 用户文档(待补充) +- [ ] 用户使用指南:如何配置和使用StarRocks引擎 +- [ ] 配置参数说明:所有相关配置项的详细说明 +- [ ] 常见问题FAQ:常见问题和解决方案 + +### 11.3 运维文档(待补充) +- [ ] 部署指南:详细部署步骤和验证方法 +- [ ] 监控运维手册:监控指标和告警处理 +- [ ] 故障排查手册:常见故障和排查方法 + +## 12. 变更清单 + +### 12.1 新增文件 +- `docs/1.17.0/requirements/aisql-starrocks-engine-switch.md` - 需求文档 +- `docs/1.17.0/design/aisql-starrocks-engine-switch-design.md` - 设计文档 + +### 12.2 修改文件 +- `linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/conf/EntranceConfiguration.scala` - 新增配置项 +- `linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TemplateConfUtils.scala` - 新增引擎类型解析方法 +- `linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/AISQLTransformInterceptor.scala` - 新增StarRocks切换逻辑 +- `linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/EntranceUtils.scala` - 扩展Doctoris调用接口 + +### 12.3 数据库变更 +无数据库变更 + +### 12.4 配置文件变更 +- `conf/linkis-entrance.properties` - 新增StarRocks相关配置项 + +## 13. 质量检查清单 + +- [ ] 代码符合项目规范(Java/Scala编码规范) +- [ ] 异常处理完整(try-catch、日志记录) +- [ ] 日志记录充分(INFO/WARN/ERROR/DEBUG) +- [ ] 单元测试覆盖(核心逻辑测试覆盖率>80%) +- [ ] 配置开关完整(功能开关、默认值配置) +- [ ] 向后兼容性检查(不影响现有功能) +- [ ] 性能测试通过(满足性能要求) +- [ ] 安全性检查(权限验证、参数校验) +- [ ] 文档完整性(需求、设计、用户、运维文档) + +## 14. 附录 + +### 14.1 相关代码文件路径 + +| 文件 | 路径 | +|------|------| +| EntranceConfiguration | linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/conf/EntranceConfiguration.scala | +| TemplateConfUtils | linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TemplateConfUtils.scala | +| AISQLTransformInterceptor | linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/AISQLTransformInterceptor.scala | +| EntranceUtils | linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/EntranceUtils.scala | +| JDBCConfiguration | linkis-engineconn-plugins/jdbc/src/main/scala/org/apache/linkis/manager/engineplugin/jdbc/conf/JDBCConfiguration.scala | + +### 14.2 参考资料 +- Apache Linkis官方文档:https://linkis.apache.org +- StarRocks官方文档:https://docs.starrocks.io +- JDBC标准文档:https://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/ + +--- + +**文档版本**:v1.0 +**创建日期**:2025-10-27 +**作者**:AI +**审核状态**:待审核 diff --git a/docs/1.17.0/requirements/aisql-starrocks-engine-switch.md b/docs/1.17.0/requirements/aisql-starrocks-engine-switch.md new file mode 100644 index 00000000000..878c5aaf369 --- /dev/null +++ b/docs/1.17.0/requirements/aisql-starrocks-engine-switch.md @@ -0,0 +1,179 @@ +# AISQL任务支持StarRocks引擎类型切换需求 + +## 需求概述 + +为AISQL类型任务增加StarRocks引擎类型切换支持,允许用户通过多种方式指定任务使用StarRocks引擎执行。 + +## 需求背景 + +### 当前现状 +1. AISQL任务目前支持Spark和Hive两种引擎类型的切换 +2. 引擎切换通过以下两种方式实现: + - 通过模板配置(ec.resource.name)中的关键字匹配自动选择引擎 + - 通过调用Doctoris服务进行智能引擎选择 +3. StarRocks引擎已通过JDBC引擎实现,可通过查询用户数据源名称的方式提交任务 + +### 存在问题 +- 当前AISQL任务无法切换到StarRocks引擎执行 +- 用户无法通过配置参数或脚本注释指定使用StarRocks引擎 + +### 业务价值 +- 支持用户灵活选择StarRocks引擎执行AISQL任务 +- 与现有Spark/Hive引擎切换机制保持一致 +- 扩展AISQL任务的引擎支持能力 + +## 功能需求 + +### 功能点1:Runtime参数方式指定StarRocks引擎 + +**需求描述**: +用户可以在任务提交时,通过runtime参数指定当前AISQL任务使用StarRocks引擎执行。 + +**实现方式**: +- 新增runtime参数键:`ec.engine.type` +- 参数值:`starrocks` +- 当检测到该参数时,优先切换到StarRocks引擎 + +**示例**: +```json +{ + "executionCode": "SELECT * FROM table", + "runType": "aisql", + "params": { + "runtime": { + "ec.engine.type": "starrocks" + } + } +} +``` + +### 功能点2:脚本注释方式指定StarRocks引擎 + +**需求描述**: +用户可以在AISQL脚本中通过注释配置参数,指定当前任务使用StarRocks引擎执行。 + +**实现方式**: +- 参考TemplateConfUtils中ec.resource.name的实现方式 +- 支持在脚本注释中添加`@set ec.engine.type=starrocks`配置 +- 支持多种注释格式(SQL/Python/Scala) + +**示例**: +```sql +---@set ec.engine.type=starrocks +SELECT * FROM starrocks_table WHERE dt = '2024-01-01' +``` + +```python +##@set ec.engine.type=starrocks +SELECT COUNT(*) FROM user_table +``` + +### 功能点3:Doctoris服务集成 + +**需求描述**: +当指定使用StarRocks引擎时,需要调用Doctoris服务,传递标识参数表明当前任务必须使用StarRocks引擎。 + +**实现要求**: +- 扩展现有Doctoris服务调用接口 +- 新增参数标识:`forceEngineType` 或 `fixedEngineType` +- 参数值:`starrocks` +- Doctoris服务根据该标识强制返回StarRocks引擎 + +## 验收标准 + +### 功能验收 +1. 通过runtime参数`ec.engine.type=starrocks`可成功切换到StarRocks引擎 +2. 通过脚本注释`@set ec.engine.type=starrocks`可成功切换到StarRocks引擎 +3. Runtime参数方式的优先级高于脚本注释方式 +4. 调用Doctoris服务时正确传递StarRocks标识参数 +5. 任务执行记录中正确记录使用的引擎类型为JDBC +6. 白名单功能正常工作: + - 白名单为空时,所有用户可以使用StarRocks引擎 + - 白名单配置用户后,只有白名单用户可以使用 + - 白名单配置部门后,只有白名单部门的用户可以使用 + - 不在白名单的用户指定StarRocks引擎时,系统忽略该配置并使用默认引擎 + +### 性能要求 +- 引擎切换逻辑不影响现有任务提交性能 +- 参数解析耗时不超过10ms + +### 兼容性要求 +- 不影响现有Spark和Hive引擎的切换功能 +- 功能开关关闭时,行为与上一版本保持一致 +- 向后兼容,不修改现有接口签名 + +### 安全要求 +- 验证用户是否有对应StarRocks数据源的访问权限 +- 引擎切换不绕过现有权限校验机制 + +## 功能点4:用户和部门白名单控制 + +**需求描述**: +为了更安全地控制StarRocks引擎切换功能的使用范围,需要增加用户和部门白名单机制。只有配置在白名单中的用户或部门才能使用StarRocks引擎切换功能。 + +**实现方式**: +- 新增配置项:`linkis.aisql.starrocks.whitelist.users`,配置允许使用的用户列表(逗号分隔) +- 新增配置项:`linkis.aisql.starrocks.whitelist.departments`,配置允许使用的部门ID列表(逗号分隔) +- 白名单检查逻辑: + - 如果白名单配置为空,则所有用户都可以使用(兼容现有行为) + - 如果白名单配置不为空,则只有白名单中的用户或部门才能使用 + - 用户检查:检查提交任务的用户是否在用户白名单中 + - 部门检查:获取提交任务用户的部门ID,检查是否在部门白名单中 + - 满足任一条件即可使用StarRocks引擎 + +**示例配置**: +```properties +# 允许使用StarRocks引擎的用户(逗号分隔) +linkis.aisql.starrocks.whitelist.users=user1,user2,admin + +# 允许使用StarRocks引擎的部门ID(逗号分隔) +linkis.aisql.starrocks.whitelist.departments=dept001,dept002 + +# 如果两个配置都为空,则所有用户都可以使用 +``` + +**行为说明**: +- 当用户不在白名单中时,即使指定了`ec.engine.type=starrocks`,系统也会忽略该配置,继续使用默认的Spark/Hive引擎选择逻辑 +- 日志中会记录白名单检查结果,方便问题排查 + +## 配置开关 + +新增配置项: +- `linkis.aisql.starrocks.switch`:StarRocks引擎切换功能开关,默认值:`false` +- `linkis.aisql.default.starrocks.engine.type`:默认StarRocks引擎类型,默认值:`jdbc-4` +- `linkis.aisql.starrocks.template.keys`:StarRocks模板关键字,默认值:`starrocks` +- `linkis.aisql.starrocks.whitelist.users`:用户白名单(逗号分隔),默认值:空(所有用户可用) +- `linkis.aisql.starrocks.whitelist.departments`:部门白名单(逗号分隔),默认值:空(所有部门可用) + +## 依赖关系 + +### 前置依赖 +- JDBC引擎插件已支持StarRocks +- StarRocks数据源管理功能已实现 + +### 影响模块 +- linkis-entrance:AISQL任务拦截器 +- linkis-entrance-conf:配置管理 +- linkis-computation-governance-common:任务协议 + +## 风险评估 + +### 技术风险 +- **风险**:StarRocks引擎可能不支持某些AISQL语法 +- **应对**:在文档中明确说明语法限制,引擎切换失败时给出明确提示 + +### 兼容性风险 +- **风险**:新增配置可能与现有配置冲突 +- **应对**:使用独立的配置键名,遵循现有配置命名规范 + +## 参考资料 + +- TemplateConfUtils实现:`linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TemplateConfUtils.scala` +- AISQLTransformInterceptor实现:`linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/AISQLTransformInterceptor.scala` +- JDBCConfiguration配置:`linkis-engineconn-plugins/jdbc/src/main/scala/org/apache/linkis/manager/engineplugin/jdbc/conf/JDBCConfiguration.scala` + +## 更新记录 + +| 版本 | 日期 | 作者 | 变更说明 | +|------|------|------|----------| +| v1.0 | 2025-10-27 | AI | 初始版本 | diff --git a/docs/ai-prompt/base-rule.md b/docs/ai-prompt/base-rule.md deleted file mode 100644 index a32b7105b57..00000000000 --- a/docs/ai-prompt/base-rule.md +++ /dev/null @@ -1,378 +0,0 @@ -# Apache Linkis AI IDE 开发规约 - -## 角色定位 -你是Apache Linkis项目的资深后端开发专家,熟练掌握: -- **核心技术栈**:Spring Boot 2.7 + Spring Cloud 2021.0.8 + MyBatis-Plus 3.5.7 -- **编程语言**:Java 8 + Scala 2.12(混合开发模式) -- **数据库**:MySQL 8.0 + Hive(通过JDBC) -- **微服务架构**:Eureka服务发现 + Gateway网关 + Feign远程调用 -- **大数据引擎**:Spark、Hive、Flink、Python、Shell等多引擎支持 - ---- - -# 项目核心信息 - -## 基础配置 -- **项目根目录**:linkis -- **基础包名**:org.apache.linkis -- **版本信息**:Apache Linkis 1.x -- **构建工具**:Maven 3.5+ -- **JDK版本**:1.8 -- **字符编码**:统一使用StandardCharsets.UTF_8 - -## 关键组件 -- **统一返回体**:`org.apache.linkis.server.Message` -- **统一异常**:`org.apache.linkis.common.exception.LinkisException` -- **配置管理**:`org.apache.linkis.common.conf.CommonVars` -- **数据库脚本**: - - DDL:`linkis-dist/package/db/linkis_ddl.sql` - - DML:`linkis-dist/package/db/linkis_dml.sql` - ---- - -# 系统架构设计 - -## 三层架构模式 -Linkis采用微服务架构,按功能职责划分为三大服务类别: - -### 1. 微服务治理服务(基础设施层) -负责微服务的基础设施支撑,包括服务发现、网关路由、配置管理等。 -- Spring Cloud Gateway:API网关服务 -- Eureka:服务注册与发现中心 -- Open Feign:声明式HTTP客户端 - -### 2. 计算治理服务(核心业务层) -负责计算任务的生命周期管理,从任务提交到执行完成的全流程控制。 -- Entrance:任务提交入口服务 -- JobHistory:任务历史记录服务 -- LinkisManager:资源管理服务 -- EngineConnManager:引擎连接管理服务 -- EngineConn:引擎连接器 - -### 3. 公共增强服务(支撑服务层) -提供跨服务的公共能力,如文件管理、数据源管理、配置管理等。 -- PublicService:公共服务 -- BML:大数据物料库 -- DataSource:数据源管理 -- Configuration:配置管理 -- ContextServer:上下文服务 -- Monitor:监控服务 - -## 服务交互模式 -``` -上层应用 -> Gateway -> Entrance -> Manager -> ECM -> EngineConn -> 底层引擎 - ↓ - 公共增强服务(BML、DataSource、Configuration等) -``` - -## 各服务模块说明 -### 微服务治理服务 -Spring Cloud Gateway -功能:API网关服务,负责请求路由转发、负载均衡、安全认证等 -主类入口:org.apache.linkis.gateway.springcloud.LinkisGatewayApplication -模块路径:linkis-spring-cloud-services/linkis-service-gateway/linkis-spring-cloud-gateway - -Eureka -功能:服务注册与发现中心,管理微服务实例的注册、发现和健康检查 -主类入口:org.apache.linkis.eureka.SpringCloudEurekaApplication -模块路径:linkis-spring-cloud-services/linkis-service-discovery/linkis-eureka - -Open Feign -功能:声明式HTTP客户端,简化微服务间的远程调用 -主类入口:集成在各个微服务模块中,无独立启动类 -模块路径:集成在linkis-commons/linkis-rpc等公共模块中 - -### 计算治理服务 -Entrance -功能:任务提交入口服务,负责任务调度、状态管控、任务信息推送等核心功能 -主类入口:org.apache.linkis.entrance.LinkisEntranceApplication -模块路径:linkis-computation-governance/linkis-entrance - -JobHistory -功能:任务历史记录服务,提供任务执行历史的查询、统计和管理功能 -主类入口:org.apache.linkis.jobhistory.LinkisJobHistoryApp -模块路径:linkis-public-enhancements/linkis-jobhistory - -LinkisManager -功能:计算治理层的管理服务,包含AppManager、ResourceManager、LabelManager等管理控制服务 -主类入口:org.apache.linkis.manager.LinkisManagerApplication -模块路径:linkis-computation-governance/linkis-manager/linkis-application-manager - -EngineConnManager -功能:引擎连接器管理服务,负责控制EngineConn的生命周期(启动、停止) -主类入口:org.apache.linkis.ecm.server.LinkisECMApplication -模块路径:linkis-computation-governance/linkis-engineconn-manager/linkis-engineconn-manager-server - -EngineConn -功能:引擎连接器,负责接收任务并提交到Spark、Hive、Flink等底层引擎执行 -主类入口:org.apache.linkis.engineconn.LinkisEngineConnApplication -模块路径:linkis-computation-governance/linkis-engineconn - -### 公共增强服务 -PublicService -功能:公共服务模块,提供统一配置管理、微服务管理等基础服务能力 -主类入口:org.apache.linkis.filesystem.LinkisPublicServiceApp -模块路径:linkis-public-enhancements/linkis-pes-publicservice - -BML -功能:大数据物料库服务(BigData Material Library),提供文件上传、下载、版本管理等功能 -主类入口:org.apache.linkis.bml.LinkisBMLApplication -模块路径:linkis-public-enhancements/linkis-bml-server - -DataSource -功能:数据源管理服务,提供统一的数据源连接、管理和元数据服务 -主类入口:org.apache.linkis.metadata.LinkisDataSourceApplication(数据源服务) -模块路径:linkis-public-enhancements/linkis-datasource - -Configuration -功能:配置管理服务,提供系统级、用户级、引擎级等多层次的配置管理 -主类入口:org.apache.linkis.configuration.LinkisConfigurationApp -模块路径:linkis-public-enhancements/linkis-configuration - -ContextServer -功能:上下文服务,支持跨引擎的资源共享、变量传递和会话管理 -主类入口:org.apache.linkis.cs.server.LinkisCSApplication -模块路径:linkis-public-enhancements/linkis-cs-server - -Monitor -功能:监控服务,提供系统性能监控、告警和运维管理功能,包括任务监控、资源监控、用户模式监控等 -主类入口:org.apache.linkis.monitor.LinksMonitorApplication -模块路径:linkis-extensions/linkis-et-monitor - ---- - -# 开发规范与约束 - -## 代码边界约束 - -### 🚫 禁止操作 -- **数据库结构**:除非明确指定,严禁修改现有表结构 -- **第三方依赖**:不允许引入新的第三方依赖库 -- **核心接口**:不得修改现有公共接口的签名 - -### ✅ 允许操作 -- **新增功能**:在不破坏现有逻辑的前提下扩展功能 -- **新增配置**:在现有配置文件中新增配置项 -- **新增表字段**:在现有表基础上新增字段 - -## 技术规范 - -### 编程语言使用 -- **Java**:主要用于REST API、Service层、Entity类、配置类 -- **Scala**:主要用于计算逻辑、RPC通信、复杂业务处理 - -### 日志规范 -```java -// 必须使用统一的Logger -private static final Logger logger = LoggerFactory.getLogger(ClassName.class); - -// 日志级别使用: -// ERROR: 系统错误、业务异常 -// WARN: 警告信息、降级处理 -// INFO: 关键业务节点、状态变更 -// DEBUG: 详细调试信息 - -logger.info("User {} starts processing task {}", username, taskId); -logger.error("Failed to process task {} for user {}", taskId, username, e); -``` - -### 配置管理规范 -- 所有配置统一使用`org.apache.linkis.common.conf.CommonVars` -- 参考示例:`org.apache.linkis.jobhistory.conf.JobhistoryConfiguration` -- 所有新需求必须添加配置开关,默认设置false -- 配置存放位置:当前模块的conf目录,一般为xxxConfiguration类 - -### 字符编码规范 -```java -// 统一使用StandardCharsets.UTF_8,禁止使用字符串"UTF-8" -import java.nio.charset.StandardCharsets; - -String content = new String(bytes, StandardCharsets.UTF_8); -Files.write(path, content.getBytes(StandardCharsets.UTF_8)); -``` - -### API设计规范 -```java -@Api(tags = "module operation") -@RestController -@RequestMapping(path = "/api/rest_j/v1/module") -public class ModuleRestfulApi { - - @ApiOperation(value = "operation", notes = "description", response = Message.class) - @RequestMapping(path = "/operation", method = RequestMethod.POST) - public Message operation(HttpServletRequest req, @RequestBody JsonNode jsonNode) { - String username = ModuleUserUtils.getOperationUser(req, "operation"); - // 业务逻辑处理 - return Message.ok("success").data("result", data); - } -} -``` - -### 异常处理规范 -```java -// 统一使用LinkisException及其子类 -try { - // 业务逻辑 -} catch (Exception e) { - logger.error("Operation failed", e); - throw new YourModuleException("Error message", e); -} -``` - ---- - -# 开发模板与示例 - -## 新功能开发模板 - -### 1. REST接口层 -```java -@Api(tags = "功能模块操作") -@RestController -@RequestMapping(path = "/api/rest_j/v1/module") -public class ModuleRestfulApi { - - @Autowired - private ModuleService moduleService; - - @ApiOperation(value = "功能操作", response = Message.class) - @RequestMapping(path = "/action", method = RequestMethod.POST) - public Message action(HttpServletRequest req, @RequestBody JsonNode jsonNode) { - String username = ModuleUserUtils.getOperationUser(req, "action"); - - // 参数解析和验证 - String param = jsonNode.get("param").asText(); - if (StringUtils.isBlank(param)) { - return Message.error("参数不能为空"); - } - - try { - Object result = moduleService.performAction(param, username); - return Message.ok("操作成功").data("result", result); - } catch (Exception e) { - logger.error("操作失败", e); - return Message.error("操作失败:" + e.getMessage()); - } - } -} -``` - -### 2. 服务层 -```java -@Service -public class ModuleServiceImpl implements ModuleService { - - private static final Logger logger = LoggerFactory.getLogger(ModuleServiceImpl.class); - - @Autowired - private ModuleMapper moduleMapper; - - @Override - @Transactional(rollbackFor = Exception.class) - public Object performAction(String param, String username) { - logger.info("User {} starts action with param: {}", username, param); - - // 业务逻辑处理 - ModuleEntity entity = new ModuleEntity(); - entity.setParam(param); - entity.setCreateUser(username); - - moduleMapper.insert(entity); - - logger.info("User {} completed action successfully", username); - return entity.getId(); - } -} -``` - -### 3. 数据访问层 -```java -@Mapper -public interface ModuleMapper { - - @Insert("INSERT INTO linkis_module_table (param, create_user, create_time) " + - "VALUES (#{param}, #{createUser}, NOW())") - @Options(useGeneratedKeys = true, keyProperty = "id") - void insert(ModuleEntity entity); - - @Select("SELECT * FROM linkis_module_table WHERE id = #{id}") - ModuleEntity selectById(@Param("id") Long id); -} -``` - -### 4. 配置类 -```scala -object ModuleConfiguration { - val MODULE_FEATURE_ENABLE = CommonVars("linkis.module.feature.enable", false) - val MODULE_TIMEOUT = CommonVars("linkis.module.timeout", 30000L) - val MODULE_BATCH_SIZE = CommonVars("linkis.module.batch.size", 1000) -} -``` - ---- - -# 需求开发流程 - -## 需求分析模板 - -### 【背景说明】 -描述业务场景、现有问题或痛点、期望解决的目标 - -### 【验收标准】 -- 功能验收点(具体、可测量) -- 性能要求(响应时间、并发数等) -- 安全要求(权限控制、数据保护) -- 兼容性要求(向后兼容) - -## 开发交付清单 - -### 变更清单 -- 新增/修改的文件路径列表 -- 数据库变更脚本(DDL/DML) -- 配置文件变更 - -### 测试验证 -- 单元测试代码 -- 集成测试用例 -- 手动测试命令(curl等) - -### 质量检查 -- [ ] 代码符合项目规范 -- [ ] 异常处理完整 -- [ ] 日志记录充分 -- [ ] 单元测试覆盖 -- [ ] 配置开关完整 -- [ ] 向后兼容性检查 - ---- - -# AI IDE开发提示 - -## 开发技巧 -1. **优先查看现有代码**:在新增功能前,先查看相似功能的实现方式 -2. **遵循现有模式**:保持与现有代码风格一致 -3. **充分测试**:编写充分的单元测试和集成测试 -4. **考虑边界情况**:处理各种异常和边界条件 - -## 常见问题及解决方案 - -### 1. 字符编码问题 -**问题**:HTTP传输过程中出现中文乱码 -**解决**:统一使用`StandardCharsets.UTF_8` - -### 2. 配置热更新问题 -**问题**:配置修改后需要重启服务 -**解决**:使用`CommonVars`并配合`@RefreshScope`注解 - -### 3. 性能优化问题 -**问题**:大批量数据处理性能差 -**解决**:采用分页处理,单次处理不超过5000条 - ---- - -**📝 重要提示** -1. 严格遵循现有架构设计,不得随意修改核心组件 -2. 新增功能必须考虑向后兼容性 -3. 关键业务逻辑必须有完整的异常处理和日志记录 -4. 所有配置项必须有合理的默认值 -5. 代码提交前必须通过本地测试验证 \ No newline at end of file diff --git a/linkis-commons/linkis-module/pom.xml b/linkis-commons/linkis-module/pom.xml index 1e4313781ec..687cc884600 100644 --- a/linkis-commons/linkis-module/pom.xml +++ b/linkis-commons/linkis-module/pom.xml @@ -144,13 +144,13 @@ com.fasterxml.woodstox woodstox-core - ${woodstox.version} + ${woodstox.core.version} org.codehaus.jettison jettison - ${jettison.version} + ${jettision.version} @@ -360,7 +360,6 @@ spring-retry 1.3.4 - diff --git a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/async/AsyncConcurrentComputationExecutor.scala b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/async/AsyncConcurrentComputationExecutor.scala index 9af394da80a..3243231d181 100644 --- a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/async/AsyncConcurrentComputationExecutor.scala +++ b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/async/AsyncConcurrentComputationExecutor.scala @@ -106,11 +106,7 @@ abstract class AsyncConcurrentComputationExecutor(override val outputPrintLimit: } { e => logger.info("failed to do with hook", e) } - if (hookedCode.length > 100) { - logger.info(s"hooked after code: ${hookedCode.substring(0, 100)} ....") - } else { - logger.info(s"hooked after code: $hookedCode ") - } + logger.info(s"AsyncConcurrentComputationExecutor code hook completed.") val localPath = EngineConnConf.getLogDir engineExecutionContext.appendStdout( LogUtils.generateInfo( diff --git a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/execute/ComputationExecutor.scala b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/execute/ComputationExecutor.scala index ee48b133096..6794172ab94 100644 --- a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/execute/ComputationExecutor.scala +++ b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/execute/ComputationExecutor.scala @@ -224,11 +224,7 @@ abstract class ComputationExecutor(val outputPrintLimit: Int = 1000) case _ => logger.info("failed to do with hook", e) } } - if (hookedCode.length > 100) { - logger.info(s"hooked after code: ${hookedCode.substring(0, 100)} ....") - } else { - logger.info(s"hooked after code: $hookedCode ") - } + logger.info(s"ComputationExecutor code hook completed.") // task params log // spark engine: at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor.executeLine log special conf diff --git a/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/executor/HiveEngineConnExecutor.scala b/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/executor/HiveEngineConnExecutor.scala index 0a52568f954..b2d0196c1e9 100644 --- a/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/executor/HiveEngineConnExecutor.scala +++ b/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/executor/HiveEngineConnExecutor.scala @@ -700,7 +700,7 @@ class HiveEngineConnExecutor( currentProps.keys.foreach { key => if (!hiveTmpConf.contains(key)) { logger.info(s"Clearing extra configuration key: $key") - sessionConf.set(key, "") + sessionConf.unset(key) } } } else {