This guide provides comprehensive instructions for developers working with MatrixOne, covering local standalone deployment, multi-CN cluster setup, monitoring, and all make dev-* commands.
- Quick Start
- Local Development Methods
- Development Commands Overview
- Standalone MatrixOne Setup
- Multi-CN Cluster Setup (Docker Compose)
- Monitoring and Metrics
- Configuration Management
- Troubleshooting
Basic Setup:
# Build and start local multi-CN cluster
make dev-build && make dev-up
# Connect via proxy (load balanced across CNs)
mysql -h 127.0.0.1 -P 6001 -u root -p111With Monitoring:
# Build and start
make dev-build && make dev-up
# Enable metrics (edit configs - see Multi-CN Workflow section)
make dev-edit-cn1 # Enable metrics in each service config
make dev-restart
# Start monitoring
make dev-up-grafana
make dev-check-grafana # Wait until ready
make dev-create-dashboard-cluster # Create dashboards
# Access Grafana at http://localhost:3000 (admin/admin)Basic Setup:
# Build and launch standalone MatrixOne
make build
./mo-service -launch ./etc/launch/launch.toml
# Connect
mysql -h 127.0.0.1 -P 6001 -u root -p111With Monitoring:
# Build
make build
# Enable metrics (edit etc/launch/cn.toml, tn.toml, log.toml)
# Set: disableMetric = false, enable-metric-to-prom = true, status-port = 7001
# Launch
./mo-service -launch ./etc/launch/launch.toml
# Start monitoring
make dev-up-grafana-local
make dev-check-grafana # Wait until ready
make dev-create-dashboard-local # Create dashboards
# Access Grafana at http://localhost:3001 (admin/admin)make dev-helpFor detailed workflows, see:
- Local Development Methods - Start here! Three ways to run MatrixOne locally with complete setup, data directories, and cleanup instructions
- Standalone MatrixOne Setup - Complete standalone workflow details
- Multi-CN Cluster Setup - Complete multi-CN workflow details
MatrixOne provides three methods for local development, each with different storage backends and use cases. Choose the method that best fits your needs.
Best for: Quick testing, simple development, learning MatrixOne
Storage: Local file system (DISK backend)
# 1. Build MatrixOne
make build
# 2. Start MatrixOne
./mo-service -launch ./etc/launch/launch.toml
# 3. Connect (in another terminal)
mysql -h 127.0.0.1 -P 6001 -u root -p111# 1. Enable metrics in config files
# Edit: etc/launch/cn.toml, etc/launch/tn.toml, etc/launch/log.toml
# Set: disableMetric = false, enable-metric-to-prom = true, status-port = 7001
# 2. Start MatrixOne
./mo-service -launch ./etc/launch/launch.toml
# 3. Start Grafana (in another terminal)
make dev-up-grafana-local
# 4. Wait for Grafana to be ready
make dev-check-grafana
# 5. Create dashboards
make dev-create-dashboard-local
# 6. Access Grafana at http://localhost:3001 (admin/admin)- MatrixOne Data:
./mo-data/(project root directory) - Metrics Data:
prometheus-local-data/(project root directory) - Grafana Data:
grafana-local-data/(project root directory)
# Stop MatrixOne process
# Press Ctrl+C in the terminal where mo-service is running
# Or find and kill the process:
ps aux | grep mo-service
kill <PID>
# Stop Grafana and Prometheus
make dev-down-grafana-local# Stop all services first
# Then remove data directories:
rm -rf ./mo-data
rm -rf prometheus-local-data
rm -rf grafana-local-dataBest for: Testing object storage, S3-compatible storage, distributed storage scenarios
Storage: MinIO (S3-compatible object storage) + local file system
# 1. Build MatrixOne
make build
# 2. Start MinIO service
make dev-up-minio-local
# 3. Start MatrixOne with MinIO storage
make launch-minio
# Or manually:
# ./mo-service -launch etc/launch-minio-local/launch.toml
# 4. Connect (in another terminal)
mysql -h 127.0.0.1 -P 6001 -u root -p111# 1. Enable metrics in config files
# Edit: etc/launch-minio-local/cn.toml, etc/launch-minio-local/tn.toml, etc/launch-minio-local/log.toml
# Set: disableMetric = false, enable-metric-to-prom = true, status-port = 7001
# 2. Start MinIO
make dev-up-minio-local
# 3. Start MatrixOne
./mo-service -launch etc/launch-minio-local/launch.toml
# 4. Start Grafana (in another terminal)
make dev-up-grafana-local
# 5. Wait for Grafana to be ready
make dev-check-grafana
# 6. Create dashboards
make dev-create-dashboard-local
# 7. Access Grafana at http://localhost:3001 (admin/admin)- MatrixOne Data:
etc/launch-minio-local/mo-data/ - MinIO Data:
etc/launch-minio-local/mo-data/minio-data/ - Metrics Data:
prometheus-local-data/(project root directory) - Grafana Data:
grafana-local-data/(project root directory)
# Stop MatrixOne process
# Press Ctrl+C in the terminal where mo-service is running
# Or find and kill the process:
ps aux | grep mo-service
kill <PID>
# Stop MinIO
make dev-down-minio-local
# Stop Grafana and Prometheus
make dev-down-grafana-local# Stop all services first
# Then remove the entire directory (contains both MatrixOne and MinIO data):
rm -rf etc/launch-minio-local/
# Also clean up monitoring data if needed:
rm -rf prometheus-local-data
rm -rf grafana-local-dataBest for: Testing multi-node scenarios, load balancing, distributed features, production-like setup
Storage: Local disk shared storage (DISK backend)
# 1. Build Docker image
make dev-build
# 2. Start multi-CN cluster
make dev-up
# 3. Connect via proxy (load balanced)
mysql -h 127.0.0.1 -P 6001 -u root -p111# 1. Start cluster with Grafana
make dev-up-grafana
# 2. Enable metrics (edit configs interactively)
make dev-edit-cn1 # Enable metrics in each service config
make dev-edit-cn2
make dev-edit-tn
make dev-edit-log
make dev-edit-proxy
# 3. Restart services to apply changes
make dev-restart
# 4. Wait for Grafana to be ready
make dev-check-grafana
# 5. Create dashboards
make dev-create-dashboard-cluster
# 6. Access Grafana at http://localhost:3000 (admin/admin)- MatrixOne Data:
mo-data/(project root directory)- Shared storage:
mo-data/shared/
- Shared storage:
- Logs:
logs/(project root directory) - Metrics Data:
prometheus-data/(project root directory) - Grafana Data:
grafana-data/(project root directory)
# Stop all cluster services (including Grafana if started with dev-up-grafana)
make dev-down
# Note: If you started Grafana separately, you can stop it with:
# cd etc/docker-multi-cn-local-disk && docker compose --profile prometheus down# Interactive cleanup (recommended - shows what will be deleted)
make dev-cleanup
# Or quick cleanup (WARNING: deletes all data without confirmation!)
make dev-clean
# This removes:
# - mo-data/ directory
# - logs/ directory
# - prometheus-data/ directory (if exists)
# - grafana-data/ directory (if exists)mo-data directories used by launch methods (Method 1 and Method 2) and multi-CN method (Method 3) are NOT compatible.
- Launch methods use
./mo-data/at project root with a different data format - Multi-CN method uses
mo-data/at project root with Docker volume mounts and shared storage
Do NOT mix them! If you switch between methods:
- Stop all services first
- Backup or remove the existing
mo-data/directory - Start fresh with the new method
Example:
# Switching from launch to multi-CN:
# 1. Stop launch method (Ctrl+C)
# 2. Backup or remove:
mv mo-data mo-data-launch-backup
# 3. Start multi-CN:
make dev-up
# Switching from multi-CN to launch:
# 1. Stop multi-CN:
make dev-down
# 2. Backup or remove:
mv mo-data mo-data-multicn-backup
# 3. Start launch method:
./mo-service -launch ./etc/launch/launch.toml| Feature | Method 1: Launch | Method 2: Launch + MinIO | Method 3: Multi-CN |
|---|---|---|---|
| Storage | Local file system | MinIO (S3-compatible) | Local shared disk |
| Nodes | Single process | Single process | Multi-CN (2+ nodes) |
| Proxy | No | No | Yes (load balancer) |
| Containerized | No | MinIO only | Yes (all services) |
| Best For | Quick testing | Object storage testing | Production-like testing |
| Data Dir | ./mo-data/ |
etc/launch-minio-local/mo-data/ |
mo-data/ |
| Grafana Port | 3001 | 3001 | 3000 |
| Stop Command | Ctrl+C or kill | Ctrl+C + make dev-down-minio-local |
make dev-down |
| Clean Command | rm -rf ./mo-data |
rm -rf etc/launch-minio-local/ |
make dev-clean |
| Command | Description |
|---|---|
make dev-build |
Build MatrixOne Docker image with local tag |
make dev-help |
Show all available dev-* commands |
| Command | Description |
|---|---|
make dev-up |
Start multi-CN cluster (default: local image) |
make dev-up-latest |
Start with official latest image from Docker Hub |
make dev-up-nightly |
Start with nightly build image |
make dev-up-test |
Start with test directory mounted at /test |
make dev-down |
Stop multi-CN cluster |
make dev-restart |
Restart all services |
make dev-ps |
Show service status |
make dev-cleanup |
Interactive cleanup (stops containers, removes data directories) |
| Command | Description |
|---|---|
make dev-restart-cn1 |
Restart CN1 only |
make dev-restart-cn2 |
Restart CN2 only |
make dev-restart-proxy |
Restart Proxy only |
make dev-restart-tn |
Restart TN only |
make dev-restart-log |
Restart Log service only |
| Command | Description |
|---|---|
make dev-up-grafana |
Start cluster with Grafana monitoring (port 3000) |
make dev-up-grafana-local |
Start Grafana dashboard for local MatrixOne (port 3001) |
make dev-check-grafana |
Check if Grafana services are ready (ports 3000, 3001) |
make dev-restart-grafana |
Restart Grafana and Prometheus (cluster mode) |
make dev-restart-grafana-local |
Restart Grafana and Prometheus (local mode) |
make dev-down-grafana-local |
Stop and remove local monitoring services |
| Command | Description |
|---|---|
make dev-logs |
View all logs (tail -f) |
make dev-logs-cn1 |
View CN1 logs |
make dev-logs-cn2 |
View CN2 logs |
make dev-logs-proxy |
View proxy logs |
make dev-logs-tn |
View TN logs |
make dev-logs-log |
View Log service logs |
make dev-logs-grafana |
View Grafana logs (cluster mode) |
make dev-logs-grafana-local |
View Grafana logs (local mode) |
| Command | Description |
|---|---|
make dev-mysql |
Connect to database via proxy (port 6001) |
make dev-mysql-cn1 |
Connect directly to CN1 (port 16001) |
make dev-mysql-cn2 |
Connect directly to CN2 (port 16002) |
| Command | Description |
|---|---|
make dev-edit-cn1 |
Edit CN1 configuration interactively |
make dev-edit-cn2 |
Edit CN2 configuration interactively |
make dev-edit-proxy |
Edit Proxy configuration interactively |
make dev-edit-tn |
Edit TN configuration interactively |
make dev-edit-log |
Edit Log service configuration interactively |
make dev-edit-common |
Edit common configuration (all services) |
make dev-config |
Generate config files from config.env |
make dev-config-example |
Create config.env.example file |
| Command | Description |
|---|---|
make dev-clean |
Stop and remove all data (WARNING: destructive!) |
make dev-cleanup |
Interactive cleanup (stops containers, removes data directories with confirmation) |
make dev-shell-cn1 |
Open shell in CN1 container |
make dev-shell-cn2 |
Open shell in CN2 container |
make dev-setup-docker-mirror |
Configure Docker registry mirror (for faster pulls) |
| Command | Description |
|---|---|
make dev-create-dashboard |
Create dashboard (default: local mode, port 3001) |
make dev-create-dashboard-local |
Create local dashboard (port 3001) |
make dev-create-dashboard-cluster |
Create cluster dashboard (port 3000, docker compose) |
make dev-create-dashboard-k8s |
Create K8S dashboard |
make dev-create-dashboard-cloud-ctrl |
Create cloud control-plane dashboard |
| Command | Description |
|---|---|
make dev-list-dashboard |
List dashboards (default: local mode, port 3001) |
make dev-list-dashboard-local |
List local dashboards (port 3001) |
make dev-list-dashboard-cluster |
List cluster dashboards (port 3000) |
make dev-list-dashboard-k8s |
List K8S dashboards |
make dev-list-dashboard-cloud-ctrl |
List cloud control-plane dashboards |
| Command | Description |
|---|---|
make dev-delete-dashboard |
Delete all dashboards in folder (default: local mode, port 3001) |
make dev-delete-dashboard-local |
Delete all local dashboards (port 3001) |
make dev-delete-dashboard-cluster |
Delete all cluster dashboards (port 3000) |
make dev-delete-dashboard-k8s |
Delete all K8S dashboards |
make dev-delete-dashboard-cloud-ctrl |
Delete all cloud control-plane dashboards |
Custom Options:
# Using make (recommended)
DASHBOARD_PORT=3001 make dev-create-dashboard
DASHBOARD_HOST=localhost make dev-create-dashboard
DASHBOARD_MODE=cloud make dev-create-dashboard
DASHBOARD_USERNAME=admin DASHBOARD_PASSWORD=admin make dev-create-dashboard
# Or directly using mo-tool
# Create dashboards
./mo-tool dashboard create --host localhost --port 3001 --mode local
./mo-tool dashboard create --mode cloud --port 3000 --datasource Prometheus
# List dashboards (to get UIDs)
./mo-tool dashboard list --mode local --port 3001
# Delete operations
./mo-tool dashboard delete --mode local --port 3001 # Delete all dashboards in folder
./mo-tool dashboard delete-dashboard --uid <uid> # Delete single dashboard by UID
./mo-tool dashboard delete-folder --mode local --port 3001 # Delete entire folder and all dashboards
# Note: Provisioned dashboards (created via config files) cannot be deleted via API
# They will be skipped with a warning when deleting foldersFor development and testing, you can run MatrixOne in standalone mode (single process with all services).
- Go 1.22 or later
- GCC/Clang compiler
- Make
Here's the complete workflow for building, launching, and monitoring standalone MatrixOne:
# Build mo-service binary
make build
# Or build with race detector and debug symbols (for debugging)
make debugOutput: The mo-service binary will be created in the project root directory.
# Start with default configuration (recommended - launches all services)
./mo-service -launch ./etc/launch/launch.tomlAlternative: Manual Service Startup
If you prefer to start services individually (useful for debugging):
# Terminal 1: Start Log service (contains HAKeeper)
./mo-service -cfg ./etc/launch/log.toml
# Terminal 2: Start TN (Transaction Node)
./mo-service -cfg ./etc/launch/tn.toml
# Terminal 3: Start CN (Compute Node)
./mo-service -cfg ./etc/launch/cn.tomlNote: When using launch.toml, all services start in a single process. The manual approach requires three separate terminals.
# Default connection (if using launch.toml)
mysql -h 127.0.0.1 -P 6001 -u root -p111
# Or if connecting directly to CN (when started manually)
mysql -h 127.0.0.1 -P 18001 -u root -p111To enable metrics collection, edit the configuration files:
For launch.toml mode:
# Edit CN configuration
vim ./etc/launch/cn.toml
# Edit TN configuration
vim ./etc/launch/tn.toml
# Edit Log service configuration
vim ./etc/launch/log.tomlIn each file, ensure the following section is present and configured:
[observability]
disableMetric = false
enable-metric-to-prom = true
status-port = 7001Then restart MatrixOne:
# Stop current instance (Ctrl+C)
# Restart with updated configuration
./mo-service -launch ./etc/launch/launch.tomlAfter enabling metrics, start the Grafana dashboard:
# Start Grafana dashboard for local standalone MatrixOne
make dev-up-grafana-local
# Wait for Grafana to initialize (2-3 minutes on first startup)
# Check if ready:
make dev-check-grafana
# Access Grafana at http://localhost:3001
# Username: admin
# Password: adminCreate Dashboards:
After Grafana is ready, create the MatrixOne dashboards:
# Create dashboards for standalone MatrixOne
make dev-create-dashboard-local
# Or with custom options:
DASHBOARD_PORT=3001 make dev-create-dashboardVerify Metrics:
You can verify metrics are working:
# Check metrics endpoint directly
curl http://localhost:7001/metrics
# Or query Prometheus (if port exposed)
curl http://localhost:9091/api/v1/query?query=mo_sql_statement_totalQuick Start (Without Metrics):
make build # Build binary
./mo-service -launch ./etc/launch/launch.toml # Launch MatrixOne
mysql -h 127.0.0.1 -P 6001 -u root -p111 # ConnectComplete Workflow (With Metrics):
# 1. Build
make build
# 2. Enable metrics in config files (edit cn.toml, tn.toml, log.toml)
# Set: disableMetric = false, enable-metric-to-prom = true, status-port = 7001
# 3. Launch
./mo-service -launch ./etc/launch/launch.toml
# 4. Start monitoring
make dev-up-grafana-local
make dev-check-grafana # Wait until ready
make dev-create-dashboard-local # Create dashboards
# 5. Connect and use
mysql -h 127.0.0.1 -P 6001 -u root -p111
# 6. Access Grafana at http://localhost:3001 (admin/admin)# Standard build
make build
# Build with race detector and debug symbols
make debug
# Clean build artifacts
make cleanThe multi-CN setup provides a complete distributed cluster with:
- Multi-CN: 2 CN nodes (scalable)
- Local Disk: Local disk shared storage (DISK backend)
- With Proxy: Load balancing proxy included
- Docker Compose: Containerized deployment, easy to manage
- Bridge Network: Bridge network with service isolation
- Enhanced Script:
start.shwith auto permissions, versioning, and dynamic mounts - No Root Files: All generated files use your user's permissions
┌─────────────────────────────────────────┐
│ MatrixOne Multi-CN Local Disk │
├─────────────────────────────────────────┤
│ │
│ ┌────────┐ ┌────────┐ ┌──────────┐ │
│ │ mo-log │ │ mo-tn │ │ mo-proxy │ │
│ │ :32001 │ │ │ │ :6001 │ │
│ └───┬────┘ └───┬────┘ └────┬─────┘ │
│ │ │ │ │
│ ┌───┴───────────┴─────────────┴────┐ │
│ │ Local Shared Disk Storage │ │
│ │ (mo-data/shared) │ │
│ └───┬───────────┬──────────────────┘ │
│ │ │ │
│ ┌───┴────┐ ┌───┴────┐ │
│ │ mo-cn1 │ │ mo-cn2 │ │
│ │ :16001 │ │ :16002 │ │
│ └────────┘ └────────┘ │
│ │
└─────────────────────────────────────────┘
| Service | Container | Port | Description |
|---|---|---|---|
| LogService | mo-log | 32001, 7001 | Log service and HAKeeper |
| TN | mo-tn | - | Transaction Node |
| CN1 | mo-cn1 | 16001 | Compute Node 1 |
| CN2 | mo-cn2 | 16002 | Compute Node 2 |
| Proxy | mo-proxy | 6001 | Load balancer proxy |
Here's the complete workflow for building, launching, and monitoring a multi-CN cluster:
# Build MatrixOne Docker image with 'local' tag
make dev-buildNote: This builds the image from source code. Build time: ~10-15 minutes depending on network and CPU.
Alternative: Use Official Images
# Skip build and use official latest image
make dev-up-latest
# Or use nightly build
make dev-up-nightly# Start cluster with local image
make dev-upWhat happens:
- Starts 2 CN nodes (mo-cn1, mo-cn2)
- Starts TN (Transaction Node)
- Starts Log service (HAKeeper)
- Starts Proxy (load balancer on port 6001)
- Auto-generates configuration files if not present
Note: On first startup, default TOML configuration files will be automatically generated. These files are not tracked by git to prevent accidental commits of your local configurations.
# Connect via proxy (load balanced across CNs)
mysql -h 127.0.0.1 -P 6001 -u root -p111
# Or connect directly to CN1
mysql -h 127.0.0.1 -P 16001 -u root -p111
# Or connect directly to CN2
mysql -h 127.0.0.1 -P 16002 -u root -p111Metrics are disabled by default. To enable:
Method 1: Interactive Configuration (Recommended)
# Edit CN1 configuration
make dev-edit-cn1
# In the editor, find and uncomment:
# [observability]
# disableMetric = false
# enable-metric-to-prom = true
# status-port = 7001
# Save and repeat for other services
make dev-edit-cn2
make dev-edit-tn
make dev-edit-log
make dev-edit-proxy
# Restart services to apply changes
make dev-restartMethod 2: Manual Configuration
# Edit configuration files directly
vim etc/docker-multi-cn-local-disk/cn1.toml
vim etc/docker-multi-cn-local-disk/cn2.toml
vim etc/docker-multi-cn-local-disk/tn.toml
vim etc/docker-multi-cn-local-disk/log.toml
vim etc/docker-multi-cn-local-disk/proxy.toml
# Add or uncomment in each file:
# [observability]
# disableMetric = false
# enable-metric-to-prom = true
# status-port = 7001
# Restart services
make dev-restartAfter enabling metrics, start the Grafana monitoring dashboard:
# Start cluster with Grafana monitoring
make dev-up-grafana
# Wait for Grafana to initialize (2-3 minutes on first startup)
# Check if ready:
make dev-check-grafana
# Access Grafana at http://localhost:3000
# Username: admin
# Password: adminCreate Dashboards:
After Grafana is ready, create the MatrixOne dashboards:
# Create dashboards for multi-CN cluster
make dev-create-dashboard-cluster
# Or with custom options:
DASHBOARD_MODE=cloud DASHBOARD_PORT=3000 make dev-create-dashboardVerify Metrics:
# Check metrics from a service
docker exec mo-cn1 curl http://localhost:7001/metrics
# Or query Prometheus (if port exposed)
curl http://localhost:9090/api/v1/query?query=mo_sql_statement_totalQuick Start (Without Metrics):
make dev-build # Build Docker image
make dev-up # Start cluster
mysql -h 127.0.0.1 -P 6001 -u root -p111 # ConnectComplete Workflow (With Metrics):
# 1. Build
make dev-build
# 2. Start cluster
make dev-up
# 3. Enable metrics (edit configs or use interactive editor)
make dev-edit-cn1 # Edit and enable metrics
make dev-edit-cn2 # Edit and enable metrics
make dev-edit-tn # Edit and enable metrics
make dev-edit-log # Edit and enable metrics
make dev-edit-proxy # Edit and enable metrics
# 4. Restart to apply metrics configuration
make dev-restart
# 5. Start monitoring
make dev-up-grafana
make dev-check-grafana # Wait until ready
make dev-create-dashboard-cluster # Create dashboards
# 6. Connect and use
mysql -h 127.0.0.1 -P 6001 -u root -p111
# 7. Access Grafana at http://localhost:3000 (admin/admin)Using Official Images (Skip Build):
# Start with latest official image
make dev-up-latest
# Or with nightly build
make dev-up-nightly
# Then follow steps 3-7 above for metrics# 1. Build Docker image
make dev-build
# 2. Start cluster
make dev-up
# 3. Connect
mysql -h 127.0.0.1 -P 6001 -u root -p111Note: On first startup, default TOML configuration files will be automatically generated. These files are not tracked by git to prevent accidental commits of your local configurations.
# Start with latest official image
make dev-up-latest
# Or use specific version
make DEV_VERSION=1.0.0 dev-up
# Or use nightly build
make dev-up-nightly# Start with test directory mounted
make dev-up-test
# In MySQL client:
mysql> source /test/distributed/cases/your_test.sql;# Mount custom directory
make DEV_MOUNT="../../test:/test:ro" dev-up
make DEV_MOUNT="/path/to/data:/data:ro" dev-up
# Combine options (version + mount)
make DEV_VERSION=latest DEV_MOUNT="../../test:/test:ro" dev-upThe start.sh script provides an enhanced wrapper with automatic permission handling:
cd etc/docker-multi-cn-local-disk
# Show help
./start.sh -h
# Default: Build and start with local image
./start.sh build
./start.sh up -d
# Use official latest version
./start.sh -v latest up -d
# Use specific version
./start.sh -v 3.0 up -d
./start.sh -v nightly up -d
# Mount test directory for source command
./start.sh -m "../../test:/test:ro" up -d
# Combine options
./start.sh -v latest -m "../../test:/test:ro" up -d
# Other commands
./start.sh down
./start.sh logs -f
./start.sh ps
./start.sh restartstart.sh Quick Reference:
| Option | Description | Example |
|---|---|---|
-v, --version |
Specify image version | ./start.sh -v latest up -d |
Default: local (built image) |
./start.sh -v nightly up -d |
|
Options: local, latest, nightly, 3.0 |
./start.sh -v 1.2.3 up -d |
|
-m, --mount |
Mount extra directory | ./start.sh -m "../../test:/test:ro" up -d |
| Useful for SQL test files | Multiple mounts: separate calls | |
-h, --help |
Show help message | ./start.sh -h |
| (no options) | Use local image + auto UID/GID | ./start.sh up -d |
Why use start.sh?
- ✅ Auto Permissions: Automatically detects your UID/GID (no root-owned files!)
- ✅ Default Local Image: Uses local build by default, no version confusion
- ✅ Easy Versioning: Switch versions with
-v latest|nightly|3.0 - ✅ Dynamic Mounts: Add test directory with
-moption - ✅ Clear Output: Shows configuration before executing
- ✅ No Manual Config: Just works out of the box
Method 1: Mount with start.sh or make (Easiest)
# Using make
make dev-up-test
# Using start.sh
./start.sh -m "../../test:/test:ro" up -d
# Connect and run
mysql -h 127.0.0.1 -P 6001 -u root -p111
mysql> source /test/distributed/cases/your_test.sql;Method 2: Pipe SQL file directly
mysql -h 127.0.0.1 -P 6001 -u root -p111 < test/distributed/cases/your_test.sqlMethod 3: Use mysql with -e option
mysql -h 127.0.0.1 -P 6001 -u root -p111 -e "$(cat test/distributed/cases/your_test.sql)"Method 4: Copy file to container
docker cp test/distributed/cases/your_test.sql mo-cn1:/tmp/test.sql
mysql -h 127.0.0.1 -P 16001 -u root -p111 -e "source /tmp/test.sql"Advantages:
- ✅ Use latest code from repository
- ✅ Custom modifications
- ✅ Debug with source code
- ✅ Test unreleased features
Build Process:
# Build from source (creates matrixorigin/matrixone:local)
make dev-build
# Or using start.sh
cd etc/docker-multi-cn-local-disk
./start.sh buildBuild Requirements:
- Docker with buildx support
- Go 1.22+
- ~10-15 minutes build time (depending on network and CPU)
Go Proxy Configuration:
The build uses Go proxy to speed up dependency downloads. Default proxies (in order):
https://goproxy.cn- Alibaba Cloudhttps://mirrors.aliyun.com/goproxy/- Aliyun Mirrorhttps://goproxy.io- Globaldirect- Fallback to source
Available Go Proxies:
| Provider | Proxy | Usage |
|---|---|---|
| Alibaba | https://goproxy.cn |
GOPROXY="https://goproxy.cn,direct" docker compose build |
| Aliyun | https://mirrors.aliyun.com/goproxy/ |
GOPROXY="https://mirrors.aliyun.com/goproxy/,direct" docker compose build |
| GoProxy.io | https://goproxy.io |
GOPROXY="https://goproxy.io,direct" docker compose build |
| Official | https://proxy.golang.org |
GOPROXY="https://proxy.golang.org,direct" docker compose build |
| Athens | https://athens.azurefd.net |
GOPROXY="https://athens.azurefd.net,direct" docker compose build |
For faster builds in different regions:
# Use regional mirror (Alibaba)
export GOPROXY="https://goproxy.cn,direct"
make dev-build
# Or combine multiple proxies for better redundancy
export GOPROXY="https://goproxy.cn,https://mirrors.aliyun.com/goproxy/,direct"
make dev-buildAdvantages:
- ✅ No build time - start immediately
- ✅ Official stable releases
- ✅ Smaller download size
- ✅ Updated regularly
Disadvantages:
⚠️ May be slow in some regions (Docker Hub access limitations)⚠️ Requires Docker mirror configuration for faster access
Available Tags:
matrixorigin/matrixone:latest- Latest stable releasematrixorigin/matrixone:nightly- Nightly buildsmatrixorigin/matrixone:v1.0.0- Specific version
Usage:
# Use latest
make dev-up-latest
# Use specific version
make DEV_VERSION=1.0.0 dev-up
# Use nightly build
make dev-up-nightlyAll containers share local disk storage:
- Data Directory:
mo-data/(under project root) - Shared Storage:
mo-data/shared/(DISK backend) - Log Directory:
logs/(service-specific log files)
File Permissions:
- When using
make dev-*or./start.sh: Files are automatically created with your user's permissions ✅ - When using
docker composedirectly: SetDOCKER_UIDandDOCKER_GIDenvironment variables - Default fallback:
501:20(macOS first user) - may need adjustment for Linux users
To add more CN nodes:
- Copy config:
cp etc/docker-multi-cn-local-disk/cn2.toml etc/docker-multi-cn-local-disk/cn3.toml - Modify UUID, ports, etc. in
cn3.toml - Add
mo-cn3service indocker-compose.yml - Start:
docker compose up -d mo-cn3
MatrixOne provides comprehensive monitoring through Prometheus and Grafana.
Metrics are disabled by default. To enable:
# Edit service configuration
make dev-edit-cn1
# In the editor, find and uncomment:
# [observability]
# disableMetric = false
# enable-metric-to-prom = true
# status-port = 7001
# Save and restart
make dev-restart-cn1Edit the configuration files directly:
etc/docker-multi-cn-local-disk/cn1.tomletc/docker-multi-cn-local-disk/cn2.tomletc/docker-multi-cn-local-disk/tn.tomletc/docker-multi-cn-local-disk/proxy.tomletc/docker-multi-cn-local-disk/log.toml
Add or uncomment:
[observability]
disableMetric = false
enable-metric-to-prom = true
status-port = 7001Then restart the services.
# Start cluster with Grafana
make dev-up-grafana
# Access Grafana
# URL: http://localhost:3000
# Username: admin
# Password: adminImportant: On first startup, Grafana needs a few minutes to initialize its database and services. You can check if Grafana is ready using:
# Check Grafana service status (recommended)
make dev-check-grafanaThis command checks both ports 3000 (cluster) and 3001 (local) and verifies that Grafana is responding to HTTP requests. Alternatively, you can manually check by accessing http://localhost:3000 - if the login page loads, Grafana is ready.
What's included:
- Prometheus: Collects metrics from all MatrixOne services
- Grafana: Visualization dashboard with pre-configured MatrixOne dashboards
- Auto-discovery: Automatically connects to all services in the cluster
Prometheus Configuration:
- Config file:
etc/docker-multi-cn-local-disk/prometheus.yml - Scrapes:
mo-cn1:7001,mo-cn2:7001,mo-tn:7001,mo-proxy:7001,mo-log:7001
If you're running MatrixOne locally (not in Docker):
# Start Grafana dashboard for local MatrixOne
make dev-up-grafana-local
# Access Grafana
# URL: http://localhost:3001
# Username: admin
# Password: adminImportant: On first startup, Grafana needs a few minutes to initialize its database and services. You can check if Grafana is ready using:
# Check Grafana service status (recommended)
make dev-check-grafanaThis command checks both ports 3000 (cluster) and 3001 (local) and verifies that Grafana is responding to HTTP requests. Alternatively, you can manually check by accessing http://localhost:3001 - if the login page loads, Grafana is ready.
What's included:
- Prometheus: Collects metrics from local MatrixOne services
- Grafana: Visualization dashboard
- Host access: Uses
host.docker.internalto reach local services
Prometheus Configuration:
- Config file:
etc/docker-multi-cn-local-disk/prometheus-local.yml - Scrapes:
host.docker.internal:7001(or your configured status-port)
Note for Linux: If host.docker.internal doesn't work, edit prometheus-local.yml and replace with your host IP address.
Important: Wait for Grafana to be ready before creating dashboards. Grafana needs time to initialize its database and services on first startup (typically 2-3 minutes).
Check if Grafana is ready:
# Recommended: Use the check command
make dev-check-grafana
# Or manually: Access the web UI - if the login page loads, it's ready
# Cluster: http://localhost:3000
# Local: http://localhost:3001After Grafana is ready, you need to create the dashboards. MatrixOne provides a convenient make command:
For Local Standalone MatrixOne (port 3001):
make dev-create-dashboard-local
# Or with custom port:
DASHBOARD_PORT=3001 make dev-create-dashboard
# Or directly using mo-tool:
./mo-tool dashboard --mode local --port 3001For Docker Compose Cluster (port 3000):
make dev-create-dashboard-cluster
# Or:
DASHBOARD_MODE=cloud DASHBOARD_PORT=3000 make dev-create-dashboard
# Or directly using mo-tool:
./mo-tool dashboard --mode cloud --port 3000Other Modes:
# K8S mode
make dev-create-dashboard-k8s
# Cloud control-plane mode
make dev-create-dashboard-cloud-ctrlCustom Options:
# Custom host and port
DASHBOARD_HOST=localhost DASHBOARD_PORT=3001 make dev-create-dashboard
# Custom credentials
DASHBOARD_USERNAME=admin DASHBOARD_PASSWORD=yourpassword make dev-create-dashboard
# Custom datasource name (for cloud/k8s modes)
DASHBOARD_MODE=cloud DASHBOARD_DATASOURCE=Prometheus make dev-create-dashboardNote: The dashboard creation tool connects to Grafana and creates all MatrixOne dashboards. Make sure Grafana is running and has finished initializing (wait 2-3 minutes after first startup) before executing this command.
- Create dashboards (see above)
- Open Grafana: http://localhost:3000 (cluster) or http://localhost:3001 (local)
- Navigate to Dashboards → Browse
- Select MatrixOne dashboards from the folder (Matrixone for cluster, Matrixone-Standalone for local)
If Prometheus port is exposed (uncomment in docker-compose.yml):
# Query total SQL statements
curl 'http://localhost:9090/api/v1/query?query=mo_sql_statement_total'
# Query CN1 CPU usage
curl 'http://localhost:9090/api/v1/query?query=process_cpu_seconds_total{instance="mo-cn1:7001"}'# Query metrics endpoint directly
curl http://localhost:7001/metrics
# Or from container
docker exec mo-cn1 curl http://localhost:7001/metricsMatrixOne exposes various metrics including:
- SQL Metrics:
mo_sql_statement_total,mo_sql_statement_duration_seconds - Transaction Metrics:
mo_txn_total,mo_txn_duration_seconds - Storage Metrics:
mo_storage_size_bytes,mo_storage_io_total - Connection Metrics:
mo_connection_total,mo_connection_active - System Metrics: CPU, memory, disk usage
See pkg/util/metric/ for complete metric definitions.
Configuration files are located in etc/docker-multi-cn-local-disk/:
cn1.toml- CN1 configurationcn2.toml- CN2 configurationtn.toml- TN configurationproxy.toml- Proxy configurationlog.toml- Log service configuration
# Edit specific service config
make dev-edit-cn1 # Opens CN1 config in your default editor
make dev-edit-cn2 # Opens CN2 config
make dev-edit-proxy # Opens Proxy config
make dev-edit-tn # Opens TN config
make dev-edit-log # Opens Log service config
make dev-edit-common # Opens common config (affects all services)The editor will:
- Open the configuration file
- Show commented examples
- Auto-generate config on save (if using
config.env)
You can use config.env for centralized configuration:
# Create config.env from example
make dev-config-example
# or: cp etc/docker-multi-cn-local-disk/config.env.example etc/docker-multi-cn-local-disk/config.env
# Edit configuration (uncomment and modify lines)
vim etc/docker-multi-cn-local-disk/config.env
# Generate config files
make dev-config
# Or start (will auto-generate TOML configs)
make dev-upHow it works:
*.tomlfiles are automatically generated fromconfig.env(or defaults if no config.env exists)- These generated files are ignored by git to avoid commit conflicts
- On first
make dev-up, default configurations are generated automatically - When
config.envexists, configurations are regenerated on every startup
Common Configuration Options:
LOG_LEVEL- debug, info, warn, error, fatal (default: info)LOG_FORMAT- console, json (default: console)MEMORY_CACHE- e.g., 512MB, 2GB, 4GB (default: 512MB)DISK_CACHE- e.g., 8GB, 20GB, 50GB (default: 8GB)DISABLE_TRACE- true, false (default: true)DISABLE_METRIC- true, false (default: true)
Service-Specific Configuration: Use service prefixes to override settings for specific services:
CN1_*- CN1 only (e.g.,CN1_MEMORY_CACHE=2GB)CN2_*- CN2 only (e.g.,CN2_MEMORY_CACHE=2GB)PROXY_*- Proxy only (e.g.,PROXY_LOG_LEVEL=debug)LOG_*- Log service only (e.g.,LOG_LOG_LEVEL=debug)TN_*- Transaction Node only (e.g.,TN_MEMORY_CACHE=4GB)
Configuration Priority:
- Service-specific (highest priority)
- Common configuration
- Default values
# Only increase cache for CN1 and CN2, keep others default
cat > etc/docker-multi-cn-local-disk/config.env << 'EOF'
CN1_MEMORY_CACHE=2GB
CN2_MEMORY_CACHE=2GB
EOF
# Apply and restart
make dev-config
make dev-restart# Enable debug logging only for proxy, others stay at info
cat > etc/docker-multi-cn-local-disk/config.env << 'EOF'
PROXY_LOG_LEVEL=debug
PROXY_LOG_FORMAT=json
EOF
# Apply
make dev-config
make dev-restart# Different cache sizes for each service
cat > etc/docker-multi-cn-local-disk/config.env << 'EOF'
CN1_MEMORY_CACHE=4GB
CN2_MEMORY_CACHE=2GB
TN_MEMORY_CACHE=1GB
EOF
# Apply
make dev-down
make dev-up# Common log level for all, but debug for proxy
cat > etc/docker-multi-cn-local-disk/config.env << 'EOF'
LOG_LEVEL=info # Default for all services
PROXY_LOG_LEVEL=debug # Override for proxy only
CN1_MEMORY_CACHE=2GB # Override for CN1 only
EOF
# Apply
make dev-config
make dev-restart# Increase cache for all services
cat > etc/docker-multi-cn-local-disk/config.env << 'EOF'
MEMORY_CACHE=4GB
DISK_CACHE=50GB
LOG_LEVEL=warn
EOF
# Apply
make dev-down
make dev-up# One-liner: increase CN1 memory cache only
echo "CN1_MEMORY_CACHE=2GB" > etc/docker-multi-cn-local-disk/config.env
make dev-config && make dev-restart
# View current configs
cat etc/docker-multi-cn-local-disk/cn1.toml | grep -A3 "\[fileservice.cache\]"[observability]
disableMetric = false
enable-metric-to-prom = true
status-port = 7001[log]
level = "debug" # Options: debug, info, warn, error[cn]
memory-capacity = "8GB" # Adjust based on available RAM[cn]
port-base = 18000 # CN1 uses 18000, CN2 uses 18001, etc.If docker pull is slow or fails:
# Configure Docker registry mirror
sudo make dev-setup-docker-mirror
# Or manually:
cd etc/docker-multi-cn-local-disk
sudo ./setup-docker-mirror.sh-
Check logs:
make dev-logs
-
Check service status:
make dev-ps
-
Verify ports are not in use:
# Check if ports are occupied lsof -i :6001 lsof -i :16001 lsof -i :16002 -
Check data directory permissions:
# Ensure mo-data and logs directories are writable ls -la mo-data logs
-
Verify metrics are enabled:
# Check if metrics endpoint responds curl http://localhost:7001/metrics -
Check Prometheus targets:
- Open Prometheus UI (if port exposed): http://localhost:9090/targets
- Verify all targets are "UP"
-
Check Grafana data source:
- Open Grafana: http://localhost:3000
- Go to Configuration → Data Sources
- Verify Prometheus connection is working
-
Restart monitoring services:
make dev-restart-grafana
-
Verify services are running:
make dev-ps
-
Check service logs:
make dev-logs-cn1
-
Verify port mapping:
docker ps | grep mo-
If you see permission errors:
# Check file ownership
ls -la mo-data logs
# Fix ownership (if needed)
sudo chown -R $(id -u):$(id -g) mo-data logsIf you need to start fresh:
# Option 1: Interactive cleanup (recommended - shows what will be deleted)
make dev-cleanup
# Option 2: Quick cleanup (WARNING: This deletes all data without confirmation!)
make dev-clean
# Then rebuild and start
make dev-build && make dev-upNote: make dev-cleanup provides an interactive interface that shows all data directories and asks for confirmation before deletion. It also handles root-owned files that may require sudo.
If make dev-build fails:
-
Check Docker is running:
docker ps
-
Check disk space:
df -h
-
Check build logs:
cd etc/docker-multi-cn-local-disk ./start.sh build -
Try different Go proxy:
# Check Docker build context with verbose output cd etc/docker-multi-cn-local-disk docker compose build --progress=plain # Try different Go proxy GOPROXY="https://goproxy.io,direct" docker compose build # Or use pre-built image (if Docker Hub is accessible) docker compose up -d # Uses matrixorigin/matrixone:latest
If you already ran containers and have root-owned files:
cd etc/docker-multi-cn-local-disk
# Stop containers
./start.sh down
# Fix permissions (need sudo because files are currently root-owned)
sudo chown -R $(id -u):$(id -g) ../../mo-data ../../logs
# Restart with correct permissions
./start.sh up -dOr clean restart (removes all data):
./start.sh down
sudo rm -rf ../../mo-data ../../logs
./start.sh up -d # Will create with your user permissionsIf you see image not found error:
# Use official image
cd etc/docker-multi-cn-local-disk
docker compose up -d
# Or build from source
docker compose build
IMAGE_NAME=matrixorigin/matrixone:local docker compose up -dIf you see "pull access denied" or timeout errors when using Docker Hub image:
Solution 1: Use local build (recommended)
make dev-build
make dev-upSolution 2: Configure Docker mirror
sudo make dev-setup-docker-mirror
# Or manually:
cd etc/docker-multi-cn-local-disk
sudo ./setup-docker-mirror.shIf containers can't communicate:
# Check network
docker network ls
docker network inspect docker-multi-cn-local-disk_matrixone-net
# Verify container names resolve
docker exec mo-cn1 ping mo-logFor performance analysis and debugging, MatrixOne supports CPU, heap, goroutine, block, and mutex profiling.
For Docker Compose Multi-CN Cluster (make dev-up):
Profiling is enabled by default for all CN and TN services with HTTP endpoints exposed:
# After running make dev-up, access profiling endpoints:
# CN1 profiling
curl http://localhost:6061/debug/pprof/heap
go tool pprof http://localhost:6061/debug/pprof/heap
# CN2 profiling
curl http://localhost:6062/debug/pprof/heap
go tool pprof http://localhost:6062/debug/pprof/heap
# TN profiling
curl http://localhost:6063/debug/pprof/heap
go tool pprof http://localhost:6063/debug/pprof/heapDefault Configuration:
- Block profiling:
rate=100(production recommended) - Mutex profiling:
fraction=100(production recommended) - HTTP profiling endpoints: Enabled on ports 6061 (CN1), 6062 (CN2), 6063 (TN)
Customize Profiling Settings:
You can control profiling behavior via environment variables:
# Disable profiling completely (no overhead)
BLOCK_PROFILE_RATE=0 MUTEX_PROFILE_FRACTION=0 make dev-up
# Maximum detail for debugging (higher overhead)
BLOCK_PROFILE_RATE=1 MUTEX_PROFILE_FRACTION=1 make dev-up
# Custom rates
BLOCK_PROFILE_RATE=1000 MUTEX_PROFILE_FRACTION=1000 make dev-upFor Standalone Mode:
# Enable block and mutex profiling
./mo-service \
-block-profile-rate=100 \
-mutex-profile-fraction=100 \
-launch ./etc/launch/launch.toml
# Or enable HTTP profiling endpoint
./mo-service -debug-http=:6060 -launch ./etc/launch/launch.toml
# Then access: http://localhost:6060/debug/pprof/Recommended Settings:
- Production:
-block-profile-rate=100 -mutex-profile-fraction=100(low overhead) - Debugging:
-block-profile-rate=1 -mutex-profile-fraction=1(detailed data)
📖 For detailed profiling documentation, see BUILD.md Performance Profiling section
| Method | Location | Features |
|---|---|---|
| Standalone | etc/launch/ |
Single node, quick test |
| Manual Multi-CN | etc/launch-multi-cn/ |
Multi-CN, manual start |
| Manual with Proxy | etc/launch-with-proxy/ |
Proxy+Multi-CN, manual |
| Docker Multi-CN | etc/docker-multi-cn-local-disk/ |
Containerized+Multi-CN+Local Storage ✨ |
- Main README: See project root
README.md - Build Guide: See
BUILD.mdfor detailed build instructions - Installation Guide: See
INSTALLATION.mdfor production deployment - Makefile Help: Run
make helpormake dev-help - Docker Compose: See
etc/docker-multi-cn-local-disk/docker-compose.ymlfor service definitions - MatrixOne Documentation: https://docs.matrixorigin.cn/
- Docker Hub - MatrixOne: https://hub.docker.com/r/matrixorigin/matrixone
Daily Development:
make dev-up # Start cluster
make dev-mysql # Connect to database
make dev-logs-cn1 # Check logs
make dev-restart-cn1 # Restart after config changeWith Monitoring:
make dev-up-grafana # Start with monitoring
# Access http://localhost:3000Testing:
make dev-up-test # Start with test directory
mysql -h 127.0.0.1 -P 6001 -u root -p111
mysql> source /test/distributed/cases/your_test.sql;Configuration Changes:
make dev-edit-cn1 # Edit config
make dev-restart-cn1 # Apply changesCleanup:
make dev-down # Stop services
make dev-clean # Remove all data (WARNING!)For more information, see the main project documentation or run make dev-help for a complete command list.