Skip to content

Commit 6e7dab4

Browse files
lukehblukehb
committed
Add E2E tests for web/metrics-client
1 parent d713972 commit 6e7dab4

9 files changed

Lines changed: 6904 additions & 0 deletions

File tree

‎.github/workflows/e2e-tests.yml‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
name: E2E Tests
2+
3+
on:
4+
push:
5+
branches: [ main, develop, web ]
6+
pull_request:
7+
branches: [ main, develop ]
8+
workflow_dispatch:
9+
10+
jobs:
11+
e2e-tests:
12+
runs-on: ubuntu-latest
13+
14+
steps:
15+
- name: Checkout code
16+
uses: actions/checkout@v4
17+
18+
- name: Set up Node.js
19+
uses: actions/setup-node@v4
20+
with:
21+
node-version: '20'
22+
cache: 'npm'
23+
cache-dependency-path: |
24+
Web/lib/package-lock.json
25+
E2ETests/package-lock.json
26+
27+
- name: Build metrics client library
28+
working-directory: Web/lib
29+
run: |
30+
npm ci
31+
npm run build
32+
npm test
33+
34+
- name: Install E2E test dependencies
35+
working-directory: E2ETests
36+
run: npm ci
37+
38+
- name: Start Docker services
39+
working-directory: Examples/Compose
40+
run: docker compose up -d --wait
41+
42+
- name: Wait for services to be ready
43+
run: |
44+
echo "Waiting for Buccaneer server..."
45+
timeout 60 bash -c 'until curl -f http://localhost:8000/ > /dev/null 2>&1; do sleep 2; done' || (echo "Buccaneer server failed to start" && exit 1)
46+
echo "Buccaneer server is ready"
47+
48+
echo "Waiting for Prometheus..."
49+
timeout 60 bash -c 'until curl -f http://localhost:9090/-/ready > /dev/null 2>&1; do sleep 2; done' || (echo "Prometheus failed to start" && exit 1)
50+
echo "Prometheus is ready"
51+
52+
- name: Check Prometheus targets
53+
run: |
54+
echo "Checking Prometheus targets..."
55+
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health, lastError: .lastError}'
56+
57+
- name: Run E2E tests
58+
working-directory: E2ETests
59+
run: npm test
60+
env:
61+
CI: true
62+
63+
- name: Show service logs on failure
64+
if: failure()
65+
working-directory: Examples/Compose
66+
run: |
67+
echo "=== Buccaneer Server Logs ==="
68+
docker compose logs buccaneerserver --tail=100
69+
echo ""
70+
echo "=== Prometheus Logs ==="
71+
docker compose logs prometheus --tail=100
72+
73+
- name: Stop Docker services
74+
if: always()
75+
working-directory: Examples/Compose
76+
run: docker compose down -v

‎Configs/loki-local-config.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,5 +26,8 @@ schema_config:
2626
prefix: index_
2727
period: 24h
2828

29+
limits_config:
30+
allow_structured_metadata: false
31+
2932
ruler:
3033
alertmanager_url: http://127.0.0.1:9093

‎E2ETests/.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Dependencies
2+
node_modules/
3+
4+
# Test artifacts
5+
coverage/
6+
7+
# IDE
8+
.vscode/
9+
.idea/

‎E2ETests/README.md‎

Lines changed: 277 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,277 @@
1+
# Buccaneer E2E Integration Tests
2+
3+
End-to-end integration tests that verify the complete metrics pipeline:
4+
5+
**TypeScript Client → Buccaneer Server → Prometheus**
6+
7+
## Overview
8+
9+
These tests push metrics using the Buccaneer TypeScript client, wait for Prometheus to scrape them, then query the Prometheus API to verify the metrics were stored correctly.
10+
11+
## Prerequisites
12+
13+
- Docker and Docker Compose installed
14+
- Node.js 20+ installed
15+
- Ports available: 8000 (Buccaneer), 9090 (Prometheus), 3000 (Grafana), 3100 (Loki), 9080 (Promtail)
16+
17+
## Quick Start
18+
19+
```bash
20+
# Install dependencies
21+
npm install
22+
23+
# Run E2E tests (automatically starts/stops Docker services)
24+
npm test
25+
```
26+
27+
## Manual Docker Management
28+
29+
```bash
30+
# Start services manually
31+
npm run docker:up
32+
33+
# Run tests without automatic Docker lifecycle
34+
jest
35+
36+
# View service logs
37+
npm run docker:logs
38+
39+
# Stop services manually
40+
npm run docker:down
41+
```
42+
43+
## How It Works
44+
45+
### Test Flow
46+
47+
1. **Setup Phase** (`beforeAll`):
48+
- Waits for Buccaneer server to be ready (up to 30 seconds)
49+
- Waits for Prometheus to be ready (up to 30 seconds)
50+
51+
2. **Test Execution**:
52+
- Creates `MetricsCollection` with test data
53+
- Sends metrics to Buccaneer server via HTTP POST
54+
- Waits for Prometheus to scrape metrics (polls every 2 seconds, up to 40 seconds)
55+
- Queries Prometheus API to retrieve metric values
56+
- Validates metrics match expected values
57+
58+
3. **Cleanup**:
59+
- Docker Compose stops and removes all containers
60+
- Volumes are cleaned up
61+
62+
### Prometheus Scraping
63+
64+
- **Scrape Interval**: 15 seconds (configured in `prometheus.yml`)
65+
- **Test Wait Logic**: Polls Prometheus every 2 seconds, max 20 attempts (40 seconds total)
66+
- **Why the wait?**: Metrics must go through: Client → Buccaneer → exposed /metrics endpoint → Prometheus scrape
67+
68+
## Test Coverage
69+
70+
The test suite covers:
71+
72+
### 1. Single-Value Metrics
73+
Tests basic metric push and retrieval:
74+
```typescript
75+
metrics.pushStat('fps', 60);
76+
// Verifies: fps = 60 in Prometheus
77+
```
78+
79+
### 2. Grouped Metrics
80+
Tests per-group metrics (e.g., per-player stats):
81+
```typescript
82+
metrics.pushStatByGroup('player_0', 'score', 100);
83+
metrics.pushStatByGroup('player_1', 'score', 200);
84+
// Verifies: score{group_id="player_0"} = 100
85+
// score{group_id="player_1"} = 200
86+
```
87+
88+
### 3. Multiple Batches
89+
Tests that multiple metric collections from the same instance work:
90+
```typescript
91+
// First batch
92+
const metrics1 = new MetricsCollection('instance-1');
93+
metrics1.pushStat('metric_a', 111);
94+
await metrics1.send(BUCCANEER_URL);
95+
96+
// Second batch
97+
const metrics2 = new MetricsCollection('instance-1');
98+
metrics2.pushStat('metric_b', 222);
99+
await metrics2.send(BUCCANEER_URL);
100+
// Verifies: Both metrics exist in Prometheus
101+
```
102+
103+
### 4. Metric Name Formatting
104+
Tests that special characters are properly converted:
105+
```typescript
106+
metrics.pushStat('Frame Time (ms)', 16.67);
107+
// Verifies: Frame_Time_ms_ = 16.67 in Prometheus
108+
```
109+
110+
## Prometheus HTTP API
111+
112+
The tests use Prometheus's HTTP API for verification:
113+
114+
### Query a Metric
115+
```bash
116+
curl "http://localhost:9090/api/v1/query?query=test_metric_123456"
117+
```
118+
119+
### Query with Labels
120+
```bash
121+
curl "http://localhost:9090/api/v1/query?query=score{group_id=\"player_0\"}"
122+
```
123+
124+
### Response Format
125+
```json
126+
{
127+
"status": "success",
128+
"data": {
129+
"resultType": "vector",
130+
"result": [
131+
{
132+
"metric": {
133+
"__name__": "test_metric_123456",
134+
"instance": "buccaneerserver:8000",
135+
"job": "buccaneer"
136+
},
137+
"value": [1638360000, "42.5"]
138+
}
139+
]
140+
}
141+
}
142+
```
143+
144+
## Troubleshooting
145+
146+
### Tests Timeout Waiting for Metrics
147+
148+
**Symptom**: Tests fail with "Metric not found after 40s"
149+
150+
**Solutions**:
151+
1. Check if Prometheus is scraping:
152+
- Open http://localhost:9090/targets
153+
- Verify Buccaneer target is "UP"
154+
155+
2. Check scrape interval:
156+
- Default is 15 seconds in `prometheus.yml`
157+
- Increase `testTimeout` in `jest.config.js` if needed
158+
159+
3. Verify metrics endpoint:
160+
- Open http://localhost:8000/metrics
161+
- Should show Prometheus format metrics
162+
163+
### Docker Services Won't Start
164+
165+
**Symptom**: Services fail to start or tests timeout during setup
166+
167+
**Solutions**:
168+
1. Check port conflicts:
169+
```bash
170+
netstat -ano | findstr "8000 9090"
171+
```
172+
173+
2. View container logs:
174+
```bash
175+
npm run docker:logs
176+
```
177+
178+
3. Manually verify services:
179+
```bash
180+
docker compose -f ../Examples/Compose/docker-compose.yml ps
181+
```
182+
183+
### Metrics Not Appearing in Prometheus
184+
185+
**Symptom**: Metrics sent but not found in Prometheus
186+
187+
**Solutions**:
188+
1. Check Buccaneer server logs:
189+
```bash
190+
docker compose -f ../Examples/Compose/docker-compose.yml logs buccaneerserver
191+
```
192+
193+
2. Verify metric was received:
194+
- Check logs for POST to `/stats`
195+
- Should see metric registration
196+
197+
3. Check Prometheus targets:
198+
- http://localhost:9090/targets
199+
- Buccaneer should be listed and UP
200+
201+
4. Manually query Prometheus:
202+
- http://localhost:9090/graph
203+
- Try querying: `{job="buccaneer"}`
204+
205+
### Test Hanging or Never Completing
206+
207+
**Symptom**: Jest process doesn't exit
208+
209+
**Solutions**:
210+
1. Kill Docker containers:
211+
```bash
212+
npm run docker:down
213+
```
214+
215+
2. Check for open handles:
216+
- Add `--detectOpenHandles` to jest command
217+
- Look for unclosed connections
218+
219+
## CI/CD Integration
220+
221+
The tests are designed to run in GitHub Actions. See `.github/workflows/e2e-tests.yml` for the workflow configuration.
222+
223+
Key points:
224+
- Services start with `--wait` flag to ensure readiness
225+
- Uses `timeout` commands for health checks
226+
- Logs are captured on failure for debugging
227+
- Cleanup runs even if tests fail (`if: always()`)
228+
229+
## Test Output
230+
231+
Successful test run shows:
232+
```
233+
=== Setting up E2E Test Environment ===
234+
235+
Checking Buccaneer server...
236+
✓ Buccaneer server is ready
237+
238+
Checking Prometheus...
239+
✓ Prometheus is ready
240+
241+
=== Environment Ready ===
242+
243+
--- Test: Single-value metrics ---
244+
245+
Sending metric: test_single_metric_1638360000 = 42.5
246+
✓ Metric sent to Buccaneer server
247+
248+
Waiting for metric "test_single_metric_1638360000" to appear in Prometheus...
249+
✓ Metric found after 16s
250+
251+
✓ Verified in Prometheus: test_single_metric_1638360000 = 42.5
252+
253+
PASS tests/metrics-pipeline.test.ts
254+
✓ should push single-value metrics and verify in Prometheus (18234 ms)
255+
✓ should push grouped metrics and verify in Prometheus (19456 ms)
256+
✓ should handle multiple metric batches from same instance (22178 ms)
257+
✓ should handle metrics with special characters in names (18903 ms)
258+
259+
Test Suites: 1 passed, 1 total
260+
Tests: 4 passed, 4 total
261+
```
262+
263+
## Contributing
264+
265+
When adding new E2E tests:
266+
267+
1. Use unique metric names (timestamp-based) to avoid conflicts
268+
2. Add descriptive console.log statements for test progress
269+
3. Use the helper functions: `waitForMetric()`, `getMetricValue()`, `queryPrometheus()`
270+
4. Set appropriate timeouts for your test scenario
271+
5. Clean up any test data if needed
272+
273+
## Related Documentation
274+
275+
- [Buccaneer Metrics Client](../Web/metrics-client/README.md)
276+
- [Prometheus Querying](https://prometheus.io/docs/prometheus/latest/querying/basics/)
277+
- [Docker Compose Configuration](../Examples/Compose/README.md)

‎E2ETests/jest.config.js‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
/** @type {import('jest').Config} */
2+
export default {
3+
preset: 'ts-jest/presets/default-esm',
4+
testEnvironment: 'node',
5+
roots: ['<rootDir>/tests'],
6+
testMatch: ['**/*.test.ts'],
7+
extensionsToTreatAsEsm: ['.ts'],
8+
moduleNameMapper: {
9+
'^(\\.{1,2}/.*)\\.js$': '$1',
10+
},
11+
transform: {
12+
'^.+\\.ts$': [
13+
'ts-jest',
14+
{
15+
useESM: true,
16+
},
17+
],
18+
},
19+
testTimeout: 60000, // 60 seconds for Docker operations and Prometheus scraping
20+
verbose: true
21+
};

0 commit comments

Comments
 (0)