Skip to content

application-validation.md

Latest commit

 

History

History
78 lines (59 loc) · 2.41 KB

application-validation.md

File metadata and controls

78 lines (59 loc) · 2.41 KB
Raw

Application validation

KernelBot can run a small, problem-owned workload against the current top 10 submissions. A problem opts in with two fields in reference-kernels:

validation:
  version: cholesky-natural-gradient-v1
  script: validation.py

The version invalidates old results when the workload changes. KernelBot loads the script beside the problem, adds it to the problem's normal source files, and runs it on B200 through the existing Modal app.

The script receives the version in KERNELBOT_VALIDATION_CONFIG and prints one JSON result:

{
  "passed_shapes": 8,
  "total_shapes": 8,
  "fully_validated": true,
  "geomean_sync_wall_speedup": 1.42,
  "results": []
}

Schedule

Every day at 22:00 America/Los_Angeles, one KernelBot replica claims each (leaderboard, GPU, contract version, local date) sweep. It snapshots the current best submission from each of the top 10 users and runs at most two Modal jobs concurrently. The database claim prevents duplicate sweeps.

Set APPLICATION_VALIDATION_ENABLED=false to disable the scheduler. A failed job is recorded as VALIDATION ERROR; a completed job is stored as X/Y VALIDATED.

Local debug

Modal already reads MODAL_ENVIRONMENT, so KernelBot does not need separate environment plumbing:

modal environment create cholesky-validation-debug
modal deploy --env cholesky-validation-debug src/runners/modal_runner_archs.py

MODAL_ENVIRONMENT=cholesky-validation-debug \
APPLICATION_VALIDATION_ENABLED=false \
python src/kernelbot/main.py --api-only --debug

Run one sweep synchronously:

kernelbot-admin --api-url http://localhost:8000 \
  validate-top10 cholesky B200 --wait

Omit --wait to enqueue the sweep and return immediately. The command reads ADMIN_TOKEN and, unless --api-url is supplied, DISCORD_CLUSTER_MANAGER_API_BASE_URL from the environment.

For a one-time backfill of every ranked user's current best submission, add --all-users:

kernelbot-admin validate-top10 cholesky B200 --all-users --only-missing

This override applies only to the manual sweep. Scheduled sweeps remain capped at the top 10 users. --only-missing skips exact submissions that already have a result for the active validation contract, making it suitable for continuing a backfill after leaderboard changes.

Roll out KernelBot's migration and runner first, then the reference-kernels contract, then the Kernelboard badge.