Skip to content

Update safe_eval config to gigaflow-matching evaluation settings #377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
eugenevinitsky wants to merge 1 commit into from
Closed

Update safe_eval config to gigaflow-matching evaluation settings #377

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 30 additions & 16 deletions pufferlib/config/ocean/drive.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,36 +228,50 @@ human_replay_num_agents = 16
enabled = True
; How often to run safe eval (in training epochs). Defaults to render_interval.
interval = 250
; Number of agents to run in the eval environment
num_agents = 64
; Number of agents to run in the eval environment (Na=50)
num_agents = 50
; Number of episodes to collect metrics over
num_episodes = 100
; episode length
episode_length = 1000
; Episode length: 9000 steps = 600s at dt=0.066
episode_length = 9000
; Map directory and count for safe eval (independent of training maps)
map_dir = "resources/drive/binaries/carla_2D"
num_maps = 8
min_goal_distance = 0.5
max_goal_distance = 1000.0
; dt=0.066 so 9000 steps = 600s
Comment on lines +235 to +242
Copy link

Copilot AI Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says “9000 steps = 600s at dt=0.066”, but 9000 * 0.066 = 594s. If the intent is a 600s episode, either adjust dt (e.g., 1/15) or episode_length, or update the comment to reflect the actual duration.

Suggested change
; Episode length: 9000 steps = 600s at dt=0.066
episode_length = 9000
; Map directory and count for safe eval (independent of training maps)
map_dir = "resources/drive/binaries/carla_2D"
num_maps = 8
min_goal_distance = 0.5
max_goal_distance = 1000.0
; dt=0.066 so 9000 steps = 600s
; Episode length: 9000 steps = 594s at dt=0.066
episode_length = 9000
; Map directory and count for safe eval (independent of training maps)
map_dir = "resources/drive/binaries/carla_2D"
num_maps = 8
min_goal_distance = 0.5
max_goal_distance = 1000.0
; dt=0.066 so 9000 steps = 594s

Copilot uses AI. Check for mistakes.
dt = 0.066
; vgoal = 3 m/s
Copy link

Copilot AI Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The config/comment indicates vgoal = 3 m/s, but with min_goal_speed = -0.01 and max_goal_speed = 3.0 the goal-speed condition effectively becomes “speed < 3.0” (no lower bound), not “speed ≈ 3”. Please clarify the intended behavior in the comment or tighten the bounds if the target is a specific goal speed.

Suggested change
; vgoal = 3 m/s
; permissible speed range: [-0.01, 3.0] m/s (v_max = 3 m/s)

Copilot uses AI. Check for mistakes.
min_goal_speed = -0.01
max_goal_speed = 3.0

; Reward conditioning values (min=max to fix the value).
; Names match the env reward_bound_* keys.
; High penalties for unsafe behavior
; Gigaflow-matching reward conditioning values (min=max to fix the value).
; αcollision = 3.0
collision = -3.0
; αboundary = 3.0
offroad = -3.0
overspeed = -1.0
; αstop-line = 1.0
traffic_light = -1.0
reverse = -0.0075
comfort = -0.1

; Standard driving rewards
goal_radius = 2.0
overspeed = -1.0
; αreverse = 5.0e-3
reverse = -0.005
; αcomfort = 0.05
comfort = -0.05

; δgoal = 10m
goal_radius = 10.0
; αl-align = 2.5e-2
lane_align = 0.025
lane_center = -0.00075
velocity = 0.005
; αl-center = 3.8e-3
lane_center = -0.0038
; αvelocity = 2.5e-3
velocity = 0.0025
; αcenter-bias = 0.0
center_bias = 0.0
; αvel-align = 1.0
vel_align = 1.0
timestep = -0.00005
; αtimestep = 2.5e-5
timestep = -0.000025

; Neutral scaling factors
throttle = 1.0
Expand Down
13 changes: 12 additions & 1 deletion pufferlib/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,18 @@ def generate_safe_eval_ini(safe_eval_config, base_ini_path="pufferlib/config/oce
"reward_conditioning": 1,
"resample_frequency": 0,
}
for key in ["episode_length", "num_agents", "min_goal_distance", "max_goal_distance", "map_dir", "num_maps"]:
for key in [
"episode_length",
"num_agents",
"min_goal_distance",
"max_goal_distance",
"map_dir",
Comment on lines +45 to +50
Copy link

Copilot AI Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says safe eval now forwards dt/goal_radius/goal speeds so safe eval episodes use the correct settings. This change only affects the temporary INI used for rendering; the actual safe-eval metrics path uses SafeEvaluator._build_eval_env_config (called from pufferl.py) and it currently only forwards map_dir/num_maps/min/max_goal_distance, so dt/goal_radius/min_goal_speed/max_goal_speed will still stay at the base [env] defaults during evaluation.

Copilot uses AI. Check for mistakes.
Comment on lines +45 to +50
Copy link

Copilot AI Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests cover generate_safe_eval_ini env overrides, but they don’t assert the newly forwarded keys (goal_radius, dt, min_goal_speed, max_goal_speed). Please extend the existing tests (e.g., in tests/test_utils.py or tests/test_generate_env_ini.py) to cover these keys so regressions in safe-eval settings are caught.

Copilot uses AI. Check for mistakes.
"num_maps",
"goal_radius",
"dt",
"min_goal_speed",
"max_goal_speed",
]:
if key in safe_eval_config:
env_overrides[key] = safe_eval_config[key]

Expand Down
Loading