Skip to content

Human in the loop#2443

Draft
IdirLISN wants to merge 1 commit into
developfrom
feature/HITL
Draft

Human in the loop#2443
IdirLISN wants to merge 1 commit into
developfrom
feature/HITL

Conversation

@IdirLISN

Copy link
Copy Markdown
Collaborator

@ mention of reviewers

@acletournel
@ObadaS
@wlln
@Didayolo

A brief description of the purpose of the changes contained in this PR.

Human in the loop feature (HITL), enables organizers to add this option for their competitions
image

If this option is enabled, each submission will require a validation on compute worker side.
image

If no validation occurs, a time out will return a submission failed status
If the compute worker administrator validates the scoring file, the submission returns a scoring to codabench.

Issues this PR resolves

It secures the scoring process.

A checklist for hand testing

  • Create a submission and click on HITL button, see screenshot.
  • Submit and check compute worker logs.
  • Check the scoring file, path displayed in compute worker logs.
  • Choose to validate or not.
  • Check Codabench interface and make sure the score is returned or not in regards of the choice made in the last step

Caution

The feature still need some polish

Checklist

  • Code review by me
  • Hand tested by me
  • I'm proud of my work
  • Code review by reviewer
  • Hand tested by reviewer
  • CircleCi tests are passing
  • Ready to merge
    S

@IdirLISN IdirLISN marked this pull request as draft June 24, 2026 12:58
@IdirLISN IdirLISN self-assigned this Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant