Human in the loop#2443
Draft
IdirLISN wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@ mention of reviewers
@acletournel
@ObadaS
@wlln
@Didayolo
A brief description of the purpose of the changes contained in this PR.
Human in the loop feature (HITL), enables organizers to add this option for their competitions

If this option is enabled, each submission will require a validation on compute worker side.

If no validation occurs, a time out will return a submission failed status
If the compute worker administrator validates the scoring file, the submission returns a scoring to codabench.
Issues this PR resolves
It secures the scoring process.
A checklist for hand testing
Caution
The feature still need some polish
Checklist
S