feat(validator): make OSS credibility volume-aware and gate-only#1359
feat(validator): make OSS credibility volume-aware and gate-only#1359plind-junior wants to merge 5 commits into
Conversation
Closes entrius#1340. Credibility (merged/(merged+closed)) did two jobs — a 0.80 hard gate and a per-PR multiplier — and both punished active contributors. A miner at 30 merged / 8 closed scored 0.79 and was rejected outright, while 3/0 earned a free 1.0; and anyone past the gate was still taxed by the same ratio on every PR (charged twice). Smoothing prior: credibility = (merged + k) / (merged + closed + 2k), k=CREDIBILITY_SMOOTHING_PRIOR=2. 3/0 -> 0.71 (no free pass), 30/8 -> 0.76 (no death sentence). MIN_CREDIBILITY retuned 0.80 -> 0.70 to match the new scale (still per-repo overridable). Gate-only: _score_eligible_repo_prs no longer multiplies earned score by credibility (credibility_multiplier=1.0). Past the gate, PRs stand on review-quality, label, and time-decay multipliers. repo_eval.credibility is still computed and reported. Scope: OSS-contribution credibility only, matching the issue's cited code. The issue-discovery analogue (calculate_issue_credibility + discovery_credibility_multiplier) has the same shape and is left as a follow-up.
|
I appreciate the work on this. Early in my Gittensor run, the maintainer of the main project I contribute to did a major rebase / fork-network switch. That left 4 of my open PRs effectively unrecoverable, so they had to be closed despite the work still being valid. It was pretty painful to get hit that hard on credibility while also having substantially more merged PRs overall. This change feels more balanced. It still filters for contributors with a real track record, but it’s a lot more forgiving for people who are consistently shipping and occasionally get caught by repo/process churn outside their control. Cheers 🔥 |
|
Mostly what I like from the issue is this:
can you scope this down to JUST that? e.g. remove credibility as the per-PR multiplier, I'm not fond of volume base credibility, it again almost tries to promote inundating maintainers by trying to just open more PRs in hopes of raising your weighted credibility tl;dr just remove credibility as a per PR multiplier, have it just gate eligibility |
Closes #1340. Credibility (merged/(merged+closed)) did two jobs — a 0.80 hard gate and a per-PR multiplier — and both punished active contributors. A miner at 30 merged / 8 closed scored 0.79 and was rejected outright, while 3/0 earned a free 1.0; and anyone past the gate was still taxed by the same ratio on every PR (charged twice).
Smoothing prior: credibility = (merged + k) / (merged + closed + 2k), k=CREDIBILITY_SMOOTHING_PRIOR=2. 3/0 -> 0.71 (no free pass), 30/8 -> 0.76 (no death sentence). MIN_CREDIBILITY retuned 0.80 -> 0.70 to match the new scale (still per-repo overridable).
Gate-only: _score_eligible_repo_prs no longer multiplies earned score by credibility (credibility_multiplier=1.0). Past the gate, PRs stand on review-quality, label, and time-decay multipliers. repo_eval.credibility is still computed and reported.
Scope: OSS-contribution credibility only, matching the issue's cited code. The issue-discovery analogue (calculate_issue_credibility + discovery_credibility_multiplier) has the same shape and is left as a follow-up.