Refine L3 promotion criteria for downstream repositories by fffrog · Pull Request #102 · pytorch/rfcs

fffrog · 2026-07-22T11:53:05Z

Summary

This PR clarifies the L3 promotion criteria for downstream repositories by:

Defining infrastructure metrics more precisely (queue time, execution time, timeout rate).
Clarifying that the pass rate refers to CI job success rate, not individual test pass rate..

fffrog · 2026-07-22T11:53:46Z

Hi @albanD @atalman, please help to take a look at this when you have a chance.

Replace the "Flaky Jobs" stat card with a "Timeout Rate" percentage card on the CRCR downstream repo page, aligning with the L3 promotion criteria proposed in pytorch/rfcs#102 which defines timeout rate < 10% as an infrastructure metric. The timeout rate is computed as timed_out / total_jobs directly in the ClickHouse query, replacing the complex flaky-job subquery. The card shows the percentage with color coding: green (0%), orange (>0%), red (≥10% — exceeds L3 threshold).

Update the Pass Rate card color coding to match the L3 promotion criteria from pytorch/rfcs#102 (job pass rate > 90%): - Green: 100% (perfect) - Orange: 90–99.9% (meets L3 threshold) - Red: < 90% (below L3 threshold) Previously the thresholds were ≥95% green, ≥80% orange, <80% red.

atalman · 2026-08-03T19:35:16Z

 > \[!NOTE\]
-> - The requirements above are an **initial reference** and may **be adjusted over time based on real-world conditions** (e.g., determining the specific values of `X`).
+> - The requirements above are an **initial reference** and may **be adjusted over time based on real-world conditions**.
 > - To maintain the PyTorch community's user experience, **downstream repos that no longer meet the requirements of their current level will be downgraded to the level that matches their actual status.**


HI @fffrog maybe we should also callout de-motion Criteria here. Also would be nice to document temporary downgrade process.

Another point to consider, and maybe add here is that we probably want to add possibility of temporary downgrades for certain repos. For example when dealing with SEV's.

Agree on that. We can also write a script to automatically upgrade or downgrade the repos so that we don't need to waste our efforts tracking whether a repo meets the criteria. WDYT?

Hi @atalman, thank you for those valuable suggestions, and I have added new sections named Promotion and Demotion, especially, add a new mechanism to achieve the temporary downgrade.

@can-gaa-hou Thank you.

Sure, based on the new mechanism of temporary downgrade, we can prepare a script to collect the necessary info to determine whether some repo should be downgraded.

Replace the "Flaky Jobs" stat card with a "Timeout Rate" percentage card on the CRCR downstream repo page, aligning with the L3 promotion criteria proposed in pytorch/rfcs#102 which defines timeout rate < 10% as an infrastructure metric. The timeout rate is computed as timed_out / total_jobs directly in the ClickHouse query, replacing the complex flaky-job subquery. The card shows the percentage with color coding: green (0%), orange (>0%), red (≥10% — exceeds L3 threshold).

Update the Pass Rate card color coding to match the L3 promotion criteria from pytorch/rfcs#102 (job pass rate > 90%): - Green: 100% (perfect) - Orange: 90–99.9% (meets L3 threshold) - Red: < 90% (below L3 threshold) Previously the thresholds were ≥95% green, ≥80% orange, <80% red.

…colors with L3 criteria (#8421) ## Summary Two changes to the CRCR per-repo dashboard page (`/crcr/{org}/{repo}`), aligning stat card thresholds with the L3 promotion criteria proposed in [pytorch/rfcs#102](pytorch/rfcs#102). ### 1. Replace Flaky Jobs card → Timeout Rate **ClickHouse query** (`crcr_backend_summary/query.sql`): - Removed the complex flaky-job subquery (correlated subquery scanning the table twice) - Added `timeout_rate = timed_out / total_jobs` **Frontend card**: - Shows percentage (e.g., `0.0%`, `1.2%`) - Sub-text: `X timed out / Y jobs` - Color: green (0%), orange (>0% but <10%), red (≥10% — exceeds L3 threshold) ### 2. Align Pass Rate color thresholds Updated to match L3 criteria (`job pass rate > 90%`): - **Green**: 100% - **Orange**: 90–99.9% (meets L3) - **Red**: < 90% (below L3) Previously: ≥95% green, ≥80% orange, <80% red. ## Context The L3 criteria in [pytorch/rfcs#102](pytorch/rfcs#102) defines: - **Timeout rate < 10%** as an infrastructure reliability signal - **Job pass rate > 90%** as a test quality signal These cards make L3 readiness directly visible on the per-repo dashboard. ## Test plan - [ ] Verify `crcr_backend_summary` query returns `timeout_rate` correctly - [ ] Verify Timeout Rate card renders with proper color coding - [ ] Verify Pass Rate card shows red when < 90%, orange when 90–99.9%, green at 100%

can-gaa-hou · 2026-08-05T03:13:41Z

+| Metric | Target | Description |
+| :--- | :--- | :--- |
+| Max execution time | < 4 h | The longest "run" phase of any single job (excludes queue wait). |
+| Avg queue time | < 30 min | Average time a job waits before a runner picks it up (excludes execution). |


Question here: for most backends, they need to build PyTorch first, then they can run their first test, which means most of their test queue time includes PyTorch building time. So the average queue time definitely exceeds 30 mins. cc @fffrog @albanD @atalman

Thank you for asking this question. I've already sought help in this channel (https://pytorch.slack.com/archives/C0BBJK4B1EJ/p1785912202488279).

Additionally, I'd like to share my personal thoughts.

I believe that avg queue time here is just a reference value, but it is indeed a necessary metric. So, what we need to discuss is how to calculate the correct value, right?

fffrog · 2026-08-05T10:58:03Z

Hi @albanD @atalman @can-gaa-hou , the new commit is ready, please help to take a look at it when you have a time. thank you in advance.

expand the criteria for Level 3

80fba2b

meta-cla Bot added the cla signed label Jul 22, 2026

fffrog commented Jul 22, 2026

View reviewed changes

Comment thread RFC-0050-Cross-Repository-CI-Relay-for-PyTorch-Out-of-Tree-Backends.md Outdated

fffrog commented Jul 22, 2026

View reviewed changes

Comment thread RFC-0050-Cross-Repository-CI-Relay-for-PyTorch-Out-of-Tree-Backends.md Outdated

can-gaa-hou mentioned this pull request Jul 28, 2026

[CRCR] Draft L3/L4 admission criteria pytorch/test-infra#8313

Open

jathu requested review from albanD and atalman July 28, 2026 16:28

albanD reviewed Jul 28, 2026

View reviewed changes

Comment thread RFC-0050-Cross-Repository-CI-Relay-for-PyTorch-Out-of-Tree-Backends.md Outdated

Comment thread RFC-0050-Cross-Repository-CI-Relay-for-PyTorch-Out-of-Tree-Backends.md Outdated

Comment thread RFC-0050-Cross-Repository-CI-Relay-for-PyTorch-Out-of-Tree-Backends.md Outdated

update criteria for L3

f9e9cdb

fffrog requested a review from albanD August 3, 2026 08:40

subinz1 mentioned this pull request Aug 3, 2026

[CRCR] Replace Flaky Jobs card with Timeout Rate on per-repo page pytorch/test-infra#8420

Closed

3 tasks

subinz1 mentioned this pull request Aug 3, 2026

[CRCR] Replace Flaky Jobs card with Timeout Rate and align Pass Rate colors with L3 criteria pytorch/test-infra#8421

Merged

3 tasks

jewelkm89 mentioned this pull request Aug 3, 2026

[CRCR][HUD] Create / align metrics in CRCR repo level page to L3 criteria pytorch/test-infra#8423

Closed

atalman requested changes Aug 3, 2026

View reviewed changes

Comment thread RFC-0050-Cross-Repository-CI-Relay-for-PyTorch-Out-of-Tree-Backends.md Outdated

atalman reviewed Aug 3, 2026

View reviewed changes

can-gaa-hou reviewed Aug 5, 2026

View reviewed changes

add the demotion and improve the criteria for L3

05e24ba

fffrog requested a review from atalman August 5, 2026 10:57

Navigation Menu

Uh oh!

Refine L3 promotion criteria for downstream repositories - #102

Refine L3 promotion criteria for downstream repositories#102
fffrog wants to merge 3 commits into
pytorch:masterfrom
fffrog:relay

fffrog commented Jul 22, 2026

Uh oh!

fffrog commented Jul 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

atalman Aug 3, 2026

Uh oh!

atalman Aug 3, 2026

Uh oh!

can-gaa-hou Aug 4, 2026

Uh oh!

fffrog Aug 5, 2026

Uh oh!

fffrog Aug 5, 2026

Uh oh!

can-gaa-hou Aug 5, 2026

Uh oh!

fffrog Aug 5, 2026

Uh oh!

fffrog commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

fffrog commented Jul 22, 2026

Summary

Uh oh!

fffrog commented Jul 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

atalman Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

atalman Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

can-gaa-hou Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

fffrog Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

fffrog Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

can-gaa-hou Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

fffrog Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

fffrog commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants