Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
1395978	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127411/merge to s	Log	dotnet/runtime#127411
1395831	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127403/merge to s	Log	dotnet/runtime#127403
1395700	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127399/merge to s	Log	dotnet/runtime#127399
2959579	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/60428/merge to s	Log	#60428
1395170	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127388/merge to s	Log	dotnet/runtime#127388
1395207	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126701/merge to s	Log	dotnet/runtime#126701
1394113	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16750/merge to s	Log	#16750
1394114	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16751/merge to s	Log	#16751
1393975	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127352/merge to s	Log	dotnet/runtime#127352
1394066	dotnet/installer	Checkout dotnet/installer@refs/pull/20856/merge to s	Log	dotnet/installer#20856
1394065	dotnet/installer	Checkout dotnet/installer@refs/pull/20855/merge to s	Log	dotnet/installer#20855
1394112	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16749/merge to s	Log	#16749
1393800	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16743/merge to s	Log	#16743
1391397	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127288/merge to s	Log	dotnet/runtime#127288
1391392	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127263/merge to s	Log	dotnet/runtime#127263
1391442	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54043/merge to s	Log	dotnet/sdk#54043
1391181	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66417/merge to s	Log	dotnet/aspnetcore#66417
1391145	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54039/merge to s	Log	dotnet/sdk#54039
1391157	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125651/merge to s	Log	dotnet/runtime#125651
1391130	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127278/merge to s	Log	dotnet/runtime#127278
1390512	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1390331	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127252/merge to s	Log	dotnet/runtime#127252
1389781	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127233/merge to s	Log	dotnet/runtime#127233
1386132	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127119/merge to s	Log	dotnet/runtime#127119
1384924	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126596/merge to s	Log	dotnet/runtime#126596
1385054	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127100/merge to s	Log	dotnet/runtime#127100
1384746	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126752/merge to s	Log	dotnet/runtime#126752
1384205	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2532/merge to s	Log	maestro-auth-test/build-result-analysis-test#2532
1384206	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2532/merge to s	Log	maestro-auth-test/build-result-analysis-test#2532
1384202	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2531/merge to s	Log	maestro-auth-test/build-result-analysis-test#2531
1384201	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2531/merge to s	Log	maestro-auth-test/build-result-analysis-test#2531
1383814	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127056/merge to s	Log	dotnet/runtime#127056
2953522	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/60202/merge to s	Log	#60202
1383137	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127032/merge to s	Log	dotnet/runtime#127032
1382907	dotnet/installer	Checkout dotnet/installer@refs/pull/20848/merge to s	Log	dotnet/installer#20848
1382747	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127024/merge to s	Log	dotnet/runtime#127024
1382359	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127018/merge to s	Log	dotnet/runtime#127018
1382103	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126959/merge to s	Log	dotnet/runtime#126959
1381750	dotnet/maui	Checkout dotnet/maui@refs/pull/34907/merge to s	Log	dotnet/maui#34907
1381977	dotnet/runtime	Checkout dotnet/runtime@refs/pull/127003/merge to s	Log	dotnet/runtime#127003
1381933	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2527/merge to s	Log	maestro-auth-test/build-result-analysis-test#2527
1381932	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2527/merge to s	Log	maestro-auth-test/build-result-analysis-test#2527
2952528	dotnet-dotnet	Checkout dotnet-dotnet@release/10.0.1xx to s	Log
1381131	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66333/merge to s	Log	dotnet/aspnetcore#66333
1381038	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126971/merge to s	Log	dotnet/runtime#126971
1380587	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126959/merge to s	Log	dotnet/runtime#126959
1379927	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53883/merge to s	Log	dotnet/sdk#53883
1379856	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125516/merge to s	Log	dotnet/runtime#125516
1379734	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126937/merge to s	Log	dotnet/runtime#126937
1378920	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66317/merge to s	Log	dotnet/aspnetcore#66317
1379062	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126912/merge to s	Log	dotnet/runtime#126912
2951001	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
1378197	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2521/merge to s	Log	maestro-auth-test/build-result-analysis-test#2521
1378196	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2521/merge to s	Log	maestro-auth-test/build-result-analysis-test#2521
1377732	dotnet/sdk	Checkout dotnet/sdk@main to s	Log
1377255	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126849/merge to s	Log	dotnet/runtime#126849
1374643	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53464/merge to s	Log	dotnet/sdk#53464
2948283	dotnet-maui	Checkout dotnet-maui@main to s	Log
1374239	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126713/merge to s	Log	dotnet/runtime#126713
1372766	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53776/merge to s	Log	dotnet/sdk#53776
1372400	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53781/merge to s	Log	dotnet/sdk#53781
1372304	dotnet/maui	Checkout dotnet/maui@inflight/current to s	Log
1372351	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126710/merge to s	Log	dotnet/runtime#126710
1371280	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66237/merge to s	Log	dotnet/aspnetcore#66237
1371282	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66237/merge to s	Log	dotnet/aspnetcore#66237
1371281	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66237/merge to s	Log	dotnet/aspnetcore#66237
1370762	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126658/merge to s	Log	dotnet/runtime#126658
1370251	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66217/merge to s	Log	dotnet/aspnetcore#66217
1369438	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1368706	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53730/merge to s	Log	dotnet/sdk#53730
1365485	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126506/merge to s	Log	dotnet/runtime#126506
1364885	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53175/merge to s	Log	dotnet/sdk#53175
1364883	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53681/merge to s	Log	dotnet/sdk#53681
1364884	dotnet/sdk	Checkout dotnet/sdk@refs/pull/53175/merge to s	Log	dotnet/sdk#53175
1364162	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124650/merge to s	Log	dotnet/runtime#124650
2942034	dotnet-aspnetcore	Checkout dotnet-aspnetcore@refs/pull/59735/merge to s	Log	#59735
1362725	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1362435	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126422/merge to s	Log	dotnet/runtime#126422
1362429	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126421/merge to s	Log	dotnet/runtime#126421
1360266	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65912/merge to s	Log	dotnet/aspnetcore#65912
1360167	dotnet/runtime	Checkout dotnet/runtime@refs/pull/125437/merge to s	Log	dotnet/runtime#125437
2939721	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/59562/merge to s	Log	#59562
1358347	dotnet/runtime	Checkout dotnet/runtime@refs/pull/124544/merge to s	Log	dotnet/runtime#124544
1357834	dotnet/maui	Checkout dotnet/maui@refs/pull/29344/merge to s	Log	dotnet/maui#29344
1355056	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66013/merge to s	Log	dotnet/aspnetcore#66013
1355085	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66025/merge to s	Log	dotnet/aspnetcore#66025
1355460	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126213/merge to s	Log	dotnet/runtime#126213
1355231	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126208/merge to s	Log	dotnet/runtime#126208
1355071	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66021/merge to s	Log	dotnet/aspnetcore#66021
1355062	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66017/merge to s	Log	dotnet/aspnetcore#66017
1355066	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66019/merge to s	Log	dotnet/aspnetcore#66019
1355074	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66023/merge to s	Log	dotnet/aspnetcore#66023
1355049	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66011/merge to s	Log	dotnet/aspnetcore#66011
1355060	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/66015/merge to s	Log	dotnet/aspnetcore#66015
1354790	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16636/merge to s	Log	#16636

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	24	95

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?