Help requested - Inconsistent behaviour between hosts

[alba02119451]

Hi everyone, I'm facing an issue where the same command has different effects on different hosts. Since I don't think it's an actual bug but an issue on my end I'm posting here, but I'm happy to move this elsewhere if it is wrong. Below, I will explain the setup and issue, then I will list everything that I have tried, and finally some suspicions/notes. I would like some pointers to help me suss out what the exact issue is, or an outright solution if you happen to have one.

The setup

I have a small collection of Windows 11 24H2 VMs with winget installed. They all have some packages in common, but some machines are also a bit more specialized. We provisioned these machines using Ansible, which orchestrates the software installation via winget. We wanted to expand our toolset by creating an Ansible role that uses winget to automatically update all installed packages. This proved to be a lot easier in principle than in practice.

The issue

While all the hosts were created at the same time, using the same VM template, and started out with the same software, a few months of manual updates may have caused some issues with updates, as these are inconsistent between machines. For example, on one machine, I can download the Microsoft.VCRedist.2015+.x64 package without issue:

Microsoft Windows [Version 10.0.26100.8246]
(c) Microsoft Corporation. All rights reserved.

C:\Users\my_user>winget download Microsoft.VCRedist.2015+.x64
Found Microsoft Visual C++ v14 Redistributable (x64) [Microsoft.VCRedist.2015+.x64] Version 14.50.35719.0
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Downloading https://download.visualstudio.microsoft.com/download/pr/6f02464a-5e9b-486d-a506-c99a17db9a83/8995548DFFFCDE7C49987029C764355612BA6850EE09A7B6F0FDDC85BDC5C280/VC_redist.x64.exe
  ██████████████████████████████  17.6 MB / 17.6 MB
Successfully verified installer hash
Installer downloaded: C:\Users\abaars\Downloads\Microsoft.VCRedist.2015+.x64_14.50.35719.0\Microsoft Visual C++ v14 Redistributable (x64)_14.50.35719.0_Machine_X64_burn_en-US.exe

However, on two other machines, it yields a 404 error:

Name                                                    Id                           Version       Available     Source
-----------------------------------------------------------------------------------------------------------------------
Microsoft Visual C++ 2015-2022 Redistributable (x86) -… Microsoft.VCRedist.2015+.x86 14.40.33816.0 14.50.35719.0 winget
Microsoft Visual C++ 2015-2022 Redistributable (x64) -… Microsoft.VCRedist.2015+.x64 14.40.33816.0 14.50.35719.0 winget
2 upgrades available.
An unexpected error occurred while executing the command: 
Download request status is not success.
0x80190194 : Not found (404).

I also have this issue with RedHat.Podman, KeePassXCTeam.KeePassXC (dependent on the VCRedist package), and Microsoft.Powershell, but not with Notepad++.Notepad++ and Postman.Postman.

What I've done/tried so far

Updating winget

My first observation was that winget was out of date on the machines that were causing these issues. The current stable release is 1.28.240. On one machine it was 1.28.220, and on the other it was 1.12.470. I have no clue how such an old client came on one of them, but alas. I suspect this is either the culprit or another symptom of the underlying problem. So, naturally, I ensured that winget was up-to-date on all machines. Another notable fact is that winget upgrade winget and winget upgrade Microsoft.AppInstaller did not work, both of them returned that there were no updates available. Downloading the newest version at https://aka.ms/getwinget and installing it did overwrite/update winget though. However, this did not change the result

Updating and resetting my sources

So, winget itself wasn't able to see new updates, leading me to believe there was something wrong with the repositories/sources itself. So I used winget source update, which didn't help. I also tried winget source reset with and without the --force flag to no avail. Strangely, checking the sources with winget source list produces the same output on a working and non-working host:

Name        Argument                                      Explicit
------------------------------------------------------------------
msstore     https://storeedgefd.dsx.mp.microsoft.com/v9.0 false
winget      https://cdn.winget.microsoft.com/cache        false
winget-font https://cdn.winget.microsoft.com/fonts        true

So, likely it's not a source issue either.

Elevated v/s non-elevated prompt

I did come across this issue and several similar ones that suggested that the issue may arise from trying to update with elevated rights. I was sadly unable to find a way to "lower" my rights in Ansible, but I did test the download of that package on a working host with a normal and administrator prompt, and the download succeeded in both cases.

Changing scope

In my search I did discover the --scope tag. This also did not help, as the example package is installed only under the machine scope, and running with that scope simply results in the same error. I did get an error for the windows terminal programme under the user scope though:

Name                  Id                        Version      Available    Source
--------------------------------------------------------------------------------
Podman Desktop 1.25.1 RedHat.Podman-Desktop     1.25.1       1.26.2       winget
Windows Terminal      Microsoft.WindowsTerminal 1.23.20211.0 1.24.10921.0 winget
2 upgrades available.
1 package(s) have pins that prevent upgrade. Use the 'winget pin' command to view and edit pins. Using the --include-pinned argument may show more results.

Installing dependencies:
This package requires the following dependencies:
 - Packages
    Microsoft.UI.Xaml.2.8 [>= 8.2306.22001.0]
(1/1) Found Windows Terminal [Microsoft.WindowsTerminal] Version 1.24.10921.0
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.

Successfully verified installer hash
Starting package install...
Installer failed with exit code: 0x80070002 : The system cannot find the file specified.

I am unsure if this is caused by the same issue.

Suspicions & notes

As stated, I suspect that the fact that winget wasn't updated is either (part of) the cause or a symptom, and the fact that it was two completely different outdated versions is very weird.

There does not seem to be a common theme between the packages that it can and cannot download. Some come from github, some come from external sources, some are 3rd party packages, some are Microsoft packages. I cannot find any reason or rhyme to why some work and some don't.

All machines are part of the same subnet, so the network causing issues is very unlikely.

So, as you can see, I've done a decent amount of searching and looking, but I just cannot find an explanation as to why this is happening or something to fix it. Is there anyone here who may have some tips, or recognizes this issue? If there is any more info I can provide, don't hestitate to let me know. Any help is appreciated :)

[florelis]

My first suggestion would be to look at the logs

I would suspect a network issue, given the 404 error. You could try changing the downloader setting.

winget upgrade winget didn't work because the winget source does not have version 1.28; I'll add that.

The sources being the same doesn't say much, since they're just the default ones. For the winget source, it may be that it is outdated in one of the machines, but you already tried winget source update.