Implement latest regex for fetch and cache by TomJo2000 · Pull Request #27734 · termux/termux-packages

TomJo2000 · 2025-12-23T15:16:33Z

closes Auto update failing for libmsgpack #27715
closes Auto update failing for dotnet9.0 #28020
closes Auto update failing for libevent #28053
closes Auto update failing for less #28111
closes Auto update failing for nemo #28153
closes Auto update failing for spirv-tools #28154
closes Auto update failing for cinnamon-menus #28155
closes Auto update failing for cinnamon-settings-daemon #28156
closes Auto update failing for cinnamon-desktop #28157
closes Auto update failing for xapp #28158
closes Auto update failing for cinnamon #28159
closes Auto update failing for mint-themes #28160
closes Auto update failing for cjs #28161
closes Auto update failing for python-xapp #28162
closes Auto update failing for liblmdb #28163
closes Auto update failing for cinnamon-control-center #28164
closes Auto update failing for muffin #28166
closes Auto update failing for cinnamon-session #28167

I need to just throw this up as a PR for further testing at this point.

The main issue with the auto-updater after the gitlab method refactor seems to have been that we never taught it how to use latest-regex in the _fetch_and_cache_tags/@all codepath.

This PR implements that and some related cleanup.
The changes should be broken out into multiple commits,
I don't have the time to do so right now,
and I need to see if this behaves as it should on the CI.

TomJo2000 · 2025-12-27T17:26:48Z

Alright everything seems to be working as expected now, except manim.
There appears to have been an incident involving a lead developer of the community fork of manim (which our package is based on) getting their accounts compromised so the project is currently in lockdown.
https://chris-besch.com/articles/github_pages_hack/

Nothing much we can do about that except wait it out, Manim's tarballs weren't effected as far as I am able to tell.
CC: @nguynkhn (as maintainer of our manim package) maybe you can provide some additional context here?

nguynkhn · 2025-12-27T19:34:39Z

Yeah that’s truly an unfortunate situation… Everything was deleted, including the GitHub organization and their other social media platforms.
They still haven’t confirmed what caused this at the moment. I think the best thing we can do right now is wait until the GitHub repository is restored or temporarily rely on an alternative source.

TomJo2000 · 2025-12-27T21:31:01Z

I've managed to part these changes out into separate commits now but continuing to work on the auto-update system may yet drive me mad.
From what I can observe it is working as it should, but there is no way I can be 100% sure of that until we see it in action and that comes with the unfortunate possibility of creating auto-update issue spam from messing up.
Everything should be working, but I'm less certain of that than I'd like to be.

I think on a meta level I'd like to get rid of the distinction between the "@ALL" and "list of packages" modes of operation in the auto-updater.
Being able to run spot checks on individual packages with issues instead of needing to go through a 10+ minute "@ALL" run to test them would significantly cut down on the madness induced by debugging issues with the auto-update system.
It would also make me significantly more confident in the results of a full "@ALL" run to know that it is fundamentally using the same codepath as a limited set run, just extended to the full package set.

API induced delirium aside, that is out of scope for this PR though and we need to just get the auto-update workflow back up and running.
I do expect additional issues from the couple days of backlog but nothing that shouldn't be manageable by babysitting the workflow until it's caught up.

TomJo2000 · 2025-12-27T21:46:50Z

That's not great...
It's just a logging problem thankfully, but I'd like the logs to be accurate

INFO: Checked 211 packages - took 1m 39s 366ms in total.
adwaita-icon-theme 49.0 => 0.18 (Error? current version is larger than latest fetched version)
adwaita-icon-theme-legacy 46.2 => 0.18 (Error? current version is larger than latest fetched version)
at-spi2-core 2.58.2 => 1.5.7 (Error? current version is larger than latest fetched version)
attr 2.5.2 => 685
cinnamon 6.6.2 => 6.6.3
dconf 0.49.0 => 0.25.10 (Error? current version is larger than latest fetched version)
desktop-file-utils 0.28 => 0.18 (Error? current version is larger than latest fetched version)
dotnet9.0 9.0.11 => 0.22.0 (Error? current version is larger than latest fetched version)
freetype 2.14.1 => 1.2.0 (Error? current version is larger than latest fetched version)
gdbm 1.26 => 10.47
gdk-pixbuf 2.44.4 => 3.1.3
gettext 0.26 => 0.25.10 (Error? current version is larger than latest fetched version)
gigolo 0.6.0 => 2.4
glib 2.86.3 => 2.17.1 (Error? current version is larger than latest fetched version)
gnome-backgrounds 49.0 => 0.25.10 (Error? current version is larger than latest fetched version)
gnome-desktop4 44.3 => 1.9.1 (Error? current version is larger than latest fetched version)
gomuks 25.11 => 25.12
gsettings-desktop-schemas 49.1 => 1.9.1 (Error? current version is larger than latest fetched version)
gsound 1.0.3 => 0.25.10 (Error? current version is larger than latest fetched version)
gstreamer 1.26.9 => 2.17.0
gvfs 1.58.0 => 0.21.5 (Error? current version is larger than latest fetched version)
harfbuzz 12.2.0 => 12.3.0
json-glib 1.10.8 => 0.3.2 (Error? current version is larger than latest fetched version)
krb5 1.22.1 => 0.22.0 (Error? current version is larger than latest fetched version)
libacl 2.3.2 => 685
libaom 3.13.1 => 0.402 (Error? current version is larger than latest fetched version)
libbluray 1.4.0 => 1.2.0 (Error? current version is larger than latest fetched version)
libdisplay-info 0.3.0 => 1.9.1
libdrm 2.4.131 => 0.25.10 (Error? current version is larger than latest fetched version)
libevent 2.1.12 => 2.2.1
libgd 2.3.3 => 1.6.0 (Error? current version is larger than latest fetched version)
libgnomekbd 3.28.1 => 14.1.1
libgsf 1.14.54 => 0.6.25 (Error? current version is larger than latest fetched version)
libice 1.1.2 => 0.3.2 (Error? current version is larger than latest fetched version)
libicu 78.1 => 1.5.7 (Error? current version is larger than latest fetched version)
libidn2 2.3.8 => 128.1
libksba 1.6.7 => 2.2.1
libmpg123 1.33.3 => 1.2.0 (Error? current version is larger than latest fetched version)
libnettle 3.10.2 => 128.1
libnghttp3 1.13.1 => 1.14.0
libnotify 0.8.7 => 14.1.1
libnspr 4.38.2 => 6.6.1
libnss 3.119.1 => 6.6.1
libopenmpt 0.8.4 => 1.2.0
libpixman 0.46.4 => 1.0.16
libpng 1.6.53 => 1.2.0 (Error? current version is larger than latest fetched version)
librsvg 2.61.3 => 1.6.0 (Error? current version is larger than latest fetched version)
libsecret 0.21.7 => 3.8.4
libsm 1.2.6 => 0.3.2 (Error? current version is larger than latest fetched version)
libsoup3 3.6.5 => 0.21.5 (Error? current version is larger than latest fetched version)
libsqlite 3.51.1 => 5.8.2
libssh 0.11.3 => 1.5.4
libstemmer 2.2.0 => 3.0.1
libtasn1 4.20.0 => 1.0.29 (Error? current version is larger than latest fetched version)
libtiff 4.7.1 => 3.1.3 (Error? current version is larger than latest fetched version)
libtool 2.5.4 => 4.19.0
libunbound 1.24.2 => 2.2.1
libuv 1.51.0 => 2.1.2
libwayland 1.24.0 => 1.7.0 (Error? current version is larger than latest fetched version)
libwayland-protocols 1.45 => 1.7.0 (Error? current version is larger than latest fetched version)
libx11 1.8.12 => 2.7.3
libxau 1.0.12 => 2.7.3
libxcb 1.17.0 => 2.7.3
libxcursor 1.2.3 => 1.7.0
libxcvt 0.1.3 => 1.5.84
libxdmcp 1.1.5 => 2.7.3
libxfixes 6.0.2 => 1.5.7 (Error? current version is larger than latest fetched version)
libxfont2 2.0.7 => 1.5.84 (Error? current version is larger than latest fetched version)
libxft 2.3.9 => 2.17.1
libxi 1.8.2 => 1.5.7 (Error? current version is larger than latest fetched version)
libxkbfile 1.1.3 => 4.19.0
libxml2 2.15.1 => 1.5.7 (Error? current version is larger than latest fetched version)
libxrandr 1.5.4 => 1.13.1
libxrender 0.9.12 => 2.17.1
libxshmfence 1.3.3 => 1.5.84
libxss 1.2.5 => 2.17.1
libxtst 1.2.5 => 1.5.7
libxxf86vm 1.1.6 => 21.1.3
libzix 0.6.2 => 6.0.2
lilv 0.26.2 => 3.1.2
littlecms 2.17 => 0.402 (Error? current version is larger than latest fetched version)
marksman 2025.11.30 => 2025.12-13
mesa 25.3.2 => 1.4.337 (Error? current version is larger than latest fetched version)
mint-themes 2.3.5 => 2.3.6
mint-y-icon-theme 1.9.0 => 1.9.1
pango 1.57.0 => 1.13.1 (Error? current version is larger than latest fetched version)
pinentry 1.3.2 => 1.0.29 (Error? current version is larger than latest fetched version)
pipewire 1.4.9 => 1.5.84
pygobject 3.54.5 => 1.29.0 (Error? current version is larger than latest fetched version)
rubberband 4.0.0 => 6.0.2
serd 0.32.6 => 6.0.2
sord 0.16.20 => 6.0.2
speexdsp 1.2.1 => 1.2.2
sratom 0.6.20 => 3.1.2
swig 4.4.1 => 2.2.1 (Error? current version is larger than latest fetched version)
texinfo 7.2 => 4.19.0 (Error? current version is larger than latest fetched version)
tree-sitter 0.25.10 => 1.51.0-1
unicode-data 17.0.0 => 0.25.10 (Error? current version is larger than latest fetched version)
valac 0.56.18 => 14.1.1
vamp-plugin-sdk 2.10 => 6.0.2
xapp-symbolic-icons 1.0.6 => 1.0.7
xcb-proto 1.17.0 => 2.7.3
xorg-util-macros 1.20.2 => 2.7.3
xorg-xkbcomp 1.5.0 => 14.1.1
xorgproto 2025.1 => 2.7.3 (Error? current version is larger than latest fetched version)
xtrans 1.6.0 => 2.7.3
xwayland 24.1.9 => 1.4.337 (Error? current version is larger than latest fetched version)
INFO: 0 failed updates.

TomJo2000 · 2025-12-28T15:58:10Z

Okay I said I wasn't gonna unify the @all codepath in this PR but it has driven me insane long enough that I actually did, and it turns out it wasn't that difficult.

We just need to accumulate a package list, eliminate anything the pre-cache caught as up-to-date and shunt the finished list back into $@ which can be done trivially with set -- "${PACKAGE_LIST[@]}".

One minor optimization we can make here is providing the package list as ${channel}/${pkg_name} since that lets it skip the channel search in the loop below.

termux-packages/scripts/bin/update-packages

Lines 418 to 433 in fb6d543

    
           for pkg in "$@"; do 
        
           	_unix_millis="$(date +%10s%3N)" 
        
           	if [ ! -d "${pkg}" ]; then # If only package name is given, try to find it's directory. 
        
           		for repo_dir in $(jq --raw-output 'del(.pkg_format) | keys | .[]' "${TERMUX_SCRIPTDIR}/repo.json"); do 
        
           			if [ -d "${repo_dir}/${pkg}" ]; then 
        
           				pkg="${repo_dir}/${pkg}" 
        
           				break 
        
           			fi 
        
           		done 
        
           	fi 
        
           	# Here `pkg` is a directory. 
        
           	! _should_update "${pkg}" && continue 
        
           	_update_dependencies "${pkg}" 
        
           	_run_update "${pkg}" 
        
           	echo "termux - took $(ms_to_human_readable $(( $(date +%10s%3N) - _unix_millis )))" 
        
           done

TomJo2000 · 2026-01-16T06:50:19Z

Alright, I'm finally out of spinning plates, so I can get back to this one.
I'll need to take a close look at the fixes to packages in here to see if they still make sense to me.
If there's one thing I wanna salvage from this PR though it's the unified @all codepath so there's not two separate codepaths for individual packages and a full auto-update run because that was causing trouble in debugging.

"line 60: [: =: unary operator expected" and some minor build script cleanup %ci:no-build

no longer needed for these packages %ci:no-build

%ci:no-build

…dk-` %ci:no-build

…_VERSION_REGEXP where necessary %ci:no-build

…TE_VERSION_REGEXP %ci:no-build

TomJo2000 · 2026-01-27T01:32:27Z

Let's see if the "days since Tom broke prod" counter gets to tick up another day.

I've crossed my t's and dotted my i's.
We should be 100% good on this PR.
But bugs have a nasty habit of showing up in prod when nothing went wrong in multiple rounds of testing.

truboxl · 2026-01-27T01:55:06Z

This PR completely ignores repology update issue, which I intend to fix

TomJo2000 · 2026-01-27T02:07:56Z

Welp, there goes my counter I guess.

robertkirkman · 2026-01-27T02:19:43Z

This PR completely ignores repology update issue, which I intend to fix

I wasn't aware of a repology update issue, could you explain it?

truboxl · 2026-01-27T02:25:10Z

This PR completely ignores repology update issue, which I intend to fix

I wasn't aware of a repology update issue, could you explain it?

Traceback (most recent call last):
  File "/home/runner/work/termux-packages/termux-packages/scripts/updates/api/dump-repology-data", line 26, in <module>
    from requests import get as requests_get
ModuleNotFoundError: No module named 'requests'

https://github.com/termux/termux-packages/actions/runs/21380867349/job/61547280850#step:4:20

robertkirkman · 2026-01-27T02:36:59Z

Oh ok, I have seen that before but I didn't know that was what you were talking about,

I didn't realize that was causing a measurable problem because, even though it has been happening for a long time, repology-based auto updates continued to succeed despite that, for example 7c279bb

but I agree it will be good for it to get fixed

TomJo2000 · 2026-01-27T02:37:21Z

I can't replicate that issue locally, nor have I seen it on the CI.
I did not touch Repology handling in this PR, so I don't see where that error could have come from.

Absolutely do look into it though, if it's happening it should be fixed.

robertkirkman · 2026-01-27T02:37:54Z

I can't replicate that issue locally, nor have I seen it on the CI. I did not touch Repology handling in this PR, so I don't see where that error could have come from.

It's not caused by this PR, it's something that has been happening for a long time before this PR

truboxl · 2026-01-27T02:45:17Z

Oh ok, I have seen that before but I didn't know that was what you were talking about,

I didn't realize that was causing a measurable problem because, even though it has been happening for a long time, repology-based auto updates continued to succeed despite that, for example 7c279bb

but I agree it will be good for it to get fixed

less has its own update function that isnt repology

I am pretty sure its 22d58b7 that broke, cc @thunder-coding

I do intend to fix but I am busy

TomJo2000 · 2026-01-27T02:46:55Z

Ah, I had temporarily changed that back to ubuntu-24.04 in a debug commit I have dropped before merge for testing with @all for auto-updates.

That was running into the 15 minute timeout.
So I may have inadvertently sidestepped the problem by doing that, hence why I never saw it in testing.

- Rebase after termux/termux-packages#27734

robertkirkman · 2026-01-28T08:42:09Z

@truboxl I think I am fixing it in this while also fixing something else,

fix(scripts/updates): fix the dependencies of the repology auto update method and use it for gitlab.gnome.org packages #28244

review would be appreciated, especially if you don't like this way of fixing it and would prefer a different way

TomJo2000 mentioned this pull request Dec 23, 2025

Refactor gitlab auto update method #27576

Merged

TomJo2000 force-pushed the implement-latest-regex-for-fetch-and-cache branch from 11b9b11 to 948f63c Compare December 27, 2025 17:59

TomJo2000 force-pushed the implement-latest-regex-for-fetch-and-cache branch from 948f63c to da2d5e8 Compare December 27, 2025 21:21

TomJo2000 marked this pull request as ready for review December 27, 2025 21:31

TomJo2000 requested review from Grimler91 and thunder-coding as code owners December 27, 2025 21:31

TomJo2000 force-pushed the implement-latest-regex-for-fetch-and-cache branch from da2d5e8 to 8ccb9b4 Compare December 27, 2025 21:35

TomJo2000 marked this pull request as draft December 27, 2025 21:53

TomJo2000 force-pushed the implement-latest-regex-for-fetch-and-cache branch from 1c7ba7a to 433085c Compare December 28, 2025 14:29

TomJo2000 mentioned this pull request Jan 10, 2026

KDE Plasma Desktop on Termux #25619

Merged

3 tasks

TomJo2000 force-pushed the implement-latest-regex-for-fetch-and-cache branch from 433085c to 5ed2360 Compare January 16, 2026 06:41

TomJo2000 force-pushed the implement-latest-regex-for-fetch-and-cache branch 4 times, most recently from 0254f19 to b280948 Compare January 17, 2026 02:21

TomJo2000 force-pushed the implement-latest-regex-for-fetch-and-cache branch 9 times, most recently from 0a657f1 to 069e335 Compare January 23, 2026 23:16

TomJo2000 added 11 commits January 27, 2026 02:26

fix(main/python-llvmlite): resolve error message during auto-update

ddec337

"line 60: [: =: unary operator expected" and some minor build script cleanup %ci:no-build

fix(main/{adms,exiftool,lib{p11,rav1e,vigra}}): remove auto-update regex

b7b4c44

no longer needed for these packages %ci:no-build

fix(main/gomuks): adapt auto-updates to new versioning scheme

478fc2c

%ci:no-build

fix(main/less): use GitHub repo for auto-update source

5252638

%ci:no-build

fix(main/libevent): auto-update - ensure tag has -stable suffix

5dfd2ed

%ci:no-build

fix(main/marksman): properly substitute version numbers for auto-updates

e625b7e

%ci:no-build

fix(main/openlist): strip tag prefix during auto-update function

52ef1b7

%ci:no-build

fix(main/spirv-llvm-translator): pin auto-updates to LLVM major version

b2ccdc0

%ci:no-build

fix(main/spirv-tools): auto-update - ensure tag begins with `vulkan-s…

0f2cde0

…dk-` %ci:no-build

fix(tree-wide): add tag prefixes and "RC guards" to TERMUX_PKG_UPDATE…

bb8f1ad

…_VERSION_REGEXP where necessary %ci:no-build

fix(tree-wide): remove ^, $ and \K anchors from TERMUX_PKG_UPDA…

a74e22a

…TE_VERSION_REGEXP %ci:no-build

TomJo2000 force-pushed the implement-latest-regex-for-fetch-and-cache branch from 0fec46d to a74e22a Compare January 27, 2026 01:26

TomJo2000 merged commit 7e93b40 into termux:master Jan 27, 2026
11 checks passed

TomJo2000 deleted the implement-latest-regex-for-fetch-and-cache branch January 27, 2026 01:32

TomJo2000 mentioned this pull request Jan 27, 2026

fix(scripts/update-packages): solve caching problem in issue lookup #28193

Merged

robertkirkman added a commit to termux-user-repository/tur that referenced this pull request Jan 27, 2026

chore(common-files/building-system-patches): rebase

df07355

- Rebase after termux/termux-packages#27734

robertkirkman mentioned this pull request Jan 28, 2026

fix(scripts/updates): fix the dependencies of the repology auto update method and use it for gitlab.gnome.org packages #28244

Merged

robertkirkman mentioned this pull request Feb 2, 2026

Auto update failing for wordgrinder #27862

Closed

TomJo2000 mentioned this pull request Feb 18, 2026

fix(scripts/termux_pkg_upgrade_version): sort LATEST_VERSION #28516

Merged

Uh oh!

Conversation

TomJo2000 commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomJo2000 commented Dec 27, 2025

Uh oh!

nguynkhn commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomJo2000 commented Dec 27, 2025

Uh oh!

TomJo2000 commented Dec 27, 2025

Uh oh!

TomJo2000 commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomJo2000 commented Jan 16, 2026

Uh oh!

TomJo2000 commented Jan 27, 2026

Uh oh!

Uh oh!

truboxl commented Jan 27, 2026

Uh oh!

TomJo2000 commented Jan 27, 2026

Uh oh!

robertkirkman commented Jan 27, 2026

Uh oh!

truboxl commented Jan 27, 2026

Uh oh!

robertkirkman commented Jan 27, 2026

Uh oh!

TomJo2000 commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertkirkman commented Jan 27, 2026

Uh oh!

truboxl commented Jan 27, 2026

Uh oh!

TomJo2000 commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertkirkman commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

TomJo2000 commented Dec 23, 2025 •

edited

Loading

nguynkhn commented Dec 27, 2025 •

edited

Loading

TomJo2000 commented Dec 28, 2025 •

edited

Loading

TomJo2000 commented Jan 27, 2026 •

edited

Loading

TomJo2000 commented Jan 27, 2026 •

edited

Loading