Skip to content

Improve filesystem repo caching#1101

Open
isc-dchui wants to merge 8 commits intomainfrom
fix-536
Open

Improve filesystem repo caching#1101
isc-dchui wants to merge 8 commits intomainfrom
fix-536

Conversation

@isc-dchui
Copy link
Collaborator

Fixes #536

Improves the filesystem repo caching by proactively rebuilding cache and improving cache building performance by about an order of magnitude. Specifically:

  • Lazy load the actual manifest of the module by deferring until install.
  • Use mtime of the module.xml as a proxy for whether a module has been modified and perform smarter cache building based on that.
  • Drastically speed up filesystem scanning using Python.
  • Installing from a filesystem repo will trigger a quick cache rebuild to pick up any additions, modifications, and deletions.
  • Add -rebuild-cache flag to repo command to manually fully rebuild (with purge) cache.

Copy link
Collaborator

@isc-jili isc-jili left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@isc-dchui This looks really good! I made a pass and few small comments

set horologTime = ##class(%Library.File).GetFileDateModified(filePath)
set currentMTime = $zdatetime(horologTime, 3)
do defObj.RefreshCacheEntry(cacheObj, filePath, currentMTime)
$$$ThrowOnError(status)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming the status is supposed to come from RefreshCacheEntry(). How is status being tracked though, since it's not getting passed to the caller of RefreshCacheEntry() here?

Copy link
Collaborator Author

@isc-dchui isc-dchui Mar 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah good catch. This is a remnant of slightly different error handling. I've also standardized the method to just set the Output arg and quit throughout.

// Compare cached mtime with current mtime
if (cacheObj.FileMTime '= currentMTime) {
do defObj.RefreshCacheEntry(cacheObj, filePath, currentMTime)
if $$$ISERR(status) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question as above for how is status being tracked for this call of RefreshCacheEntry()?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed that remnant too.

set server = ##class(%IPM.Repo.Definition).ServerDefinitionKeyOpen(pRepoName,,.tSC)
$$$ThrowOnError(tSC)
if server.%IsA("%IPM.Repo.Filesystem.Definition") {
write !,"Last full cache rebuild for '", pRepoName, "' was on ", server.CacheLastRebuilt, " (UTC). Run 'repo -n "_pRepoName_" -rebuild-cache' to refresh the cache.",!
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find the wording here a bit confusing. Is "Run 'repo -n "pRepoName" -rebuild-cache' to refresh the cache" supposed to be just informative? Or is it asking the user to do that as an action item before proceeding further?
ie. Could potentially confuse users if all they're interested in is seeing which modules are available

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just informative. I've modified it to hopefully indicate that better.

@isc-dchui isc-dchui requested a review from isc-jili March 16, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Package Manager needs updated cache of repos before installation

2 participants