Skip to content

Conversation

@cubap
Copy link
Member

@cubap cubap commented Feb 9, 2026

This pull request introduces a major refactor across multiple components to standardize and improve how IIIF resources (such as canvases, manifests, and annotation pages) are fetched and hydrated. The new approach replaces direct network requests with a centralized vault utility, which caches and handles resource resolution, including fallback logic to hydrate missing resources from project manifests. This results in more robust error handling, improved performance through caching, and consistent resource access patterns.

Resource fetching and hydration improvements:

  • Replaced direct fetch calls with vault.get for canvases, annotation pages, manifests, and related IIIF resources in all relevant components (e.g., line-parser.js, plain.js, default-transcribe/index.js, column-selector/index.js, continue-working/index.js, read-only-transcribe/index.js, simple-transcription/index.js, transcription-block/index.js, transcription/index.js, line-image/index.js). This ensures resources are cached and resolved consistently. [1] [2] [3] [4] [5] [6]

  • Added fallback logic to hydrate missing resources: if a resource isn't found in the cache, the code attempts to prefetch project manifests using vault.prefetchDocuments, then retries the resource fetch. This pattern is now used throughout the codebase for canvases and annotation pages. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]

Error handling and user feedback:

  • Improved error handling and user feedback for unresolved resources: components now display user-friendly messages or fallback images when canvases or annotation pages cannot be resolved, instead of throwing raw errors or leaving the UI blank. [1] [2] [3] [4] [5]

Codebase consistency and maintainability:

  • Centralized IIIF resource access by importing the shared vault utility in all relevant files, removing redundant instantiations and direct fetch logic. [1] [2] [3] [4] [5]

  • Updated urlFromIdAndType in js/utils.js to return empty strings for IIIF resource types (canvas, manifest, collection) that should not be constructed as internal URLs, preventing erroneous fetch attempts.

Minor improvements:

  • Fixed a typo in a comment ("recieve" → "receive") for clarity.

These changes collectively make the application more resilient, performant, and maintainable when working with IIIF resources.

image

cubap added 6 commits February 6, 2026 09:59
Introduce a seed-based hydration path and helper (hydrateFromObject) so an already-provided item object can be used if the network fetch fails. Reorder logic to define hydration first, use the seed fallback on non-OK responses and exceptions, and return the hydrated object. Also adjust calls to this.get and this.set to pass the embedded object and the correct type. Local caching to localStorage is preserved.
Add Vault.prefetchDocuments to eagerly cache manifest documents and use it when a canvas or annotation page cannot be resolved from the vault. Components updated (continue-working, transcription-block, transcription interface) now attempt to hydrate all project manifests and retry vault lookups before falling back to placeholders. getImage was converted to async and now uses vault.get (with prefetch) instead of direct fetch, with explicit fallbacks to 404/no-image assets. These changes reduce failed network fetches and improve resolution of IIIF v2/v3 canvases and pages.
Replace ad-hoc fetches with the centralized vault API across multiple components and add a manifest prefetch fallback when an item is not immediately resolvable. Components updated: annotorious-annotator/line-parser, column-selector, continue-working, default-transcribe, legacy-annotator/plain, line-image, simple-transcription — they now import and use the shared vault, attempt vault.prefetchDocuments(TPEN.activeProject.manifest) when a resource is missing, and retry resolving. Default-transcribe and others now use vault.get for annotation/page/contentresource resolution and show clearer failure messages.

Major Vault improvements (js/vault.js):
- Normalizes IDs (removes fragment, expands 24-char Mongo hex IDs to TPEN.RERUMURL/id/...).
- Deduplicates concurrent fetches with inFlightPromises to avoid duplicate network requests.
- Improved hydration: walks embedded objects, queues and caches known IIIF resource types, skips common non-resource properties, normalizes and caches discovered URLs/IDs, and stores data keyed by detected resource type when available.
- More resilient fetch handling (returns null on failure if a seed exists) and safer localStorage writes keyed to resource type.
- PrefetchDocuments now calls vault.get with a best-effort type and collects errors.
- Removed the debug runVaultTest export.

These changes make resource resolution more consistent and robust, reduce duplicate network loads, and allow components to recover by hydrating manifests when resources are initially missing.
Treat canvas/manifest/collection IDs as external/embedded (return empty URL) and simplify _normalizeId to only strip fragment identifiers. Add a _processIIIFResource helper to detect IIIF resources, call get() for them, and return a minimal {id,type,label} representation while avoiding duplicate fetches via the visited set. Update _fetchAndHydrate to invoke this helper for array items and non-array object values. Remove previous behavior that auto-prefixed Mongo hex IDs with TPEN.RERUMURL and the older string-URL detection/fetch logic to better handle embedded IIIF manifests and avoid incorrect fetching.
@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2026

@cubap
Copy link
Member Author

cubap commented Feb 9, 2026

This is intended to replace #434 and #435 closing the features connected there.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors IIIF resource resolution across the UI by routing canvas/manifest/annotation-page fetching through a shared vault utility, aiming to standardize hydration, caching, and fallback behavior (e.g., prefetching manifests to resolve embedded resources).

Changes:

  • Enhanced js/vault.js with in-flight de-duping, deeper hydration logic, and a prefetchDocuments() helper.
  • Updated multiple components/interfaces to replace direct fetch() usage with vault.get() plus manifest-prefetch fallback logic.
  • Adjusted urlFromIdAndType() to avoid constructing internal URLs for IIIF resource types like canvases/manifests/collections.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
js/vault.js Adds in-flight request de-duping, hydration traversal, and manifest prefetch helper.
js/utils.js Prevents internal URL construction for IIIF canvas/manifest/collection IDs.
interfaces/transcription/index.js Switches image/canvas/page resolution to vault.get() with manifest-prefetch fallback.
components/transcription-block/index.js Uses vault.get() + prefetch fallback for annotation pages.
components/simple-transcription/index.js Uses vault.get() + prefetch fallback for pages/canvases; minor state cleanup.
components/read-only-transcribe/index.js Replaces manifest fetch() with vault.get() for static manifest loading.
components/line-image/index.js Resolves canvases via vault.get() on canvas-change events.
components/legacy-annotator/plain.js Resolves annotation pages/canvases via vault.get() with manifest-prefetch fallback.
components/default-transcribe/index.js Replaces external Vault usage with shared vault and resolves annotations/resources via vault.get().
components/continue-working/index.js Resolves canvases via vault.get() with manifest-prefetch fallback.
components/column-selector/index.js Resolves annotation pages via vault.get() with manifest-prefetch fallback.
components/annotorious-annotator/line-parser.js Uses shared vault for canvas resolution; fixes a comment typo.
Comments suppressed due to low confidence (1)

components/default-transcribe/index.js:91

  • After the two vault.get attempts, lineElem.line can still be null/undefined, but the code unconditionally dereferences lineElem.line.body[0] and lineElem.line.target..., which will throw and abort rendering the whole page. Add a guard/skip for lines that fail to resolve (and optionally surface a user-facing message).
            lineElem.line = await vault.get(l.id, 'annotation')
            if (!lineElem.line) {
                lineElem.line = await vault.get(l.id, 'annotation', true)
            }
            lineElem.line.body[0] = await vault.get(lineElem.line.body[0].id, 'contentresource')
            lineElem.setAttribute('tpen-line-id', l.id)
            lineImg.setAttribute('tpen-line-id', l.id)
            lineImg.setAttribute('iiif-canvas', lineElem.line.target.source.id)
            lineImg.setAttribute('region', lineElem.line.target.selector.value)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Contributor

Copilot AI commented Feb 9, 2026

@cubap I've opened a new pull request, #437, to work on those changes. Once the pull request is ready, I'll request review from you.

Copy link
Contributor

Copilot AI commented Feb 9, 2026

@cubap I've opened a new pull request, #438, to work on those changes. Once the pull request is ready, I'll request review from you.

cubap and others added 2 commits February 8, 2026 22:56
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Contributor

Copilot AI commented Feb 9, 2026

@cubap I've opened a new pull request, #439, to work on those changes. Once the pull request is ready, I'll request review from you.

Copy link
Contributor

Copilot AI commented Feb 9, 2026

@cubap I've opened a new pull request, #440, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot AI and others added 4 commits February 8, 2026 23:03
#437)

* Initial plan

* Add guard to prevent fetch with empty URI

Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>
…ype parameters (#438)

* Initial plan

* Add prefetchManifests and prefetchCollections aliases and update all usages

Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>

* Refactor prefetch methods to use docType parameter instead of duplicating code

Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>
* Initial plan

* Fix N+1 fetch issue by caching embedded IIIF resources directly

Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>

* Improve comments explaining embedded object detection logic

Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>

* Improve embedded object detection with specific content properties

Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>

* Allow annotations with body OR target to be cached as embedded

Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>
* Initial plan

* Fix IIIF v2 prefix handling in type normalization

Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>

* Move prefix pattern to class constant and document resource types

Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>

* Revert prefix stripping; explicitly add IIIF v2 prefixed types

Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>

* Clarify documentation for agent and annotationlist types

Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>
@cubap cubap marked this pull request as draft February 9, 2026 05:38
@cubap
Copy link
Member Author

cubap commented Feb 9, 2026

Worth noting:

This does not have anywhere yet it uses the "force" parameter that may be wanted for loading pages and layers that we expect will be overwritten. The tests on this so far were just to meet the tester demo.

@thehabes thehabes changed the base branch from main to 417-strengthen-attempts-to-load-data February 9, 2026 15:37
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

}

_normalizeType(type) {
return (type ?? '').toString().toLowerCase() || 'none'
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_normalizeType preserves IIIF v2 prefixes (e.g., sc:Canvassc:canvas). That means resources fetched from v2 endpoints will be cached under sc:* buckets, but most callers request unprefixed types ('canvas', 'manifest'), so cache hits and getWithFallback() after prefetchManifests() can fail to resolve embedded v2 resources. Consider canonicalizing v2 prefixes (e.g., strip sc: / oa: in _normalizeType, and map oa:annotationlist to annotationpage) so lookups work across v2/v3 consistently.

Suggested change
return (type ?? '').toString().toLowerCase() || 'none'
// Normalize IIIF types across Presentation API v2 (prefixed) and v3 (unprefixed)
// Examples:
// "Canvas" -> "canvas"
// "sc:Canvas" -> "canvas"
// "oa:AnnotationList" -> "annotationpage"
const raw = (type ?? '').toString().trim().toLowerCase()
if (!raw) return 'none'
// Strip common IIIF v2 prefixes (Shared Canvas / Open Annotation)
let normalized = raw
if (normalized.startsWith('sc:') || normalized.startsWith('oa:')) {
normalized = normalized.slice(normalized.indexOf(':') + 1)
}
// Map v2-specific type names to their v3 canonical equivalents
if (normalized === 'annotationlist') {
normalized = 'annotationpage'
}
return normalized

Copilot uses AI. Check for mistakes.
Comment on lines +306 to +309
for (const canvasRef of manifest.items) {
const canvas = await vault.get(canvasRef, 'canvas')
if (!canvas) continue
const imgUrl = canvas.items?.[0]?.items?.find(i => i.motivation === "painting")?.body?.id
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loop fetches canvases sequentially (await inside for...of). For manifests with many canvases, this will be noticeably slower even with caching. Consider resolving all canvases concurrently (e.g., Promise.all(manifest.items.map(ref => vault.get(ref, 'canvas')))), then iterating over the resolved results.

Copilot uses AI. Check for mistakes.

this.renderCleanup.onElement(this.shadowRoot.querySelector("#autoParseBtn"), "click", async () => {
// TODO this needs testing before we are ready for users to use it
return
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Auto Parse click handler immediately returns before the try block, leaving a large unreachable block of code registered on every render. If the feature is intentionally disabled, it would be clearer to not register the listener (and/or hide/remove the button) behind a feature flag; otherwise remove the early return so the handler can run.

Suggested change
return

Copilot uses AI. Check for mistakes.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nah it's supposed to be dead like that for now until we want to use it.

@thehabes thehabes merged commit 4a208a5 into 417-strengthen-attempts-to-load-data Feb 11, 2026
8 checks passed
cubap added a commit that referenced this pull request Feb 12, 2026
* Hydrate from seed when fetch fails

Introduce a seed-based hydration path and helper (hydrateFromObject) so an already-provided item object can be used if the network fetch fails. Reorder logic to define hydration first, use the seed fallback on non-OK responses and exceptions, and return the hydrated object. Also adjust calls to this.get and this.set to pass the embedded object and the correct type. Local caching to localStorage is preserved.

* runVaulttest

* vault all fetches

* Prefetch manifests before resolving canvases

Add Vault.prefetchDocuments to eagerly cache manifest documents and use it when a canvas or annotation page cannot be resolved from the vault. Components updated (continue-working, transcription-block, transcription interface) now attempt to hydrate all project manifests and retry vault lookups before falling back to placeholders. getImage was converted to async and now uses vault.get (with prefetch) instead of direct fetch, with explicit fallbacks to 404/no-image assets. These changes reduce failed network fetches and improve resolution of IIIF v2/v3 canvases and pages.

* Use vault and prefetch manifests; enhance Vault

Replace ad-hoc fetches with the centralized vault API across multiple components and add a manifest prefetch fallback when an item is not immediately resolvable. Components updated: annotorious-annotator/line-parser, column-selector, continue-working, default-transcribe, legacy-annotator/plain, line-image, simple-transcription — they now import and use the shared vault, attempt vault.prefetchDocuments(TPEN.activeProject.manifest) when a resource is missing, and retry resolving. Default-transcribe and others now use vault.get for annotation/page/contentresource resolution and show clearer failure messages.

Major Vault improvements (js/vault.js):
- Normalizes IDs (removes fragment, expands 24-char Mongo hex IDs to TPEN.RERUMURL/id/...).
- Deduplicates concurrent fetches with inFlightPromises to avoid duplicate network requests.
- Improved hydration: walks embedded objects, queues and caches known IIIF resource types, skips common non-resource properties, normalizes and caches discovered URLs/IDs, and stores data keyed by detected resource type when available.
- More resilient fetch handling (returns null on failure if a seed exists) and safer localStorage writes keyed to resource type.
- PrefetchDocuments now calls vault.get with a best-effort type and collects errors.
- Removed the debug runVaultTest export.

These changes make resource resolution more consistent and robust, reduce duplicate network loads, and allow components to recover by hydrating manifests when resources are initially missing.

* Process IIIF resources and simplify ID normalization

Treat canvas/manifest/collection IDs as external/embedded (return empty URL) and simplify _normalizeId to only strip fragment identifiers. Add a _processIIIFResource helper to detect IIIF resources, call get() for them, and return a minimal {id,type,label} representation while avoiding duplicate fetches via the visited set. Update _fetchAndHydrate to invoke this helper for array items and non-array object values. Remove previous behavior that auto-prefixed Mongo hex IDs with TPEN.RERUMURL and the older string-URL detection/fetch logic to better handle embedded IIIF manifests and avoid incorrect fetching.

* Update interfaces/transcription/index.js



* Update components/read-only-transcribe/index.js



* Guard against empty URI in vault fetch to prevent unnecessary requests (#437)

* Initial plan

* Add guard to prevent fetch with empty URI



---------




* Add prefetchManifests and prefetchCollections aliases with explicit type parameters (#438)

* Initial plan

* Add prefetchManifests and prefetchCollections aliases and update all usages



* Refactor prefetch methods to use docType parameter instead of duplicating code



---------




* Eliminate N+1 fetches for embedded IIIF resources in vault (#439)

* Initial plan

* Fix N+1 fetch issue by caching embedded IIIF resources directly



* Improve comments explaining embedded object detection logic



* Improve embedded object detection with specific content properties



* Allow annotations with body OR target to be cached as embedded



---------




* Add explicit IIIF v2 prefixed types to resource set (#440)

* Initial plan

* Fix IIIF v2 prefix handling in type normalization



* Move prefix pattern to class constant and document resource types



* Revert prefix stripping; explicitly add IIIF v2 prefixed types



* Clarify documentation for agent and annotationlist types



---------




* Update vault.js

* deprecate these

* strip out TPEN.js and vault.js

This component should not include these dependencies.

* Import vault in read-only-transcribe component

Add an import for the vault module (../../js/vault.js) in components/read-only-transcribe/index.js so the component can access vault-related functionality.

* This will work here (#443)

* Early guard to avoid NPEs when this.#transcriptions cannot be used (#444)

* Address Issues 13-17 from static review: Fix vault consistency, error handling, and performance issues (#442)

* Initial plan

* Address Issues 13-17: Fix vault consistency, error handling, and performance issues



---------




* Fix vault.js issues 8-11: Hoist constants, fix noCache, clone data, add getWithFallback helper (#445)

Issues Fixed:
- Issue 8: Move SKIP_PROPERTIES and IIIF_RESOURCE_TYPES to module scope to prevent recreation on every fetch (28 + 13 = 41 object allocations eliminated per fetch)
- Issue 9: Extract vault.getWithFallback() helper method to consolidate repeated 5-line fallback pattern used in 10+ locations
- Issue 10: Fix noCache parameter to bypass in-memory store, not just localStorage. When noCache=true, caller gets fresh fetch, not stale cached copy
- Issue 11: Clone data with structuredClone() before BFS mutation to prevent corrupting caller's original object via shared reference

Performance Impact:
- Eliminates unnecessary Set allocations (Issue 8)
- Reduces code duplication and maintenance burden (Issue 9)
- Fixes cache bypass behavior (Issue 10)
- Prevents data corruption bugs (Issue 11)

* Update Conditionals to avoid NPEs (#447)

* update conditions to fix pre existing errors

* update conditions to fix pre existing errors

* Use vault.getWithFallback for fetching (#449)

Replace repeated vault.get + prefetchManifests patterns with a single vault.getWithFallback utility to simplify resource resolution and avoid duplicated manifest-hydration logic. Affected components: annotorious-annotator/line-parser.js, column-selector/index.js, continue-working/index.js, default-transcribe/index.js, legacy-annotator/plain.js, simple-transcription/index.js, transcription-block/index.js, interfaces/transcription/index.js. Preserves existing behavior (including the optional strict flag) while centralizing the fallback to project manifests.

* refactor manage columns to use vault (#450)

* refactor manage columns to use vault

* refactor manage columns to use vault

* hotfix for error messaging

* No 'Auto Parse' for now

* No 'Auto Parse' for now

* Fix so CONTRIBUTOR users see the link into the /annotator interface

* See transcription text in .transcription-input on the transcription interface.

* Don't change @deprecated components

* Changes while reviewing

* Changes during review

* Changes during review

* zooming on view transcription interface

* Don't let view transcription interface be larger than the viewport and cause scrolling

* double right-click to zoom out

* double right-click to zoom out

* small change for accessibility

* small change for accessibility.  Add some consistency around how these buttons show or not.

* small change for accessibility.  Add some consistency around how these buttons show or not.

* whoops make it prettier

* changes while reviewing

* remove references to elements that are no longer used

* Extra defensive for an extra chance to find the image

* some cleanup

* Changes while reviewing

* Changes while reviewing

* Changes while reviewing

* Changes while reviewing

---------

Co-authored-by: Patrick Cuba <cubap@slu.edu>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: cubap <1119165+cubap@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants