Strengthen Attempts to Load Data

.claude/CLAUDE.md

@@ -44,7 +44,7 @@ TPEN3 Interfaces is a JavaScript library that provides web components and API cl
 │   ├── index.css            # Main stylesheet
 │   ├── manage/              # Management page styles
 │   └── collaborators/       # Collaborator styles
-├── js/                      # Utility scripts (redirect, vault)
+├── js/                      # Utility scripts (redirect, vault, utils)
 ├── assets/                  # Static assets (icons, images, logos)
 ├── _classes/                # Jekyll class documentation
 ├── _cookbook/               # Jekyll cookbook examples
@@ -169,6 +169,43 @@ CONFIG.TPEN3URL     // TPEN 3 main URL
 // 4. Defaults to 'dev'
 ```
 
+### Vault (`js/vault.js`)
+Singleton caching layer for IIIF resources. **This is a custom utility, not the `@iiif/helpers` Vault.**
+All components should use the vault instead of raw `fetch()` for IIIF resources (canvases, manifests, annotation pages, annotations, etc.).
+
+```javascript
+import vault from '../../js/vault.js'
+
+// Core methods:
+vault.get(item, itemType, noCache)                       // Fetch with dual-layer cache (memory + localStorage)
+vault.set(item, itemType)                                // Store in both caches
+vault.delete(item, itemType)                             // Remove from both caches
+vault.clear(itemType)                                    // Purge all of a given type
+vault.all()                                              // Get all cached resources
+
+// Fallback methods:
+vault.getWithFallback(item, itemType, manifestUrls, noCache) // get() + prefetch manifests on miss
+vault.prefetchManifests(urls)                                // Batch prefetch manifests
+vault.prefetchDocuments(items, docType)                      // Batch prefetch any document type
+vault.prefetchCollections(items)                             // Batch prefetch collections
+```
+
+**Key behaviors:**
+- **Dual-layer cache**: in-memory `Map` for speed + `localStorage` for persistence across reloads
+- **BFS hydration**: when a resource is fetched, embedded IIIF sub-resources are recursively cached individually via `structuredClone()`. The parent stores minimal stubs (`{id, type, label}`) for non-array properties.
+- **In-flight deduplication**: concurrent `get()` calls for the same resource share a single network request
+- **Seed fallback**: when `item` is an object and the network fetch fails, the object is hydrated and cached as a fallback
+- **IIIF v2 + v3 types**: recognizes both unprefixed (v3: `Canvas`, `Manifest`) and prefixed (v2: `sc:Canvas`, `sc:Manifest`) types
+- **`noCache` flag**: bypasses in-memory and localStorage lookups to force a fresh fetch
+
+### URL Resolution (`js/utils.js`)
+`urlFromIdAndType(id, type, { projectId, pageId, layerId })` resolves resource IDs to fetchable URLs.
+
+- Full URLs (http/https) are returned as-is
+- TPEN resources (`annotationpage`, `annotation`, `annotationcollection`) are resolved to the services API
+- **External IIIF types (`canvas`, `manifest`, `collection`) return `null`** — these must already be full URLs or will be found as embedded data via manifest prefetch
+- Returns `null` (not empty string) when resolution is impossible
+
 ## API Endpoints
 
 The API follows RESTful conventions. Base URL is determined by environment (dev: `https://dev.api.t-pen.org`, prod: `https://api.t-pen.org`).
@@ -458,14 +495,7 @@ node --test api/__tests__/project.test.js  # Run specific test
 ## Configuration & Deployment
 
 ### Environment Configuration
-Configuration is managed via `api/config.js`:
-- **Development** (`dev`): Uses dev.api.t-pen.org, devstore.rerum.io
-- **Production** (`prod`): Uses api.t-pen.org, store.rerum.io
-
-Set environment via:
-1. `globalThis.TPEN_ENV = 'prod'` before importing
-2. `<meta name="tpen-env" content="prod">` in HTML
-3. `process.env.TPEN_ENV` for server-side
+See [Configuration (`api/config.js`)](#configuration-apiconfigjs) above for environment detection and URLs.
 
 ### Jekyll Configuration (`_config.yml`)
 ```yaml
@@ -524,7 +554,31 @@ Automated deployment pipeline:
 
 ## Common Patterns & Best Practices
 
-### Error Handling
+### IIIF Resource Fetching
+Always use `vault` for IIIF resources. Never use raw `fetch()` for canvases, manifests, annotation pages, or annotations.
+
+```javascript
+import vault from '../../js/vault.js'
+
+// Simple fetch — checks cache, then network
+const canvas = await vault.get(canvasURI, 'canvas')
+
+// Fetch with manifest fallback — if not found, prefetches project manifests
+// and retries (useful when canvas/page might be embedded in manifest)
+const page = await vault.getWithFallback(pageID, 'annotationpage', TPEN.activeProject?.manifest, true)
+const canvas = await vault.getWithFallback(canvasID, 'canvas', TPEN.activeProject?.manifest)
+
+// Force fresh fetch (bypass cache) with noCache=true
+const freshPage = await vault.get(pageID, 'annotationpage', true)
+
+// Resolve annotations that may be stubs (have target but no body)
+let annotation = item
+if (!annotation?.body) {
+    annotation = await vault.get(item, 'annotation', true)
+}
+```
+
+### Error Handling (TPEN API)
 ```javascript
 try {
     const response = await fetch(url)
@@ -571,18 +625,19 @@ try {
 1. **Authentication is Critical**: Always ensure authentication is properly handled before making API calls
 2. **Web Components Lifecycle**: Respect the component lifecycle - initialization in connectedCallback, cleanup in disconnectedCallback
 3. **Event Bubbling**: Use bubbles: true and composed: true for events that need to cross shadow DOM boundaries
-4. **IIIF Standards**: When working with manifests, follow IIIF Presentation API standards
+4. **IIIF Resources via Vault**: Always use `vault.get()` or `vault.getWithFallback()` for IIIF resources — never raw `fetch()`. The vault handles caching, fallback to manifest data, and IIIF v2/v3 compatibility
 5. **Project Context**: Projects are the central organizing unit - users create projects, add manifests, and collaborate
 6. **Error Messages**: Provide clear, user-friendly error messages, not raw API errors
 7. **Testing**: Always write tests for new functionality, especially for API interactions
 8. **CSS Encapsulation**: Remember that shadow DOM encapsulates styles - use CSS custom properties for theming
 
 ### Common Issues to Avoid:
-- Don't forget to attach authentication to fetch requests
+- Don't forget to attach authentication to fetch requests for TPEN API calls
 - Don't manipulate DOM directly in components - use render() method
 - Don't store sensitive data in localStorage beyond auth tokens
 - Don't make synchronous API calls - always use async/await
 - Don't forget to clean up event listeners in disconnectedCallback
+- Don't import `@iiif/helpers` Vault — `components/default-transcribe` still uses it and is `@deprecated`
 
 ### Development Tips:
 - Use the browser's Developer Tools to inspect web components

components/annotorious-annotator/line-parser.js

@@ -18,6 +18,7 @@ import { detectTextLinesCombined } from "./detect-lines.js"
 import { v4 as uuidv4 } from "https://cdn.skypack.dev/uuid@9.0.1"
 import { CleanupRegistry } from '../../utilities/CleanupRegistry.js'
 import { onProjectReady } from '../../utilities/projectReady.js'
+import vault from '../../js/vault.js'
 import '../page-selector/index.js'
 
 class AnnotoriousAnnotator extends HTMLElement {
@@ -282,7 +283,7 @@ class AnnotoriousAnnotator extends HTMLElement {
         tpen-page-selector {
           position: absolute;
           display: none;
-          top: 60px;
+          top: 10px;
           right: 10px;
           z-index: 9;
         }
@@ -378,6 +379,8 @@ class AnnotoriousAnnotator extends HTMLElement {
       this.renderCleanup.run()
 
       this.renderCleanup.onElement(this.shadowRoot.querySelector("#autoParseBtn"), "click", async () => {
+        // TODO this needs testing before we are ready for users to use it
+        return
         try {
           if (typeof cv === "undefined") {
             await new Promise((resolve, reject) => {
@@ -553,7 +556,7 @@ class AnnotoriousAnnotator extends HTMLElement {
 
   /**
    * Fetch a Canvas URI and check that it is a Canvas object.  Pass it forward to render the Image into the interface.
-   * Be prepared to recieve presentation api 2+
+   * Be prepared to receive presentation api 2+
    *
    * FIXME
    * Give users a path when Canvas URIs do not resolve or resolve to something unexpected.
@@ -562,20 +565,15 @@ class AnnotoriousAnnotator extends HTMLElement {
    */
   async processCanvas(uri) {
     if (!uri) return
-      // TODO Vault me?
-    const resolvedCanvas = await fetch(uri)
-      .then(r => {
-        if (!r.ok) throw r
-        return r.json()
-      })
-      .catch(e => {
-        this.shadowRoot.innerHTML = `
-          <h3>Canvas Error</h3>
-          <p>The Canvas within this Page could not be loaded.</p>
-          <p> ${e.status ?? e.code}: ${e.statusText ?? e.message} </p>
-        `
-        throw e
-      })
+    let resolvedCanvas = await vault.getWithFallback(uri, 'canvas', TPEN.activeProject?.manifest)
+    if (!resolvedCanvas) {
+      this.shadowRoot.innerHTML = `
+        <h3>Canvas Error</h3>
+        <p>The Canvas within this Page could not be loaded.</p>
+        <p>The Canvas could not be resolved or is invalid.</p>
+      `
+      return
+    }
     const context = resolvedCanvas["@context"]
     if (!context?.includes("iiif.io/api/presentation/3/context.json")) {
       console.warn("The Canvas object did not have the IIIF Presentation API 3 context and may not be parseable.")
@@ -612,14 +610,14 @@ class AnnotoriousAnnotator extends HTMLElement {
     const canvasID = resolvedCanvas["@id"] ?? resolvedCanvas.id
     this.#canvasID = canvasID
     let fullImage = resolvedCanvas?.items?.[0]?.items?.[0]?.body?.id
-    if (!fullImage) fullImage = resolvedCanvas?.images?.[0]?.resource?.["@id"]
+    if (!fullImage) fullImage = resolvedCanvas?.images?.[0]?.resource?.["@id"] ?? resolvedCanvas?.images?.[0]?.resource?.id
     let imageService = resolvedCanvas?.items?.[0]?.items?.[0]?.body?.service?.id
     if (!imageService) imageService = resolvedCanvas?.images?.[0]?.resource?.service?.["@id"]
     if (!fullImage) {
       throw new Error("Cannot Resolve Canvas Image", { "cause": "The Image is 404 or unresolvable." })
     }
     let imgx = resolvedCanvas?.items?.[0]?.items?.[0]?.body?.width
-    if (!imgx) imgx = resolvedCanvas?.images[0]?.resource?.width
+    if (!imgx) imgx = resolvedCanvas?.images?.[0]?.resource?.width
     let imgy = resolvedCanvas?.items?.[0]?.items?.[0]?.body?.height
     if (!imgy) imgy = resolvedCanvas?.images?.[0]?.resource?.height
     this.#imageDims = [imgx, imgy]
@@ -689,6 +687,20 @@ class AnnotoriousAnnotator extends HTMLElement {
         dblClickToZoom: true
       }
     })
+
+    // Double right-click to zoom out
+    let lastRightClickTime = 0
+    this.renderCleanup.onElement(this.#annotoriousContainer, 'contextmenu', (e) => {
+      e.preventDefault()
+      const now = Date.now()
+      if (now - lastRightClickTime < 400) {
+        this.#osd.viewport.zoomBy(0.5)
+        lastRightClickTime = 0
+      } else {
+        lastRightClickTime = now
+      }
+    })
+
     // Link to transcribe if they have view permissions for it
     if (CheckPermissions.checkViewAccess("LINE", "TEXT") || CheckPermissions.checkEditAccess("LINE", "TEXT")) {
       let parsingRedirectButton = new OpenSeadragon.Button({
@@ -871,7 +883,8 @@ class AnnotoriousAnnotator extends HTMLElement {
     this.#annotoriousInstance.setAnnotations(allAnnotations, false)
     this.#annotoriousContainer.style.backgroundImage = "none"
     this.shadowRoot.getElementById("tools-container").style.display = "block"
-    this.shadowRoot.querySelector("#autoParseBtn").style.display = "block"
+    // TODO This needs testing before we are ready for users to use it.
+    // this.shadowRoot.querySelector("#autoParseBtn").style.display = "block"
     const pageSelector = this.shadowRoot.querySelector("tpen-page-selector")
     if (pageSelector) {
       pageSelector.style.display = "block"

components/annotorious-annotator/plain.js

@@ -2,7 +2,7 @@
  * A plain Annotorious Annotator that can draw Rectangles.  It is able to draw Polygons, but this ability is not exposed to the user.
  * It assigns all Annotations to the provided AnnotationPage (does not make or track more than one page at a time)
  *
- * It is exposed to the user at /interfaces/annotator/index.html and so is our 'default annotator'.
+ * It is exposed to the user at /interfaces/annotator/index.html.
  *
  * The Annotation generation UI is powered by Annotorious.  The TPEN3 team hereby acknowledges
  * and thanks the Annotorious development team for this open source software.

components/column-selector/index.js

@@ -68,7 +68,8 @@ export default class ColumnSelector extends HTMLElement {
             return { ...col, label: isAuto ? `Unnamed ${i + 1}` : col.label }
         })
 
-        this.#page = await vault.get(pageId, 'annotationpage', true)
+        this.#page = await vault.getWithFallback(pageId, 'annotationpage', TPEN.activeProject?.manifest, true)
+        if (!this.#page) return
         const { orderedItems, columnsInPage, allColumnLines } = orderPageItemsByColumns(
             { columns: this.columns, items: page?.items },
             this.#page

components/continue-working/index.js

@@ -1,4 +1,5 @@
 import TPEN from '../../api/TPEN.js'
+import vault from '../../js/vault.js'
 import { stringFromDate } from '/js/utils.js'
 import { CleanupRegistry } from '../../utilities/CleanupRegistry.js'
 
@@ -172,31 +173,22 @@ class ContinueWorking extends HTMLElement {
     async getProjectThumbnail(project, annotationPageId) {
         try {
             if (!annotationPageId) return this.generateProjectPlaceholder(project)
-            const annotationPage = await fetch(`${TPEN.servicesURL}/project/${project._id}/page/${annotationPageId}`).then(r => r.json())
+            const annotationPage = await vault.get(
+                `${TPEN.servicesURL}/project/${project._id}/page/${annotationPageId}`,
+                'annotationpage',
+                true
+            )
+            if (!annotationPage) return this.generateProjectPlaceholder(project)
             const canvasId = annotationPage.target
             if (!canvasId) return this.generateProjectPlaceholder(project)
 
             let canvas, isV3
-            try {
-                canvas = await fetch(canvasId).then(r => r.json())
-                const context = canvas['@context']
-                isV3 = Array.isArray(context)
-                    ? context.some(ctx => typeof ctx === 'string' && ctx.includes('iiif.io/api/presentation/3'))
-                    : typeof context === 'string' && context.includes('iiif.io/api/presentation/3')
-            } catch {
-                // Fetch manifest
-                const manifestUrl = project.manifest?.[0]
-                if (!manifestUrl) return this.generateProjectPlaceholder(project)
-
-                const manifest = await fetch(manifestUrl).then(r => r.json())
-                const context = manifest['@context']
-                isV3 = Array.isArray(context)
-                    ? context.some(ctx => typeof ctx === 'string' && ctx.includes('iiif.io/api/presentation/3'))
-                    : typeof context === 'string' && context.includes('iiif.io/api/presentation/3')
-                const canvases = isV3 ? manifest.items : manifest.sequences?.[0]?.canvases
-                canvas = canvases?.find(c => (isV3 ? c.id : c['@id']) === canvasId)
-                if (!canvas) return this.generateProjectPlaceholder(project)
-            }
+            canvas = await vault.getWithFallback(canvasId, 'canvas', project.manifest)
+
+            if (!canvas) return this.generateProjectPlaceholder(project)
+
+            // Structure-based detection
+            isV3 = Array.isArray(canvas.items) || canvas.type === "Canvas"
 
             // Get thumbnail from canvas
             let thumbnailUrl = canvas.thumbnail?.id ?? canvas.thumbnail?.['@id'] ?? canvas.thumbnail

components/create-column/index.js

@@ -2,6 +2,7 @@ import TPEN from "../../api/TPEN.js"
 import CheckPermissions from "../check-permissions/checkPermissions.js"
 import { onProjectReady } from "../../utilities/projectReady.js"
 import { CleanupRegistry } from "../../utilities/CleanupRegistry.js"
+import vault from '../../js/vault.js'
 
 /**
  * TpenCreateColumn - Interface for creating and managing columns on annotation pages.
@@ -563,27 +564,14 @@ class TpenCreateColumn extends HTMLElement {
         return { x: xywh[0], y: xywh[1], w: xywh[2], h: xywh[3] }
     }
 
-    isValidUrl(str) {
-        try {
-            new URL(str)
-            return true
-        } catch {
-            return false
-        }
-    }
-
-    async getSpecificTypeData(type) {
-        if (!type) throw new Error("No IIIF resource provided")
-        if (typeof type === "string" && this.isValidUrl(type)) {
-            const res = await fetch(type, { cache: "no-store" })
-            if (!res.ok) throw new Error(`Fetch failed: ${res.status}`)
-            return await res.json()
-        }
-    }
-
     async fetchPageViewerData(pageID = null) {
-        const annotationPageData = pageID ? await this.getSpecificTypeData(pageID) : null
-        const canvasData = await this.getSpecificTypeData(annotationPageData.target)
+        if (!pageID) throw new Error("No page ID provided")
+        const annotationPageData = await vault.getWithFallback(pageID, 'annotationpage', TPEN.activeProject?.manifest, true)
+        if (!annotationPageData) throw new Error("Failed to load annotation page")
+        const canvasData = await vault.getWithFallback(
+            annotationPageData.target, 'canvas', TPEN.activeProject?.manifest
+        )
+        if (!canvasData) throw new Error("Failed to load canvas data")
         return await this.processDirectCanvasData(canvasData, annotationPageData)
     }
 
@@ -597,8 +585,11 @@ class TpenCreateColumn extends HTMLElement {
         if (!annotationPageData?.items) return []
         const results = await Promise.all(annotationPageData.items.map(async anno => {
             try {
-                const res = await fetch(anno.id, { cache: "no-store" })
-                const data = await res.json()
+                let data = anno
+                if (!data?.target) {
+                    data = await vault.get(anno.id ?? anno, 'annotation', true)
+                }
+                if (!data) return null
                 return { target: data?.target?.selector?.value ?? data?.target, lineId: data?.id }
             } catch { return null }
         }))

components/default-transcribe/index.js

@@ -2,6 +2,8 @@
  * TpenTranscriptionElement - Default transcription view component.
  * Displays line text and images for an annotation page.
  * @element tpen-transcription
+ *
+ * @deprecated in favor of tpen-simple-transcription.
  */
 import { userMessage, encodeContentState } from "../iiif-tools/index.js"
 import "../line-image/index.js"

components/legacy-annotator/plain.js

@@ -6,6 +6,8 @@
  *
  * It is exposed to the user through /interfaces/annotator/legacy.html
  * @element tpen-legacy-annotator
+ *
+ * @deprecated in favor of tpen-plain-annotator.
  */
 
 import { eventDispatcher } from '../../api/events.js'