[Shopify] Migrate Shopify Connector to Expiring Offline Access Tokens#9327
[Shopify] Migrate Shopify Connector to Expiring Offline Access Tokens#9327onbuyuka wants to merge 35 commits into
Conversation
Adds support for Shopify expiring offline access tokens (public apps must
migrate by 2027-01-01):
- Persist token/refresh expiry on "Shpfy Registered Store New" and store the
refresh token in IsolatedStorage.
- Request expiring tokens on install (expiring=1), refresh before expiry, and
migrate legacy non-expiring tokens via token exchange (best-effort).
- Orchestrate on-demand from "Shpfy Communication Mgt." (GetAccessToken) with a
reactive 401 refresh-and-retry.
- Scheduled backstop job ("Shpfy Token Refresh" + per-shop worker) to keep
tokens and 90-day refresh tokens alive; registered via installer/upgrade.
- Shop Card reconnect notification when the refresh token has expired.
- Tests for the refresh-token expiry decision logic.
NOTE: "Shpfy Token Dev Tools" (page 30440) is temporary test scaffolding and
must be removed before merge.
Fixes AB#637954
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
- Remove the temporary "Shpfy Token Dev Tools" page (30440) and its permission-set entry. - Clear the new telemetry event IDs (set to '') so they can be assigned by the tagging script. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Refresh the living docs (al-docs update) to cover slice 637954: - business-logic.md: new "Authentication and token lifecycle" section. - data-model.md: Shpfy Registered Store New token/expiry storage + ER entry. - patterns.md: Job Queue dispatcher/worker per-shop isolation pattern. - CLAUDE.md (app + Base): auth overview and "things to know". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Remove the LogTokenTelemetry helper and inline Session.LogMessage('', ...) at
each call site so the telemetry tagging script can assign event IDs (it only
fills literal Session.LogMessage first-args). Matches the existing connector
convention. Tags left empty pending the script.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Fill the token lifecycle telemetry event IDs (0000UIV-0000UJ1) via the tagging script. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
- Mark SaveInstalledToken [NonDebuggable] (it holds the token-bearing response body). - Give the token-refresh backstop its own Job Queue category (SHPFYAUTH) so it is not serialized behind long-running SHPFY sync jobs (pricing/inventory) and can refresh 1-hour access tokens promptly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Copilot PR ReviewIteration 11 · Outcome: completed Knowledge source: https://github.com/microsoft/BCQuality@186d8a131465475c79244d994acb872cd5c0d4bf Findings by domainFindings split into Knowledge-backed (cite a BCQuality article) and Agent (the agent's own judgement, no matching BCQuality rule).
Totals: 2 knowledge-backed · 0 agent findings. Orchestrator pre-filter (2 file(s) excluded)
Findings produced by the AL review agent v1.7.3. Reply 👎 on any inline comment to flag false positives. |
- Encrypt the refresh token at rest: IsolatedStorage.SetEncrypted instead of Set
(90-day credential; Get transparently decrypts).
- Mark ResponseHasAccessToken [NonDebuggable] (handles the token-bearing body).
- Remove the redundant per-iteration Commit() in the token-refresh dispatcher
loop; the per-shop worker (Codeunit.Run) already commits on success.
- Guard Codeunit.Run("Job Queue - Enqueue") in ScheduleRefreshJob so an enqueue
failure is logged instead of aborting the install/upgrade.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
SaveTokenResponse (new in this PR) stores the Shopify offline access token via RegisteredStoreNew.SetAccessToken(), which persists it with IsolatedStorage.Set (unencrypted at rest), while the refresh token introduced by this same PR is correctly stored via SetRefreshToken() using IsolatedStorage.SetEncrypted.Both are OAuth credentials for the same store and belong in the same encrypted class. This inconsistency becomes load-bearing now: every store going forward will have an encrypted refresh token sitting next to an unencrypted, equally sensitive access token in the same record. Change SetAccessToken (Shpfy Registered Store New table) to use SetEncrypted for parity with the refresh token. Knowledge: Line mapping was unavailable, so this was posted as an issue comment. 👍 useful · ❤️ especially valuable · 👎 wrong - reply with why |
Fill 0000UJ6 (enqueue-failure warning) via the tagging script. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
A legacy store has no refresh token, so EnsureValidAccessToken never hits the fast path and re-attempts migration on every API call -- in a sync loop over many records a failing migration means many redundant lock+HTTP+commit round trips. Add a "Last Migration Attempt" timestamp on Shpfy Registered Store New and only re-attempt migration once per hour (ShouldAttemptMigration/TryMigrate, shared by EnsureValidAccessToken and ForceTokenRefresh). Docs updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
- Do not interpolate GetLastErrorText() into the telemetry message string (LogRefreshFailure and the schedule-failure log). Use a generic message, move the error detail and shop code to custom dimensions, and classify the event as CustomerContent so the platform can handle it appropriately. - Move the shared telemetry Labels to the codeunit object scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Codeunit 139635 is already used by "E-Doc. Receive Files" in the EDocument test app; BCApps validates test object IDs globally. Move Shpfy Token Refresh Test to 139613, which is free repo-wide within the Shopify test ID ranges. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
…riant API Test) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
…l loudly on token exchange - Circuit-breaker cooldown: ForceTokenRefresh committed the Last Force Refresh At marker only after RefreshAccessToken returned, so a terminal reconnect error rolled it back and every subsequent 401 re-entered the refresh. Commit the cooldown before the refresh attempt so the throttle persists on the failing path. - Reconnect ErrorInfo: set Title and DetailedMessage alongside the existing Reconnect action so the actionable error has a clear heading and supporting detail. - Initial OAuth token exchange: GetToken exited silently on a non-success status or a payload without an access token, so Request Access/reconnect failed with no feedback. Raise a specific error instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
…all/upgrade
Enqueuing a Job Queue Entry runs Codeunit.Run("Job Queue - Enqueue"), which
implicitly commits on success (and ScheduleTask commits to register the task).
Implicit commits are not allowed inside install/upgrade triggers, so scheduling
the backstop from OnInstallAppPerCompany / OnUpgradePerCompany crashed a fresh
install with "the transaction is stopped".
Move scheduling out of the install/upgrade transaction entirely: schedule the
recurring job lazily from ShpfyCommunicationMgt.GetAccessToken (the single
chokepoint for all connector API usage), once per session via a SingleInstance
guard. This runs in a normal committable runtime context and uniformly covers
install, upgrade, new companies, and pre-existing shops. Removed the install and
upgrade scheduling calls and the now-unused upgrade tag.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
IsolatedStorage.SetEncrypted throws "An encryption key is required" when no encryption key is configured (e.g. on-prem). Guard both SetAccessToken and SetRefreshToken with EncryptionEnabled(), falling back to IsolatedStorage.Set otherwise - the same pattern the base app OAuth token storage uses. On SaaS (where encryption is always on) tokens remain encrypted at rest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
…ransaction
EnsureBackstopScheduled ran from GetAccessToken at the start of ExecuteWebRequest,
so its Codeunit.Run("Job Queue - Enqueue") could execute while the caller still had
an open write transaction (nesting / implicit-commit hazard). Move the once-per-
session scheduling to the end of ExecuteWebRequest, right after the request's own
Commit(), where no write transaction is open. The flag is now only set once the
scheduling actually runs, so an early-erroring first request retries on the next.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
…e API path Move the once-only backstop scheduling out of ShpfyCommunicationMgt entirely and into ShpfyShopCard.OnOpenPage. The API request path no longer runs any job-queue query or Codeunit.Run enqueue - scheduling now happens in the page/UI context when an enabled Shop Card is opened. ScheduleRefreshJob stays idempotent (exits when the job already exists). Removes the per-session EnsureBackstopScheduled hook and its flag from the SingleInstance communication codeunit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Scheduling from the Shop Card open would recreate the Job Queue Entry every time the page opened, so an administrator could not permanently remove it. Add EnsureBackstopScheduled, which schedules the job at most once per company using an upgrade tag as a persistent marker: once scheduled, the tag is set and the job is never silently recreated - if an admin deletes it, it stays deleted. The tag is set only on a successful enqueue (a transient failure retries on a later open). The tag is intentionally NOT registered in OnGetPerCompanyUpgradeTags so newly created companies still schedule on first Shop Card open. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
- Drop the upgrade-tag "scheduled once" marker; ScheduleRefreshJob keeps only the existing-entry check, so new shops are always covered and an admin can disable the backstop by setting the Job Queue Entry On Hold. Make it a plain procedure (the Boolean return was only needed by the removed tag caller) and call it directly from the Shop Card. - Trim verbose explanatory comments in the token codeunits to concise notes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
The Shpfy Token Refresh job is scheduled when an enabled Shopify Shop Card is opened, not on install/upgrade; update business-logic.md to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
- Test fixture: RegisterAccessTokenForShop now seeds a healthy expiring token
(refresh token + future expiries) instead of a legacy access-token-only store.
After this feature GetAccessToken calls EnsureValidAccessToken on every request;
a legacy-style fixture made API-call tests attempt migration -> GetClientId,
which errors in the non-SaaS CI environment ("only supported in SaaS"). A healthy
expiring token makes EnsureValidAccessToken a no-op (no HTTP, no GetClientId).
- Docs: correct base CLAUDE.md - the backstop Job Queue entry is scheduled from the
Shop Card on open, not registered on install/upgrade.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
RefreshAccessToken is shared by the proactive on-demand path (EnsureValidAccessToken, under a table lock), the backstop worker (also via EnsureValidAccessToken), and the reactive 401 handler (ForceTokenRefresh, unlocked). The in-lock 3-attempt retry loop (Sleep 1+2+3s + up to 3 round trips) sat on the hot path of every API call, serializing concurrent sessions in the same company across the token request. - Add a MaxAttempts argument to RefreshAccessToken. - EnsureValidAccessToken (proactive + backstop) passes 1: the token is still valid within the 5-minute buffer, so a transient failure is left for the next call or the backstop. Lock hold drops to ~1 round trip without weakening the single-refresh guarantee. - ForceTokenRefresh (reactive, unlocked) keeps 3: on a 401 the token is already dead, and these retries are free (no lock) and valuable for recovery. - Guard the backoff with Attempt < MaxAttempts so the final iteration no longer sleeps. - Docs: business-logic.md now describes the on-demand refresh as a single attempt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
| if not ErrorInfo.CustomDimensions.ContainsKey(StoreDimensionTok) then | ||
| exit; | ||
| Store := ErrorInfo.CustomDimensions.Get(StoreDimensionTok).ToLower(); | ||
| Shop.SetRange(Enabled, true); |
There was a problem hiding this comment.
ReconnectFromError scans every enabled shop with FindSet() and then tests Shop.GetStoreName() inside the loop. That store predicate can be pushed into the database with a filter on the shop URL instead of reading all enabled shops in AL before reconnecting the one matching store.
Knowledge:
👍 useful · ❤️ especially valuable · 👎 wrong - reply with why · AL review agent v1.12.4
| SetupStore(RegisteredStoreNew, true, DaysFromNow(1)); | ||
|
|
||
| // [THEN] The refresh token is present. | ||
| LibraryAssert.IsTrue(RegisteredStoreNew.HasRefreshToken(), 'Refresh token should be stored.'); |
There was a problem hiding this comment.
RefreshTokenRoundTrips never checks that the token returned by GetRefreshToken is the same token written in SetupStore; it only asserts that some non-empty secret is returned. A broken implementation that returns the wrong token would still pass. Capture the generated refresh token and assert the exact round-tripped value.
👍 useful · ❤️ especially valuable · 👎 wrong - reply with why · AL review agent v1.12.4
Background
As of December 2025, Shopify supports expiring offline access tokens with refresh-token rotation. Public apps created before 2026-04-01 must migrate to expiring tokens by 2027-01-01; after that date, REST/GraphQL Admin API requests made with non-expiring tokens are rejected. The Shopify Connector currently uses non-expiring offline tokens and must be migrated.
Token characteristics (Shopify):
expires_in: 3600)refresh_token_expires_in: 7776000)Refs: About offline access tokens - Migrating from non-expiring to expiring tokens
Approach
Token validity is centralized in a single orchestrator,
EnsureValidAccessToken(Store), invoked fromShpfy Communication Mgt.GetAccessToken- so every API call (interactive and background) transparently migrates a legacy token or refreshes an expiring one before use. A scheduled backstop job additionally keeps idle shops access/refresh tokens alive and completes migrations for shops that never make an on-demand call.Changes
Token storage -
Shpfy Registered Store New(table 30138)Token Expires At/Refresh Token Expires At; refresh token kept in IsolatedStorage.Acquisition, refresh & migration -
Shpfy Authentication Mgt.(codeunit 30199)expiring=1).EnsureValidAccessTokenorchestrates migrate-if-legacy / refresh-if-near-expiry, serialized per shop.RefreshAccessToken(grant_type=refresh_token) with transient retry and a terminal reconnect error.MigrateToExpiringToken(one-time token exchange, best-effort - keeps the working token on failure).ForceTokenRefresh,IsRefreshTokenExpired, and a Shop Card reconnect notification.Consumption -
Shpfy Communication Mgt.GetAccessTokeninvokes the orchestrator; reactive 401 -> refresh -> retry inExecuteWebRequest.Scheduled backstop -
Shpfy Token Refresh(30431) +Shpfy Token Refresh Shop(30432)TableNo = Job Queue Entry) iterates enabled shops and runs the per-shop worker (TableNo = Shpfy Shop) viaCodeunit.Runfor isolation; recurring Job Queue entry registered via installer + upgrade.UX -
Shpfy Shop CardDocs & tests
Shpfy Token Refresh Testunit tests; living docs refreshed (business-logic, data-model, patterns, CLAUDE.md).Telemetry
Emitted for migration success/failure, refresh success/transient/terminal and backstop per-shop failures, under event IDs
0000UIV-0000UJ1.Test plan
Shpfy Token Refresh Testunit tests for the expiry decisions.Fixes AB#637954