Skip to content

[Shopify] Migrate Shopify Connector to Expiring Offline Access Tokens#9327

Open
onbuyuka wants to merge 35 commits into
mainfrom
bugs/637954-migrate-shopify-expiring-tokens
Open

[Shopify] Migrate Shopify Connector to Expiring Offline Access Tokens#9327
onbuyuka wants to merge 35 commits into
mainfrom
bugs/637954-migrate-shopify-expiring-tokens

Conversation

@onbuyuka

@onbuyuka onbuyuka commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Background

As of December 2025, Shopify supports expiring offline access tokens with refresh-token rotation. Public apps created before 2026-04-01 must migrate to expiring tokens by 2027-01-01; after that date, REST/GraphQL Admin API requests made with non-expiring tokens are rejected. The Shopify Connector currently uses non-expiring offline tokens and must be migrated.

Token characteristics (Shopify):

  • Access token lifetime: 1 hour (expires_in: 3600)
  • Refresh token lifetime: 90 days (refresh_token_expires_in: 7776000)
  • Migration is a one-time, irreversible token exchange per shop (the original non-expiring token is revoked on success)
  • Only one refreshable expiring token per app + store

Refs: About offline access tokens - Migrating from non-expiring to expiring tokens

Approach

Token validity is centralized in a single orchestrator, EnsureValidAccessToken(Store), invoked from Shpfy Communication Mgt.GetAccessToken - so every API call (interactive and background) transparently migrates a legacy token or refreshes an expiring one before use. A scheduled backstop job additionally keeps idle shops access/refresh tokens alive and completes migrations for shops that never make an on-demand call.

Changes

Token storage - Shpfy Registered Store New (table 30138)

  • New fields Token Expires At / Refresh Token Expires At; refresh token kept in IsolatedStorage.

Acquisition, refresh & migration - Shpfy Authentication Mgt. (codeunit 30199)

  • New installs request expiring tokens (expiring=1).
  • EnsureValidAccessToken orchestrates migrate-if-legacy / refresh-if-near-expiry, serialized per shop.
  • RefreshAccessToken (grant_type=refresh_token) with transient retry and a terminal reconnect error.
  • MigrateToExpiringToken (one-time token exchange, best-effort - keeps the working token on failure).
  • ForceTokenRefresh, IsRefreshTokenExpired, and a Shop Card reconnect notification.

Consumption - Shpfy Communication Mgt.

  • GetAccessToken invokes the orchestrator; reactive 401 -> refresh -> retry in ExecuteWebRequest.

Scheduled backstop - Shpfy Token Refresh (30431) + Shpfy Token Refresh Shop (30432)

  • Dispatcher (TableNo = Job Queue Entry) iterates enabled shops and runs the per-shop worker (TableNo = Shpfy Shop) via Codeunit.Run for isolation; recurring Job Queue entry registered via installer + upgrade.

UX - Shpfy Shop Card

  • Reconnect notification when the refresh token has expired.

Docs & tests

  • Shpfy Token Refresh Test unit tests; living docs refreshed (business-logic, data-model, patterns, CLAUDE.md).

Telemetry

Emitted for migration success/failure, refresh success/transient/terminal and backstop per-shop failures, under event IDs 0000UIV-0000UJ1.

Test plan

  • Shopify App project compiles clean.
  • Shpfy Token Refresh Test unit tests for the expiry decisions.
  • Install / refresh / migration / backstop / notification scenarios validated locally.

Fixes AB#637954

Adds support for Shopify expiring offline access tokens (public apps must
migrate by 2027-01-01):

- Persist token/refresh expiry on "Shpfy Registered Store New" and store the
  refresh token in IsolatedStorage.
- Request expiring tokens on install (expiring=1), refresh before expiry, and
  migrate legacy non-expiring tokens via token exchange (best-effort).
- Orchestrate on-demand from "Shpfy Communication Mgt." (GetAccessToken) with a
  reactive 401 refresh-and-retry.
- Scheduled backstop job ("Shpfy Token Refresh" + per-shop worker) to keep
  tokens and 90-day refresh tokens alive; registered via installer/upgrade.
- Shop Card reconnect notification when the refresh token has expired.
- Tests for the refresh-token expiry decision logic.

NOTE: "Shpfy Token Dev Tools" (page 30440) is temporary test scaffolding and
must be removed before merge.

Fixes AB#637954

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
@github-actions github-actions Bot added the AL: Apps (W1) Add-on apps for W1 label Jul 11, 2026
@github-actions github-actions Bot added this to the Version 29.0 milestone Jul 11, 2026
onbuyuka and others added 4 commits July 11, 2026 17:06
- Remove the temporary "Shpfy Token Dev Tools" page (30440) and its permission-set entry.
- Clear the new telemetry event IDs (set to '') so they can be assigned by the tagging script.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Refresh the living docs (al-docs update) to cover slice 637954:
- business-logic.md: new "Authentication and token lifecycle" section.
- data-model.md: Shpfy Registered Store New token/expiry storage + ER entry.
- patterns.md: Job Queue dispatcher/worker per-shop isolation pattern.
- CLAUDE.md (app + Base): auth overview and "things to know".

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Remove the LogTokenTelemetry helper and inline Session.LogMessage('', ...) at
each call site so the telemetry tagging script can assign event IDs (it only
fills literal Session.LogMessage first-args). Matches the existing connector
convention. Tags left empty pending the script.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Fill the token lifecycle telemetry event IDs (0000UIV-0000UJ1) via the tagging script.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
- Mark SaveInstalledToken [NonDebuggable] (it holds the token-bearing response body).
- Give the token-refresh backstop its own Job Queue category (SHPFYAUTH) so it is
  not serialized behind long-running SHPFY sync jobs (pricing/inventory) and can
  refresh 1-hour access tokens promptly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
@onbuyuka
onbuyuka marked this pull request as ready for review July 13, 2026 06:57
@onbuyuka
onbuyuka requested a review from a team July 13, 2026 06:58
Comment thread src/Apps/W1/Shopify/App/src/Integration/Codeunits/ShpfyTokenRefresh.Codeunit.al Outdated
Comment thread src/Apps/W1/Shopify/App/src/Integration/Tables/ShpfyRegisteredStoreNew.Table.al Outdated
Comment thread src/Apps/W1/Shopify/App/src/Integration/Codeunits/ShpfyTokenRefresh.Codeunit.al Outdated
@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Copilot PR Review

Iteration 11 · Outcome: completed

Knowledge source: https://github.com/microsoft/BCQuality@186d8a131465475c79244d994acb872cd5c0d4bf

Findings by domain

Findings split into Knowledge-backed (cite a BCQuality article) and Agent (the agent's own judgement, no matching BCQuality rule).

Domain Findings Knowledge-backed Agent Inline Fallback
Error Handling 1 1 0 1 0
Performance 1 1 0 0 0

Totals: 2 knowledge-backed · 0 agent findings.

Orchestrator pre-filter (2 file(s) excluded)

  • layer-disabled (knowledge) : 2 file(s)

Findings produced by the AL review agent v1.7.3. Reply 👎 on any inline comment to flag false positives.

darjoo
darjoo previously requested changes Jul 13, 2026
- Encrypt the refresh token at rest: IsolatedStorage.SetEncrypted instead of Set
  (90-day credential; Get transparently decrypts).
- Mark ResponseHasAccessToken [NonDebuggable] (handles the token-bearing body).
- Remove the redundant per-iteration Commit() in the token-refresh dispatcher
  loop; the per-shop worker (Codeunit.Run) already commits on success.
- Guard Codeunit.Run("Job Queue - Enqueue") in ScheduleRefreshJob so an enqueue
  failure is logged instead of aborting the install/upgrade.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Comment thread src/Apps/W1/Shopify/App/src/Base/Codeunits/ShpfyCommunicationMgt.Codeunit.al Outdated
@github-actions

Copy link
Copy Markdown
Contributor

$\textbf{🟠\ High\ Severity\ —\ Security} \quad \color{gray}{\texttt{\small Iteration\ 2}}$

SaveTokenResponse (new in this PR) stores the Shopify offline access token via RegisteredStoreNew.SetAccessToken(), which persists it with IsolatedStorage.Set (unencrypted at rest), while the refresh token introduced by this same PR is correctly stored via SetRefreshToken() using IsolatedStorage.SetEncrypted.

Both are OAuth credentials for the same store and belong in the same encrypted class. This inconsistency becomes load-bearing now: every store going forward will have an encrypted refresh token sitting next to an unencrypted, equally sensitive access token in the same record. Change SetAccessToken (Shpfy Registered Store New table) to use SetEncrypted for parity with the refresh token.

Knowledge:

Line mapping was unavailable, so this was posted as an issue comment.

👍 useful · ❤️ especially valuable · 👎 wrong - reply with why

onbuyuka and others added 2 commits July 13, 2026 10:50
Fill 0000UJ6 (enqueue-failure warning) via the tagging script.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
A legacy store has no refresh token, so EnsureValidAccessToken never hits the
fast path and re-attempts migration on every API call -- in a sync loop over
many records a failing migration means many redundant lock+HTTP+commit round
trips. Add a "Last Migration Attempt" timestamp on Shpfy Registered Store New
and only re-attempt migration once per hour (ShouldAttemptMigration/TryMigrate,
shared by EnsureValidAccessToken and ForceTokenRefresh). Docs updated.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Comment thread src/Apps/W1/Shopify/App/src/Integration/Codeunits/ShpfyTokenRefresh.Codeunit.al Outdated
Comment thread src/Apps/W1/Shopify/App/src/Integration/Codeunits/ShpfyTokenRefresh.Codeunit.al Outdated
Comment thread src/Apps/W1/Shopify/App/src/Integration/Codeunits/ShpfyTokenRefresh.Codeunit.al Outdated
onbuyuka and others added 3 commits July 13, 2026 11:12
- Do not interpolate GetLastErrorText() into the telemetry message string
  (LogRefreshFailure and the schedule-failure log). Use a generic message,
  move the error detail and shop code to custom dimensions, and classify the
  event as CustomerContent so the platform can handle it appropriately.
- Move the shared telemetry Labels to the codeunit object scope.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Codeunit 139635 is already used by "E-Doc. Receive Files" in the EDocument
test app; BCApps validates test object IDs globally. Move Shpfy Token Refresh
Test to 139613, which is free repo-wide within the Shopify test ID ranges.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
…riant API Test)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
onbuyuka and others added 3 commits July 20, 2026 14:34
…l loudly on token exchange

- Circuit-breaker cooldown: ForceTokenRefresh committed the Last Force Refresh At
  marker only after RefreshAccessToken returned, so a terminal reconnect error
  rolled it back and every subsequent 401 re-entered the refresh. Commit the
  cooldown before the refresh attempt so the throttle persists on the failing path.
- Reconnect ErrorInfo: set Title and DetailedMessage alongside the existing
  Reconnect action so the actionable error has a clear heading and supporting detail.
- Initial OAuth token exchange: GetToken exited silently on a non-success status or
  a payload without an access token, so Request Access/reconnect failed with no
  feedback. Raise a specific error instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
…all/upgrade

Enqueuing a Job Queue Entry runs Codeunit.Run("Job Queue - Enqueue"), which
implicitly commits on success (and ScheduleTask commits to register the task).
Implicit commits are not allowed inside install/upgrade triggers, so scheduling
the backstop from OnInstallAppPerCompany / OnUpgradePerCompany crashed a fresh
install with "the transaction is stopped".

Move scheduling out of the install/upgrade transaction entirely: schedule the
recurring job lazily from ShpfyCommunicationMgt.GetAccessToken (the single
chokepoint for all connector API usage), once per session via a SingleInstance
guard. This runs in a normal committable runtime context and uniformly covers
install, upgrade, new companies, and pre-existing shops. Removed the install and
upgrade scheduling calls and the now-unused upgrade tag.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
IsolatedStorage.SetEncrypted throws "An encryption key is required" when no
encryption key is configured (e.g. on-prem). Guard both SetAccessToken and
SetRefreshToken with EncryptionEnabled(), falling back to IsolatedStorage.Set
otherwise - the same pattern the base app OAuth token storage uses. On SaaS
(where encryption is always on) tokens remain encrypted at rest.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
@onbuyuka
onbuyuka enabled auto-merge July 20, 2026 13:20
@onbuyuka
onbuyuka requested a review from darjoo July 20, 2026 13:23
Comment thread src/Apps/W1/Shopify/App/src/Integration/Codeunits/ShpfyTokenRefresh.Codeunit.al Outdated
onbuyuka and others added 3 commits July 20, 2026 15:46
…ransaction

EnsureBackstopScheduled ran from GetAccessToken at the start of ExecuteWebRequest,
so its Codeunit.Run("Job Queue - Enqueue") could execute while the caller still had
an open write transaction (nesting / implicit-commit hazard). Move the once-per-
session scheduling to the end of ExecuteWebRequest, right after the request's own
Commit(), where no write transaction is open. The flag is now only set once the
scheduling actually runs, so an early-erroring first request retries on the next.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
…e API path

Move the once-only backstop scheduling out of ShpfyCommunicationMgt entirely and
into ShpfyShopCard.OnOpenPage. The API request path no longer runs any job-queue
query or Codeunit.Run enqueue - scheduling now happens in the page/UI context when
an enabled Shop Card is opened. ScheduleRefreshJob stays idempotent (exits when the
job already exists). Removes the per-session EnsureBackstopScheduled hook and its
flag from the SingleInstance communication codeunit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Scheduling from the Shop Card open would recreate the Job Queue Entry every time
the page opened, so an administrator could not permanently remove it. Add
EnsureBackstopScheduled, which schedules the job at most once per company using an
upgrade tag as a persistent marker: once scheduled, the tag is set and the job is
never silently recreated - if an admin deletes it, it stays deleted. The tag is set
only on a successful enqueue (a transient failure retries on a later open). The tag
is intentionally NOT registered in OnGetPerCompanyUpgradeTags so newly created
companies still schedule on first Shop Card open.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Comment thread src/Apps/W1/Shopify/App/src/Base/Pages/ShpfyShopCard.Page.al Outdated
Comment thread src/Apps/W1/Shopify/App/src/Base/Pages/ShpfyShopCard.Page.al Outdated
- Drop the upgrade-tag "scheduled once" marker; ScheduleRefreshJob keeps only the
  existing-entry check, so new shops are always covered and an admin can disable the
  backstop by setting the Job Queue Entry On Hold. Make it a plain procedure (the
  Boolean return was only needed by the removed tag caller) and call it directly
  from the Shop Card.
- Trim verbose explanatory comments in the token codeunits to concise notes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
The Shpfy Token Refresh job is scheduled when an enabled Shopify Shop Card is
opened, not on install/upgrade; update business-logic.md to match.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
darjoo
darjoo previously approved these changes Jul 21, 2026
- Test fixture: RegisterAccessTokenForShop now seeds a healthy expiring token
  (refresh token + future expiries) instead of a legacy access-token-only store.
  After this feature GetAccessToken calls EnsureValidAccessToken on every request;
  a legacy-style fixture made API-call tests attempt migration -> GetClientId,
  which errors in the non-SaaS CI environment ("only supported in SaaS"). A healthy
  expiring token makes EnsureValidAccessToken a no-op (no HTTP, no GetClientId).
- Docs: correct base CLAUDE.md - the backstop Job Queue entry is scheduled from the
  Shop Card on open, not registered on install/upgrade.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
RefreshAccessToken is shared by the proactive on-demand path (EnsureValidAccessToken,
under a table lock), the backstop worker (also via EnsureValidAccessToken), and the
reactive 401 handler (ForceTokenRefresh, unlocked). The in-lock 3-attempt retry loop
(Sleep 1+2+3s + up to 3 round trips) sat on the hot path of every API call, serializing
concurrent sessions in the same company across the token request.

- Add a MaxAttempts argument to RefreshAccessToken.
- EnsureValidAccessToken (proactive + backstop) passes 1: the token is still valid within
  the 5-minute buffer, so a transient failure is left for the next call or the backstop.
  Lock hold drops to ~1 round trip without weakening the single-refresh guarantee.
- ForceTokenRefresh (reactive, unlocked) keeps 3: on a 401 the token is already dead, and
  these retries are free (no lock) and valuable for recovery.
- Guard the backoff with Attempt < MaxAttempts so the final iteration no longer sleeps.
- Docs: business-logic.md now describes the on-demand refresh as a single attempt.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
if not ErrorInfo.CustomDimensions.ContainsKey(StoreDimensionTok) then
exit;
Store := ErrorInfo.CustomDimensions.Get(StoreDimensionTok).ToLower();
Shop.SetRange(Enabled, true);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$\textbf{🟠\ High\ Severity\ —\ Performance}$

ReconnectFromError scans every enabled shop with FindSet() and then tests Shop.GetStoreName() inside the loop. That store predicate can be pushed into the database with a filter on the shop URL instead of reading all enabled shops in AL before reconnecting the one matching store.

Knowledge:

👍 useful · ❤️ especially valuable · 👎 wrong - reply with why · AL review agent v1.12.4

SetupStore(RegisteredStoreNew, true, DaysFromNow(1));

// [THEN] The refresh token is present.
LibraryAssert.IsTrue(RegisteredStoreNew.HasRefreshToken(), 'Refresh token should be stored.');

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$\textbf{🟡\ Medium\ Severity\ —\ Testing}$

RefreshTokenRoundTrips never checks that the token returned by GetRefreshToken is the same token written in SetupStore; it only asserts that some non-empty secret is returned. A broken implementation that returns the wrong token would still pass. Capture the generated refresh token and assert the exact round-tripped value.

👍 useful · ❤️ especially valuable · 👎 wrong - reply with why · AL review agent v1.12.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AL: Apps (W1) Add-on apps for W1 Integration GitHub request for Integration area

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants