Skip to content

Commit 7a23adc

Browse files
author
Magnus Hartvig Grønbech
committed
Transactions and Receipts Storage Docs
Add AL documentation for the Transaction Storage compliance archival app: - CLAUDE.md: App overview, Azure Blob Storage archival, incremental export - docs/data-model.md: 6 tables, export state tracking, task audit trail - docs/business-logic.md: GL posting trigger, 29-table export, Azure upload, retry - docs/extensibility.md: Event subscribers, closed system design - docs/patterns.md: Incremental extraction, binary search, distributed scheduling
1 parent dba2f88 commit 7a23adc

5 files changed

Lines changed: 605 additions & 0 deletions

File tree

‎src/Apps/W1/TransactionStorage/App/CLAUDE.md‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# Transaction Storage
2+
3+
Compliance archival system that exports Business Central transaction data and
4+
document attachments to Azure Blob Storage. Designed for regulatory requirements
5+
(e.g., Denmark Bookkeeping Act) requiring machine-readable transaction preservation.
6+
7+
## Quick reference
8+
9+
**App ID:** `c832ba23-ab01-45f5-8cb2-bfdf061a7a8c`
10+
**Object range:** 6200-6250
11+
**Core objects:** 14 (6 tables, 6 codeunits, 1 page, 1 enum)
12+
**Dependencies:** None
13+
**Target markets:** SaaS production environments with compliance requirements
14+
15+
## How it works
16+
17+
### Trigger mechanism
18+
19+
The system uses an event-driven approach to initiate exports:
20+
21+
1. **GL posting** (OnAfterGLFinishPosting) sets a flag indicating transactions occurred
22+
2. **User logout** (OnAfterCompanyClose) detects the flag and schedules a background export task
23+
3. **Background task** runs asynchronously, typically during scheduled time slots (2:00-4:40 AM)
24+
25+
### Incremental export
26+
27+
The system tracks `last_handled_datetime` per table using `SystemModifiedAt` timestamps.
28+
Only new or modified records since the last export are included. This prevents
29+
re-exporting unchanged data.
30+
31+
### Data scope
32+
33+
Exports **29 tables** covering the full transaction audit trail:
34+
35+
- **Ledger entries:** G/L Entry, VAT Entry, Customer/Vendor Ledger, Item Ledger, FA Ledger
36+
- **Documents:** Sales/Purchase Invoice/Credit Memo headers and lines
37+
- **Reminders:** Reminder/Finance Charge headers and lines
38+
- **Master data:** GL Account, Customer, Vendor, Bank, Fixed Asset (referenced by transactions)
39+
40+
### Chunking and limits
41+
42+
- Records exported in chunks of **50,050 records** to manage memory
43+
- Binary search algorithm limits each table to **200,000 records per run**
44+
- This prevents runaway exports on large datasets
45+
46+
### Azure upload
47+
48+
- **Dual endpoints:** JSON text endpoint for transactions + base64 document endpoint for attachments
49+
- **Authentication:** OAuth2 certificate auth with secrets from Azure Key Vault
50+
- **Blob container:** Formatted Company Registration Number (CVR)
51+
- **Blob path:** `{AAD-Tenant-ID}_{Env-Name}/{YYYYMMDD}/`
52+
- **Retention:** 6 years (or 6 years from fiscal year-end)
53+
54+
### Document attachments
55+
56+
Incoming document attachments linked to GL entries are exported as base64-encoded
57+
files. Maximum size: **100MB per attachment**.
58+
59+
### Error handling
60+
61+
- **Retry logic:** Up to 4 attempts with 5-15 minute random delay
62+
- **Timeout detection:** Handles long-running operations
63+
- **OutOfMemory handling:** Graceful degradation on memory exhaustion
64+
- **Critical alerts:** 7-day consecutive failure monitoring
65+
66+
### Scheduling
67+
68+
- **Distributed scheduling:** Tenant export times spread across 2:00-4:40 AM
69+
- **Time slot calculation:** Based on tenant ID hex digits to prevent server load spikes
70+
71+
## Structure
72+
73+
All objects are in the flat `src/` folder:
74+
75+
- **Tables (6):** Setup, export status tracking, failure monitoring, task scheduling
76+
- **Codeunits (6):** Export orchestration, Azure upload, retry logic, scheduling, event subscribers
77+
- **Page (1):** Setup page for configuration
78+
- **Enum (1):** Export status values
79+
- **Permissions:** Permission set definitions in `Permissions/` folder
80+
81+
## Documentation
82+
83+
Reference materials are in the parent `TransactionStorage/` folder:
84+
85+
- **README.md** -- Overview and compliance context
86+
- **Business Central Transactions and Receipts API Specification.md** -- API contract for Azure endpoint
87+
88+
## Things to know
89+
90+
### Country-specific behavior
91+
92+
- **Denmark:** First run date = 2024-01-01 (when regulation took effect)
93+
- Other countries may have different start dates or opt-in configuration
94+
95+
### Environment filtering
96+
97+
The system **only runs in SaaS production** environments. It skips:
98+
99+
- Demo companies
100+
- Evaluation companies
101+
- Sandbox environments
102+
103+
This prevents test data from polluting compliance archives.
104+
105+
### Background tasks
106+
107+
Exports run as Task Scheduler background tasks. They do not block user sessions
108+
and can be monitored via the Task Scheduler page. If a task fails, the retry
109+
logic reschedules it automatically.
110+
111+
### Azure Functions dependency
112+
113+
The system requires a deployed Azure Function endpoint to receive exported data.
114+
The endpoint URL, certificate thumbprint, and Key Vault details are configured
115+
in the setup table. Without valid Azure configuration, exports will fail silently
116+
until corrected.
117+
118+
### SystemModifiedAt tracking
119+
120+
The incremental export relies on `SystemModifiedAt` timestamps, which are
121+
automatically maintained by the platform. Do not manually modify these fields
122+
or the export will skip records.

‎src/Apps/W1/TransactionStorage/App/docs/business-logic.md‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
# Business Logic
2+
3+
## Overview
4+
5+
The Transaction Storage extension implements an event-driven architecture that captures Business Central transactions, schedules background exports, and uploads data to Azure Blob Storage for long-term archival.
6+
7+
## Trigger Mechanism
8+
9+
**TransStoragePostingState** (SingleInstance codeunit) decouples posting from export:
10+
11+
- Subscribes to **OnAfterGLFinishPosting** (Gen. Jnl.-Post Line) -- sets internal flag when GL entries are posted
12+
- Subscribes to **OnAfterCompanyClose** (LogInManagement) -- checks flag at user logout; if set, calls TransStorageScheduleTask to schedule export
13+
- Non-blocking design -- posting completes immediately without waiting for export
14+
15+
## Scheduling Logic
16+
17+
**TransStorageScheduleTask** validates environment and creates a background task:
18+
19+
1. **Environment validation** -- ensures SaaS production environment (skips on-prem, demo, evaluation)
20+
2. **Duplicate prevention** -- checks if a valid scheduled task already exists; if so, exits without creating duplicate
21+
3. **Retry reset** -- sets NoOfTasksAttempts to 3 in TransactStorageSetup
22+
4. **Start time calculation** -- reads Earliest Start Time from setup; if that time has passed today, schedules for tomorrow; otherwise schedules for today
23+
5. **Tenant distribution** -- CalcTenantExportStartTime() uses first 2 hex characters of AAD tenant ID to distribute start times across 2:00-4:40 AM window (prevents thundering herd)
24+
6. **Task creation** -- creates TaskScheduler record with:
25+
- Run codeunit: TransactStorageExport
26+
- Error handler: TransStorageErrorHandler
27+
- Scheduled start time
28+
29+
## Export Process
30+
31+
**TransactStorageExport.OnRun()** delegates to **TransactStorageExportData.ExportData()**:
32+
33+
1. **Table iteration** -- processes 29 hard-coded tables (GL Entry, Sales Header, Sales Line, Purchase Header, etc.)
34+
2. **Date range filtering** -- for each table, calls SetRangeOnDataTable():
35+
- SystemModifiedAt between last_handled_datetime (from TransactStorageTableEntry) and task_start_datetime
36+
- Volume limiting via CalcFilterRecordToDateTime() if record count exceeds 200,000
37+
3. **Record collection** -- CollectDataFromTable() iterates records:
38+
- Converts each record to JSON object (field whitelist per table)
39+
- Chunks JSON into 50,050-record arrays stored in TransStorageExportData temp table (blob parts)
40+
4. **Document collection** -- for GL Entry records, collects related incoming documents:
41+
- Filters by posting date and document number
42+
- Converts attached documents to base64
43+
5. **Master data collection** -- HandleTableFieldSet() identifies foreign key references:
44+
- Collects referenced Customers, Vendors, GL Accounts, etc.
45+
- Ensures archive is self-contained with all referenced master data
46+
6. **Upload** -- calls TransactionStorageABS.ArchiveTransactionsToABS():
47+
- Uploads JSON text (transaction data + master data)
48+
- Uploads base64 documents
49+
- Uploads execution log
50+
- Uploads metadata file
51+
52+
## Volume Limiting
53+
54+
**CalcFilterRecordToDateTime()** implements binary search to cap record count at 200,000:
55+
56+
1. **First pass** -- halves date range until record count <= 200,000 (day-level granularity)
57+
2. **Second pass** -- refines end date to millisecond precision to maximize records within limit
58+
3. **Result** -- returns adjusted SystemModifiedAt filter; unprocessed records handled in next scheduled run
59+
60+
## Azure Upload
61+
62+
**TransactionStorageABS** uploads to Azure Blob Storage:
63+
64+
- **Endpoints** -- separate Azure Function endpoints for JSON text and base64 documents
65+
- **Secrets** -- retrieved from Key Vault (configured in TransactStorageSetup)
66+
- **Container** -- formatted CVR number (company registration number)
67+
- **Blob folder structure** -- {TenantID}_{EnvironmentName}/{YYYYMMDD}
68+
- **Retention policy** -- 6 years or fiscal year-end + 6 years (configurable via Deletion Date Expression in setup)
69+
70+
## Retry Logic
71+
72+
**TransStorageErrorHandler** captures task errors and decides whether to reschedule:
73+
74+
1. **Attempt exhaustion** -- if 4 attempts consumed, logs critical alert and stops
75+
2. **Timeout** -- no reschedule (prevents infinite retry loop on slow queries)
76+
3. **OutOfMemory** -- no reschedule (prevents server instability)
77+
4. **Duplicate prevention** -- checks if valid task exists before rescheduling
78+
5. **Reschedule** -- if retriable error, schedules new task 5-15 minutes later (random backoff), decrements NoOfTasksAttempts
79+
6. **Consecutive failure monitoring** -- if 7 consecutive first-attempt failures detected, logs critical alert
80+
81+
## Flow Diagram
82+
83+
```
84+
┌─────────────────────┐
85+
│ GL Posting Event │
86+
│ (Gen. Jnl.-Post) │
87+
└──────────┬──────────┘
88+
│
89+
v
90+
┌─────────────────────┐
91+
│ TransStoragePosting │
92+
│ State (flag set) │
93+
└──────────┬──────────┘
94+
│
95+
v
96+
┌─────────────────────┐
97+
│ User Logout │
98+
│ (OnAfterCompanyClose)│
99+
└──────────┬──────────┘
100+
│
101+
v
102+
┌─────────────────────┐
103+
│ TransStorageSchedule│
104+
│ Task (create task) │
105+
└──────────┬──────────┘
106+
│
107+
v
108+
┌─────────────────────┐
109+
│ TaskScheduler (2-4AM│
110+
│ next day, run code-│
111+
│ unit:TransactStorage│
112+
│ Export) │
113+
└──────────┬──────────┘
114+
│
115+
v
116+
┌─────────────────────┐
117+
│ TransactStorageExport│
118+
│ .OnRun() │
119+
└──────────┬──────────┘
120+
│
121+
v
122+
┌─────────────────────┐
123+
│ TransactStorageExport│
124+
│ Data.ExportData() │
125+
│ (29 tables, filter by│
126+
│ SystemModifiedAt, │
127+
│ volume limiting) │
128+
└──────────┬──────────┘
129+
│
130+
v
131+
┌─────────────────────┐
132+
│ CollectDataFromTable│
133+
│ (JSON chunking, │
134+
│ 50,050 records/chunk)│
135+
└──────────┬──────────┘
136+
│
137+
v
138+
┌─────────────────────┐
139+
│ Collect documents │
140+
│ (incoming docs by │
141+
│ posting date + doc │
142+
│ no, base64 encode) │
143+
└──────────┬──────────┘
144+
│
145+
v
146+
┌─────────────────────┐
147+
│ Collect master data │
148+
│ (customers, vendors, │
149+
│ GL accounts via FK │
150+
│ references) │
151+
└──────────┬──────────┘
152+
│
153+
v
154+
┌─────────────────────┐
155+
│ TransactionStorageABS│
156+
│ .ArchiveTransactions │
157+
│ ToABS() │
158+
│ (upload JSON, docs, │
159+
│ log, metadata) │
160+
└──────────┬──────────┘
161+
│
162+
v
163+
┌─────────────────────┐
164+
│ Azure Blob Storage │
165+
│ (container: CVR, │
166+
│ folder: TenantID_ │
167+
│ EnvName/YYYYMMDD, │
168+
│ retention: 6 years) │
169+
└─────────────────────┘
170+
171+
ERROR PATH:
172+
┌─────────────────────┐
173+
│ TransStorageError │
174+
│ Handler (on task │
175+
│ error) │
176+
└──────────┬──────────┘
177+
│
178+
v
179+
┌───────┴───────┐
180+
│ Retriable? │
181+
└───┬───────┬───┘
182+
│ │
183+
YES NO
184+
│ │
185+
v v
186+
Reschedule Log critical
187+
(5-15 min alert, stop
188+
backoff, retry
189+
decrement
190+
attempts)
191+
```

‎src/Apps/W1/TransactionStorage/App/docs/data-model.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Transaction Storage Data Model
2+
3+
The Transaction Storage extension uses six core tables to manage scheduled exports of transaction data to Azure Blob Storage. The model separates concerns: configuration, execution tracking, retry state, per-table incremental progress, and temporary staging.
4+
5+
## Entity Relationships
6+
7+
```mermaid
8+
erDiagram
9+
TransactionStorageSetup ||--o{ TransactStorageTaskEntry : "schedules"
10+
TransactStorageExportState ||--|| TransactionStorageSetup : "controls retries for"
11+
TransactStorageTableEntry }o--|| TransactionStorageSetup : "configured by"
12+
TransStorageExportData }o--|| TransactStorageTaskEntry : "stages data for"
13+
14+
TransactionStorageSetup {
15+
Time EarliestStartTime
16+
Integer MaxNumberOfHours
17+
}
18+
19+
TransactStorageTaskEntry {
20+
Guid TaskID
21+
Enum Status
22+
DateTime StartingDateTime
23+
DateTime EndingDateTime
24+
}
25+
26+
TransactStorageExportState {
27+
Integer NumberOfAttempts
28+
Date FirstRunDate
29+
}
30+
31+
TransactStorageTableEntry {
32+
Integer TableID
33+
DateTime LastHandledDateTime
34+
Boolean ExportedToABS
35+
}
36+
37+
TransStorageExportData {
38+
Integer TableID
39+
Integer Part
40+
Blob Content
41+
}
42+
```
43+
44+
## Configuration Layer
45+
46+
**TransactionStorageSetup (6201)** is a singleton that controls when exports can run. It defines an earliest start time (default 02:00) and a maximum window of hours (default 3, minimum 3). On insert, the system calculates a distributed export time for the tenant to spread load across the time window. This prevents all tenants from starting exports simultaneously.
47+
48+
## Execution Tracking
49+
50+
**TransactStorageTaskEntry (6202)** maintains an audit trail of every export task. Each entry records a task ID (GUID), execution status (via the TransStorageExportStatus enum), start and end times, and error details if the task failed. The error call stack is stored in a blob field and can be retrieved or displayed via dedicated procedures. The table tracks whether each execution is a first attempt and when it was originally scheduled.
51+
52+
## Retry State
53+
54+
**TransactStorageExportState (6203)** is a singleton that tracks retry attempts. It starts with a configurable number of attempts (default 3) and decrements on each failure. The First Run Date field is country-aware -- for Denmark, it defaults to 2024-01-01. A ResetSetup procedure allows the retry counter to be reset when needed.
55+
56+
## Per-Table Progress
57+
58+
**TransactStorageTableEntry (6204)** tracks the export state for each table being monitored. Each record uses the table ID as its primary key and includes a flow field for the table name. The table maintains incremental export state via Filter Record To DT and Last Handled Date/Time fields, allowing the system to resume from where it left off. It records the number of exported records, any record filters applied (up to 2048 characters), whether the data has been exported to Azure Blob Storage, and the blob name used.
59+
60+
## Temporary Staging
61+
62+
**TransStorageExportData (6205)** is a temporary table used during export operations. It has a composite primary key of Table ID and Part, allowing large datasets to be chunked into multiple JSON arrays. The Content blob field stores each chunk, and Record Count tracks how many records are in that part. Data is added via an Add procedure and cleared between runs.
63+
64+
## Status Enumeration
65+
66+
**TransStorageExportStatus (6201)** defines five states: None, Scheduled, Started, Completed, and Failed. This enumeration is not extensible, ensuring consistent status tracking across the export lifecycle.
67+
68+
## Data Flow
69+
70+
During an export cycle, the setup table determines when execution can begin. The export state singleton controls retry logic. For each configured table in the table entry records, the system reads incremental changes, stages them in temporary export data chunks, and writes them to Azure Blob Storage. The task entry table records each execution attempt with full error details if failures occur.

0 commit comments

Comments
 (0)