feat: Add enterprise token consumption schema documentation

DanielB945 · claude · DanielB945 · commit 350bdcb7e86d · 2026-03-15T23:23:21.000+02:00
## Changes

### New Documentation
- **shared/enterprise-token-pools.md**: Comprehensive guide to shared vs non-shared token pools
  - Shared token pool logic (is_shared_token_pool flag)
  - Non-shared pool logic (McCann_NY, Cylndr, code users)
  - Enterprise organization classification
  - Complete credit limit monitoring queries
  - Code redemption tracking
  - Future user cap feature documentation

### Schema Updates (shared/bq-schema.md)
- **backend_griffin_market_purchased_consumables**: Token purchases and initial balances
  - Key fields: consumable_id, ltid, organization_id, is_shared_token_pool, quantity
  - Shared vs non-shared pool logic
  - Example queries for credit limit monitoring

- **backend_griffin_market_consumed_consumables**: Token consumption transactions
  - Key field: consumed_by_ltid (critical for shared pool attribution)
  - Consumption tracking by user

- **backend_griffin_market_unified_payment_source**: Payment sources and code redemptions
  - complimentary_code field for tracking code users
  - Code redemption query examples

- **backend_griffin_shared_subscription_group**: Shared subscription groups
  - has_shared_token_pool flag
  - Seat configurations and consumable offerings

### Reference Updates (CLAUDE.md)
- Added enterprise-token-pools.md to shared knowledge table
- Read before any enterprise token/billing analysis

## Context

These tables are critical for:
- Credit limit monitoring (Report 4 in daily reports)
- Enterprise account health tracking
- Token consumption analysis
- Code redemption tracking

## Key Insights

**Shared Token Pools (most enterprise orgs):**
- Records at ORG level: ltid = group_id
- Use consumed_by_ltid to track individual user consumption
- Examples: Indegene, HearWell_BeWell, Miroma, Deriv, McCann_Paris

**Non-Shared Pools:**
- Records at USER level: ltid = individual user
- Examples: McCann_NY, Cylndr Studios (SSO), code users

**McCann Special Case:**
- McCann_NY: Separate entity, non-shared pool
- McCann_Paris: Catch-all for other McCann offices, shared pool

Co-Authored-By: Claude Sonnet 4.5 &lt;noreply@anthropic.com&gt;
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -38,6 +38,7 @@ Every agent reads from these files. They are the single source of truth — do n
 | `shared/bq-schema.md` | BQ tables, columns, joins, segmentation queries | Writing ANY SQL |
 | `shared/event-registry.yaml` | Known events per feature, types, status | Referencing ANY event |
 | `shared/metric-standards.md` | How every metric is calculated (with SQL) | Defining ANY metric |
+| `shared/enterprise-token-pools.md` | Token pool logic, shared vs non-shared, credit limits | Enterprise token/billing analysis |
 | `shared/gpu-cost-query-templates.md` | 11 GPU cost queries (DoD, WoW, anomaly detection, breakdowns) | Analyzing GPU/infrastructure costs |
 | `shared/gpu-cost-analysis-patterns.md` | Cost analysis workflows, benchmarks, investigation playbooks | Interpreting GPU cost data |
 
diff --git a/shared/bq-schema.md b/shared/bq-schema.md
@@ -358,6 +358,224 @@ Clustered by: to_date, griffin_tier_name
 | base_fortress_user_login | Login events (Fortress auth) | bq_partitiondate |
 | base_join_ltxstudio_web_process_started_ended | Joined start + end with duration | bq_partitiondate |
 
+### Enterprise Token & Billing Tables (backend_griffin_market)
+
+> Use these tables to track token purchases, consumption, and credit limit monitoring for enterprise accounts.
+> See `shared/enterprise-token-pools.md` for detailed logic on shared vs non-shared token pools.
+
+| Table | Description | Partition |
+|-------|-------------|-----------|
+| base_backend_griffin_market_purchased_consumables | Token purchases (initial balance) | None |
+| base_backend_griffin_market_consumed_consumables | Token consumption (transactions) | None |
+| base_backend_griffin_market_unified_payment_source | Payment sources & code redemptions | None |
+| base_backend_griffin_shared_subscription_group | Shared subscription groups & seat details | None |
+
+#### backend_griffin_market_purchased_consumables
+
+**Token purchases and initial balances.** One row per purchased consumable (token package).
+
+```
+Table: `ltx-dwh-prod-processed.base.backend_griffin_market_purchased_consumables`
+Grain: One row per consumable_id
+```
+
+**Key Fields:**
+
+| Column | Type | Description |
+|--------|------|-------------|
+| consumable_id | STRING | Unique identifier of this consumable |
+| consumable_type | STRING | "Coin" or "YOUniverse_pack" |
+| ltid | STRING | Owner of consumable (NULL when org-level, see is_shared_token_pool) |
+| organization_id | STRING | Organization ID (when consumable belongs to org) |
+| payment_source_id | STRING | Where this consumable came from |
+| quantity | INT | Units purchased |
+| group_id | STRING | Shared subscription group ID |
+| **is_shared_token_pool** | **BOOL** | **If TRUE: ltid = group_id (org-level record)** |
+| is_cap | BOOL | Whether this is a cap allocation |
+| is_unlimited | BOOL | Unlimited tokens flag |
+| expiration_ms | TIMESTAMP | When this consumable expires |
+| payment_id | STRING | Only for single product purchases (not bundles) |
+| revoked_at_ms | TIMESTAMP | Revocation timestamp (NULL if not revoked) |
+| revocation_reason | STRING | Reason for revocation |
+| created_at_ms | TIMESTAMP | Creation timestamp |
+| updated_at_ms | TIMESTAMP | Last update timestamp |
+
+**Critical Logic:**
+
+```sql
+-- Shared token pool (most enterprise orgs)
+IF is_shared_token_pool = TRUE:
+  - ltid = group_id (this is an ORG-level record)
+  - All users in the group consume from this pool
+  - See consumed_consumables.consumed_by_ltid for individual user consumption
+
+-- Non-shared pool (McCann, Cylndr SSO users, code users)
+IF is_shared_token_pool = FALSE:
+  - ltid = individual user
+  - Only this user can consume from this balance
+```
+
+#### backend_griffin_market_consumed_consumables
+
+**Token consumption transactions.** One row per consumption event.
+
+```
+Table: `ltx-dwh-prod-processed.base.backend_griffin_market_consumed_consumables`
+Grain: One row per consumption event
+```
+
+**Key Fields:**
+
+| Column | Type | Description |
+|--------|------|-------------|
+| consumable_id | STRING | References purchased_consumables.consumable_id |
+| consumable_type | STRING | "Coin" or "YOUniverse_pack" |
+| ltid | STRING | Owner of the purchased consumable (NULL when org-level) |
+| organization_id | STRING | Organization ID (when org-level) |
+| **consumed_by_ltid** | **STRING** | **Who actually consumed (key for shared pools)** |
+| product_consumed_id | STRING | Product ID that was consumed |
+| product_consumed_description | STRING | Product description |
+| quantity | INT | Units consumed |
+| group_id | STRING | Shared subscription group ID |
+| **is_shared_token_pool** | **BOOL** | **If TRUE: ltid = group_id** |
+| is_cap | BOOL | Whether from cap allocation |
+| is_unlimited | BOOL | Unlimited tokens flag |
+| created_at_ms | TIMESTAMP | Consumption timestamp |
+| updated_at_ms | TIMESTAMP | Last update timestamp |
+
+**Critical Logic:**
+
+```sql
+-- For shared token pools
+IF is_shared_token_pool = TRUE:
+  - ltid = group_id (matches purchased_consumables)
+  - consumed_by_ltid = individual user who consumed
+  - Use consumed_by_ltid to attribute consumption to users
+
+-- For non-shared pools
+IF is_shared_token_pool = FALSE:
+  - ltid = individual user
+  - consumed_by_ltid may be same as ltid or NULL
+```
+
+**Example Query — Credit Limit by Org:**
+
+```sql
+WITH shared_pool_orgs AS (
+  SELECT
+    p.ltid AS group_id,  -- This is group_id when is_shared_token_pool = true
+    p.organization_id,
+    SUM(p.quantity) AS total_purchased,
+    SUM(CASE WHEN c.quantity IS NOT NULL THEN c.quantity ELSE 0 END) AS total_consumed,
+    COUNT(DISTINCT c.consumed_by_ltid) AS users_consuming
+  FROM `ltx-dwh-prod-processed.base.backend_griffin_market_purchased_consumables` p
+  LEFT JOIN `ltx-dwh-prod-processed.base.backend_griffin_market_consumed_consumables` c
+    ON p.consumable_id = c.consumable_id
+  WHERE p.is_shared_token_pool = TRUE
+    AND p.revoked_at_ms IS NULL
+  GROUP BY 1, 2
+),
+
+non_shared_pool_users AS (
+  SELECT
+    p.ltid,
+    p.organization_id,
+    SUM(p.quantity) AS total_purchased,
+    SUM(CASE WHEN c.quantity IS NOT NULL THEN c.quantity ELSE 0 END) AS total_consumed
+  FROM `ltx-dwh-prod-processed.base.backend_griffin_market_purchased_consumables` p
+  LEFT JOIN `ltx-dwh-prod-processed.base.backend_griffin_market_consumed_consumables` c
+    ON p.consumable_id = c.consumable_id
+  WHERE p.is_shared_token_pool = FALSE
+    AND p.revoked_at_ms IS NULL
+  GROUP BY 1, 2
+)
+
+SELECT * FROM shared_pool_orgs
+UNION ALL
+SELECT ltid AS group_id, organization_id, total_purchased, total_consumed, 1 AS users_consuming
+FROM non_shared_pool_users
+```
+
+#### backend_griffin_market_unified_payment_source
+
+**Payment sources and complimentary code redemptions.** One row per payment source.
+
+```
+Table: `ltx-dwh-prod-processed.base.backend_griffin_market_unified_payment_source`
+Grain: One row per payment_source_id
+```
+
+**Key Fields:**
+
+| Column | Type | Description |
+|--------|------|-------------|
+| payment_source_id | STRING | Unique identifier |
+| full_payment_source_id | STRING | Identifier with vendor prefix |
+| ltid | STRING | User who owns this payment source |
+| vendor | STRING | "adyencc", "appstore", "stripe", "paypal" |
+| **complimentary_code** | **STRING** | **Complimentary code redeemed (for code tracking)** |
+| is_sandbox | BOOL | Sandbox flag |
+| quote_id | STRING | Original quote ID |
+| renewal_details | RECORD | Auto-renew info (complex nested object) |
+| geolocation | RECORD | Country, region, city, postal code |
+| purchasing_device_details | RECORD | App, version, platform |
+| credit_card_details | RECORD | Card info (when applicable) |
+| created_at_ms | TIMESTAMP | Creation timestamp |
+| updated_at_ms | TIMESTAMP | Last update timestamp |
+
+**Example Query — Code Redemption Tracking:**
+
+```sql
+SELECT
+  u.ltid,
+  u.complimentary_code,
+  u.created_at_ms AS redemption_ts,
+  p.quantity AS tokens_granted,
+  p.consumable_type
+FROM `ltx-dwh-prod-processed.base.backend_griffin_market_unified_payment_source` u
+INNER JOIN `ltx-dwh-prod-processed.base.backend_griffin_market_purchased_consumables` p
+  ON u.payment_source_id = p.payment_source_id
+WHERE u.complimentary_code IS NOT NULL
+ORDER BY u.created_at_ms DESC
+```
+
+#### backend_griffin_shared_subscription_group
+
+**Shared subscription groups and seat configurations.** One row per group.
+
+```
+Table: `ltx-dwh-prod-processed.base.backend_griffin_shared_subscription_group`
+Grain: One row per group_id (item_identifier)
+```
+
+**Key Fields:**
+
+| Column | Type | Description |
+|--------|------|-------------|
+| seats | ARRAY<RECORD> | Seat types, quantities, consumable offerings |
+| apps | ARRAY<STRING> | "ltxstudio", "facetune", "popular-pays.brands" |
+| platforms | ARRAY<STRING> | "w" (web), "a" (android), "i" (iOS) |
+| admin_ltid | STRING | Group admin user ID (NULL if pending) |
+| **has_shared_token_pool** | **BOOL** | **Shared pool managed under admin's ltid** |
+| product_id | STRING | Associated product |
+| payment_source_id | STRING | Associated payment source |
+| expiration_ms | TIMESTAMP | When group expires |
+| revoked_at_ms | TIMESTAMP | Revocation timestamp (NULL if active) |
+| created_at_ms | TIMESTAMP | Creation timestamp |
+| updated_at_ms | TIMESTAMP | Last update timestamp |
+
+**Seats Array Structure:**
+
+```sql
+-- seats[].type: "admin", "regular"
+-- seats[].quantity: Max seats allowed
+-- seats[].feature_tags[]: Enabled features
+-- seats[].consumable_offerings[].consumable_type: "Coin"
+-- seats[].consumable_offerings[].quantity: Tokens per cycle
+-- seats[].consumable_offerings[].timespan: ISO-8601 duration
+-- seats[].consumable_offerings[].is_unlimited: Unlimited flag
+```
+
 ### Common Base Columns
 
 Most base tables share these columns:
diff --git a/shared/enterprise-token-pools.md b/shared/enterprise-token-pools.md
