Embed DB addressing in sys_environment; add org/env scoping

Merge physical database mapping into sys_environment and introduce organizationId/environmentId scoping across metadata and control-plane code. sys_environment_database object/table removed; environment rows now store database_url, database_driver, storage_limit_mb and provisioned_at. DatabaseLoader, MetadataManager, plugin and history cleanup updated to use organization_id + env_id (env_id = null = platform-global). sys_metadata/sys_metadata_history schemas and indexes updated to include organization_id/env_id; sys_database_credential now references environment_id and its indexes updated. HTTP dispatcher, provisioning service, migration, and tests adjusted to the new model to simplify lookups, reduce joins, and support per-environment isolation.

Author: hotlong

docs/adr/0002-environment-database-isolation.md

@@ -34,29 +34,36 @@ We upgrade the multi-tenant architecture from **per-organization database** to *
 
 Registers environments and how to reach them — **never** stores business data:
 
-| Table                     | Purpose                                                    |
-|---------------------------|------------------------------------------------------------|
-| `sys_environment`         | One row per environment — `(organization_id, slug)` UNIQUE |
-| `sys_environment_database`| Physical DB addressing (1:1 with `sys_environment`)        |
-| `sys_database_credential` | Rotatable encrypted secrets (N:1 with `sys_environment_database`) |
-| `sys_environment_member`  | Per-environment RBAC (`(environment_id, user_id)` UNIQUE)  |
+| Table                       | Purpose                                                                          |
+|-----------------------------|----------------------------------------------------------------------------------|
+| `sys_environment`           | One row per environment — `(organization_id, slug)` UNIQUE. **Includes** physical DB addressing fields (`database_url`, `database_driver`, `storage_limit_mb`, `provisioned_at`). |
+| `sys_database_credential`   | Rotatable encrypted secrets (N:1 with `sys_environment`)                         |
+| `sys_environment_member`    | Per-environment RBAC (`(environment_id, user_id)` UNIQUE)                        |
+| `sys_package_installation`  | Which packages are installed in which environment (includes `environment_id`)     |
+| `sys_metadata`              | Schema metadata for all environments (includes `organization_id` + `env_id`)     |
+
+> **Design note:** `sys_environment_database` has been merged into `sys_environment`.
+> Physical DB addressing is a 1:1 relationship — a dedicated table was unnecessary.
+> This reduces joins and simplifies the provisioning flow.
+
+> **Design note:** `sys_metadata` and `sys_package_installation` live in the **Control Plane**,
+> not in environment DBs. Metadata is environment-scoped via `env_id` (NULL = platform-global).
+> This follows the Power Apps model where the management plane tracks all schema and packages.
 
 ### Data Plane (one database per environment)
 
 Each environment owns its own physical database containing:
 
-- All `sys_` data-plane objects — `sys_package_installation`, `sys_solution_history`, …
 - All business objects — `account`, `contact`, user tables, …
-- **Zero** `environment_id` columns. The environment is **implicit** in the connection.
+- **Zero** system tables or `environment_id` columns. The environment is **implicit** in the connection.
 
 ### Session → Routing
 
 `better-auth` sessions carry a single `active_environment_id`. The tenant router resolves:
 
 ```
 session.active_environment_id
-    → sys_environment (→ organization_id)
-    → sys_environment_database (url, driver, region)
+    → sys_environment (url, driver, region — single row lookup)
     → sys_database_credential (active secret, decrypted)
     → data-plane driver
 ```
@@ -68,8 +75,8 @@ Switching environments ⇒ swapping DB connections. There is no in-process filte
 `EnvironmentProvisioningService` (new) exposes:
 
 - `provisionOrganization(req)` — atomically creates the org's **default** environment and its physical DB (replaces `provisionTenant`).
-- `provisionEnvironment(req)` — allocates any subsequent `dev` / `test` / `sandbox` / `preview` environment, each with its own DB and credential row.
-- `rotateCredential(envDbId, plaintext)` — issues a new `active` credential and revokes the previous one.
+- `provisionEnvironment(req)` — allocates any subsequent `dev` / `test` / `sandbox` / `preview` environment, each with its own DB and credential row. DB addressing fields are written directly onto the `sys_environment` row.
+- `rotateCredential(environmentId, plaintext)` — issues a new `active` credential and revokes the previous one.
 
 Physical-DB allocation is delegated to pluggable `EnvironmentDatabaseAdapter` implementations (initially `turso`; `libsql` / `sqlite` / `postgres` drop in without core changes).
 
@@ -117,7 +124,7 @@ The migration is **non-destructive** and **idempotent**: each legacy org's datab
 ## References
 
 - `packages/spec/src/cloud/environment.zod.ts` — protocol schemas
-- `packages/services/service-tenant/src/objects/sys-environment*.object.ts` — control-plane objects
+- `packages/services/service-tenant/src/objects/sys-environment.object.ts` — merged control-plane environment object (includes DB addressing)
 - `packages/services/service-tenant/src/environment-provisioning.ts` — provisioning service
 - `packages/services/service-tenant/migrations/v4-to-v5-env-migration.ts` — migration skeleton
 - Power Platform environments: <https://learn.microsoft.com/power-platform/admin/environments-overview>

packages/metadata/src/loaders/database-loader.test.ts

@@ -342,26 +342,27 @@ describe('DatabaseLoader', () => {
   });
 
   describe('multi-tenant isolation', () => {
-    it('should filter by tenantId when configured', async () => {
+    it('should filter by organizationId and environmentId when configured', async () => {
       const tenantLoader = new DatabaseLoader({
         driver: mockDriver,
-        tenantId: 'tenant-1',
+        organizationId: 'org-1',
+        environmentId: 'env-1',
       });
 
       await tenantLoader.save('object', 'account', { name: 'account' });
 
-      // The create call should include tenant_id
+      // The create call should include organization_id and env_id
       expect(mockDriver.create).toHaveBeenCalledWith(
         'sys_metadata',
-        expect.objectContaining({ tenant_id: 'tenant-1' })
+        expect.objectContaining({ organization_id: 'org-1', env_id: 'env-1' })
       );
 
-      // The find calls should filter by tenant_id
+      // The find calls should filter by organization_id and env_id
       await tenantLoader.load('object', 'account');
       expect(mockDriver.findOne).toHaveBeenCalledWith(
         'sys_metadata',
         expect.objectContaining({
-          where: expect.objectContaining({ tenant_id: 'tenant-1' }),
+          where: expect.objectContaining({ organization_id: 'org-1', env_id: 'env-1' }),
         })
       );
     });

packages/metadata/src/loaders/database-loader.ts

@@ -46,8 +46,11 @@ export interface DatabaseLoaderOptions {
   /** The table name to store history records (default: 'sys_metadata_history') */
   historyTableName?: string;
 
-  /** Tenant ID for multi-tenant isolation */
-  tenantId?: string;
+  /** Organization ID for multi-tenant isolation */
+  organizationId?: string;
+
+  /** Environment ID — null = platform-global, set = env-scoped */
+  environmentId?: string;
 
   /** Enable history tracking (default: true) */
   trackHistory?: boolean;
@@ -76,7 +79,8 @@ export class DatabaseLoader implements MetadataLoader {
   private engine?: IDataEngine;
   private tableName: string;
   private historyTableName: string;
-  private tenantId?: string;
+  private organizationId?: string;
+  private environmentId?: string;
   private trackHistory: boolean;
   private schemaReady = false;
   private historySchemaReady = false;
@@ -89,7 +93,8 @@ export class DatabaseLoader implements MetadataLoader {
     this.engine = options.engine;
     this.tableName = options.tableName ?? 'sys_metadata';
     this.historyTableName = options.historyTableName ?? 'sys_metadata_history';
-    this.tenantId = options.tenantId;
+    this.organizationId = options.organizationId;
+    this.environmentId = options.environmentId;
     this.trackHistory = options.trackHistory !== false; // Default to true
   }
 
@@ -193,16 +198,19 @@ export class DatabaseLoader implements MetadataLoader {
 
   /**
    * Build base filter conditions for queries.
-   * Always includes tenantId when configured.
+   * Filters by organizationId when configured; env_id when environmentId is set,
+   * or null (platform-global) when not set.
    */
   private baseFilter(type: string, name?: string): Record<string, unknown> {
     const filter: Record<string, unknown> = { type };
     if (name !== undefined) {
       filter.name = name;
     }
-    if (this.tenantId) {
-      filter.tenant_id = this.tenantId;
+    if (this.organizationId) {
+      filter.organization_id = this.organizationId;
     }
+    // When environmentId is set, scope to that env; otherwise query platform-global (env_id = null).
+    filter.env_id = this.environmentId ?? null;
     return filter;
   }
 
@@ -258,7 +266,8 @@ export class DatabaseLoader implements MetadataLoader {
       changeNote,
       recordedBy,
       recordedAt: now,
-      ...(this.tenantId ? { tenantId: this.tenantId } : {}),
+      ...(this.organizationId ? { organizationId: this.organizationId } : {}),
+      ...(this.environmentId !== undefined ? { environmentId: this.environmentId } : {}),
     };
 
     try {
@@ -275,7 +284,8 @@ export class DatabaseLoader implements MetadataLoader {
         change_note: historyRecord.changeNote,
         recorded_by: historyRecord.recordedBy,
         recorded_at: historyRecord.recordedAt,
-        ...(this.tenantId ? { tenant_id: this.tenantId } : {}),
+        ...(this.organizationId ? { organization_id: this.organizationId } : {}),
+        ...(this.environmentId !== undefined ? { env_id: this.environmentId } : {}),
       });
     } catch (error) {
       // Log error but don't fail the main operation
@@ -314,7 +324,8 @@ export class DatabaseLoader implements MetadataLoader {
       strategy: (row.strategy as MetadataRecord['strategy']) ?? 'merge',
       owner: row.owner as string | undefined,
       state: (row.state as MetadataRecord['state']) ?? 'active',
-      tenantId: row.tenant_id as string | undefined,
+      organizationId: row.organization_id as string | undefined,
+      environmentId: row.env_id as string | undefined,
       version: (row.version as number) ?? 1,
       checksum: row.checksum as string | undefined,
       source: row.source as MetadataRecord['source'],
@@ -468,9 +479,10 @@ export class DatabaseLoader implements MetadataLoader {
       metadata_id: metadataRow.id,
       version,
     };
-    if (this.tenantId) {
-      filter.tenant_id = this.tenantId;
+    if (this.organizationId) {
+      filter.organization_id = this.organizationId;
     }
+    filter.env_id = this.environmentId ?? null;
 
     const row = await this._findOne(this.historyTableName, {
       where: filter,
@@ -488,7 +500,8 @@ export class DatabaseLoader implements MetadataLoader {
       checksum: row.checksum as string,
       previousChecksum: row.previous_checksum as string | undefined,
       changeNote: row.change_note as string | undefined,
-      tenantId: row.tenant_id as string | undefined,
+      organizationId: row.organization_id as string | undefined,
+      environmentId: row.env_id as string | undefined,
       recordedBy: row.recorded_by as string | undefined,
       recordedAt: row.recorded_at as string,
     };
@@ -520,7 +533,8 @@ export class DatabaseLoader implements MetadataLoader {
 
     // Find the metadata record
     const filter: Record<string, unknown> = { type, name };
-    if (this.tenantId) filter.tenant_id = this.tenantId;
+    if (this.organizationId) filter.organization_id = this.organizationId;
+    filter.env_id = this.environmentId ?? null;
 
     const metadataRecord = await this._findOne(this.tableName, { where: filter });
     if (!metadataRecord) {
@@ -531,7 +545,8 @@ export class DatabaseLoader implements MetadataLoader {
     const historyFilter: Record<string, unknown> = {
       metadata_id: metadataRecord.id,
     };
-    if (this.tenantId) historyFilter.tenant_id = this.tenantId;
+    if (this.organizationId) historyFilter.organization_id = this.organizationId;
+    historyFilter.env_id = this.environmentId ?? null;
     if (options?.operationType) historyFilter.operation_type = options.operationType;
     if (options?.since) historyFilter.recorded_at = { $gte: options.since };
     if (options?.until) {
@@ -574,7 +589,8 @@ export class DatabaseLoader implements MetadataLoader {
         checksum: row.checksum as string,
         previousChecksum: row.previous_checksum as string | undefined,
         changeNote: row.change_note as string | undefined,
-        tenantId: row.tenant_id as string | undefined,
+        organizationId: row.organization_id as string | undefined,
+        environmentId: row.env_id as string | undefined,
         recordedBy: row.recorded_by as string | undefined,
         recordedAt: row.recorded_at as string,
       };
@@ -711,7 +727,8 @@ export class DatabaseLoader implements MetadataLoader {
           state: 'active',
           version: 1,
           source: 'database',
-          ...(this.tenantId ? { tenant_id: this.tenantId } : {}),
+          ...(this.organizationId ? { organization_id: this.organizationId } : {}),
+          ...(this.environmentId !== undefined ? { env_id: this.environmentId } : { env_id: null }),
           created_at: now,
           updated_at: now,
         });

packages/metadata/src/metadata-manager.ts

@@ -133,12 +133,16 @@ export class MetadataManager implements IMetadataService {
    * Can be called at any time to enable database storage (e.g. after kernel resolves the driver).
    *
    * @param driver - An IDataDriver instance for database operations
+   * @param organizationId - Organization ID for multi-tenant isolation
+   * @param environmentId - Environment ID (undefined = platform-global)
    */
-  setDatabaseDriver(driver: IDataDriver): void {
+  setDatabaseDriver(driver: IDataDriver, organizationId?: string, environmentId?: string): void {
     const tableName = this.config.tableName ?? 'sys_metadata';
     const dbLoader = new DatabaseLoader({
       driver,
       tableName,
+      organizationId,
+      environmentId,
     });
     this.registerLoader(dbLoader);
     this.logger.info('DatabaseLoader configured', { datasource: this.config.datasource, tableName });
@@ -151,12 +155,16 @@ export class MetadataManager implements IMetadataService {
    * No manual driver resolution needed.
    *
    * @param engine - An IDataEngine instance (typically the ObjectQL service)
+   * @param organizationId - Organization ID for multi-tenant isolation
+   * @param environmentId - Environment ID (undefined = platform-global)
    */
-  setDataEngine(engine: IDataEngine): void {
+  setDataEngine(engine: IDataEngine, organizationId?: string, environmentId?: string): void {
     const tableName = this.config.tableName ?? 'sys_metadata';
     const dbLoader = new DatabaseLoader({
       engine,
       tableName,
+      organizationId,
+      environmentId,
     });
     this.registerLoader(dbLoader);
     this.logger.info('DatabaseLoader configured via DataEngine', { tableName });

packages/metadata/src/objects/sys-metadata-history.object.ts

@@ -105,12 +105,22 @@ export const SysMetadataHistoryObject = ObjectSchema.create({
       description: 'Description of what changed in this version',
     }),
 
-    /** Tenant ID for multi-tenant isolation */
-    tenant_id: Field.text({
-      label: 'Tenant ID',
+    /** Organization ID for multi-tenant isolation */
+    organization_id: Field.text({
+      label: 'Organization ID',
       required: false,
       readonly: true,
       maxLength: 255,
+      description: 'Organization identifier for multi-tenant isolation.',
+    }),
+
+    /** Environment ID — null = platform-global, set = env-scoped */
+    env_id: Field.text({
+      label: 'Environment ID',
+      required: false,
+      readonly: true,
+      maxLength: 255,
+      description: 'Scopes this history entry to a specific environment.',
     }),
 
     /** User who made this change */
@@ -135,7 +145,8 @@ export const SysMetadataHistoryObject = ObjectSchema.create({
     { fields: ['type', 'name'] },
     { fields: ['recorded_at'] },
     { fields: ['operation_type'] },
-    { fields: ['tenant_id'] },
+    { fields: ['organization_id'] },
+    { fields: ['env_id'] },
   ],
 
   enable: {

packages/metadata/src/objects/sys-metadata.object.ts

@@ -110,11 +110,20 @@ export const SysMetadataObject = ObjectSchema.create({
       defaultValue: 'active',
     }),
 
-    /** Tenant ID for multi-tenant isolation */
-    tenant_id: Field.text({
-      label: 'Tenant ID',
+    /** Organization ID for multi-tenant isolation */
+    organization_id: Field.text({
+      label: 'Organization ID',
       required: false,
       maxLength: 255,
+      description: 'Organization identifier for multi-tenant isolation.',
+    }),
+
+    /** Environment ID — null = platform-global, set = env-scoped */
+    env_id: Field.text({
+      label: 'Environment ID',
+      required: false,
+      maxLength: 255,
+      description: 'Scopes this metadata to a specific environment. Null = platform-global.',
     }),
 
     /** Version number for optimistic concurrency */
@@ -171,9 +180,10 @@ export const SysMetadataObject = ObjectSchema.create({
   },
 
   indexes: [
-    { fields: ['type', 'name'], unique: true },
+    { fields: ['type', 'name', 'env_id'], unique: true },
     { fields: ['type', 'scope'] },
-    { fields: ['tenant_id'] },
+    { fields: ['organization_id'] },
+    { fields: ['env_id'] },
     { fields: ['state'] },
     { fields: ['namespace'] },
   ],

packages/metadata/src/plugin.ts

@@ -11,6 +11,10 @@ export interface MetadataPluginOptions {
     rootDir?: string;
     watch?: boolean;
     config?: Partial<MetadataPluginConfig>;
+    /** Organization ID for multi-tenant metadata isolation (passed to DatabaseLoader). */
+    organizationId?: string;
+    /** Environment ID — undefined = platform-global, set = env-scoped metadata. */
+    environmentId?: string;
 }
 
 export class MetadataPlugin implements Plugin {
@@ -114,7 +118,7 @@ export class MetadataPlugin implements Plugin {
             const ql = ctx.getService<any>('objectql');
             if (ql) {
                 ctx.logger.info('[MetadataPlugin] Bridging ObjectQL engine to MetadataManager');
-                this.manager.setDataEngine(ql);
+                this.manager.setDataEngine(ql, this.options.organizationId, this.options.environmentId);
             }
         } catch {
             ctx.logger.debug('[MetadataPlugin] ObjectQL not available — database persistence disabled');