Research: Dynamic Configuration

https://github.com/vmware/versatile-data-kit/issues/1201 outlines one possible solution for dynamic configuration

The solution is focused to allow administrators to set configuration options dynamically without needing to redeploy jobs. Basicallly also solve https://github.com/vmware/versatile-data-kit/issues/1722

The below research focuses on enabling users to do the same thing. Let's consider reusing Properties and Secrets API that already exists in the control service to enable the same thing This would simplify architecture as we will not be adding new components but reusing existing ones

Requirements

Definition:

• User properties: arbitrary key/value pairs user can set , the value could be complex structure (with limited size 1KB) , user use them to keep state (last process time) or secrets (API keys)
• System configuration: Specific and documented configuration operations by Versatile Data Kit SDK. They control different aspects of the SDK

1. User should be able to user Properties and Secretes API to set arbitrary key/value pairs , the value could be complex structure (with limited size 1KB)
2. User should be able to set properties per team if they have permissions for that team
3. Administrators should be able to

To support both user settings and system settings, there are several approaches we can consider.

Option 1: Extend Existing Endpoints

Extend the existing API to include a flag indicating whether the secret is a system setting or user setting. PUT API

{
  "type": "user", // or "system"
  "secrets": {"my-custom-key": "value"}
}

GET API

{
  "user": {"my-custom-key": "value"},
  "system": {"db-password": "value"}
}

Option 2: Separate Endpoints

Create separate endpoints for managing system settings and user settings.

• /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/user-secrets
• /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/system-secrets

Option 3: URL Parameters

Use URL parameters to indicate the type of settings.

/data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/secrets?type=system

Option 4: Role-based Access

If system settings should only be editable by administrators, we can implement role-based access control. Any secret added by an admin could automatically be considered a system setting.

Option 5: Namespaced Secrets

Use a naming convention to distinguish between user and system settings. This could be implemented in the backend logic or required in the API request payload.

PUT API

{
  "secrets": {"user.my-custom-key": "value", "system.db-password": "value"}
}

GET API

{
  "user.my-custom-key": "value",
  "system.db-password": "value"
}

The scores range from 1 to 5, with 5 being the best

Criteria	Extend Existing Endpoints (1)	Separate Endpoints (2)	URL Parameters (3)	Role-based Access (4)	Namespaced Secrets (5)
Ease of Implementation	5	3	4	1	4
Backward Compatibility	4	2	5	4	5
User Experience	3	4	3	4	2
Flexibility	2	4	5	2	4
Security	3	4	3	5	3
Maintainability	3	5	4	3	3
Final Score	20	22	24	19	21