diff --git a/guides/data-apps.mdx b/guides/data-apps.mdx index 83c19192..299aa965 100644 --- a/guides/data-apps.mdx +++ b/guides/data-apps.mdx @@ -61,6 +61,8 @@ Every prompt you send creates a new version of the app. The agent keeps its work If a build is taking too long or going in the wrong direction, you can cancel it and try a different prompt. +Versions that were uploaded with a [custom dependency set](#declared-dependencies) show a small package chip in the version timeline. Hover the chip to see the `name@version` list that was installed in the build sandbox for that version — handy for confirming what changed when you carry an app forward or restore an earlier version. + ### Restoring an earlier version If a recent prompt took the app in the wrong direction, you can roll back to any earlier ready version and keep iterating from there. @@ -322,11 +324,52 @@ You can also set `LIGHTDASH_URL` and `LIGHTDASH_API_KEY` instead of running `lig Before you rely on data apps as code in a workflow, keep the following in mind: - **Enterprise only.** Both the source and destination instances must have data apps enabled, including the build sandbox. See [Self-hosting Data apps](/guides/data-apps/self-hosting) for the configuration required on self-hosted instances. -- **Fixed dependency set.** The sandbox rebuilds against a pre-installed library set. You can edit any source file, but you can't add libraries that the sandbox doesn't already provide, and edits to root config files (`package.json`, `vite.config.js`, `tsconfig.json`, etc.) have no effect on the deployed app. +- **Declared dependencies only.** The sandbox rebuilds against a trusted template. You can declare extra npm packages by editing `package.json` (see [Declared dependencies](#declared-dependencies)), but all other root config files (`vite.config.js`, `tsconfig.json`, `tailwind.config.js`, etc.) are read-only — edits to them have no effect on the deployed app. - **Semantic layer coupling.** A moved app's queries run against the target project's fields **by name**. If a field the app references is missing in the target project, the upload still succeeds and the build still finishes — but the app surfaces query errors at runtime when a user opens it. Check the app in the destination after uploading. - **Concurrent build rate limit.** Each project has a cap on how many app builds can run at once. If you upload a large batch and exceed the cap, the server returns HTTP 429 and the CLI reports the error for that app. Re-run the upload for the failed apps once earlier builds complete. - **Bulk download cap.** `--include-apps` returns at most the first 10 apps in a space. Use `--apps ` to fetch more or to reach apps that aren't in a space. +### Declared dependencies + +An app can declare its own npm dependencies alongside the sandbox's built-in library set. This lets a locally-authored app pull in libraries the template doesn't ship — for example, adding `date-fns` to a small ecommerce dashboard for date formatting, or a niche chart library for a one-off visualization. + +Custom dependencies are gated because adding third-party npm packages is a supply-chain capability. Three checks must all pass for an upload to declare dependencies: + +1. **Instance flag on** — `LIGHTDASH_APP_CUSTOM_DEPENDENCIES_ENABLED=true` (see [Self-hosting Data apps](/guides/data-apps/self-hosting#custom-dependencies)). +2. **Organization flag on** — the `enable-data-app-custom-dependencies` feature flag, enabled per organization. On single-org self-hosted instances an admin can enable it globally via `LIGHTDASH_ENABLE_FEATURE_FLAGS`. +3. **User has `manage:DataAppDependency` scope** — a dedicated admin-only scope required specifically to add or change custom dependencies. Everyone with `create:DataApp` or `manage:DataApp` can still upload template-only apps; they just can't declare new dependencies. + +If any of these is missing, the upload is rejected at the API with a clear error. Template-only uploads (no declared dependencies) are unaffected. + +Additional upload-time security screens may also run before your version is accepted (see [Upload-time dependency checks](/guides/data-apps/self-hosting#upload-time-dependency-checks)): a known-malware feed screen (OSV, default on) and an optional minimum release-age guard. + +**Adding a dependency locally** + +Work inside a downloaded app folder (see [Downloading an app](#downloading-an-app)): + +```bash +cd lightdash/apps/ecom-dashboard +pnpm add date-fns +``` + +`pnpm add` updates both `package.json` and `pnpm-lock.yaml`. Commit both — uploads reject new dependencies without a matching lockfile. If you edit `package.json` by hand, run `pnpm install --lockfile-only` to regenerate the lockfile. + +Rules the CLI validates on upload: + +- **Registry packages only.** Plain semver versions (`"date-fns": "^3.0.0"`); no git, file, tarball, or URL specs. +- **Up to 60 direct dependencies.** +- **`pnpm-lock.yaml` must be committed** and consistent with `package.json`. +- **`.npmrc` sets `ignore-scripts=true`.** Downloaded apps may be authored by someone else — this stops their dependencies' install/lifecycle scripts from running on your machine when you `pnpm install` locally. Leave it in place. Explicit `pnpm dev` / `pnpm build` still work. The pinned `@lightdash/query-sdk` is exempt from pnpm's release-age policies because its releases are often less than three days old. +- **Root config outside `package.json` and `pnpm-lock.yaml` stays read-only** — the server rebuilds against its trusted template. + +The AI builder and in-app UI cannot change the dependency set. Declared dependencies are set by editing `package.json` and re-uploading with `lightdash upload --apps`. + +On upload the CLI warns which packages will be installed in the build sandbox. Once the version is built, its `name@version` set appears as a package chip on the version timeline, and the assistant bubble in the chat UI shows the same chip so you can confirm what was actually installed. + +**Blob-URL workers** + +The app-serving Content Security Policy allows `blob:` sources in `script-src`, `worker-src`, and `child-src`. This means libraries that spin up workers or animation runtimes from blob URLs — for example `canvas-confetti`, `deck.gl`, or `comlink` — work in a deployed app without any extra configuration. + ## Data context and the app model Data apps don't ship with their own copy of your data. They run inside a sandboxed iframe with no direct access to the Lightdash API, so every query the app wants to run is proxied through the Lightdash UI hosting the iframe. diff --git a/guides/data-apps/self-hosting.mdx b/guides/data-apps/self-hosting.mdx index 5936d253..1587beef 100644 --- a/guides/data-apps/self-hosting.mdx +++ b/guides/data-apps/self-hosting.mdx @@ -59,6 +59,33 @@ Restart the backend. The "Data apps" entry will appear in the **New** menu for u | --- | --- | --- | | `E2B_TEMPLATE_TAG` | Running Lightdash version (e.g. `0.2917.1`) | Pins the sandbox image to a specific tag of the E2B template. Each Lightdash release publishes a matching tag so the backend and sandbox stay in sync. You can override this to roll back to a previous build (e.g. `E2B_TEMPLATE_TAG=0.2916.0`) or set it to an empty string to use the template's `default` tag. Most operators don't need to touch this. | +### Custom dependencies + +Data apps can [declare their own npm dependencies](/guides/data-apps#declared-dependencies) alongside the sandbox template's built-in library set. Because installing third-party packages is a supply-chain capability, the feature is gated at three layers — all three must pass for an upload to declare dependencies: + +1. **Instance env flag** (this page): `LIGHTDASH_APP_CUSTOM_DEPENDENCIES_ENABLED=true`. Off by default. Also acts as an instance-wide kill-switch — see below. +2. **Per-organization feature flag**: `enable-data-app-custom-dependencies`. Lets Lightdash Cloud enable the feature for specific orgs without instance-wide exposure. On single-org self-hosted instances, add it to `LIGHTDASH_ENABLE_FEATURE_FLAGS` to turn it on globally. +3. **Admin-only scope**: `manage:DataAppDependency` (see [Permissions](#permissions)). Required to add or change custom dependencies specifically; template-only uploads are unaffected. + +Template-only uploads always succeed regardless of these settings. + +| Variable | Default | Purpose | +| --- | --- | --- | +| `LIGHTDASH_APP_CUSTOM_DEPENDENCIES_ENABLED` | `false` | Master switch for custom dependencies. When `false`, uploads that declare a non-empty dependency set are rejected at the API, and builds or AI iterations of versions that already have stored custom dependencies also refuse to run — flipping the toggle back to `false` is a hard stop that also blocks sandbox restores of previously-approved dependency sets. Template-only uploads are always accepted. | +| `LIGHTDASH_APP_DEPENDENCY_REGISTRY_HOSTS` | `registry.npmjs.org` | Comma-separated list of npm registry hostnames added to the sandbox egress allowlist when a version has custom dependencies. Use this to point at an internal mirror. Template-only builds never gain these hosts. | +| `LIGHTDASH_APP_DEPENDENCY_INSTALL_TIMEOUT_MS` | `120000` (2 minutes) | Timeout in milliseconds for the `pnpm install` step that runs before the Vite build when a version has custom dependencies. Raise this if your registry is slow, or lower it to fail faster in CI-style setups. | + +Turning `LIGHTDASH_APP_CUSTOM_DEPENDENCIES_ENABLED` back off is a full kill-switch: it stops installs on every path, including new uploads, AI-iterate carry-forward builds, and sandbox restores of versions whose dependency sets were previously approved. Use it if you need to disable third-party installs on the instance without redeploying app source. + +### Upload-time dependency checks + +Two optional guards run on uploads that pass the gates above. Both apply to the resolved lockfile (direct + transitive), so a malicious transitive dependency is caught even if the top-level package looks fine. + +| Variable | Default | Purpose | +| --- | --- | --- | +| `LIGHTDASH_APP_DEPENDENCY_MALWARE_CHECK_ENABLED` | `true` | Screens every resolved package in the lockfile against the [OSV](https://osv.dev/) malicious-packages feed and rejects the upload if any version carries a `MAL-` advisory. Precise (near-zero false positives) and defaults on for that reason. **Fails closed** — if `api.osv.dev` can't be reached the upload is rejected. Set to `false` on air-gapped instances or during an OSV outage to keep uploading. | +| `LIGHTDASH_APP_DEPENDENCY_MIN_RELEASE_AGE_DAYS` | `0` (off) | Minimum age in days a resolved dependency's version must have since publication on npm. Guards against freshly published, potentially compromised versions before advisory feeds catch up. Off by default because it adds registry round-trips and can block legitimate recent releases; `3` is a sensible starting point if you want it on. Fails closed when a publish date can't be verified. | + ## Costs Self-hosting Data apps means you pay E2B and your Claude provider directly: @@ -72,6 +99,8 @@ Both providers expose usage dashboards. We recommend setting spend limits on bot Data apps follow the same space-based permission model as charts and dashboards. The relevant scopes (`view:DataApp`, `create:DataApp`, `manage:DataApp`) are bundled into the default system roles - but on enterprise instances using custom roles, you'll need to grant them explicitly. See [Custom roles](/references/workspace/custom-roles) for details. +Adding [custom npm dependencies](#custom-dependencies) to an app requires an additional scope, `manage:DataAppDependency`, which is granted to admins only in the default role set. This is deliberate — declaring third-party packages is a supply-chain capability distinct from uploading a template-only app. Grant it to a custom role only when you're comfortable with those users choosing which packages your build sandbox pulls from your configured registry. + ## Troubleshooting **The "Data apps" entry doesn't appear in the New menu.** diff --git a/self-host/customize-deployment/environment-variables.mdx b/self-host/customize-deployment/environment-variables.mdx index 4030f3ef..f5b944a5 100644 --- a/self-host/customize-deployment/environment-variables.mdx +++ b/self-host/customize-deployment/environment-variables.mdx @@ -664,6 +664,11 @@ Configure the sandboxed runtime that powers [data apps](/guides/data-apps/self-h | `E2B_TEMPLATE_TAG` | E2B template tag. Defaults to the running Lightdash version so the template matches the release. | | `E2B_AI_WRITEBACK_TEMPLATE_NAME` | E2B template used for AI writeback sandboxes. (default=`lightdash-ai-writeback`) | | `E2B_AI_WRITEBACK_TEMPLATE_TAG` | E2B AI writeback template tag. Defaults to the running Lightdash version. | +| `LIGHTDASH_APP_CUSTOM_DEPENDENCIES_ENABLED` | Instance-wide switch for [custom app dependencies](/guides/data-apps#declared-dependencies). When `false`, uploads with declared dependencies are rejected and builds/iterations of versions with stored custom deps refuse to run (including sandbox restores). Template-only uploads are unaffected. (default=false) | +| `LIGHTDASH_APP_DEPENDENCY_REGISTRY_HOSTS` | Comma-separated npm registry hostnames added to the sandbox egress allowlist when a version has custom dependencies. Point at an internal mirror if needed. (default=`registry.npmjs.org`) | +| `LIGHTDASH_APP_DEPENDENCY_INSTALL_TIMEOUT_MS` | Timeout in milliseconds for the `pnpm install` step that runs before the Vite build when a version has custom dependencies. (default=120000) | +| `LIGHTDASH_APP_DEPENDENCY_MALWARE_CHECK_ENABLED` | Screens the resolved lockfile of custom-dependency uploads against the OSV malicious-packages feed and rejects the upload on any `MAL-` advisory. Fails closed — set to `false` on air-gapped instances or during an OSV outage. (default=true) | +| `LIGHTDASH_APP_DEPENDENCY_MIN_RELEASE_AGE_DAYS` | Minimum age in days a resolved dependency version must have since publication on npm; `0` disables. Guards against freshly published, potentially compromised versions. Fails closed when a publish date can't be verified. (default=0) | ## Managed agent