diff --git a/documentation/configuration/http-server.md b/documentation/configuration/http-server.md index 65b14ef09..f28a6cd51 100644 --- a/documentation/configuration/http-server.md +++ b/documentation/configuration/http-server.md @@ -32,6 +32,16 @@ Enable or disable the HTTP server. Sets the clock to always return zero. Used for internal testing. +### http.ingest.max.request.size + +- **Default**: `5M` +- **Reloadable**: no + +Maximum request body size for the `/ingest` endpoint used by +[payload transforms](/docs/ingestion/payload-transforms/). The entire request +body is held in memory during processing. Requests exceeding this limit receive +an HTTP 413 (Payload Too Large) response. + ### http.server.keep.alive - **Default**: `true` diff --git a/documentation/ingestion/payload-transforms.md b/documentation/ingestion/payload-transforms.md new file mode 100644 index 000000000..a30e5bc96 --- /dev/null +++ b/documentation/ingestion/payload-transforms.md @@ -0,0 +1,278 @@ +--- +title: Payload transforms +sidebar_label: Payload Transforms +description: + Guide to payload transforms in QuestDB, which parse and insert HTTP payloads + into tables using SQL expressions without middleware. +--- + +Payload transforms define how incoming HTTP payloads are parsed, transformed, and +inserted into a QuestDB table. You define a transform once with a SQL `SELECT` +expression, then POST data directly to QuestDB. The +[`payload()`](/docs/query/functions/meta/#payload) function provides access to +the raw HTTP request body within the transform. No middleware or intermediary +service is required. + +Use cases include webhook ingestion, IoT device data, and external API responses +where you want to skip building a dedicated ingestion service. + +For full SQL syntax, see +[CREATE PAYLOAD TRANSFORM](/docs/query/sql/create-payload-transform/), +[DROP PAYLOAD TRANSFORM](/docs/query/sql/drop-payload-transform/), and +[SHOW PAYLOAD TRANSFORMS](/docs/query/sql/show/#show-payload-transforms). + +## Example: Coinbase order book snapshots + +Store order book snapshots from the +[Coinbase book API](https://api.exchange.coinbase.com/products/BTC-USD/book?level=2). +With `level=2`, the API returns up to 50 aggregated price levels per side: + +```json +{ + "sequence": 125688480181, + "bids": [["69678.77","0.00007525",2], ["69676.36","0.00000022",1], ...], + "asks": [["69678.78","0.35468555",6], ["69679.99","0.00071759",1], ...], + "time": "2026-04-06T11:52:14.454632476Z", + "auction_mode": false +} +``` + +Each bid/ask entry contains the price, quantity, and number of orders at that +level. The `time` field provides a nanosecond-precision exchange timestamp. + +Create a target table with full depth arrays plus top-of-book prices, and a +transform that extracts them from the payload: + +```questdb-sql title="Table and transform definition" +CREATE TABLE coinbase_order_book ( + timestamp TIMESTAMP, + symbol SYMBOL, + bids DOUBLE[][], + asks DOUBLE[][], + best_bid DOUBLE, + best_ask DOUBLE +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; + +CREATE PAYLOAD TRANSFORM coinbase_book_api +INTO coinbase_order_book +DLQ dlq_errors PARTITION BY DAY TTL 7 DAYS +AS DECLARE OVERRIDABLE @symbol := 'BTC-USD' +SELECT + json_extract(payload(), '$.time')::TIMESTAMP AS timestamp, + @symbol AS symbol, + json_extract(payload(), '$.bids')::DOUBLE[][] AS bids, + json_extract(payload(), '$.asks')::DOUBLE[][] AS asks, + json_extract(payload(), '$.bids[0][0]')::DOUBLE AS best_bid, + json_extract(payload(), '$.asks[0][0]')::DOUBLE AS best_ask; +``` + +Fetch 50 levels of depth and ingest the snapshot: + +```shell title="POST a payload" +curl -s "https://api.exchange.coinbase.com/products/BTC-USD/book?level=2" | \ + curl -X POST "http://localhost:9000/ingest?transform=coinbase_book_api" -d @- +``` + +Response: + +```json +{"status": "ok", "rows_inserted": 1} +``` + +### Overriding variables + +The `@symbol` variable is declared `OVERRIDABLE`, so you can override it per +request via URL query parameters: + +```shell title="Override a variable" +curl -s "https://api.exchange.coinbase.com/products/ETH-USD/book?level=2" | \ + curl -X POST "http://localhost:9000/ingest?transform=coinbase_book_api&symbol=ETH-USD" -d @- +``` + +Any URL query parameter other than `transform` is matched to a +`DECLARE OVERRIDABLE` variable by name. Variables not marked `OVERRIDABLE` +cannot be overridden - attempting to do so returns an error. + +## Example: Coinbase trades with UNNEST + +The [Coinbase trades API](https://api.exchange.coinbase.com/products/BTC-USD/trades?limit=100) +returns a JSON array of recent trades. + +```json +[ + {"trade_id": 994619709, "side": "sell", "size": "0.00000100", + "price": "69839.36000000", "time": "2026-04-06T10:32:55.517183Z"}, + {"trade_id": 994619708, "side": "buy", "size": "0.00000006", + "price": "69839.35000000", "time": "2026-04-06T10:32:55.418434Z"}, + ... +] +``` + +The transform uses [JSON UNNEST](/docs/query/sql/unnest/#json-unnest) to expand +the array into individual rows, one per trade. Each request may return trades +already seen in a previous request, so the target table enables +[deduplication](/docs/concepts/deduplication/) to handle overlapping results +safely: + +```questdb-sql title="Table with deduplication and transform" +CREATE TABLE coinbase_trades ( + timestamp TIMESTAMP, + symbol SYMBOL, + trade_id LONG, + price DOUBLE, + size DOUBLE, + side SYMBOL +) TIMESTAMP(timestamp) PARTITION BY DAY WAL +DEDUP UPSERT KEYS(timestamp, symbol, side); + +CREATE PAYLOAD TRANSFORM coinbase_trades_api +INTO coinbase_trades +DLQ dlq_errors PARTITION BY DAY TTL 7 DAYS +AS DECLARE OVERRIDABLE @symbol := 'BTC-USD' +SELECT + u.time AS timestamp, + @symbol AS symbol, + u.trade_id, + u.price, + u.size, + u.side +FROM UNNEST( + payload() COLUMNS( + trade_id LONG, + price DOUBLE, + size DOUBLE, + side VARCHAR, + time TIMESTAMP + ) +) u; +``` + +Fetch the latest 100 trades and ingest them: + +```shell title="Ingest trades" +curl -s "https://api.exchange.coinbase.com/products/BTC-USD/trades?limit=100" | \ + curl -X POST "http://localhost:9000/ingest?transform=coinbase_trades_api" -d @- +``` + +If any trades were already ingested from a previous request, deduplication +discards the duplicates automatically. + +### Inspecting failed payloads + +When a payload fails (bad JSON, type mismatch, missing columns), QuestDB writes +the original payload, the error stage, and the error message to the DLQ table +configured in the transform: + +```questdb-sql title="Query the DLQ" +SELECT ts, transform_name, stage, error FROM dlq_errors; +``` + +| ts | transform_name | stage | error | +| :--- | :--- | :--- | :--- | +| 2026-03-23T14:00:00.000000Z | coinbase_book_api | transform | column not found in target table [column=extra] | +| 2026-03-23T14:01:00.000000Z | coinbase_trades_api | transform | bad JSON payload | + +Multiple transforms can share the same DLQ table. See +[CREATE PAYLOAD TRANSFORM](/docs/query/sql/create-payload-transform/#dead-letter-queue-schema) +for the full DLQ schema. + +## HTTP endpoint + +**POST** `/ingest` + +### Query parameters + +| Parameter | Required | Description | +| :--- | :--- | :--- | +| `transform` | Yes | Name of the payload transform to execute | +| Any other | No | Overrides a `DECLARE OVERRIDABLE` variable by name | + +The request body is the raw payload, accessible via +[`payload()`](/docs/query/functions/meta/#payload) in the transform SQL. + +### Responses + +Success: + +```json +{"status": "ok", "rows_inserted": 1} +``` + +Error: + +```json +{"status": "error", "message": "..."} +``` + +## Permissions + +In QuestDB Open Source, any user with access to the HTTP endpoint can create +transforms and invoke `/ingest`. + +:::note Enterprise + +In [QuestDB Enterprise](/enterprise/) deployments with +[RBAC](/docs/security/rbac/) enabled, the following grants are required: + +| Action | Required grants | +| :----- | :-------------- | +| Create a transform | `CREATE PAYLOAD TRANSFORM` and `INSERT` on the target table (and DLQ table, if configured) | +| Replace a transform (`OR REPLACE`) | `CREATE PAYLOAD TRANSFORM` and `DROP PAYLOAD TRANSFORM` | +| Drop a transform | `DROP PAYLOAD TRANSFORM` | +| Invoke `/ingest` | `HTTP` endpoint grant and `INSERT` on the target table | + +```questdb-sql title="Typical Enterprise setup" +-- Admin who manages transforms +GRANT CREATE PAYLOAD TRANSFORM, DROP PAYLOAD TRANSFORM TO ingest_admin; +GRANT INSERT ON coinbase_order_book, coinbase_trades, dlq_errors TO ingest_admin; + +-- Service account that calls /ingest +GRANT HTTP TO ingest_service; +GRANT INSERT ON coinbase_order_book, coinbase_trades TO ingest_service; +``` + +::: + +## Request size limit + +The `/ingest` endpoint rejects request bodies that exceed a configurable maximum +size. The default limit is 5 MB. To change it, set the +`http.ingest.max.request.size` property in `server.conf`: + +```ini title="server.conf" +http.ingest.max.request.size=10M +``` + +Requests exceeding the limit receive an HTTP 413 (Payload Too Large) response. +The entire request body is held in memory during processing, so set this limit +based on available memory and expected payload sizes. + +## Limitations + +- **Single payload per request** - Each HTTP request executes the transform + once. That execution may produce multiple rows. Sending multiple + independent payload documents in a single request is not supported. +- **Per-request SQL compilation** - Transform SQL is compiled on every request. + This is acceptable for low-rate ingestion workloads. Compiled-plan caching is + a planned optimization. +- **No table references** - The transform SELECT must not reference existing + tables. It can only use functions and expressions, including CTEs. +- **SELECT only** - Only `SELECT` statements are allowed. `INSERT`, `UPDATE`, + and other statements are rejected at creation time. +- **Schema drift** - Column names and types are validated against the target + table at creation time. Schema changes to the target table after creating a + transform may cause runtime errors. +- **Concurrent DDL** - `CREATE`, `DROP`, and `OR REPLACE` for the same + transform name are not serialized. If two sessions operate on the same + transform name concurrently, the outcome is last-writer-wins. + +:::info Related documentation +- [CREATE PAYLOAD TRANSFORM](/docs/query/sql/create-payload-transform/) +- [DROP PAYLOAD TRANSFORM](/docs/query/sql/drop-payload-transform/) +- [SHOW PAYLOAD TRANSFORMS](/docs/query/sql/show/#show-payload-transforms) +- [UNNEST](/docs/query/sql/unnest/) +- [`payload()` function](/docs/query/functions/meta/#payload) +- [JSON functions](/docs/query/functions/json/) +- [REST API](/docs/query/rest-api/) +- [Role-Based Access Control (RBAC)](/docs/security/rbac/) +::: diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index dae212033..dedeacd66 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -204,6 +204,44 @@ memory_metrics(); | MMAP_O3 | 0 | | NATIVE_O3 | 96 | +## payload + +`payload()` returns the raw HTTP request body as a `VARCHAR`. It is only +available inside +[payload transform](/docs/ingestion/payload-transforms/) queries - calling it in +any other context produces an error. + +**Arguments:** + +- `payload()` does not require arguments. + +**Return value:** + +Returns the HTTP request body as a `VARCHAR` value. + +**Examples:** + +Typically used with `json_extract()` to parse JSON payloads: + +```questdb-sql title="Extract fields from a JSON payload" +CREATE PAYLOAD TRANSFORM sensor_ingest +INTO sensor_data +AS SELECT + now() AS ts, + json_extract(payload(), '$.device_id')::VARCHAR AS device_id, + json_extract(payload(), '$.temperature')::DOUBLE AS temperature; +``` + +Can also be stored directly as a raw string: + +```questdb-sql title="Store the raw payload" +CREATE PAYLOAD TRANSFORM raw_events +INTO event_log +AS SELECT + now() AS ts, + payload() AS raw_body; +``` + ## query_activity **Arguments:** diff --git a/documentation/query/rest-api.md b/documentation/query/rest-api.md index 3d3ee40e0..94529801a 100644 --- a/documentation/query/rest-api.md +++ b/documentation/query/rest-api.md @@ -27,6 +27,7 @@ The [Web Console](/docs/getting-started/web-console/overview/) is the official W - [`/imp`](#imp---import-data) for importing data from `.CSV` files - [`/exec`](#exec---execute-queries) to execute a SQL statement - [`/exp`](#exp---export-data) to export data +- [`/ingest`](#ingest---payload-transforms) to ingest data via payload transforms ## Examples @@ -38,6 +39,7 @@ insert-capable entrypoints: | :----------------------------------------- | :---------- | :-------------------------------------- | :-------------------------------------------------------- | | [`/imp`](#imp-uploading-tabular-data) | POST | Import CSV data | [Reference](/docs/query/rest-api/#imp---import-data) | | [`/exec?query=..`](#exec-sql-insert-query) | GET | Run SQL Query returning JSON result set | [Reference](/docs/query/rest-api/#exec---execute-queries) | +| [`/ingest?transform=..`](#ingest---payload-transforms) | POST | Execute a payload transform | [Reference](/docs/query/rest-api/#ingest---payload-transforms) | For details such as content type, query parameters and more, refer to the [REST API](/docs/query/rest-api/) docs. @@ -696,6 +698,61 @@ curl -G \ http://localhost:9000/exp > trades_bloom.parquet ``` +## /ingest - Payload transforms + +`/ingest` executes a [payload transform](/docs/ingestion/payload-transforms/) +against the raw HTTP request body and inserts the resulting rows into the +transform's target table. + +### Overview + +`/ingest` expects an HTTP POST request with the raw payload as the request body. +The `Content-Type` header is not enforced - the body is passed as-is to the +transform's `payload()` function. + +#### Parameters + +| Parameter | Required | Description | +| :--- | :--- | :--- | +| `transform` | Yes | Name of the payload transform to execute | +| Any other | No | Overrides a `DECLARE OVERRIDABLE` variable by name | + +#### Responses + +Success (HTTP 200): + +```json +{"status": "ok", "rows_inserted": 1} +``` + +Error (HTTP 400 or 413): + +```json +{"status": "error", "message": "..."} +``` + +Requests exceeding the configured `http.ingest.max.request.size` (default 5 MB) +receive an HTTP 413 (Payload Too Large) response. + +### Example + +```shell title="Ingest a JSON payload" +curl -s "https://api.exchange.coinbase.com/products/BTC-USD/book?level=2" | \ + curl -X POST "http://localhost:9000/ingest?transform=coinbase_book_api" -d @- +``` + +For full documentation on creating and managing transforms, see +[Payload transforms](/docs/ingestion/payload-transforms/). + +:::note Enterprise + +In [QuestDB Enterprise](/enterprise/) deployments with +[RBAC](/docs/security/rbac/) enabled, the caller must hold the `HTTP` endpoint +grant and `INSERT` permission on the target table. + +::: + + ## Error responses ### Malformed queries diff --git a/documentation/query/sql/create-payload-transform.md b/documentation/query/sql/create-payload-transform.md new file mode 100644 index 000000000..74c7a0241 --- /dev/null +++ b/documentation/query/sql/create-payload-transform.md @@ -0,0 +1,184 @@ +--- +title: CREATE PAYLOAD TRANSFORM +sidebar_label: CREATE PAYLOAD TRANSFORM +description: + Documentation for the CREATE PAYLOAD TRANSFORM SQL keyword in QuestDB. +--- + +Creates a payload transform that defines how incoming HTTP payloads are parsed, +transformed, and inserted into a target table. Once created, data is ingested by +POSTing to the [`/ingest` endpoint](/docs/ingestion/payload-transforms/#http-endpoint). + +## Syntax + +``` +CREATE [ OR REPLACE ] PAYLOAD TRANSFORM transformName +INTO targetTable +[ DLQ dlqTable [ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) ] [ TTL n timeUnit ] ] +AS [ DECLARE [ OVERRIDABLE ] @var := value [, [ OVERRIDABLE ] @var2 := value2 ... ] ] +SELECT ... +``` + +Where: +- `timeUnit`: `HOURS | DAYS | WEEKS | MONTHS | YEARS` +- The `SELECT` must not reference existing tables - it can only use functions + and expressions, including CTEs +- Use [`payload()`](/docs/query/functions/meta/#payload) to access the raw HTTP + request body as a `VARCHAR` + +## Parameters + +| Parameter | Description | +| --------- | ----------- | +| `transformName` | Name for the payload transform | +| `OR REPLACE` | Replace existing transform with the same name | +| `targetTable` | Table to insert rows into | +| `DLQ dlqTable` | Route failed payloads to a dead-letter queue table | +| `PARTITION BY` | Partitioning unit for the DLQ table (if QuestDB creates it) | +| `TTL` | Retention period for DLQ rows | +| `DECLARE` | Define variables used in the SELECT | +| `OVERRIDABLE` | Allow variable to be overridden via URL query parameters | + +## Column mapping + +SELECT output column names must match column names in the target table. Columns +are matched by name, not position. You do not need to produce all columns - any +columns not included in the SELECT receive their default values. + +## Examples + +### Basic transform + +```questdb-sql title="Create a table and a transform for Coinbase order book snapshots" +CREATE TABLE coinbase_order_book ( + timestamp TIMESTAMP, + symbol SYMBOL, + bids DOUBLE[][], + asks DOUBLE[][], + best_bid DOUBLE, + best_ask DOUBLE +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; + +CREATE PAYLOAD TRANSFORM coinbase_book_api +INTO coinbase_order_book +AS DECLARE OVERRIDABLE @symbol := 'BTC-USD' +SELECT + json_extract(payload(), '$.time')::TIMESTAMP AS timestamp, + @symbol AS symbol, + json_extract(payload(), '$.bids')::DOUBLE[][] AS bids, + json_extract(payload(), '$.asks')::DOUBLE[][] AS asks, + json_extract(payload(), '$.bids[0][0]')::DOUBLE AS best_bid, + json_extract(payload(), '$.asks[0][0]')::DOUBLE AS best_ask; +``` + +### With dead-letter queue + +```questdb-sql title="Transform with DLQ and 7-day retention" +CREATE PAYLOAD TRANSFORM coinbase_book_api +INTO coinbase_order_book +DLQ dlq_errors PARTITION BY DAY TTL 7 DAYS +AS DECLARE OVERRIDABLE @symbol := 'BTC-USD' +SELECT + json_extract(payload(), '$.time')::TIMESTAMP AS timestamp, + @symbol AS symbol, + json_extract(payload(), '$.bids')::DOUBLE[][] AS bids, + json_extract(payload(), '$.asks')::DOUBLE[][] AS asks, + json_extract(payload(), '$.bids[0][0]')::DOUBLE AS best_bid, + json_extract(payload(), '$.asks[0][0]')::DOUBLE AS best_ask; +``` + +### Replace an existing transform + +```questdb-sql title="Replace a transform definition" +CREATE OR REPLACE PAYLOAD TRANSFORM coinbase_book_api +INTO coinbase_order_book +AS SELECT + json_extract(payload(), '$.time')::TIMESTAMP AS timestamp, + 'BTC-USD' AS symbol, + json_extract(payload(), '$.bids')::DOUBLE[][] AS bids, + json_extract(payload(), '$.asks')::DOUBLE[][] AS asks, + json_extract(payload(), '$.bids[0][0]')::DOUBLE AS best_bid, + json_extract(payload(), '$.asks[0][0]')::DOUBLE AS best_ask; +``` + +### Multiple overridable variables + +```questdb-sql title="Two overridable variables with defaults" +CREATE PAYLOAD TRANSFORM sensor_ingest +INTO sensor_data +AS DECLARE OVERRIDABLE @source := 'default', OVERRIDABLE @region := 'us-east' +SELECT + now() AS ts, + @source AS source, + @region AS region, + json_extract(payload(), '$.temperature')::DOUBLE AS temperature; +``` + +## Validation + +QuestDB validates the transform at creation time: + +| Check | Description | +| ----- | ----------- | +| Column names | Every SELECT output column must exist in the target table | +| Column types | Each output type must be convertible to the target column type, following `INSERT AS SELECT` rules | +| DLQ schema | If the DLQ table already exists, its schema must match the expected DLQ layout | + +Validation errors report the position of the offending column expression in the +SELECT. + +## Dead-letter queue schema + +When a DLQ is configured and a transform error occurs, QuestDB writes a row with +the following columns: + +| Column | Type | Description | +| :----- | :--- | :---------- | +| `ts` | TIMESTAMP | When the error occurred (designated timestamp) | +| `transform_name` | SYMBOL | Name of the transform that failed | +| `payload` | VARCHAR | The original HTTP body | +| `query` | VARCHAR | The transform's SELECT SQL | +| `stage` | SYMBOL | Processing stage where the error occurred | +| `error` | VARCHAR | Error message | + +Multiple transforms can share the same DLQ table. The HTTP response still +returns an error so the caller knows the request failed. + +## Permissions + +| Context | Requirement | +| ------- | ----------- | +| Target table | The `/ingest` caller must have INSERT permission, checked at request time | +| DLQ table | The DDL caller must have INSERT permission, checked at creation time. Runtime DLQ writes use the system security context | + +:::note Enterprise + +In [QuestDB Enterprise](/enterprise/) deployments with +[RBAC](/docs/security/rbac/) enabled, the user creating the transform must hold +the `CREATE PAYLOAD TRANSFORM` grant. When using `OR REPLACE` on a transform +that already exists, the user must also hold `DROP PAYLOAD TRANSFORM`. The +`/ingest` caller must hold the `HTTP` endpoint grant and `INSERT` permission on +the target table (and on the DLQ table, if configured). + +```questdb-sql +-- Create-only +GRANT CREATE PAYLOAD TRANSFORM TO ingest_admin; + +-- If using OR REPLACE +GRANT CREATE PAYLOAD TRANSFORM, DROP PAYLOAD TRANSFORM TO ingest_admin; + +GRANT HTTP TO ingest_service; +GRANT INSERT ON coinbase_order_book TO ingest_service; +GRANT INSERT ON dlq_errors TO ingest_service; +``` + +::: + +:::info Related documentation +- [DROP PAYLOAD TRANSFORM](/docs/query/sql/drop-payload-transform/) +- [SHOW PAYLOAD TRANSFORMS](/docs/query/sql/show/#show-payload-transforms) +- [Payload transforms overview](/docs/ingestion/payload-transforms/) +- [`payload()` function](/docs/query/functions/meta/#payload) +- [JSON functions](/docs/query/functions/json/) +- [Role-Based Access Control (RBAC)](/docs/security/rbac/) +::: diff --git a/documentation/query/sql/drop-payload-transform.md b/documentation/query/sql/drop-payload-transform.md new file mode 100644 index 000000000..509bffc06 --- /dev/null +++ b/documentation/query/sql/drop-payload-transform.md @@ -0,0 +1,61 @@ +--- +title: DROP PAYLOAD TRANSFORM +sidebar_label: DROP PAYLOAD TRANSFORM +description: + Documentation for the DROP PAYLOAD TRANSFORM SQL keyword in QuestDB. +--- + +Permanently deletes a payload transform definition. The target table and any DLQ +table are not affected. + +## Syntax + +``` +DROP PAYLOAD TRANSFORM [ IF EXISTS ] transformName +``` + +## Parameters + +| Parameter | Description | +| --------- | ----------- | +| `transformName` | Name of the payload transform to drop | +| `IF EXISTS` | Suppress error if the transform does not exist | + +## Examples + +```questdb-sql title="Drop a payload transform" +DROP PAYLOAD TRANSFORM coinbase_book_api; +``` + +```questdb-sql title="Drop only if exists (no error if missing)" +DROP PAYLOAD TRANSFORM IF EXISTS coinbase_book_api; +``` + +## Behavior + +| Aspect | Description | +| ------ | ----------- | +| Target table | Not affected - existing data remains | +| DLQ table | Not affected - existing error rows remain | +| Active requests | In-flight `/ingest` requests may still complete | + +## Permissions + +:::note Enterprise + +In [QuestDB Enterprise](/enterprise/) deployments with +[RBAC](/docs/security/rbac/) enabled, the user must hold the +`DROP PAYLOAD TRANSFORM` grant. + +```questdb-sql +GRANT DROP PAYLOAD TRANSFORM TO ingest_admin; +``` + +::: + +:::info Related documentation +- [CREATE PAYLOAD TRANSFORM](/docs/query/sql/create-payload-transform/) +- [SHOW PAYLOAD TRANSFORMS](/docs/query/sql/show/#show-payload-transforms) +- [Payload transforms overview](/docs/ingestion/payload-transforms/) +- [Role-Based Access Control (RBAC)](/docs/security/rbac/) +::: diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 6976ce129..17dcadde2 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -4,13 +4,10 @@ sidebar_label: SHOW description: SHOW SQL keyword reference documentation. --- -This keyword provides table, column, and partition information including -metadata. The `SHOW` keyword is useful for checking the -[designated timestamp setting](/docs/concepts/designated-timestamp/) column, the -[partition attachment settings](/docs/query/sql/alter-table-attach-partition/), -and partition storage size on disk. +`SHOW` returns metadata about tables, columns, partitions, transforms, +configuration, and users. -## Syntax +## Available statements ```questdb-sql SHOW { TABLES @@ -18,6 +15,7 @@ SHOW { TABLES | PARTITIONS FROM tableName | CREATE TABLE tableName | CREATE VIEW viewName + | PAYLOAD TRANSFORMS | USER [userName] | USERS | GROUPS [userName] @@ -28,49 +26,37 @@ SHOW { TABLES | PARAMETERS }; ``` -## Description - -- `SHOW TABLES` returns all the tables. -- `SHOW COLUMNS` returns all the columns and their metadata for the selected - table. -- `SHOW PARTITIONS` returns the partition information for the selected table. -- `SHOW CREATE TABLE` returns a DDL query that allows you to recreate the table. -- `SHOW CREATE VIEW` returns a DDL query that allows you to recreate a view. -- `SHOW USER` shows user secret (enterprise-only) -- `SHOW GROUPS` shows all groups the user belongs or all groups in the system - (enterprise-only) -- `SHOW USERS` shows all users (enterprise-only) -- `SHOW SERVICE ACCOUNT` displays details of a service account (enterprise-only) -- `SHOW SERVICE ACCOUNTS` displays all service accounts or those assigned to the - user/group (enterprise-only) -- `SHOW PERMISSIONS` displays permissions of user, group or service account - (enterprise-only) -- `SHOW SERVER_VERSION` displays PostgreSQL compatibility version -- `SHOW PARAMETERS` shows configuration keys and their matching `env_var_name`, - their values and the source of the value - -## Examples - -### SHOW TABLES - -```questdb-sql title="show tables" demo -SHOW TABLES; +- [`SHOW COLUMNS`](#show-columns) - column metadata for a table +- [`SHOW CREATE TABLE`](#show-create-table) - DDL to recreate a table +- [`SHOW CREATE VIEW`](#show-create-view) - DDL to recreate a view +- [`SHOW GROUPS`](#show-groups) - groups a user belongs to, or all groups (enterprise) +- [`SHOW PARAMETERS`](#show-parameters) - configuration keys, values, and sources +- [`SHOW PARTITIONS`](#show-partitions) - partition information for a table +- [`SHOW PAYLOAD TRANSFORMS`](#show-payload-transforms) - all defined payload transforms +- [`SHOW PERMISSIONS`](#show-permissions) - permissions for a user, group, or service account (enterprise) +- [`SHOW SERVER_VERSION`](#show-server_version) - PostgreSQL compatibility version +- [`SHOW SERVICE ACCOUNT`](#show-service-account) - details of a service account (enterprise) +- [`SHOW SERVICE ACCOUNTS`](#show-service-accounts) - all service accounts, or those assigned to a user/group (enterprise) +- [`SHOW TABLES`](#show-tables) - all tables +- [`SHOW USER`](#show-user) - user authentication details (enterprise) +- [`SHOW USERS`](#show-users) - all users (enterprise) + +## SHOW COLUMNS + +### Syntax + +``` +SHOW COLUMNS FROM tableName ``` -| table_name | -| --------------- | -| ethblocks_json | -| trades | -| weather | -| AAPL_orderbook | -| trips | +Returns all columns and their metadata for the selected table. -### SHOW COLUMNS +### Example -```questdb-sql title="show columns" demo +```questdb-sql title="Show columns" demo SHOW COLUMNS FROM trades; - ``` + | column | type | indexed | indexBlockCapacity | symbolCached | symbolCapacity | symbolTableSize | designated | upsertKey | | --------- | --------- | ------- | ------------------ | ------------ | -------------- | --------------- | ---------- | --------- | | symbol | SYMBOL | false | 0 | true | 256 | 42 | false | false | @@ -79,9 +65,19 @@ SHOW COLUMNS FROM trades; | amount | DOUBLE | false | 0 | false | 0 | 0 | false | false | | timestamp | TIMESTAMP | false | 0 | false | 0 | 0 | true | false | -### SHOW CREATE TABLE +## SHOW CREATE TABLE + +### Syntax -```questdb-sql title="retrieving table ddl" demo +``` +SHOW CREATE TABLE tableName +``` + +Returns a DDL query that allows you to recreate the table. + +### Example + +```questdb-sql title="Show create table" demo SHOW CREATE TABLE trades; ``` @@ -163,9 +159,19 @@ This clause assigns permissions for the table to that user. If permissions should be assigned to a different user, please modify this clause appropriately. -### SHOW CREATE VIEW +## SHOW CREATE VIEW -```questdb-sql title="retrieving view ddl" +### Syntax + +``` +SHOW CREATE VIEW viewName +``` + +Returns a DDL query that allows you to recreate a view. + +### Example + +```questdb-sql title="Show create view" SHOW CREATE VIEW my_view; ``` @@ -176,30 +182,48 @@ SHOW CREATE VIEW my_view; This returns the `CREATE VIEW` statement that would recreate the view, including any `DECLARE` parameters if the view is parameterized. -### SHOW PARTITIONS +## SHOW GROUPS + +### Syntax + +``` +SHOW GROUPS [ entityName ] +``` + +Shows all groups in the system, or all groups a user belongs to. Enterprise only. + +### Examples ```questdb-sql -SHOW PARTITIONS FROM my_table; +SHOW GROUPS; ``` -| index | partitionBy | name | minTimestamp | maxTimestamp | numRows | diskSize | diskSizeHuman | readOnly | active | attached | detached | attachable | hasParquetGenerated | isParquet | parquetFileSize | -| ----- | ----------- | -------- | --------------------- | --------------------- | ------- | -------- | ------------- | -------- | ------ | -------- | -------- | ---------- | ------------------- | --------- | --------------- | -| 0 | WEEK | 2022-W52 | 2023-01-01 00:36:00.0 | 2023-01-01 23:24:00.0 | 39 | 98304 | 96.0 KiB | false | false | true | false | false | false | false | -1 | -| 1 | WEEK | 2023-W01 | 2023-01-02 00:00:00.0 | 2023-01-08 23:24:00.0 | 280 | 98304 | 96.0 KiB | false | false | true | false | false | false | false | -1 | -| 2 | WEEK | 2023-W02 | 2023-01-09 00:00:00.0 | 2023-01-15 23:24:00.0 | 280 | 98304 | 96.0 KiB | false | false | true | false | false | false | false | -1 | -| 3 | WEEK | 2023-W03 | 2023-01-16 00:00:00.0 | 2023-01-18 12:00:00.0 | 101 | 83902464 | 80.0 MiB | false | true | true | false | false | false | false | -1 | +```questdb-sql +SHOW GROUPS john; +``` -See [`table_partitions()`](/docs/query/functions/meta/#table_partitions) for the -full column list, including `hasParquetGenerated`, `isParquet`, and -`parquetFileSize`. +| name | +| ---------- | +| management | + +## SHOW PARAMETERS -### SHOW PARAMETERS +### Syntax + +``` +SHOW PARAMETERS +``` + +Shows configuration keys and their matching `env_var_name`, their values, and +the source of the value. + +### Example ```questdb-sql SHOW PARAMETERS; ``` -The output demonstrates: +The output columns: - `property_path`: the configuration key - `env_var_name`: the matching env var for the key @@ -218,7 +242,6 @@ The output demonstrates: | pg.readonly.password | QDB_PG_READONLY_PASSWORD | **** | default | true | true | | http.password | QDB_HTTP_PASSWORD | **** | default | true | false | - You can optionally chain `SHOW PARAMETERS` with other clauses: ```questdb-sql @@ -235,59 +258,140 @@ You can optionally chain `SHOW PARAMETERS` with other clauses: (SHOW PARAMETERS) WHERE value_source <> 'default'; ``` -### SHOW USER +## SHOW PARTITIONS -```questdb-sql -SHOW USER; --as john +### Syntax + +``` +SHOW PARTITIONS FROM tableName ``` -or +Returns partition information for the selected table. + +### Example ```questdb-sql -SHOW USER john; +SHOW PARTITIONS FROM my_table; ``` -| auth_type | enabled | -| ---------- | ------- | -| Password | false | -| JWK Token | false | -| REST Token | false | +| index | partitionBy | name | minTimestamp | maxTimestamp | numRows | diskSize | diskSizeHuman | readOnly | active | attached | detached | attachable | hasParquetGenerated | isParquet | parquetFileSize | +| ----- | ----------- | -------- | --------------------- | --------------------- | ------- | -------- | ------------- | -------- | ------ | -------- | -------- | ---------- | ------------------- | --------- | --------------- | +| 0 | WEEK | 2022-W52 | 2023-01-01 00:36:00.0 | 2023-01-01 23:24:00.0 | 39 | 98304 | 96.0 KiB | false | false | true | false | false | false | false | -1 | +| 1 | WEEK | 2023-W01 | 2023-01-02 00:00:00.0 | 2023-01-08 23:24:00.0 | 280 | 98304 | 96.0 KiB | false | false | true | false | false | false | false | -1 | +| 2 | WEEK | 2023-W02 | 2023-01-09 00:00:00.0 | 2023-01-15 23:24:00.0 | 280 | 98304 | 96.0 KiB | false | false | true | false | false | false | false | -1 | +| 3 | WEEK | 2023-W03 | 2023-01-16 00:00:00.0 | 2023-01-18 12:00:00.0 | 101 | 83902464 | 80.0 MiB | false | true | true | false | false | false | false | -1 | -### SHOW USERS +See [`table_partitions()`](/docs/query/functions/meta/#table_partitions) for the +full column list, including `hasParquetGenerated`, `isParquet`, and +`parquetFileSize`. -```questdb-sql -SHOW USERS; +## SHOW PAYLOAD TRANSFORMS + +### Syntax + +``` +SHOW PAYLOAD TRANSFORMS ``` -| name | -| ----- | -| admin | -| john | +Lists all defined [payload transforms](/docs/ingestion/payload-transforms/). -### SHOW GROUPS +### Example -```questdb-sql -SHOW GROUPS; +```questdb-sql title="List all payload transforms" +SHOW PAYLOAD TRANSFORMS; +``` + +| name | target_table | dlq_table | query | +| :--- | :--- | :--- | :--- | +| coinbase_book_api | coinbase_order_book | dlq_errors | DECLARE OVERRIDABLE @symbol := 'BTC-USD' SELECT json_extract(payload(), '$.time')::TIMESTAMP AS timestamp, ... | +| coinbase_trades_api | coinbase_trades | dlq_errors | DECLARE OVERRIDABLE @symbol := 'BTC-USD' SELECT u.time AS timestamp, ... FROM UNNEST(payload() COLUMNS(...)) u | + +## SHOW PERMISSIONS + +### Syntax + +``` +SHOW PERMISSIONS [ entityName ] +``` + +Displays permissions of a user, group, or service account. Enterprise only. + +Without an argument, shows permissions for the current user. + +### Examples + +```questdb-sql title="Current user" +SHOW PERMISSIONS; +``` + +| permission | table_name | column_name | grant_option | origin | +| ---------- | ---------- | ----------- | ------------ | ------ | +| SELECT | | | t | G | + +```questdb-sql title="Specific user" +SHOW PERMISSIONS admin; +``` + +| permission | table_name | column_name | grant_option | origin | +| ---------- | ---------- | ----------- | ------------ | ------ | +| SELECT | | | t | G | +| INSERT | orders | | f | G | +| UPDATE | order_itme | quantity | f | G | + +```questdb-sql title="Group" +SHOW PERMISSIONS admin_group; +``` + +| permission | table_name | column_name | grant_option | origin | +| ---------- | ---------- | ----------- | ------------ | ------ | +| INSERT | orders | | f | G | + +```questdb-sql title="Service account" +SHOW PERMISSIONS ilp_ingestion; ``` -or +| permission | table_name | column_name | grant_option | origin | +| ---------- | ---------- | ----------- | ------------ | ------ | +| SELECT | | | t | G | +| INSERT | | | f | G | +| UPDATE | | | f | G | + +## SHOW SERVER_VERSION + +### Syntax + +``` +SHOW SERVER_VERSION +``` + +Shows PostgreSQL compatibility version. + +### Example ```questdb-sql -SHOW GROUPS john; +SHOW SERVER_VERSION; ``` -| name | -| ---------- | -| management | +| server_version | +| -------------- | +| 12.3 (questdb) | + +## SHOW SERVICE ACCOUNT + +### Syntax + +``` +SHOW SERVICE ACCOUNT [ accountName ] +``` + +Displays details of a service account. Enterprise only. -### SHOW SERVICE ACCOUNT +### Examples ```questdb-sql SHOW SERVICE ACCOUNT; ``` -or - ```questdb-sql SHOW SERVICE ACCOUNT ilp_ingestion; ``` @@ -298,7 +402,18 @@ SHOW SERVICE ACCOUNT ilp_ingestion; | JWK Token | false | | REST Token | false | -### SHOW SERVICE ACCOUNTS +## SHOW SERVICE ACCOUNTS + +### Syntax + +``` +SHOW SERVICE ACCOUNTS [ entityName ] +``` + +Displays all service accounts, or those assigned to a user or group. Enterprise +only. + +### Examples ```questdb-sql SHOW SERVICE ACCOUNTS; @@ -325,63 +440,76 @@ SHOW SERVICE ACCOUNTS admin_group; | ---------- | | svc1_admin | -### SHOW PERMISSIONS FOR CURRENT USER +## SHOW TABLES + +### Syntax -```questdb-sql -SHOW PERMISSIONS; +``` +SHOW TABLES ``` -| permission | table_name | column_name | grant_option | origin | -| ---------- | ---------- | ----------- | ------------ | ------ | -| SELECT | | | t | G | +Returns all tables. -### SHOW PERMISSIONS user +### Example -```questdb-sql -SHOW PERMISSIONS admin; +```questdb-sql title="Show tables" demo +SHOW TABLES; ``` -| permission | table_name | column_name | grant_option | origin | -| ---------- | ---------- | ----------- | ------------ | ------ | -| SELECT | | | t | G | -| INSERT | orders | | f | G | -| UPDATE | order_itme | quantity | f | G | +| table_name | +| --------------- | +| ethblocks_json | +| trades | +| weather | +| AAPL_orderbook | +| trips | -### SHOW PERMISSIONS +## SHOW USER -#### For a group +### Syntax -```questdb-sql -SHOW PERMISSIONS admin_group; +``` +SHOW USER [ userName ] ``` -| permission | table_name | column_name | grant_option | origin | -| ---------- | ---------- | ----------- | ------------ | ------ | -| INSERT | orders | | f | G | +Shows user authentication details. Enterprise only. -#### For a service account +### Examples ```questdb-sql -SHOW PERMISSIONS ilp_ingestion; +SHOW USER; --as john ``` -| permission | table_name | column_name | grant_option | origin | -| ---------- | ---------- | ----------- | ------------ | ------ | -| SELECT | | | t | G | -| INSERT | | | f | G | -| UPDATE | | | f | G | +```questdb-sql +SHOW USER john; +``` -### SHOW SERVER_VERSION +| auth_type | enabled | +| ---------- | ------- | +| Password | false | +| JWK Token | false | +| REST Token | false | -Shows PostgreSQL compatibility version. +## SHOW USERS + +### Syntax + +``` +SHOW USERS +``` + +Shows all users. Enterprise only. + +### Example ```questdb-sql -SHOW SERVER_VERSION; +SHOW USERS; ``` -| server_version | -| -------------- | -| 12.3 (questdb) | +| name | +| ----- | +| admin | +| john | ## See also diff --git a/documentation/query/sql/unnest.md b/documentation/query/sql/unnest.md index de677f13e..70d8896ec 100644 --- a/documentation/query/sql/unnest.md +++ b/documentation/query/sql/unnest.md @@ -164,6 +164,16 @@ JSON `UNNEST` expands a JSON array (stored as `VARCHAR`) into rows with explicitly typed columns. The `COLUMNS(...)` clause distinguishes JSON `UNNEST` from array `UNNEST`. +:::tip Automate API ingestion + +Combine JSON UNNEST with +[payload transforms](/docs/ingestion/payload-transforms/) to ingest data from +external APIs directly into QuestDB - no middleware required. See the +[Coinbase trades example](/docs/ingestion/payload-transforms/#example-coinbase-trades-with-unnest) +for a complete walkthrough. + +::: + ### Syntax ```questdb-sql diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index b82e3eff7..deb8cfe32 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -566,8 +566,9 @@ SELECT * FROM all_permissions(); | ATTACH PARTITION | Database | Table | Attach partitions | | BACKUP DATABASE | Database | Create database backups | | CANCEL ANY COPY | Database | Cancel COPY operations | -| CREATE TABLE | Database | Create tables | | CREATE MATERIALIZED VIEW | Database | Create materialized views | +| [CREATE PAYLOAD TRANSFORM](/docs/query/sql/create-payload-transform/) | Database | Create [payload transforms](/docs/ingestion/payload-transforms/) | +| CREATE TABLE | Database | Create tables | | DEDUP ENABLE | Database | Table | Enable deduplication | | DEDUP DISABLE | Database | Table | Disable deduplication | | DETACH PARTITION | Database | Table | Detach partitions | @@ -575,8 +576,9 @@ SELECT * FROM all_permissions(); | DROP COLUMN | Database | Table | Column | Drop columns | | DROP INDEX | Database | Table | Column | Drop indexes | | DROP PARTITION | Database | Table | Drop partitions | -| DROP TABLE | Database | Table | Drop tables | | DROP MATERIALIZED VIEW | Database | Table | Drop materialized views | +| [DROP PAYLOAD TRANSFORM](/docs/query/sql/drop-payload-transform/) | Database | Drop [payload transforms](/docs/ingestion/payload-transforms/) | +| DROP TABLE | Database | Table | Drop tables | | ENABLE STORAGE POLICY | Database | Table | Enable storage policies | | INSERT | Database | Table | Insert data | | REFRESH MATERIALIZED VIEW | Database | Table | Refresh materialized views | @@ -654,4 +656,4 @@ SELECT * FROM all_permissions(); - [SHOW GROUPS](/docs/query/sql/show/#show-groups) - [SHOW SERVICE ACCOUNT](/docs/query/sql/show/#show-service-account) - [SHOW SERVICE ACCOUNTS](/docs/query/sql/show/#show-service-accounts) -- [SHOW PERMISSIONS](/docs/query/sql/show/#show-permissions-for-current-user) +- [SHOW PERMISSIONS](/docs/query/sql/show/#show-permissions) diff --git a/documentation/sidebars.js b/documentation/sidebars.js index ea45c1353..c6c8d4d9e 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -153,6 +153,7 @@ module.exports = { ], }, "ingestion/import-csv", + "ingestion/payload-transforms", ], }, @@ -339,6 +340,7 @@ module.exports = { id: "query/sql/acl/create-service-account", type: "doc", }, + "query/sql/create-payload-transform", "query/sql/create-table", { id: "query/sql/acl/create-user", @@ -356,6 +358,7 @@ module.exports = { type: "doc", }, "query/sql/drop-mat-view", + "query/sql/drop-payload-transform", { id: "query/sql/acl/drop-service-account", type: "doc",