diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 92dd94bb7..4010bf03e 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -607,5 +607,5 @@ the replica's view was not fully up-to-date. materialized view (QuestDB Enterprise) - **Configuration** - - [Materialized views configs](/docs/configuration/overview/#materialized-views): + - [Materialized views configs](/docs/configuration/materialized-views/): Server configuration options for materialized views from `server.conf` diff --git a/documentation/concepts/write-ahead-log.md b/documentation/concepts/write-ahead-log.md index 9917ea07f..a99a236ce 100644 --- a/documentation/concepts/write-ahead-log.md +++ b/documentation/concepts/write-ahead-log.md @@ -121,7 +121,7 @@ enabling high write throughput. WAL behavior can be tuned via server configuration: - `cairo.wal.enabled.default` — WAL enabled by default (default: `true`) -- Parallel threads for WAL application — see [WAL configuration](/docs/configuration/overview/#wal-table-configurations) +- Parallel threads for WAL application — see [WAL configuration](/docs/configuration/wal/) To convert an existing table between WAL and non-WAL: diff --git a/documentation/configuration/cairo-engine.md b/documentation/configuration/cairo-engine.md new file mode 100644 index 000000000..b7f1ba9de --- /dev/null +++ b/documentation/configuration/cairo-engine.md @@ -0,0 +1,1006 @@ +--- +title: Cairo engine +description: Configuration settings for the Cairo SQL engine in QuestDB. +--- + +The Cairo engine is the core storage and query engine in QuestDB. These settings +control how data is written, read, indexed, and queried. Most defaults work well +for typical workloads, but tuning may be needed for high-throughput ingestion, +large analytical queries, or specific storage configurations. + +## General + +### cairo.date.locale + +- **Default**: `en` +- **Reloadable**: no + +The locale used to handle date types. + +### cairo.root + +- **Default**: `db` +- **Reloadable**: no + +Directory for storing database tables and metadata. This directory is relative +to the server root directory provided at startup. + +### cairo.system.table.prefix + +- **Default**: `sys.` +- **Reloadable**: no + +Prefix for QuestDB internal data storage tables. These tables are hidden from +the web console. + +### cairo.timestamp.locale + +- **Default**: `en` +- **Reloadable**: no + +The locale used to handle timestamp types. + +### config.reload.enabled + +- **Default**: `true` +- **Reloadable**: no + +When `false`, disables the `reload_config()` SQL function. + +### query.timeout.sec + +- **Default**: `60` +- **Reloadable**: no + +A global timeout in seconds for long-running queries. Per-query overrides are +available via the HTTP header +[`Statement-Timeout`](/docs/query/rest-api/#headers) or the Postgres +[`options`](/docs/query/pgwire/overview/) +connection property. + +## Commit and write behavior + +### cairo.commit.mode + +- **Default**: `nosync` +- **Reloadable**: no + +How changes are flushed to disk upon commit. Options: + +- `nosync`: no explicit flush (relies on OS page cache) +- `async`: flush call is scheduled but returns immediately +- `sync`: waits for flush on appended column files to complete + +### cairo.max.uncommitted.rows + +- **Default**: `500000` +- **Reloadable**: no + +Maximum number of uncommitted rows per table. When pending rows reach this +threshold, a commit is issued automatically. + +### cairo.wal.enabled.default + +- **Default**: `true` +- **Reloadable**: no + +Whether WAL tables are the default when using `CREATE TABLE`. + +## Writer settings + +### cairo.system.writer.data.append.page.size + +- **Default**: `256k` +- **Reloadable**: no + +mmap sliding page size that the table writer uses to append data for each +column, specifically for system tables. + +### cairo.writer.alter.busy.wait.timeout + +- **Default**: `500` +- **Reloadable**: no + +Maximum wait timeout in milliseconds for `ALTER TABLE` statements executed +via REST or PostgreSQL wire protocol when execution is asynchronous. + +### cairo.writer.command.queue.capacity + +- **Default**: `32` +- **Reloadable**: no + +Maximum capacity of the writer ALTER TABLE and replication command queue. +Shared between all tables. + +### cairo.writer.data.append.page.size + +- **Default**: `16M` +- **Reloadable**: no + +mmap sliding page size that the table writer uses to append data for each +column. + +### cairo.writer.data.index.key.append.page.size + +- **Default**: `512K` +- **Reloadable**: no + +mmap page size for appending index key data. Key data is the number of +distinct symbol values times 4 bytes. + +### cairo.writer.data.index.value.append.page.size + +- **Default**: `16M` +- **Reloadable**: no + +mmap page size for appending index value data. + +### cairo.writer.misc.append.page.size + +- **Default**: `4K` +- **Reloadable**: no + +mmap page size for mapping small files. Default is the OS page size (4KB +on Linux, 64KB on Windows, 16KB on macOS Apple Silicon). Overriding this +rounds up to the nearest multiple of the OS page size. + +### cairo.writer.tick.rows.count + +- **Default**: `1024` +- **Reloadable**: no + +How often the writer checks its command queue during busy writes, measured +in rows written. + +## Reader and writer pools + +### cairo.idle.check.interval + +- **Default**: `300000` +- **Reloadable**: no + +Frequency of the writer maintenance job in milliseconds. + +### cairo.inactive.reader.ttl + +- **Default**: `120000` +- **Reloadable**: no + +Time-to-live in milliseconds before closing inactive readers. + +### cairo.inactive.writer.ttl + +- **Default**: `600000` +- **Reloadable**: no + +Time-to-live in milliseconds before closing inactive writers. + +### cairo.reader.pool.max.segments + +- **Default**: `10` +- **Reloadable**: no + +Number of segments in the table reader pool. Each segment holds up to 32 +readers. + +### cairo.wal.inactive.writer.ttl + +- **Default**: `120000` +- **Reloadable**: no + +Time-to-live in milliseconds before closing inactive WAL writers. + +### cairo.wal.writer.pool.max.segments + +- **Default**: `10` +- **Reloadable**: no + +Number of segments in the WAL writer pool. Each segment holds up to 32 +writers. + +## Out-of-order (O3) ingestion + +These settings control the in-memory buffer used for out-of-order data +ingestion. The buffer size is determined dynamically based on the shape of +incoming data, within the bounds set here. + +### cairo.o3.column.memory.size + +- **Default**: `256k` +- **Reloadable**: no + +Memory page size per column for O3 operations. O3 uses 2x this value per +column (so the default effective size is 512KB per column). + +### cairo.o3.last.partition.max.splits + +- **Default**: `20` +- **Reloadable**: no + +Number of partition pieces allowed before the last piece is merged back into +the physical partition. + +### cairo.o3.max.lag + +- **Default**: `10 minutes` +- **Reloadable**: no + +Upper limit for the in-memory O3 buffer size, in milliseconds. + +### cairo.o3.min.lag + +- **Default**: `1 second` +- **Reloadable**: no + +Lower limit for the in-memory O3 buffer size, in milliseconds. + +### cairo.o3.partition.purge.list.initial.capacity + +- **Default**: `1` +- **Reloadable**: no + +Initial allocation for the partition purge job. Extended automatically at +runtime. + +### cairo.o3.partition.split.min.size + +- **Default**: `50MB` +- **Reloadable**: no + +Estimated partition size on disk. This is one of the conditions that triggers +[auto-partitioning](/docs/getting-started/capacity-planning/). + +## Symbol and indexing + +### cairo.default.symbol.cache.flag + +- **Default**: `true` +- **Reloadable**: no + +When `true`, symbol values are cached on the Java heap instead of being +looked up in database files. + +### cairo.default.symbol.capacity + +- **Default**: `256` +- **Reloadable**: no + +Approximate capacity for `SYMBOL` columns. Should equal the number of unique +symbol values stored in the table. Getting this value significantly wrong +causes performance degradation. Must be a power of 2. + +### cairo.index.value.block.size + +- **Default**: `256` +- **Reloadable**: no + +Approximation of the number of rows for a single index key. Must be a power +of 2. + +### cairo.parallel.index.threshold + +- **Default**: `100000` +- **Reloadable**: no + +Minimum number of rows before parallel indexation is used. + +### cairo.parallel.indexing.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables parallel indexation. Works in conjunction with +`cairo.parallel.index.threshold`. + +### cairo.spin.lock.timeout + +- **Default**: `1000` +- **Reloadable**: no + +Timeout in milliseconds when attempting to acquire BitmapIndexReaders. + +### cairo.work.steal.timeout.nanos + +- **Default**: `10000` +- **Reloadable**: no + +Latch await timeout in nanoseconds for stealing indexing work from other +threads. + +## File operations + +### cairo.file.descriptor.cache.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables or disables the file descriptor cache. + +### cairo.file.operation.retry.count + +- **Default**: `30` +- **Reloadable**: no + +Number of attempts to open files. + +### cairo.max.swap.file.count + +- **Default**: `30` +- **Reloadable**: no + +Number of attempts to open swap files. + +### cairo.mkdir.mode + +- **Default**: `509` +- **Reloadable**: no + +File permission mode for new directories. + +### cairo.volumes + +- **Default**: none +- **Reloadable**: no + +A comma-separated list of `alias -> root-path` pairs defining allowed volumes +for use in +[CREATE TABLE IN VOLUME](/docs/query/sql/create-table/#table-target-volume) +statements. + +## Snapshot settings + +### cairo.snapshot.instance.id + +- **Default**: empty string +- **Reloadable**: no + +Instance ID to include in disk snapshots. + +### cairo.snapshot.recovery.enabled + +- **Default**: `true` +- **Reloadable**: no + +When `false`, disables snapshot recovery on database start. + +## SQL compiler pools + +Internal object pool sizes for the SQL compiler. Increasing these reduces +garbage collection pressure at the cost of higher baseline memory usage. + +### cairo.character.store.capacity + +- **Default**: `1024` +- **Reloadable**: no + +Size of the CharacterStore. + +### cairo.character.store.sequence.pool.capacity + +- **Default**: `64` +- **Reloadable**: no + +Size of the CharacterSequence pool. + +### cairo.column.pool.capacity + +- **Default**: `4096` +- **Reloadable**: no + +Size of the Column pool in the SQL compiler. + +### cairo.expression.pool.capacity + +- **Default**: `8192` +- **Reloadable**: no + +Size of the ExpressionNode pool in the SQL compiler. + +### cairo.lexer.pool.capacity + +- **Default**: `2048` +- **Reloadable**: no + +Size of the FloatingSequence pool in GenericLexer. + +### cairo.model.pool.capacity + +- **Default**: `1024` +- **Reloadable**: no + +Size of the QueryModel pool in the SQL compiler. + +### cairo.sql.analytic.column.pool.capacity + +- **Default**: `64` +- **Reloadable**: no + +Size of the AnalyticColumn pool in the SQL parser. + +### cairo.sql.column.cast.model.pool.capacity + +- **Default**: `16` +- **Reloadable**: no + +Size of the CreateTableModel pool in the SQL parser. + +### cairo.sql.copy.model.pool.capacity + +- **Default**: `32` +- **Reloadable**: no + +Size of the CopyModel pool in the SQL parser. + +### cairo.sql.insert.model.pool.capacity + +- **Default**: `64` +- **Reloadable**: no + +Size of the InsertModel pool in the SQL parser. + +### cairo.sql.join.context.pool.capacity + +- **Default**: `64` +- **Reloadable**: no + +Size of the JoinContext pool in the SQL compiler. + +### cairo.sql.query.registry.pool.size + +- **Default**: auto +- **Reloadable**: no + +Pre-sizes the internal data structure that stores active query executions. +Automatically chosen based on the number of shared worker threads. + +### cairo.sql.rename.table.model.pool.capacity + +- **Default**: `16` +- **Reloadable**: no + +Size of the RenameTableModel pool in the SQL parser. + +### cairo.sql.with.clause.model.pool.capacity + +- **Default**: `128` +- **Reloadable**: no + +Size of the WithClauseModel pool in the SQL parser. + +## SQL map settings + +These settings control the hash maps used internally for GROUP BY, JOIN, +and other operations that build intermediate result sets. + +### cairo.compact.map.load.factor + +- **Default**: `0.7` +- **Reloadable**: no + +Load factor for CompactMaps. + +### cairo.default.map.type + +- **Default**: `fast` +- **Reloadable**: no + +Type of map used. Options: `fast` (speed at the expense of storage) or +`compact`. + +### cairo.fast.map.load.factor + +- **Default**: `0.5` +- **Reloadable**: no + +Load factor for FastMaps. + +### cairo.sql.map.key.capacity + +- **Default**: `2M` +- **Reloadable**: no + +Key capacity in FastMap and CompactMap. + +### cairo.sql.map.max.pages + +- **Default**: `2^31` +- **Reloadable**: no + +Maximum memory pages for CompactMap. + +### cairo.sql.map.max.resizes + +- **Default**: `2^31` +- **Reloadable**: no + +Maximum number of map resizes in FastMap and CompactMap before a resource +limit exception is thrown. Each resize doubles the previous size. + +### cairo.sql.map.page.size + +- **Default**: `4m` +- **Reloadable**: no + +Memory page size for FastMap and CompactMap. + +### cairo.sql.unordered.map.max.entry.size + +- **Default**: `24` +- **Reloadable**: no + +Threshold in bytes for switching from a single-buffer hash table (unordered) +to a hash table with a separate heap for entries (ordered). + +## SQL sort and join + +Memory settings for sort operations and hash joins. + +### cairo.sql.hash.join.light.value.max.pages + +- **Default**: `2^31` +- **Reloadable**: no + +Maximum pages of the slave chain in light hash joins. + +### cairo.sql.hash.join.light.value.page.size + +- **Default**: `1048576` +- **Reloadable**: no + +Memory page size of the slave chain in light hash joins. + +### cairo.sql.hash.join.value.max.pages + +- **Default**: `2^31` +- **Reloadable**: no + +Maximum pages of the slave chain in full hash joins. + +### cairo.sql.hash.join.value.page.size + +- **Default**: `16777216` +- **Reloadable**: no + +Memory page size of the slave chain in full hash joins. + +### cairo.sql.join.metadata.max.resizes + +- **Default**: `2^31` +- **Reloadable**: no + +Maximum number of map resizes in JoinMetadata before a resource limit +exception is thrown. Each resize doubles the previous size. + +### cairo.sql.join.metadata.page.size + +- **Default**: `16384` +- **Reloadable**: no + +Memory page size for the JoinMetadata file. + +### cairo.sql.latest.by.row.count + +- **Default**: `1000` +- **Reloadable**: no + +Number of rows for LATEST BY. + +### cairo.sql.sort.key.max.pages + +- **Default**: `2^31` +- **Reloadable**: no + +Maximum pages for storing keys in LongTreeChain before a resource limit +exception is thrown. + +### cairo.sql.sort.key.page.size + +- **Default**: `4M` +- **Reloadable**: no + +Memory page size for storing keys in LongTreeChain. + +### cairo.sql.sort.light.value.max.pages + +- **Default**: `2^31` +- **Reloadable**: no + +Maximum pages for storing values in LongTreeChain. + +### cairo.sql.sort.light.value.page.size + +- **Default**: `1048576` +- **Reloadable**: no + +Memory page size for storing values in LongTreeChain. + +### cairo.sql.sort.value.max.pages + +- **Default**: `2^31` +- **Reloadable**: no + +Maximum pages for storing values in SortedRecordCursorFactory. + +### cairo.sql.sort.value.page.size + +- **Default**: `16777216` +- **Reloadable**: no + +Memory page size for storing values in SortedRecordCursorFactory. + +## Page frames + +### cairo.sql.page.frame.max.rows + +- **Default**: `1000000` +- **Reloadable**: no + +Maximum number of rows in page frames used in SQL queries. + +### cairo.sql.page.frame.min.rows + +- **Default**: `1000` +- **Reloadable**: no + +Minimum number of rows in page frames used in SQL queries. + +## JIT compilation + +These settings control Just-In-Time compilation of SQL filter expressions. +JIT compilation can significantly speed up queries with simple filter +predicates. + +### cairo.sql.jit.bind.vars.memory.max.pages + +- **Default**: `8` +- **Reloadable**: no + +Maximum memory pages for storing bind variable values in JIT compiled filters. + +### cairo.sql.jit.bind.vars.memory.page.size + +- **Default**: `4K` +- **Reloadable**: no + +Memory page size for storing bind variable values in JIT compiled filters. + +### cairo.sql.jit.debug.enabled + +- **Default**: `false` +- **Reloadable**: no + +When enabled, prints generated assembly to `stdout`. + +### cairo.sql.jit.ir.memory.max.pages + +- **Default**: `8` +- **Reloadable**: no + +Maximum memory pages for storing intermediate representation during JIT +compilation. + +### cairo.sql.jit.ir.memory.page.size + +- **Default**: `8K` +- **Reloadable**: no + +Memory page size for storing intermediate representation during JIT +compilation. + +### cairo.sql.jit.max.in.list.size.threshold + +- **Default**: `10` +- **Reloadable**: no + +If an `IN` predicate list exceeds this length, JIT compilation is skipped for +that query. + +### cairo.sql.jit.mode + +- **Default**: `on` +- **Reloadable**: no + +JIT compilation for SQL queries. Set to `off` to disable. + +### cairo.sql.jit.page.address.cache.threshold + +- **Default**: `1M` +- **Reloadable**: no + +Minimum cache size to shrink the page address cache after query execution. + +## GROUP BY + +### cairo.sql.groupby.allocator.default.chunk.size + +- **Default**: `128k` +- **Reloadable**: no + +Default size for memory buffers in the GROUP BY function native memory +allocator. + +### cairo.sql.groupby.allocator.max.chunk.size + +- **Default**: `4gb` +- **Reloadable**: no + +Maximum allowed native memory allocation for GROUP BY functions. + +### cairo.sql.parallel.groupby.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables parallel GROUP BY execution. Requires at least 4 shared worker +threads. + +### cairo.sql.parallel.groupby.merge.shard.queue.capacity + +- **Default**: auto +- **Reloadable**: no + +Merge queue capacity for parallel GROUP BY. Used for parallel tasks that +merge shard hash tables. + +### cairo.sql.parallel.groupby.sharding.threshold + +- **Default**: `100000` +- **Reloadable**: no + +Row count threshold for parallel GROUP BY to shard the hash table holding +the aggregates. + +## SAMPLE BY + +### cairo.sql.sampleby.default.alignment.calendar + +- **Default**: `0` +- **Reloadable**: no + +Default SAMPLE BY alignment behavior. `true` corresponds to ALIGN TO +CALENDAR, `false` corresponds to ALIGN TO FIRST OBSERVATION. + +### cairo.sql.sampleby.page.size + +- **Default**: `0` +- **Reloadable**: no + +SAMPLE BY index query page size (maximum values returned in a single scan). +`0` means to use the symbol block capacity. + +## Window functions + +### cairo.sql.analytic.initial.range.buffer.size + +- **Default**: `32` +- **Reloadable**: no + +Window function buffer size in record counts. Pre-sizes the buffer for +every window function execution. + +### cairo.sql.window.max.recursion + +- **Default**: `128` +- **Reloadable**: no + +Prevents stack overflow errors when evaluating complex nested SQL. The value +is the approximate number of nested SELECT clauses allowed. + +## Batch operations + +### cairo.create.as.select.retry.count + +- **Default**: `5` +- **Reloadable**: no + +Number of times table creation or insertion will be attempted. + +### cairo.sql.copy.buffer.size + +- **Default**: `2M` +- **Reloadable**: no + +Size of buffer used when copying tables. + +### cairo.sql.create.table.model.batch.size + +- **Default**: `1000000` +- **Reloadable**: no + +Batch size for non-atomic CREATE AS SELECT statements. + +### cairo.sql.insert.model.batch.size + +- **Default**: `1000000` +- **Reloadable**: no + +Batch size for non-atomic INSERT INTO SELECT statements. + +## Type casting and formatting + +### cairo.sql.copy.formats.file + +- **Default**: `/text_loader.json` +- **Reloadable**: no + +Name of the file containing user-defined date and timestamp formats. + +### cairo.sql.double.cast.scale + +- **Default**: `12` +- **Reloadable**: no + +Maximum number of decimal places for types cast as doubles. + +### cairo.sql.float.cast.scale + +- **Default**: `4` +- **Reloadable**: no + +Maximum number of decimal places for types cast as floats. + +## JSON UNNEST + +### cairo.json.unnest.max.value.size + +- **Default**: `4096` +- **Reloadable**: no + +Maximum byte size of a single VARCHAR or TIMESTAMP field value extracted +during JSON [UNNEST](/docs/query/sql/unnest/). Numeric types (DOUBLE, LONG, +INT, SHORT, BOOLEAN) are unaffected. Each VARCHAR/TIMESTAMP column allocates +`2 * maxValueSize` bytes of native memory per active UNNEST cursor, so +increase with care. + +## Random function memory + +### cairo.rnd.memory.max.pages + +- **Default**: `128` +- **Reloadable**: no + +Maximum number of pages for memory used by `rnd_` functions. Supports +`rnd_str()` and `rnd_symbol()`. + +### cairo.rnd.memory.page.size + +- **Default**: `8K` +- **Reloadable**: no + +Memory page size used by `rnd_` functions. Supports `rnd_str()` and +`rnd_symbol()`. + +## Parquet encoding + +Settings for Parquet-encoded partitions, used by storage policies and +COPY TO exports. + +### cairo.partition.encoder.parquet.bloom.filter.fpp + +- **Default**: `0.01` +- **Reloadable**: no + +Default bloom filter false positive probability for in-place partition +encoding. Lower values produce larger but more accurate filters. Range: +0.0 to 1.0. + +### cairo.partition.encoder.parquet.compression.codec + +- **Default**: `ZSTD` +- **Reloadable**: no + +Default compression codec for parquet-encoded partitions. Alternatives +include `LZ4_RAW` and `SNAPPY`. + +### cairo.partition.encoder.parquet.compression.level + +- **Default**: `9` (ZSTD), `0` (otherwise) +- **Reloadable**: no + +Default compression level for parquet-encoded partitions. Dependent on +the underlying compression codec. + +### cairo.partition.encoder.parquet.data.page.size + +- **Default**: `1048576` +- **Reloadable**: no + +Default page size for parquet-encoded partitions. + +### cairo.partition.encoder.parquet.min.compression.ratio + +- **Default**: `1.2` +- **Reloadable**: no + +Minimum compression ratio (uncompressed / compressed) for Parquet pages. +When a compressed page does not meet this threshold, it is stored +uncompressed instead. A value of `0.0` disables the check. + +### cairo.partition.encoder.parquet.raw.array.encoding.enabled + +- **Default**: `false` +- **Reloadable**: no + +When `true`, exports arrays in QuestDB-native binary format (less +compatible). When `false`, uses Parquet-native format (more compatible). + +### cairo.partition.encoder.parquet.row.group.size + +- **Default**: `100000` +- **Reloadable**: no + +Default row-group size for parquet-encoded partitions. + +### cairo.partition.encoder.parquet.statistics.enabled + +- **Default**: `true` +- **Reloadable**: no + +Controls whether statistics are included in parquet-encoded partitions. + +### cairo.partition.encoder.parquet.version + +- **Default**: `1` +- **Reloadable**: no + +Output Parquet version for parquet-encoded partitions. Can be `1` or `2`. + +### cairo.sql.parquet.row.group.pruning.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables row group pruning for queries on Parquet partitions. When enabled, +QuestDB uses min/max statistics, bloom filters, and null counts to skip row +groups that cannot match the query filter. + +## Column purge + +These settings control the background job that cleans up stale column files +after UPDATE statements. + +### cairo.sql.column.purge.queue.capacity + +- **Default**: `128` +- **Reloadable**: no + +Purge column version job queue capacity. Increase if column versions are not +automatically cleaned up after UPDATE statements. Reduce to decrease initial +memory footprint. + +### cairo.sql.column.purge.retry.delay + +- **Default**: `10000` +- **Reloadable**: no + +Initial delay in microseconds before re-trying purge of stale column files. + +### cairo.sql.column.purge.retry.delay.limit + +- **Default**: `60000000` +- **Reloadable**: no + +Delay limit in microseconds. Once reached, the retry delay remains constant. + +### cairo.sql.column.purge.retry.delay.multiplier + +- **Default**: `10.0` +- **Reloadable**: no + +Multiplier used to increase retry delay with each iteration. + +### cairo.sql.column.purge.retry.limit.days + +- **Default**: `31` +- **Reloadable**: no + +Number of days the purge system will continue retrying before giving up on +stale column files. + +### cairo.sql.column.purge.task.pool.capacity + +- **Default**: `256` +- **Reloadable**: no + +Column version task object pool capacity. Increase to reduce GC, reduce to +decrease memory footprint. diff --git a/documentation/configuration/command-line-options.md b/documentation/configuration/command-line-options.md deleted file mode 100644 index afc5d55d2..000000000 --- a/documentation/configuration/command-line-options.md +++ /dev/null @@ -1,291 +0,0 @@ ---- -title: Command-line options -description: Command-line options reference documentation. ---- - -QuestDB may be started, stopped and passed configuration options from the -command line. On Windows, the QuestDB server can also start an -[interactive session](#interactive-session-windows). - -## Options - -The following sections describe the options that may be passed to QuestDB when -starting the server from the command line. - - - -import Tabs from "@theme/Tabs" -import TabItem from "@theme/TabItem" - - - - - - - -```shell -./questdb.sh [start|stop|status] [-d dir] [-f] [-n] [-t tag] -``` - - - - - -```shell -questdb [start|stop|status] [-d dir] [-f] [-n] [-t tag] -``` - - - - - -```shell -questdb.exe [start|stop|status|install|remove] \ - [-d dir] [-f] [-j JAVA_HOME] [-t tag] -``` - - - - - -### Start - -`start` - starts QuestDB as a service. - -| Option | Description | -| ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `-d` | Expects a `dir` directory value which is a folder that will be used as QuestDB's root directory. For more information and the default values, see the [default root](#default-root-directory-1) section below. | -| `-t` | Expects a `tag` string value which will be as a tag for the service. This option allows users to run several QuestDB services and manage them separately. If this option is omitted, the default tag will be `questdb`. | -| `-f` | Force re-deploying the [Web Console](/docs/getting-started/web-console/overview/). Without this option, the [Web Console](/docs/getting-started/web-console/overview/) is cached and deployed only when missing. | -| `-n` | Do not respond to the HUP signal. This keeps QuestDB alive after you close the terminal window where you started it. | -| `-j` | **Windows only!** This option allows to specify a path to `JAVA_HOME`. | - -:::note - -- When running multiple QuestDB services, a tag must be used to disambiguate - between services for `start` and `stop` commands. There will be conflicting - ports and root directories if only the tag flag is specified when starting - multiple services. Each new service should have its own config file or should - be started with separate port and root directory options. - -- When running QuestDB as Windows service you can check status in both: - - Windows Event Viewer - look for events with "QuestDB" source in Windows Logs - | Application . - - service log file - `$dataDir\log\service-%Y-%m-%dT%H-%M-%S.txt` (default is - `C:\Windows\System32\qdbroot\log\service-%Y-%m-%dT%H-%M-%S.txt` ) - -::: - - - - - - - - - - -```shell -./questdb.sh start [-d dir] [-f] [-n] [-t tag] -``` - - - - - -```shell -questdb start [-d dir] [-f] [-n] [-t tag] -``` - - - - - -```shell -questdb.exe start [-d dir] [-f] [-j JAVA_HOME] [-t tag] -``` - - - - - -#### Default root directory - -By default, QuestDB's [root directory](/docs/concepts/deep-dive/root-directory-structure/) -will be the following: - - - - - - - - - -```shell -$HOME/.questdb -``` - - - - - -Path on Macs with Apple Silicon (M1 or M2) chip: - -```shell -/opt/homebrew/var/questdb -``` - -Path on Macs with Intel chip: - -```shell -/usr/local/var/questdb -``` - - - - - -```shell -C:\Windows\System32\qdbroot -``` - - - - - -### Stop - -`stop` - stops a service. - -| Option | Description | -| ------ | ------------------------------------------------------------------------------------------------------------------ | -| `-t` | Expects a `tag` string value which to stop a service by tag. If this is omitted, the default tag will be `questdb` | - - - - - - - - - -```shell -./questdb.sh stop -``` - - - - - -```shell -questdb stop -``` - - - - - -```shell -questdb.exe stop -``` - - - - - -### Status - -`status` - shows the status for a service. - -| Option | Description | -| ------ | -------------------------------------------------------------------------------------------------------------- | -| `-t` | Expects a `tag` string value which to stop a service by tag. If this is omitted, the default will be `questdb` | - - - - - - - - - -```shell -./questdb.sh status -``` - - - - - -```shell -questdb status -``` - - - - - -```shell -questdb.exe status -``` - - - - - -### Install (Windows) - -`install` - installs the Windows QuestDB service. The service will start -automatically at startup. - -```shell -questdb.exe install -``` - -### Remove (Windows) - -`remove` - removes the Windows QuestDB service. It will no longer start at -startup. - -```shell -questdb.exe remove -``` - -## Interactive session (Windows) - -You can start QuestDB interactively by running `questdb.exe`. This will launch -QuestDB interactively in the active `Shell` window. QuestDB will be stopped when -the Shell is closed. - -### Default root directory - -When started interactively, QuestDB's root directory defaults to the `current` -directory. - -### Stop - -To stop, press Ctrl+C in the terminal or close it -directly. diff --git a/documentation/configuration/configuration-utils/_cairo.config.json b/documentation/configuration/configuration-utils/_cairo.config.json deleted file mode 100644 index bed859b73..000000000 --- a/documentation/configuration/configuration-utils/_cairo.config.json +++ /dev/null @@ -1,494 +0,0 @@ -{ - "config.reload.enabled": { - "default": "true", - "description": "When `false`, disables reload_config() SQL function." - }, - "query.timeout.sec": { - "default": "60", - "description": "A global timeout (in seconds) for long-running queries. Timeout for each query can override the default by setting HTTP header [`Statement-Timeout`](/docs/query/rest-api/#headers) or Postgres [`options`](/docs/query/pgwire/overview/#list-of-supported-connection-properties)." - }, - "cairo.max.uncommitted.rows": { - "default": "500000", - "description": "Maximum number of uncommitted rows per table, when the number of pending rows reaches this parameter on a table, a commit will be issued." - }, - "cairo.o3.max.lag": { - "default": "10 minutes", - "description": "The maximum size of in-memory buffer in milliseconds. The buffer is allocated dynamically through analyzing the shape of the incoming data, and `o3MaxLag` is the upper limit." - }, - "cairo.o3.min.lag": { - "default": "1 second", - "description": "The minimum size of in-memory buffer in milliseconds. The buffer is allocated dynamically through analyzing the shape of the incoming data, and `o3MinLag` is the lower limit." - }, - "cairo.snapshot.instance.id": { - "default": "empty string", - "description": "Instance id to be included into disk snapshots." - }, - "cairo.snapshot.recovery.enabled": { - "default": "true", - "description": "When `false`, disables snapshot recovery on database start." - }, - "cairo.root": { - "default": "db", - "description": "Directory for storing db tables and metadata. This directory is inside the server root directory provided at startup." - }, - "cairo.commit.mode": { - "default": "nosync", - "description": "How changes to table are flushed to disk upon commit. Choices: `nosync`, `async` (flush call schedules update, returns immediately), `sync` (waits for flush on the appended column files to complete)." - }, - "cairo.rnd.memory.max.pages": { - "default": "128", - "description": "Sets the max number of pages for memory used by `rnd_` functions. Supports `rnd_str()` and `rnd_symbol()`." - }, - "cairo.rnd.memory.page.size": { - "default": "8K", - "description": "Sets the memory page size used by `rnd_` functions. Supports `rnd_str()` and `rnd_symbol()`." - }, - "cairo.create.as.select.retry.count": { - "default": "5", - "description": "Number of types table creation or insertion will be attempted." - }, - "cairo.default.map.type": { - "default": "fast", - "description": "Type of map used. Options: `fast` (speed at the expense of storage), `compact`." - }, - "cairo.default.symbol.cache.flag": { - "default": "true", - "description": "When `true`, symbol values will be cached on Java heap instead of being looked up in the database files." - }, - "cairo.default.symbol.capacity": { - "default": "256", - "description": "Specifies approximate capacity for `SYMBOL` columns. It should be equal to number of unique symbol values stored in the table and getting this value badly wrong will cause performance degradation. Must be power of 2." - }, - "cairo.file.operation.retry.count": { - "default": "30", - "description": "Number of attempts to open files." - }, - "cairo.idle.check.interval": { - "default": "300000", - "description": "Frequency of writer maintenance job in milliseconds." - }, - "cairo.inactive.reader.ttl": { - "default": "120000", - "description": "TTL (Time-To-Live) to close inactive readers in milliseconds." - }, - "cairo.wal.inactive.writer.ttl": { - "default": "120000", - "description": "TTL (Time-To-Live) to close inactive WAL writers in milliseconds." - }, - "cairo.inactive.writer.ttl": { - "default": "600000", - "description": "TTL (Time-To-Live) to close inactive writers in milliseconds." - }, - "cairo.index.value.block.size": { - "default": "256", - "description": "Approximation of number of rows for a single index key, must be power of 2." - }, - "cairo.max.swap.file.count": { - "default": "30", - "description": "Number of attempts to open swap files." - }, - "cairo.mkdir.mode": { - "default": "509", - "description": "File permission mode for new directories." - }, - "cairo.parallel.index.threshold": { - "default": "100000", - "description": "Minimum number of rows before allowing use of parallel indexation." - }, - "cairo.reader.pool.max.segments": { - "default": "10", - "description": "Number of segments in the table reader pool. Each segment holds up to 32 readers." - }, - "cairo.wal.writer.pool.max.segments": { - "default": "10", - "description": "Number of segments in the WAL writer pool. Each segment holds up to 32 writers." - }, - "cairo.spin.lock.timeout": { - "default": "1000", - "description": "Timeout when attempting to get BitmapIndexReaders in millisecond." - }, - "cairo.character.store.capacity": { - "default": "1024", - "description": "Size of the CharacterStore." - }, - "cairo.character.store.sequence.pool.capacity": { - "default": "64", - "description": "Size of the CharacterSequence pool." - }, - "cairo.column.pool.capacity": { - "default": "4096", - "description": "Size of the Column pool in the SqlCompiler." - }, - "cairo.compact.map.load.factor": { - "default": "0.7", - "description": "Load factor for CompactMaps." - }, - "cairo.expression.pool.capacity": { - "default": "8192", - "description": "Size of the ExpressionNode pool in SqlCompiler." - }, - "cairo.fast.map.load.factor": { - "default": "0.5", - "description": "Load factor for all FastMaps." - }, - "cairo.sql.join.context.pool.capacity": { - "default": "64", - "description": "Size of the JoinContext pool in SqlCompiler." - }, - "cairo.lexer.pool.capacity": { - "default": "2048", - "description": "Size of FloatingSequence pool in GenericLexer." - }, - "cairo.sql.map.key.capacity": { - "default": "2M", - "description": "Key capacity in FastMap and CompactMap." - }, - "cairo.sql.map.max.resizes": { - "default": "2^31", - "description": "Number of map resizes in FastMap and CompactMap before a resource limit exception is thrown, each resize doubles the previous size." - }, - "cairo.sql.map.page.size": { - "default": "4m", - "description": "Memory page size for FastMap and CompactMap." - }, - "cairo.sql.map.max.pages": { - "default": "2^31", - "description": "Memory max pages for CompactMap." - }, - "cairo.model.pool.capacity": { - "default": "1024", - "description": "Size of the QueryModel pool in the SqlCompiler." - }, - "cairo.sql.sort.key.page.size": { - "default": "4M", - "description": "Memory page size for storing keys in LongTreeChain." - }, - "cairo.sql.sort.key.max.pages": { - "default": "2^31", - "description": "Max number of pages for storing keys in LongTreeChain before a resource limit exception is thrown." - }, - "cairo.sql.sort.light.value.page.size": { - "default": "1048576", - "description": "Memory page size for storing values in LongTreeChain." - }, - "cairo.sql.sort.light.value.max.pages": { - "default": "2^31", - "description": "Max pages for storing values in LongTreeChain." - }, - "cairo.sql.hash.join.value.page.size": { - "default": "16777216", - "description": "Memory page size of the slave chain in full hash joins." - }, - "cairo.sql.hash.join.value.max.pages": { - "default": "2^31", - "description": "Max pages of the slave chain in full hash joins." - }, - "cairo.sql.latest.by.row.count": { - "default": "1000", - "description": "Number of rows for LATEST BY." - }, - "cairo.sql.hash.join.light.value.page.size": { - "default": "1048576", - "description": "Memory page size of the slave chain in light hash joins." - }, - "cairo.sql.hash.join.light.value.max.pages": { - "default": "2^31", - "description": "Max pages of the slave chain in light hash joins." - }, - "cairo.sql.sort.value.page.size": { - "default": "16777216", - "description": "Memory page size of file storing values in SortedRecordCursorFactory." - }, - "cairo.sql.sort.value.max.pages": { - "default": "2^31", - "description": "Max pages of file storing values in SortedRecordCursorFactory." - }, - "cairo.work.steal.timeout.nanos": { - "default": "10000", - "description": "Latch await timeout in nanos for stealing indexing work from other threads." - }, - "cairo.parallel.indexing.enabled": { - "default": "true", - "description": "Allows parallel indexation. Works in conjunction with cairo.parallel.index.threshold." - }, - "cairo.sql.join.metadata.page.size": { - "default": "16384", - "description": "Memory page size for JoinMetadata file." - }, - "cairo.sql.join.metadata.max.resizes": { - "default": "2^31", - "description": "Number of map resizes in JoinMetadata before a resource limit exception is thrown, each resize doubles the previous size." - }, - "cairo.sql.analytic.column.pool.capacity": { - "default": "64", - "description": "Size of AnalyticColumn pool in SqlParser." - }, - "cairo.sql.create.table.model.batch.size": { - "default": "1000000", - "description": "Batch size for non-atomic CREATE AS SELECT statements." - }, - "cairo.sql.column.cast.model.pool.capacity": { - "default": "16", - "description": "Size of CreateTableModel pool in SqlParser." - }, - "cairo.sql.rename.table.model.pool.capacity": { - "default": "16", - "description": "Size of RenameTableModel pool in SqlParser." - }, - "cairo.sql.with.clause.model.pool.capacity": { - "default": "128", - "description": "Size of WithClauseModel pool in SqlParser." - }, - "cairo.sql.insert.model.pool.capacity": { - "default": "64", - "description": "Size of InsertModel pool in SqlParser." - }, - "cairo.sql.insert.model.batch.size": { - "default": "1000000", - "description": "Batch size for non-atomic INSERT INTO SELECT statements." - }, - "cairo.sql.copy.model.pool.capacity": { - "default": "32", - "description": "Size of CopyModel pool in SqlParser." - }, - "cairo.sql.copy.buffer.size": { - "default": "2M", - "description": "Size of buffer used when copying tables." - }, - "cairo.sql.double.cast.scale": { - "default": "12", - "description": "Maximum number of decimal places that types cast as doubles have." - }, - "cairo.sql.float.cast.scale": { - "default": "4", - "description": "Maximum number of decimal places that types cast as floats have." - }, - "cairo.sql.copy.formats.file": { - "default": "/text_loader.json", - "description": "Name of file with user's set of date and timestamp formats." - }, - "cairo.sql.jit.mode": { - "default": "on", - "description": "JIT compilation for SQL queries. May be disabled by setting this value to `off`." - }, - "cairo.sql.jit.debug.enabled": { - "default": "false", - "description": "Sets debug flag for JIT compilation. When enabled, assembly will be printed into `stdout`." - }, - "cairo.sql.jit.max.in.list.size.threshold": { - "default": "10", - "description": "Controls whether or not JIT compilation will be used for a query that uses the IN predicate. If the IN list is longer than this threshold, JIT compilation will be cancelled." - }, - "cairo.sql.jit.bind.vars.memory.page.size": { - "default": "4K", - "description": "Sets the memory page size for storing bind variable values for JIT compiled filter." - }, - "cairo.sql.jit.bind.vars.memory.max.pages": { - "default": "8", - "description": "Sets the max memory pages for storing bind variable values for JIT compiled filter." - }, - "cairo.sql.jit.page.address.cache.threshold": { - "default": "1M", - "description": "Sets minimum cache size to shrink page address cache after query execution." - }, - "cairo.sql.jit.ir.memory.page.size": { - "default": "8K", - "description": "Sets the memory page size for storing IR for JIT compilation." - }, - "cairo.sql.jit.ir.memory.max.pages": { - "default": "8", - "description": "Sets max memory pages for storing IR for JIT compilation." - }, - "cairo.json.unnest.max.value.size": { - "default": "4096", - "description": "Maximum byte size of a single VARCHAR or TIMESTAMP field value extracted during JSON [UNNEST](/docs/query/sql/unnest/). Numeric types (DOUBLE, LONG, INT, SHORT, BOOLEAN) are unaffected. Each VARCHAR/TIMESTAMP column allocates `2 * maxValueSize` bytes of native memory per active UNNEST cursor, so increase with care." - }, - "cairo.sql.page.frame.min.rows": { - "default": "1000", - "description": "Sets the minimum number of rows in page frames used in SQL queries." - }, - "cairo.sql.page.frame.max.rows": { - "default": "1000000", - "description": "Sets the maximum number of rows in page frames used in SQL. queries" - }, - "cairo.sql.sampleby.page.size": { - "default": "0", - "description": "SampleBy index query page size. Max values returned in single scan. 0 is default, and it means to use symbol block capacity." - }, - "cairo.sql.sampleby.default.alignment.calendar": { - "default": "0", - "description": "SampleBy default alignment behaviour. true corresponds to ALIGN TO CALENDAR, false corresponds to ALIGN TO FIRST OBSERVATION." - }, - "cairo.date.locale": { - "default": "en", - "description": "The locale to handle date types." - }, - "cairo.timestamp.locale": { - "default": "en", - "description": "The locale to handle timestamp types." - }, - "cairo.o3.column.memory.size": { - "default": "256k", - "description": "Memory page size per column for O3 operations. Please be aware O3 will use 2x of the set value per column (therefore a default of 2x256kb)." - }, - "cairo.writer.data.append.page.size": { - "default": "16M", - "description": "mmap sliding page size that table writer uses to append data for each column." - }, - "cairo.writer.data.index.key.append.page.size": { - "default": "512K", - "description": "mmap page size for appending index key data; key data is number of distinct symbol values times 4 bytes." - }, - "cairo.writer.data.index.value.append.page.size": { - "default": "16M", - "description": "mmap page size for appending value data." - }, - "cairo.writer.misc.append.page.size": { - "default": "4K", - "description": "mmap page size for mapping small files, default value is OS page size (4k Linux, 64K windows, 16k OSX M1). Overriding this rounds to the nearest (greater) multiple of the OS page size." - }, - "cairo.writer.command.queue.capacity": { - "default": "32", - "description": "Maximum writer ALTER TABLE and replication command capacity. Shared between all the tables." - }, - "cairo.writer.tick.rows.count": { - "default": "1024", - "description": "Row count to check writer command queue after on busy writing, e.g. tick after X rows written." - }, - "cairo.writer.alter.busy.wait.timeout": { - "default": "500", - "description": "Maximum wait timeout in milliseconds for `ALTER TABLE` SQL statement run via REST and PostgreSQL Wire Protocol interfaces when statement execution is `ASYNCHRONOUS`." - }, - "cairo.sql.column.purge.queue.capacity": { - "default": "128", - "description": "Purge column version job queue. Increase the size if column version not automatically cleanup after execution of UPDATE SQL statement. Reduce to decrease initial memory footprint." - }, - "cairo.sql.column.purge.task.pool.capacity": { - "default": "256", - "description": "Column version task object pool capacity. Increase to reduce GC, reduce to decrease memory footprint." - }, - "cairo.sql.column.purge.retry.delay": { - "default": "10000", - "description": "Initial delay (μs) before re-trying purge of stale column files." - }, - "cairo.sql.column.purge.retry.delay.multiplier": { - "default": "10.0", - "description": "Multiplier used to increases retry delay with each iteration." - }, - "cairo.sql.column.purge.retry.delay.limit": { - "default": "60000000", - "description": "Delay limit (μs), upon reaching which, the re-try delay remains constant." - }, - "cairo.sql.column.purge.retry.limit.days": { - "default": "31", - "description": "Number of days purge system will continue to re-try deleting stale column files before giving up." - }, - "cairo.volumes": { - "default": "-", - "description": "A comma separated list of _alias -> root-path_ pairs defining allowed volumes to be used in [CREATE TABLE IN VOLUME](/docs/query/sql/create-table/#table-target-volume) statements." - }, - "cairo.system.table.prefix": { - "default": "sys.", - "description": "Prefix of the tables used for QuestDB internal data storage. These tables are hidden from QuestDB web console." - }, - "cairo.wal.enabled.default": { - "default": "true", - "description": "Setting defining whether WAL table is the default when using `CREATE TABLE`." - }, - "cairo.o3.partition.split.min.size": { - "default": "50MB", - "description": "The estimated partition size on disk. This setting is one of the conditions to trigger [auto-partitioning](/docs/getting-started/capacity-planning/#auto-partitioning)." - }, - "cairo.o3.last.partition.max.splits": { - "default": "20", - "description": "The number of partition pieces allowed before the last partition piece is merged back to the physical partition." - }, - "cairo.o3.partition.purge.list.initial.capacity": { - "default": "1", - "description": "Number of partition expected on average. Initial value for purge allocation job, extended in runtime automatically." - }, - "cairo.sql.parallel.groupby.enabled": { - "default": "true", - "description": "Enables parallel GROUP BY execution; requires at least 4 shared worker threads." - }, - "cairo.sql.parallel.groupby.merge.shard.queue.capacity": { - "default": "", - "description": "Merge queue capacity for parallel GROUP BY; used for parallel tasks that merge shard hash tables." - }, - "cairo.sql.parallel.groupby.sharding.threshold": { - "default": "100000", - "description": "Threshold for parallel GROUP BY to shard the hash table holding the aggregates." - }, - "cairo.sql.groupby.allocator.default.chunk.size": { - "default": "128k", - "description": "Default size for memory buffers in GROUP BY function native memory allocator." - }, - "cairo.sql.groupby.allocator.max.chunk.size": { - "default": "4gb", - "description": "Maximum allowed native memory allocation for GROUP BY functions." - }, - "cairo.sql.unordered.map.max.entry.size": { - "default": "24", - "description": "Threshold in bytes for switching from single memory buffer hash table (unordered) to a hash table with separate heap for entries (ordered)." - }, - "cairo.sql.window.max.recursion": { - "default": "128", - "description": "Prevents stack overflow errors when evaluating complex nested SQLs. The value is an approximate number of nested SELECT clauses." - }, - "cairo.sql.query.registry.pool.size": { - "default": "", - "description": "Pre-sizes the internal data structure that stores active query executions. The value is chosen automatically based on the number of threads in the shared worker pool." - }, - "cairo.sql.analytic.initial.range.buffer.size": { - "default": "32", - "description": "Window function buffer size in record counts. Pre-sizes buffer for every windows function execution to contain window records." - }, - "cairo.system.writer.data.append.page.size": { - "default": "256k", - "description": "mmap sliding page size that TableWriter uses to append data for each column specifically for System tables." - }, - "cairo.file.descriptor.cache.enabled": { - "default": "true", - "description": "enables or disables the file-descriptor cache" - }, - "cairo.partition.encoder.parquet.raw.array.encoding.enabled": { - "default": "false", - "description": "determines whether to export arrays in QuestDB-native binary format (true, less compatible) or Parquet-native format (false, more compatible)." - }, - "cairo.partition.encoder.parquet.version": { - "default": "1", - "description": "Output parquet version to use for parquet-encoded partitions. Can be 1 or 2." - }, - "cairo.partition.encoder.parquet.statistics.enabled": { - "default": "true", - "description": "Controls whether or not statistics are included in parquet-encoded partitions." - }, - "cairo.partition.encoder.parquet.compression.codec": { - "default": "ZSTD", - "description": "Sets the default compression codec for parquet-encoded partitions. Alternatives include `LZ4_RAW`, `SNAPPY`." - }, - "cairo.partition.encoder.parquet.compression.level": { - "default": "9 (ZSTD), 0 (otherwise)", - "description": "Sets the default compression level for parquet-encoded partitions. Dependent on underlying compression codec." - }, - "cairo.partition.encoder.parquet.row.group.size": { - "default": "100000", - "description": "Sets the default row-group size for parquet-encoded partitions." - }, - "cairo.partition.encoder.parquet.data.page.size": { - "default": "1048576", - "description": "Sets the default page size for parquet-encoded partitions." - }, - "cairo.partition.encoder.parquet.min.compression.ratio": { - "default": "1.2", - "description": "Minimum compression ratio (uncompressed_size / compressed_size) for Parquet pages. When a compressed page does not meet this threshold, it is stored uncompressed instead. A value of 0.0 disables the check." - }, - "cairo.sql.parquet.row.group.pruning.enabled": { - "default": "true", - "description": "Enables or disables row group pruning for queries on Parquet partitions. When enabled, QuestDB uses min/max statistics, bloom filters, and null counts to skip row groups that cannot match the query filter." - }, - "cairo.partition.encoder.parquet.bloom.filter.fpp": { - "default": "0.01", - "description": "Default bloom filter false positive probability (FPP) for in-place partition encoding. Lower values produce larger but more accurate filters. Range: 0.0 to 1.0." - } -} diff --git a/documentation/configuration/configuration-utils/_config-validation.config.json b/documentation/configuration/configuration-utils/_config-validation.config.json deleted file mode 100644 index ae0d73d5e..000000000 --- a/documentation/configuration/configuration-utils/_config-validation.config.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "config.validation.strict": { - "default": "false", - "description": "When enabled, startup fails if there are configuration errors." - } -} diff --git a/documentation/configuration/configuration-utils/_csv-import.config.json b/documentation/configuration/configuration-utils/_csv-import.config.json deleted file mode 100644 index fdfd0262e..000000000 --- a/documentation/configuration/configuration-utils/_csv-import.config.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "cairo.sql.copy.root": { - "default": "import", - "description": "Input root directory for CSV imports via `COPY` SQL and for Parquet file reading. This path must not overlap with other directory (e.g. db, conf) of running instance, otherwise import may delete or overwrite existing files. Relative paths are resolved against the server root directory." - }, - "cairo.sql.copy.work.root": { - "default": "null", - "description": "Temporary import file directory. Defaults to `root_directory/tmp` if not set explicitly." - }, - "cairo.iouring.enabled": { - "default": "true", - "description": "Enable or disable io_uring implementation. Applicable to newer Linux kernels only. Can be used to switch io_uring interface usage off if there's a kernel bug affecting it." - }, - "cairo.sql.copy.buffer.size": { - "default": "2 MiB", - "description": "Size of read buffers used in import." - }, - "cairo.sql.copy.log.retention.days": { - "default": "3", - "description": "Number of days to keep import messages in `sys.text_import_log`." - }, - "cairo.sql.copy.max.index.chunk.size": { - "default": "100M", - "description": "Maximum size of index chunk file used to limit total memory requirements of import. Indexing phase should use roughly `thread_count * cairo.sql.copy.max.index.chunk.size` of memory." - }, - "cairo.sql.copy.queue.capacity": { - "default": "32", - "description": "Size of copy task queue. Should be increased if there's more than 32 import workers." - } -} diff --git a/documentation/configuration/configuration-utils/_http.config.json b/documentation/configuration/configuration-utils/_http.config.json deleted file mode 100644 index b92ad0dce..000000000 --- a/documentation/configuration/configuration-utils/_http.config.json +++ /dev/null @@ -1,270 +0,0 @@ -{ - "http.enabled": { - "default": "true", - "description": "Enable or disable HTTP server." - }, - "http.bind.to": { - "default": "0.0.0.0:9000", - "description": "IP address and port of HTTP server. A value of `0` means that the HTTP server will bind to all network interfaces. You can specify IP address of any individual network interface on your system." - }, - "http.user": { - "default": "N/A", - "description": "Username for HTTP Basic Authentication in QuestDB Open Source. QuestDB Enterprise Edition supports more advanced authentication mechanisms: RBAC" - }, - "http.password": { - "default": "N/A", - "description": "Password for HTTP Basic Authentication in QuestDB Open Source. QuestDB Enterprise Edition supports more advanced authentication mechanisms: RBAC" - }, - "http.net.connection.limit": { - "default": "64", - "description": "The maximum number permitted for simultaneous TCP connection to the HTTP server. The rationale of the value is to control server memory consumption." - }, - "http.query.connection.limit": { - "default": "none", - "description": "Soft limit for simultaneous HTTP query connections. When breached, new connections will be rejected but existing connections won't be closed immediately as long as http.net.connection.limit is not exceeded." - }, - "http.ilp.connection.limit": { - "default": "none", - "description": "Soft limit for simultaneous ILP connections. When breached, new connections will be rejected but existing connections won't be closed immediately as long as http.net.connection.limit is not exceeded." - }, - "http.net.connection.timeout": { - "default": "300000", - "description": "TCP connection idle timeout in milliseconds. Connection is closed by HTTP server when this timeout lapses." - }, - "http.net.connection.sndbuf": { - "default": "2M", - "description": "Maximum send buffer size on each TCP socket. If this value is `-1`, the socket send buffer size remains unchanged from the OS defaults." - }, - "http.net.connection.rcvbuf": { - "default": "2M", - "description": "Maximum receive buffer size on each TCP socket. If this value is `-1`, the socket receive buffer size remains unchanged from the OS defaults." - }, - "http.net.connection.hint": { - "default": "false", - "description": "Windows specific flag to overcome OS limitations on TCP backlog size" - }, - "http.net.connection.queue.timeout": { - "default": "5000", - "description": "Amount of time in milliseconds a connection can wait in the listen backlog queue before it is refused. Connections will be aggressively removed from the backlog until the active connection limit is breached." - }, - "http.net.bind.to": { - "default": "0.0.0.0:9000", - "description": "IP address and port of HTTP server." - }, - "http.connection.pool.initial.capacity": { - "default": "4", - "description": "Initial size of pool of reusable objects that hold connection state. The pool should be configured to maximum realistic load so that it does not resize at runtime." - }, - "http.connection.string.pool.capacity": { - "default": "128", - "description": "Initial size of the string pool shared by the HTTP header and multipart content parsers." - }, - "http.multipart.header.buffer.size": { - "default": "512", - "description": "Buffer size in bytes used by the HTTP multipart content parser.", - "reloadable": true - }, - "http.multipart.idle.spin.count": { - "default": "10000", - "description": "How long the code accumulates incoming data chunks for column and delimiter analysis." - }, - "http.receive.buffer.size": { - "default": "1M", - "description": "Size of receive buffer.", - "reloadable": true - }, - "http.request.header.buffer.size": { - "default": "64K", - "description": "Size of internal buffer allocated for HTTP request headers. The value is rounded up to the nearest power of 2. When HTTP requests contain headers that exceed the buffer size server will disconnect the client with HTTP error in server log.", - "reloadable": true - }, - "http.worker.count": { - "default": "0", - "description": "Number of threads in private worker pool. When `0`, HTTP server will be using shared worker pool of the server. Values above `0` switch on private pool." - }, - "http.worker.affinity": { - "default": "", - "description": "Comma separated list of CPU core indexes. The number of items in this list must be equal to the worker count." - }, - "http.worker.haltOnError": { - "default": "false", - "description": "**Changing the default value is strongly discouraged**. Flag that indicates if the worker thread must stop when an unexpected error occurs." - }, - "http.send.buffer.size": { - "default": "2M", - "description": "Size of the internal send buffer. Larger buffer sizes result in fewer I/O interruptions the server is making at the expense of memory usage per connection. There is a limit of send buffer size after which increasing it stops being useful in terms of performance. 2MB seems to be optimal value.", - "reloadable": true - }, - "http.static.index.file.name": { - "default": "index.html", - "description": "Name of index file for the Web Console." - }, - "http.frozen.clock": { - "default": "false", - "description": "Sets the clock to always return zero. This configuration parameter is used for internal testing." - }, - "http.allow.deflate.before.send": { - "default": "false", - "description": "Flag that indicates if Gzip compression of outgoing data is allowed." - }, - "http.keep-alive.timeout": { - "default": "5", - "description": "Used together with `http.keep-alive.max` to set the value of HTTP `Keep-Alive` response header. This instructs browser to keep TCP connection open. Has to be `0` when `http.version` is set to `HTTP/1.0`." - }, - "http.keep-alive.max": { - "default": "10000", - "description": "See `http.keep-alive.timeout`. Has to be `0` when `http.version` is set to `HTTP/1.0`." - }, - "http.static.public.directory": { - "default": "public", - "description": "The name of directory for public web site." - }, - "http.text.date.adapter.pool.capacity": { - "default": "16", - "description": "Size of date adapter pool. This should be set to the anticipated maximum number of `DATE` fields a text input can have. The pool is assigned to connection state and is reused alongside of connection state object." - }, - "http.text.json.cache.limit": { - "default": "16384", - "description": "JSON parser cache limit. Cache is used to compose JSON elements that have been broken up by TCP protocol. This value limits the maximum length of individual tag or tag value." - }, - "http.text.json.cache.size": { - "default": "8192", - "description": "Initial size of JSON parser cache. The value must not exceed `http.text.json.cache.limit` and should be set to avoid cache resizes at runtime." - }, - "http.text.max.required.delimiter.stddev": { - "default": "0.1222d", - "description": "The maximum standard deviation value for the algorithm that calculates text file delimiter. Usually when text parser cannot recognise the delimiter it will log the calculated and maximum standard deviation for the delimiter candidate." - }, - "http.text.max.required.line.length.stddev": { - "default": "0.8", - "description": "Maximum standard deviation value for the algorithm that classifies input as text or binary. For the values above configured stddev input will be considered binary." - }, - "http.text.metadata.string.pool.capacity": { - "default": "128", - "description": "The initial size of pool for objects that wrap individual elements of metadata JSON, such as column names, date pattern strings and locale values." - }, - "http.text.roll.buffer.limit": { - "default": "4M", - "description": "The limit of text roll buffer. See `http.text.roll.buffer.size` for description." - }, - "http.text.roll.buffer.size": { - "default": "1024", - "description": "Roll buffer is a structure in the text parser that holds a copy of a line that has been broken up by TCP. The size should be set to the maximum length of text line in text input." - }, - "http.text.analysis.max.lines": { - "default": "1000", - "description": "Number of lines to read on CSV import for heuristics which determine column names & types. Lower line numbers may detect CSV schemas quicker, but possibly with less accuracy. 1000 lines is the maximum for this value." - }, - "http.text.lexer.string.pool.capacity": { - "default": "64", - "description": "The initial capacity of string fool, which wraps `STRING` column types in text input. The value should correspond to the maximum anticipated number of STRING columns in text input." - }, - "http.text.timestamp.adapter.pool.capacity": { - "default": "64", - "description": "Size of timestamp adapter pool. This should be set to the anticipated maximum number of `TIMESTAMP` fields a text input can have. The pool is assigned to connection state and is reused alongside of connection state object." - }, - "http.text.utf8.sink.size": { - "default": "4096", - "description": "Initial size of UTF-8 adapter sink. The value should correspond the maximum individual field value length in text input." - }, - "http.json.query.connection.check.frequency": { - "default": "1000000", - "description": "**Changing the default value is strongly discouraged**. The value to throttle check if client socket has been disconnected." - }, - "http.json.query.float.scale": { - "default": "4", - "description": "The scale value of string representation of `FLOAT` values." - }, - "http.json.query.double.scale": { - "default": "12", - "description": "The scale value of string representation of `DOUBLE` values." - }, - "http.query.cache.enabled": { - "default": "true", - "description": "Enable or disable the query cache. Cache capacity is `number_of_blocks * number_of_rows`." - }, - "http.query.cache.block.count": { - "default": "4", - "description": "Number of blocks for the query cache." - }, - "http.query.cache.row.count": { - "default": "16", - "description": "Number of rows for the query cache." - }, - "http.security.readonly": { - "default": "false", - "description": "Forces HTTP read only mode when `true`, disabling commands which modify the data or data structure, e.g. INSERT, UPDATE, or CREATE TABLE." - }, - "http.security.max.response.rows": { - "default": "2^63-1", - "description": "Limit the number of response rows over HTTP." - }, - "http.security.interrupt.on.closed.connection": { - "default": "true", - "description": "Switch to enable termination of SQL processing if the HTTP connection is closed. The mechanism affects performance so the connection is only checked after `circuit.breaker.throttle` calls are made to the check method. The mechanism also reads from the input stream and discards it since some HTTP clients send this as a keep alive in between requests, `circuit.breaker.buffer.size` denotes the size of the buffer for this." - }, - "http.pessimistic.health.check.enabled": { - "default": "false", - "description": "When enabled, the health check returns HTTP 500 for any unhandled errors since the server started." - }, - "circuit.breaker.throttle": { - "default": "2000000", - "description": "Number of internal iterations such as loops over data before checking if the HTTP connection is still open" - }, - "circuit.breaker.buffer.size": { - "default": "32", - "description": "Size of buffer to read from HTTP connection. If this buffer returns zero and the HTTP client is no longer sending data, SQL processing will be terminated." - }, - "http.server.keep.alive": { - "default": "true", - "description": "If set to `false`, the server will disconnect the client after completion of each request." - }, - "http.version": { - "default": "HTTP/1.1", - "description": "Protocol version, other supported value is `HTTP/1.0`." - }, - "http.context.web.console": { - "default": "/", - "description": "Context path for the Web Console. If other REST services remain on the default context paths they will move to the same context path as the Web Console. InfluxDB Line Protocol (ILP) HTTP services are not affected and remain on their default paths. When default context paths are changed, moving the Web Console will not affect the configured paths. QuestDB creates copies of services on the Web Console paths so that both the Web Console and custom services remain operational." - }, - "http.context.import": { - "default": "/imp", - "description": "Context path of the file import service." - }, - "http.context.table.status": { - "default": "/chk", - "description": "Context path for the table statusservice used by the Import UI in the Web Console." - }, - "http.context.export": { - "default": "/exp", - "description": "Context path for the SQL result CSV export service." - }, - "http.context.settings": { - "default": "/settings", - "description": "Context path for the service which provides server-side settings to the Web Console." - }, - "http.context.execute": { - "default": "/exec", - "description": "Context path for the SQL execution service." - }, - "http.context.warnings": { - "default": "/warnings", - "description": "Context path for the Web Console specific service." - }, - "http.context.ilp": { - "default": "/write,/api/v2/write", - "description": "Context paths for the Influx Line Protocol (ILP) HTTP services. These are not used by the Web Console." - }, - "http.context.ilp.ping": { - "default": "/ping", - "description": "Context path for the Influx Line Protocol (ILP) ping endpoint." - }, - "http.redirect.count": { - "default": "1", - "description": "Number of HTTP redirects. All redirects are 301 - Moved Permanently." - }, - "http.redirect.1": { - "default": "/ -> /index.html", - "description": "Example redirect configuration. Format is 'source -> destination'." - } -} diff --git a/documentation/configuration/configuration-utils/_iam.config.json b/documentation/configuration/configuration-utils/_iam.config.json deleted file mode 100644 index 7303a04ce..000000000 --- a/documentation/configuration/configuration-utils/_iam.config.json +++ /dev/null @@ -1,87 +0,0 @@ -{ - "acl.enabled": { - "default": "true", - "description": "Enables/disables Identity and Access Management." - }, - "acl.admin.user.enabled": { - "default": "true", - "description": "Enables/disables the built-in admin user." - }, - "acl.admin.user": { - "default": "admin", - "description": "Name of the built-in admin user." - }, - "acl.admin.password": { - "default": "quest", - "description": "The password of the built-in admin user.", - "reloadable": true - }, - "acl.basic.auth.realm.enabled": { - "default": "false", - "description": "When enabled the browser's basic auth popup window is used instead of the Web Console's login screen. Only present for backwards compatibility." - }, - "acl.entity.name.max.length": { - "default": "255", - "description": "Maximum length of user, group and service account names." - }, - "acl.password.hash.iteration.count": { - "default": "100000", - "description": "QuestDB Enterprise never stores passwords in plain text, it stores password hashes only. This is the number of hash iterations used in password hashing. Higher means safer, almost never should be changed." - }, - "acl.rest.token.refresh.threshold": { - "default": "10", - "description": "When a REST token is created in REFRESH mode, its TTL is extended on every successful authentication, unless the last successful authentication was within this threshold. This setting removes unnecessary overhead of continuously refreshing REST tokens if they are used often. The value is expressed in seconds." - }, - "tls.enabled": { - "default": "false", - "description": "Enables/disables TLS encryption globally for all QuestDB interfaces (HTTP endpoints, ILP over TCP)." - }, - "tls.cert.path": { - "default": "", - "description": "Path to certificate used for TLS encryption globally. The certificate should be DER-encoded and saved in PEM format." - }, - "tls.private.key.path": { - "default": "", - "description": "Path to private key used for TLS encryption globally." - }, - "http.tls.enabled": { - "default": "false", - "description": "Enables/disables TLS encryption for the HTTP server only." - }, - "http.tls.cert.path": { - "default": "", - "description": "Path to certificate used for TLS encryption for the HTTP server only. The certificate should be DER-encoded and saved in PEM format." - }, - "http.tls.private.key.path": { - "default": "", - "description": "Path to private key used for TLS encryption for the HTTP server only." - }, - "http.min.tls.enabled": { - "default": "false", - "description": "Enables/disables TLS encryption for the minimal HTTP server only." - }, - "http.min.tls.cert.path": { - "default": "", - "description": "Path to certificate used for TLS encryption for the minimal HTTP server only. The certificate should be DER-encoded and saved in PEM format." - }, - "http.min.tls.private.key.path": { - "default": "", - "description": "Path to private key used for TLS encryption for the minimal HTTP server only." - }, - "line.tcp.tls.enabled": { - "default": "false", - "description": "Enables/disables TLS encryption for ILP over TCP only." - }, - "line.tcp.tls.cert.path": { - "default": "", - "description": "Path to certificate used for TLS encryption for ILP over TCP only. The certificate should be DER-encoded and saved in PEM format." - }, - "line.tcp.tls.private.key.path": { - "default": "", - "description": "Path to private key used for TLS encryption for ILP over TCP only." - }, - "line.tcp.acl.enabled": { - "default": "true", - "description": "Enables/disables authentication for the ILP over TCP endpoint only." - } -} \ No newline at end of file diff --git a/documentation/configuration/configuration-utils/_log.config.json b/documentation/configuration/configuration-utils/_log.config.json deleted file mode 100644 index d6b42f0be..000000000 --- a/documentation/configuration/configuration-utils/_log.config.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "log.level.verbose": { - "default": "false", - "description": "Converts short-hand log level indicators (E, C, I) into long-hand (ERROR, CRITICAL, INFO)" - }, - "log.timezone": { - "default": "UTC", - "description": "Sets the timezone for log timestamps. Can be a timezone ID such as 'Antarctica/McMurdo', 'SystemDefault' to use system timezone, or the default UTC with 'Z' suffix" - } -} \ No newline at end of file diff --git a/documentation/configuration/configuration-utils/_mat-view.config.json b/documentation/configuration/configuration-utils/_mat-view.config.json deleted file mode 100644 index dff9aaa35..000000000 --- a/documentation/configuration/configuration-utils/_mat-view.config.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "cairo.mat.view.enabled": { - "default": "true", - "description": "Enables or disables SQL support and refresh job for materialized views." - }, - "cairo.mat.view.parallel.sql.enabled": { - "default": "true", - "description": "When disabled, SQL executed by the materialized view refresh job will always run single-threaded." - }, - "mat.view.refresh.worker.count": { - "default": "0", - "description": "Number of dedicated worker threads assigned to refresh materialized views. When `0`, the jobs will use the shared pool." - }, - "mat.view.refresh.worker.affinity": { - "default": "Equal to the CPU core count", - "description": "Comma separated list of numerical CPU core indexes." - }, - "mat.view.refresh.worker.haltOnError": { - "default": "false", - "description": "Flag that indicates if the worker thread must stop when an unexpected error occurs." - } -} diff --git a/documentation/configuration/configuration-utils/_oidc.config.json b/documentation/configuration/configuration-utils/_oidc.config.json deleted file mode 100644 index e084545c8..000000000 --- a/documentation/configuration/configuration-utils/_oidc.config.json +++ /dev/null @@ -1,98 +0,0 @@ -{ - "acl.oidc.enabled": { - "default": "false", - "description": "Enables/disables OIDC authentication. When enabled, few other configuration options must also be set." - }, - "acl.oidc.pkce.enabled": { - "default": "true", - "description": "Enables/disables PKCE for the Authorization Code Flow. This should always be enabled in a production environment, the Web Console is not fully secure without it." - }, - "acl.oidc.ropc.flow.enabled": { - "default": "false", - "description": "Enables/disables Resource Owner Password Credentials flow. When enabled, this flow also has to be configured in the OIDC Provider." - }, - "acl.oidc.configuration.url": { - "default": "", - "description": "URL where the OpenID Provider's configuration information cna be loaded in json format, should always end with `/.well-known/openid-configuration`." - }, - "acl.oidc.host": { - "default": "", - "description": "OIDC provider hostname. Required when OIDC is enabled, unless the OIDC configuration URL is set." - }, - "acl.oidc.port": { - "default": 443, - "description": "OIDC provider port number." - }, - "acl.oidc.tls.enabled": { - "default": "true", - "description": "Whether the OIDC provider requires a secure connection or not. It is highly unlikely in a production environment, but if the OpenID Provider endpoints do not require a secure connection, this option can be set to `false`." - }, - "acl.oidc.tls.validation.enabled": { - "default": "true", - "description": "Enables/disables TLS certificate validation. If you are working with self-signed certificates that you would like QuestDB to trust, disable this option. Validation is strongly recommended in production environments. QuestDB will check that the certificate is valid, and that it is issued for the server to which it connects." - }, - "acl.oidc.tls.keystore.path": { - "default": "", - "description": "Path to a keystore file that contains trusted Certificate Authorities. Will be used when validating the certificate of the OIDC provider. Not required if your OIDC provider's certificate is signed by a public CA." - }, - "acl.oidc.tls.keystore.password": { - "default": "", - "description": "Keystore password, required if there is a keystore file and it is password protected." - }, - "acl.oidc.http.timeout": { - "default": 30000, - "description": "OIDC provider HTTP request timeout in milliseconds." - }, - "acl.oidc.client.id": { - "default": "", - "description": "Client name assigned to QuestDB in the OIDC server, required when OIDC is enabled." - }, - "acl.oidc.audience": { - "default": "", - "description": "OAuth2 audience as set on the tokens issued by the OIDC Provider, defaults to the client id." - }, - "acl.oidc.redirect.uri": { - "default": "", - "description": "The redirect URI tells the OIDC server where to redirect the user after successful authentication. If not set, the Web Console defaults it to the location where it was loaded from (`window.location.href`)." - }, - "acl.oidc.scope": { - "default": "openid", - "description": "The OIDC server should ask consent for the list of scopes provided in this property. The scope `openid` is mandatory, and always should be included." - }, - "acl.oidc.public.keys.endpoint": { - "default": "/pf/JWKS", - "description": "JSON Web Key Set (JWKS) Endpoint, the default value should work for the Ping Identity Platform. This endpoint provides the list of public keys can be used to decode and validate ID tokens issued by the OIDC Provider." - }, - "acl.oidc.authorization.endpoint": { - "default": "/as/authorization.oauth2", - "description": "OIDC Authorization Endpoint, the default value should work for the Ping Identity Platform." - }, - "acl.oidc.token.endpoint": { - "default": "/as/token.oauth2", - "description": "OIDC Token Endpoint, the default value should work for the Ping Identity Platform." - }, - "acl.oidc.userinfo.endpoint": { - "default": "/idp/userinfo.openid", - "description": "OIDC User Info Endpoint, the default value should work for the Ping Identity Platform. Used to retrieve additional user information which contains the user's group memberships." - }, - "acl.oidc.groups.encoded.in.token": { - "default": "false", - "description": "Should be set to false, if the OIDC Provider is configured to encode the group memberships of the user into the id token. When set to true, QuestDB will look for the groups in the token instead of calling the User Info endpoint." - }, - "acl.oidc.sub.claim": { - "default": "sub", - "description": "The name of the claim in the user information, which contains the name of the user. Could be a username, the user's full name or email. It will be displayed in the Web Console, and logged for audit purposes." - }, - "acl.oidc.groups.claim": { - "default": "groups", - "description": "The name of the custom claim in the user information, which contains the group memberships of the user." - }, - "acl.oidc.cache.ttl": { - "default": 30000, - "description": "User info cache entry TTL (time to live) in milliseconds, default value is 30 seconds. For improved performance QuestDB caches user info responses for each valid access token, this settings drives how often the access token should be validated and the user info updated." - }, - "acl.oidc.pg.token.as.password.enabled": { - "default": "false", - "description": "When enabled, the PGWire endpoint supports OIDC authentication. The OAuth2 token should be sent in the password field, while the username field should contain the string `_sso`, or left empty if that is an option." - } -} diff --git a/documentation/configuration/configuration-utils/_parallel-sql.config.json b/documentation/configuration/configuration-utils/_parallel-sql.config.json deleted file mode 100644 index 39a5d6c16..000000000 --- a/documentation/configuration/configuration-utils/_parallel-sql.config.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "cairo.sql.parallel.filter.enabled": { - "default": "true", - "description": "Enable or disable parallel SQL filter execution. JIT compilation takes place only when this setting is enabled." - }, - "cairo.sql.parallel.filter.pretouch.enabled": { - "default": "true", - "description": "Enable column pre-touch as part of the parallel SQL filter execution, to improve query performance for large tables." - }, - "cairo.page.frame.shard.count": { - "default": "4", - "description": "Number of shards for both dispatch and reduce queues. Shards reduce queue contention between SQL statements that are executed concurrently." - }, - "cairo.page.frame.reduce.queue.capacity": { - "default": "64", - "description": "Reduce queue is used for data processing and should be large enough to supply tasks for worker threads (shared worked pool)." - }, - "cairo.page.frame.rowid.list.capacity": { - "default": "256", - "description": "Row ID list initial capacity for each slot of the reduce queue. Larger values reduce memory allocation rate, but increase minimal RSS size." - }, - "cairo.page.frame.column.list.capacity": { - "default": "16", - "description": "Column list capacity for each slot of the reduce queue. Used by JIT-compiled filter functions. Larger values reduce memory allocation rate, but increase minimal RSS size." - } -} diff --git a/documentation/configuration/configuration-utils/_parquet-export.config.json b/documentation/configuration/configuration-utils/_parquet-export.config.json deleted file mode 100644 index 970bd2992..000000000 --- a/documentation/configuration/configuration-utils/_parquet-export.config.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "cairo.sql.copy.export.root": { - "default": "export", - "description": "Root directory for parquet exports via `COPY-TO` SQL. This path must not overlap with other directory (e.g. db, conf) of running instance, otherwise export may delete or overwrite existing files. Relative paths are resolved against the server root directory." - }, - "cairo.parquet.export.bloom.filter.fpp": { - "default": "0.01", - "description": "Default bloom filter false positive probability (FPP) for Parquet exports via `COPY-TO` and REST `/exp`. Lower values produce larger but more accurate filters. Range: 0.0 to 1.0." - }, - "cairo.parquet.export.statistics.enabled": { - "default": "true", - "description": "Enables or disables generation of column statistics (min/max, null count) in Parquet exports." - } -} diff --git a/documentation/configuration/configuration-utils/_postgres.config.json b/documentation/configuration/configuration-utils/_postgres.config.json deleted file mode 100644 index d08c568a1..000000000 --- a/documentation/configuration/configuration-utils/_postgres.config.json +++ /dev/null @@ -1,159 +0,0 @@ -{ - "pg.enabled": { - "default": "true", - "description": "Configuration for enabling or disabling the Postres interface." - }, - "pg.net.bind.to": { - "default": "0.0.0.0:8812", - "description": "IP address and port of Postgres wire protocol server. 0 means that the server will bind to all network interfaces. You can specify IP address of any individual network interface on your system." - }, - "pg.net.connection.limit": { - "default": "64", - "description": "The maximum number permitted for simultaneous Postgres connections to the server. This value is intended to control server memory consumption.", - "reloadable": true - }, - "pg.net.connection.timeout": { - "default": "300000", - "description": "Connection idle timeout in milliseconds. Connections are closed by the server when this timeout lapses." - }, - "pg.net.connection.rcvbuf": { - "default": "-1", - "description": "Maximum send buffer size on each TCP socket. If value is -1 socket send buffer remains unchanged from OS default." - }, - "pg.net.connection.sndbuf": { - "default": "-1", - "description": "Maximum receive buffer size on each TCP socket. If value is -1, the socket receive buffer remains unchanged from OS default." - }, - "pg.net.connection.hint": { - "default": "false", - "description": "Windows specific flag to overcome OS limitations on TCP backlog size" - }, - "pg.net.connection.queue.timeout": { - "default": "300000", - "description": "Amount of time in milliseconds a connection can wait in the listen backlog queue before it is refused. Connections will be aggressively removed from the backlog until the active connection limit is breached." - }, - "pg.security.readonly": { - "default": "false", - "description": "Forces PostgreSQL Wire Protocol read only mode when `true`, disabling commands which modify the data or data structure, e.g. INSERT, UPDATE, or CREATE TABLE." - }, - "pg.character.store.capacity": { - "default": "4096", - "description": "Size of the CharacterStore." - }, - "pg.character.store.pool.capacity": { - "default": "64", - "description": "Size of the CharacterStore pool capacity." - }, - "pg.connection.pool.capacity": { - "default": "64", - "description": "The maximum amount of pooled connections this interface may have." - }, - "pg.password": { - "default": "quest", - "description": "Postgres database password.", - "reloadable": true - }, - "pg.user": { - "default": "admin", - "description": "Postgres database username.", - "reloadable": true - }, - "pg.readonly.user.enabled": { - "default": "false", - "description": "Enable or disable Postgres database read-only user account. When enabled, this additional user can be used to open read-only connections to the database.", - "reloadable": true - }, - "pg.readonly.password": { - "default": "quest", - "description": "Postgres database read-only user password.", - "reloadable": true - }, - "pg.readonly.user": { - "default": "user", - "description": "Postgres database read-only user username.", - "reloadable": true - }, - "pg.select.cache.enabled": { - "default": "true", - "description": "Enable or disable the SELECT query cache. Cache capacity is `number_of_blocks * number_of_rows`." - }, - "pg.select.cache.block.count": { - "default": "16", - "description": "Number of blocks to cache SELECT query execution plan against text to speed up execution." - }, - "pg.select.cache.row.count": { - "default": "16", - "description": "Number of rows to cache for SELECT query execution plan against text to speed up execution." - }, - "pg.insert.cache.enabled": { - "default": "true", - "description": "Enable or disable the INSERT query cache. Cache capacity is `number_of_blocks * number_of_rows`." - }, - "pg.insert.cache.block.count": { - "default": "8", - "description": "Number of blocks to cache INSERT query execution plan against text to speed up execution." - }, - "pg.insert.cache.row.count": { - "default": "8", - "description": "Number of rows to cache for INSERT query execution plan against text to speed up execution." - }, - "pg.update.cache.enabled": { - "default": "true", - "description": "Enable or disable the UPDATE query cache. Cache capacity is `number_of_blocks * number_of_rows`." - }, - "pg.update.cache.block.count": { - "default": "8", - "description": "Number of blocks to cache UPDATE query execution plan against text to speed up execution." - }, - "pg.update.cache.row.count": { - "default": "8", - "description": "Number of rows to cache for UPDATE query execution plan against text to speed up execution." - }, - "pg.max.blob.size.on.query": { - "default": "512k", - "description": "For binary values, clients will receive an error when requesting blob sizes above this value." - }, - "pg.recv.buffer.size": { - "default": "1M", - "description": "Size of the buffer for receiving data.", - "reloadable": true - }, - "pg.send.buffer.size": { - "default": "1M", - "description": "Size of the buffer for sending data.", - "reloadable": true - }, - "pg.date.locale": { - "default": "en", - "description": "The locale to handle date types." - }, - "pg.timestamp.locale": { - "default": "en", - "description": "The locale to handle timestamp types." - }, - "pg.worker.count": { - "default": "0", - "description": "Number of dedicated worker threads assigned to handle PostgreSQL Wire Protocol queries. When `0`, the jobs will use the shared pool." - }, - "pg.worker.affinity": { - "default": "", - "description": "Comma-separated list of thread numbers which should be pinned for Postgres ingestion. Example `pg.worker.affinity=1,2,3`." - }, - "pg.halt.on.error": { - "default": "false", - "description": "Whether ingestion should stop upon internal error." - }, - "pg.daemon.pool": { - "default": "true", - "description": "Defines whether to run all PostgreSQL Wire Protocol worker threads in daemon mode (`true`) or not (`false`)." - }, - "pg.binary.param.count.capacity": { - "default": "2", - "description": "Size of the initial capacity for the pool used for binary bind variables." - }, - "pg.named.statement.limit": { - "default": "64", - "description": "Size of the named statement pool.", - "reloadable": true - } -} \ No newline at end of file diff --git a/documentation/configuration/configuration-utils/_replication.config.json b/documentation/configuration/configuration-utils/_replication.config.json deleted file mode 100644 index a42104620..000000000 --- a/documentation/configuration/configuration-utils/_replication.config.json +++ /dev/null @@ -1,150 +0,0 @@ -{ - "replication.role": { - "default": "none", - "description": "Defaults to `none` for stand-alone instances. To enable replication set to one of: `primary`, `replica`." - }, - "replication.object.store": { - "default": "", - "description": "A configuration string that allows connecting to an object store. The format is **scheme::key1=value;key2=value2;…**. The various keys and values are detailed in a later section. Ignored if replication is disabled. No default given variability." - }, - "cairo.wal.segment.rollover.size": { - "default": "2097152", - "description": "The size of the WAL segment before it is rolled over. Default is `2MiB`. However, defaults to `0` unless `replication.role=primary` is set." - }, - "cairo.writer.command.queue.capacity": { - "default": "32", - "description": "Maximum writer ALTER TABLE and replication command capacity. Shared between all the tables." - }, - "replication.primary.throttle.window.duration": { - "default": "10000", - "description": "The millisecond duration of the sliding window used to process replication batches. Default is `10000` ms." - }, - "replication.requests.max.concurrent": { - "default": "0", - "description": "A limit to the number of concurrent object store requests. The default is `0` for unlimited." - }, - "replication.requests.retry.attempts": { - "default": "3", - "description": "Maximum number of times to retry a failed object store request before logging an error and reattempting later after a delay. Default is `3`." - }, - "replication.requests.retry.interval": { - "default": "200", - "description": "How long to wait before retrying a failed operation. Default is `200` ms." - }, - "replication.primary.compression.threads": { - "default": "calculated", - "description": "Max number of threads used to perform file compression operations before uploading to the object store. The default value is calculated as half the number of CPU cores." - }, - "replication.primary.compression.level": { - "default": "1", - "description": "Zstd compression level. Defaults to `1`. Valid values are from 1 to 22." - }, - "replication.replica.poll.interval": { - "default": "1000", - "description": "Millisecond polling rate of a replica instance to check for the availability of new changes." - }, - "replication.primary.sequencer.part.txn.count": { - "default": "5000", - "description": "Sets the txn chunking size for each compressed batch. Smaller is better for constrained networks (but more costly)." - }, - "replication.primary.checksum=service-dependent": { - "default": "service-dependent", - "description": "Where a checksum should be calculated for each uploaded artifact. Required for some object stores. Other options: never, always" - }, - "replication.primary.upload.truncated": { - "default": "true", - "description": "Skip trailing, empty column data inside a WAL column file." - }, - "replication.requests.buffer.size": { - "default": "32768", - "description": "Buffer size used for object-storage downloads." - }, - "replication.summary.interval": { - "default": "1m", - "description": "Frequency for printing replication progress summary in the logs." - }, - "replication.metrics.per.table": { - "default": "true", - "description": "Enable per-table replication metrics on the prometheus metrics endpoint." - }, - "replication.metrics.dropped.table.poll.count": { - "default": "10", - "description": "How many scrapes of prometheus metrics endpoint before dropped tables will no longer appear." - }, - "replication.requests.max.batch.size.fast": { - "default": "64", - "description": "Number of parallel requests allowed during the 'fast' process (non-resource constrained)." - }, - "replication.requests.max.batch.size.slow": { - "default": "2", - "description": "Number of parallel requests allowed during the 'slow' process (error/resource constrained path)." - }, - "replication.requests.base.timeout": { - "default": "10s", - "description": "Replication upload/download request timeout." - }, - "replication.requests.min.throughput": { - "default": "262144", - "description": "Expected minimum network speed for replication transfers. Used to expand the timeout and account for network delays." - }, - "native.async.io.threads": { - "default": "cpuCount", - "description": "The number of async (network) io threads used for replication (and in the future cold storage). The default should be appropriate for most use cases." - }, - "native.max.blocking.threads": { - "default": "cpuCount * 4", - "description": "Maximum number of threads for parallel blocking disk IO read/write operations for replication (and other). These threads are ephemeral: They are spawned per need and shut down after a short duration if no longer in use. These are not cpu-bound threads, hence the relative large number. The default should be appropriate for most use cases." - }, - "replication.primary.cleaner.enabled": { - "default": "true", - "description": "Master switch for the WAL cleaner." - }, - "replication.primary.cleaner.interval": { - "default": "10m", - "description": "Time between cleanup cycles. Range: 1s – 24h." - }, - "replication.primary.cleaner.checkpoint.source": { - "default": "true", - "description": "Use checkpoint history as a cleanup trigger source." - }, - "replication.primary.cleaner.backup.window.count": { - "default": "backup.cleanup.keep.latest.n or 5", - "description": "Minimum complete backups/checkpoints per instance before cleanup starts. Defaults to `backup.cleanup.keep.latest.n` if backups are enabled, otherwise `5`." - }, - "replication.primary.cleaner.delete.concurrency": { - "default": "4 – 12 (auto)", - "description": "Concurrent deletion tasks. Derived from `replication.requests.max.concurrent`. Range: 4 – 32." - }, - "replication.primary.cleaner.max.requests.per.second": { - "default": "service-dependent", - "description": "Rate limit for object store delete requests. Set to `0` for unlimited. Range: 0 – 10000." - }, - "replication.primary.cleaner.progress.write.interval": { - "default": "5s", - "description": "How often progress is persisted during a cleanup cycle. Lower values mean less re-work after a crash but more writes. Range: 100ms – 60s." - }, - "replication.primary.cleaner.dropped.table.cooloff": { - "default": "1h", - "description": "Wait time after `DROP TABLE` before removing the table's data from object storage. Guards against clock skew." - }, - "replication.primary.cleaner.retry.attempts": { - "default": "20", - "description": "Retries for transient object store failures during cleanup. Range: 0 – 100." - }, - "replication.primary.cleaner.retry.interval": { - "default": "2s", - "description": "Delay between cleanup retries. Range: 0 – 5m." - }, - "checkpoint.history.enabled": { - "default": "true (when replication is enabled)", - "description": "Enable the checkpoint history tracker. Requires replication." - }, - "checkpoint.history.keep.count": { - "default": "100", - "description": "Maximum checkpoint records retained per instance." - }, - "checkpoint.history.long.retry.interval": { - "default": "1m", - "description": "Retry interval for syncing checkpoint history to the object store after burst retries fail." - } -} diff --git a/documentation/configuration/configuration-utils/_shared-worker.config.json b/documentation/configuration/configuration-utils/_shared-worker.config.json deleted file mode 100644 index 1f7e301fc..000000000 --- a/documentation/configuration/configuration-utils/_shared-worker.config.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "shared.network.worker.count": { - "default": "max(2, CPU count - 2) if CPU count > 32, max(2, CPU count - 1) if CPU count > 16, otherwise max(2, CPU count)", - "description": "Number of worker threads for the network pool, which handles HTTP, PostgreSQL, and ILP server I/O. Increasing this number will increase network I/O parallelism at the expense of CPU resources." - }, - "shared.network.worker.affinity": { - "default": "none", - "description": "Comma-delimited list of CPU ids, one per thread specified in `shared.network.worker.count`. By default, threads have no CPU affinity." - }, - "shared.query.worker.count": { - "default": "max(2, CPU count - 2) if CPU count > 32, max(2, CPU count - 1) if CPU count > 16, otherwise max(2, CPU count)", - "description": "Number of worker threads for the query pool, which executes parallel query operations (filters, group-by). Increasing this number will increase query parallelism at the expense of CPU resources." - }, - "shared.query.worker.affinity": { - "default": "none", - "description": "Comma-delimited list of CPU ids, one per thread specified in `shared.query.worker.count`. By default, threads have no CPU affinity." - }, - "shared.write.worker.count": { - "default": "max(2, CPU count - 2) if CPU count > 32, max(2, CPU count - 1) if CPU count > 16, otherwise max(2, CPU count)", - "description": "Number of worker threads for the write pool, which manages WAL apply jobs, table writes, materialized view refresh, and housekeeping tasks. Increasing this number will increase write parallelism at the expense of CPU resources." - }, - "shared.write.worker.affinity": { - "default": "none", - "description": "Comma-delimited list of CPU ids, one per thread specified in `shared.write.worker.count`. By default, threads have no CPU affinity." - }, - "shared.worker.haltOnError": { - "default": "false", - "description": "Flag that indicates if the worker thread must stop when an unexpected error occurs." - } -} diff --git a/documentation/configuration/configuration-utils/_storage-policy.config.json b/documentation/configuration/configuration-utils/_storage-policy.config.json deleted file mode 100644 index 91042fc20..000000000 --- a/documentation/configuration/configuration-utils/_storage-policy.config.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "storage.policy.check.interval": { - "default": "15m", - "description": "How often QuestDB scans for partitions to process." - }, - "storage.policy.retry.interval": { - "default": "1m", - "description": "Retry interval for failed storage policy tasks." - }, - "storage.policy.max.reschedule.count": { - "default": "20", - "description": "Maximum number of retries before abandoning a storage policy task." - }, - "storage.policy.writer.wait.timeout": { - "default": "30s", - "description": "Timeout for acquiring the table writer during storage policy operations." - }, - "storage.policy.worker.count": { - "default": "2", - "description": "Number of storage policy worker threads. Setting to `0` disables the feature." - }, - "storage.policy.worker.affinity": { - "default": "-1", - "description": "CPU affinity for each storage policy worker thread (comma-separated list). `-1` means no affinity." - }, - "storage.policy.worker.sleep.timeout": { - "default": "100ms", - "description": "Sleep duration when a storage policy worker has no tasks to process." - } -} diff --git a/documentation/configuration/configuration-utils/_tcp.config.json b/documentation/configuration/configuration-utils/_tcp.config.json deleted file mode 100644 index 8e337fe59..000000000 --- a/documentation/configuration/configuration-utils/_tcp.config.json +++ /dev/null @@ -1,115 +0,0 @@ -{ - "line.tcp.enabled": { - "default": "true", - "description": "Enable or disable line protocol over TCP." - }, - "line.tcp.net.bind.to": { - "default": "0.0.0.0:9009", - "description": "IP address of the network interface to bind listener to and port. By default, TCP receiver listens on all network interfaces." - }, - "line.tcp.net.connection.limit": { - "default": "256", - "description": "The maximum number permitted for simultaneous connections to the server. This value is intended to control server memory consumption.", - "reloadable": true - }, - "line.tcp.net.connection.timeout": { - "default": "300000", - "description": "Connection idle timeout in milliseconds. Connections are closed by the server when this timeout lapses." - }, - "line.tcp.net.connection.hint": { - "default": "false", - "description": "Windows specific flag to overcome OS limitations on TCP backlog size" - }, - "line.tcp.net.connection.rcvbuf": { - "default": "-1", - "description": "Maximum buffer receive size on each TCP socket. If value is -1, the socket receive buffer remains unchanged from OS default." - }, - "line.tcp.net.connection.queue.timeout": { - "default": "5000", - "description": "Amount of time in milliseconds a connection can wait in the listen backlog queue before its refused. Connections will be aggressively removed from the backlog until the active connection limit is breached." - }, - "line.tcp.auth.db.path": { - "default": "", - "description": "Path which points to the authentication db file." - }, - "line.tcp.connection.pool.capacity": { - "default": "64", - "description": "The maximum amount of pooled connections this interface may have." - }, - "line.tcp.timestamp": { - "default": "n", - "description": "Input timestamp resolution. Possible values are `n`, `u`, `ms`, `s` and `h`." - }, - "line.tcp.msg.buffer.size": { - "default": "32768", - "description": "Size of the buffer read from queue. Maximum size of write request, regardless of the number of measurements." - }, - "line.tcp.maintenance.job.interval": { - "default": "1000", - "description": "Maximum amount of time (in milliseconds) between maintenance jobs committing any uncommitted data on inactive tables." - }, - "line.tcp.min.idle.ms.before.writer.release": { - "default": "500", - "description": "Minimum amount of idle time (in milliseconds) before a table writer is released." - }, - "line.tcp.commit.interval.fraction": { - "default": "0.5", - "description": "Commit lag fraction. Used to calculate commit interval for the table according to the following formula: `commit_interval = commit_lag ∗ fraction`. The calculated commit interval defines how long uncommitted data will need to remain uncommitted." - }, - "line.tcp.commit.interval.default": { - "default": "1000", - "description": "Default commit interval in milliseconds." - }, - "line.tcp.max.measurement.size": { - "default": "32768", - "description": "Maximum size of any measurement." - }, - "line.tcp.writer.worker.count": { - "default": "", - "description": "Number of dedicated I/O worker threads assigned to write data to tables. When `0`, the writer jobs will use the shared pool." - }, - "line.tcp.writer.worker.affinity": { - "default": "", - "description": "Comma-separated list of thread numbers which should be pinned for line protocol ingestion over TCP. CPU core indexes are 0-based." - }, - "line.tcp.writer.worker.sleep.threshold": { - "default": "1000", - "description": "Amount of subsequent loop iterations with no work done before the worker goes to sleep." - }, - "line.tcp.writer.worker.yield.threshold": { - "default": "10", - "description": "Amount of subsequent loop iterations with no work done before the worker thread yields." - }, - "line.tcp.writer.queue.capacity": { - "default": "128", - "description": "Size of the queue between the IO jobs and the writer jobs, each queue entry represents a measurement." - }, - "line.tcp.writer.halt.on.error": { - "default": "false", - "description": "Flag that indicates if the worker thread must stop when an unexpected error occurs." - }, - "line.tcp.io.worker.count": { - "default": "", - "description": "Number of dedicated I/O worker threads assigned to parse TCP input. When `0`, the writer jobs will use the shared pool." - }, - "line.tcp.io.worker.affinity": { - "default": "", - "description": "Comma-separated list of thread numbers which should be pinned for line protocol ingestion over TCP. CPU core indexes are 0-based." - }, - "line.tcp.io.worker.sleep.threshold": { - "default": "1000", - "description": "Amount of subsequent loop iterations with no work done before the worker goes to sleep." - }, - "line.tcp.io.worker.yield.threshold": { - "default": "10", - "description": "Amount of subsequent loop iterations with no work done before the worker thread yields." - }, - "line.tcp.disconnect.on.error": { - "default": "true", - "description": "Disconnect TCP socket that sends malformed messages." - }, - "line.tcp.acl.enabled": { - "default": "true", - "description": "Enable or disable Access Control List (ACL) authentication for InfluxDB Line Protocol over TCP. Enterprise only." - } -} diff --git a/documentation/configuration/configuration-utils/_telemetry.config.json b/documentation/configuration/configuration-utils/_telemetry.config.json deleted file mode 100644 index e8a951b9b..000000000 --- a/documentation/configuration/configuration-utils/_telemetry.config.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "telemetry.enabled": { - "default": "true", - "description": "Enable or disable anonymous usage metrics collection." - }, - "telemetry.hide.tables": { - "default": "false", - "description": "Hides telemetry tables from `select * from tables()` output. As a result, telemetry tables will not be visible in the Web Console table view." - }, - "telemetry.queue.capacity": { - "default": "512", - "description": "Capacity of the internal telemetry queue, which is the gateway of all telemetry events. This queue capacity does not require tweaking." - } -} diff --git a/documentation/configuration/configuration-utils/_udp.config.json b/documentation/configuration/configuration-utils/_udp.config.json deleted file mode 100644 index 59ffeb30d..000000000 --- a/documentation/configuration/configuration-utils/_udp.config.json +++ /dev/null @@ -1,50 +0,0 @@ -{ - "line.udp.join": { - "default": "232.1.2.3", - "description": "Multicast address receiver joins. This values is ignored when receiver is in \"unicast\" mode." - }, - "line.udp.bind.to": { - "default": "0.0.0.0:9009", - "description": "IP address of the network interface to bind listener to and port. By default UDP receiver listens on all network interfaces." - }, - "line.udp.commit.rate": { - "default": "1000000", - "description": "For packet bursts the number of continuously received messages after which receiver will force commit. Receiver will commit irrespective of this parameter when there are no messages." - }, - "line.udp.msg.buffer.size": { - "default": "2048", - "description": "Buffer used to receive single message. This value should be roughly equal to your MTU size." - }, - "line.udp.msg.count": { - "default": "10000", - "description": "Only for Linux. On Linux, QuestDB will use the `recvmmsg()` system call. This is the max number of messages to receive at once." - }, - "line.udp.receive.buffer.size": { - "default": "8388608", - "description": "UDP socket buffer size. Larger size of the buffer will help reduce message loss during bursts." - }, - "line.udp.enabled": { - "default": "false", - "description": "Enable or disable UDP receiver." - }, - "line.udp.own.thread": { - "default": "false", - "description": "When `true`, UDP receiver will use its own thread and busy spin that for performance reasons. \"false\" makes receiver use worker threads that do everything else in QuestDB." - }, - "line.udp.own.thread.affinity": { - "default": "-1", - "description": "-1 does not set thread affinity. OS will schedule thread and it will be liable to run on random cores and jump between the. 0 or higher pins thread to give core. This property is only valid when UDP receiver uses own thread." - }, - "line.udp.unicast": { - "default": "false", - "description": "When `true`, UDP will use unicast. Otherwise multicast." - }, - "line.udp.timestamp": { - "default": "n", - "description": "Input timestamp resolution. Possible values are `n`, `u`, `ms`, `s` and `h`." - }, - "line.udp.commit.mode": { - "default": "nosync", - "description": "Commit durability. Available values are `nosync`, `sync` and `async`." - } -} diff --git a/documentation/configuration/configuration-utils/_wal.config.json b/documentation/configuration/configuration-utils/_wal.config.json deleted file mode 100644 index c4eb75536..000000000 --- a/documentation/configuration/configuration-utils/_wal.config.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "wal.apply.worker.count": { - "default": "equal to the CPU core count", - "description": "Number of dedicated worker threads assigned to handle WAL table data." - }, - "wal.apply.worker.affinity": { - "default": "equal to the CPU core count", - "description": "Comma separated list of CPU core indexes." - }, - "wal.apply.worker.haltOnError": { - "default": "false", - "description": "Flag that indicates if the worker thread must stop when an unexpected error occurs." - }, - "cairo.wal.purge.interval": { - "default": "30000", - "description": "Period in ms of how often WAL-applied files are cleaned up from the disk" - }, - "cairo.wal.segment.rollover.row.count": { - "default": "200000", - "description": "Row count of how many rows are written to the same WAL segment before starting a new segment. Triggers in conjunction with `cairo.wal.segment.rollover.size` (whichever is first)." - }, - "cairo.wal.squash.uncommitted.rows.multiplier": { - "default": "20.0", - "description": "Multiplier to cairo.max.uncommitted.rows to calculate the limit of rows that can be kept invisible when writing to WAL table under heavy load, when multiple transactions are to be applied. It is used to reduce the number Out-Of-Order (O3) commits when O3 commits are unavoidable by squashing multiple commits together. Setting it very low can increase O3 commit frequency and decrease the throughput. Setting it too high may cause excessive memory usage and increase the latency." - }, - "cairo.wal.max.lag.txn.count": { - "default": "20", - "description": "Maximum number of transactions that can be kept invisible when writing to WAL table. Once the number is reached, full commit occurs. If not set, defaults to the rounded value of cairo.wal.squash.uncommitted.rows.multiplier." - }, - "cairo.wal.apply.parallel.sql.enabled": { - "default": "true", - "description": "When disabled, SQL executed by the WAL apply job will always run single-threaded." - } -} diff --git a/documentation/configuration/copy-settings.md b/documentation/configuration/copy-settings.md new file mode 100644 index 000000000..8a74f781e --- /dev/null +++ b/documentation/configuration/copy-settings.md @@ -0,0 +1,130 @@ +--- +title: COPY settings +description: Configuration settings for CSV import and Parquet export in QuestDB. +--- + +These settings control the `COPY` SQL statement for importing CSV files and +exporting Parquet files. Import and export use separate root directories to +isolate source files from output. + +## Import + +Settings for `COPY FROM` (CSV import): + +### cairo.iouring.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable or disable the io_uring implementation. Applicable to newer Linux +kernels only. Can be used to switch io_uring off if a kernel bug affects it. + +### cairo.sql.copy.buffer.size + +- **Default**: `2 MiB` +- **Reloadable**: no + +Size of read buffers used during import. + +### cairo.sql.copy.log.retention.days + +- **Default**: `3` +- **Reloadable**: no + +Number of days to keep import messages in `sys.text_import_log`. + +### cairo.sql.copy.max.index.chunk.size + +- **Default**: `100M` +- **Reloadable**: no + +Maximum size of an index chunk file, used to limit total memory during import. +The indexing phase uses roughly `thread_count * cairo.sql.copy.max.index.chunk.size` +of memory. + +### cairo.sql.copy.queue.capacity + +- **Default**: `32` +- **Reloadable**: no + +Size of the copy task queue. Increase if there are more than 32 import workers. + +### cairo.sql.copy.root + +- **Default**: `import` +- **Reloadable**: no + +Input root directory for CSV imports via `COPY` SQL and for Parquet file +reading. This path must not overlap with other directories (e.g. `db`, `conf`) +of the running instance, otherwise import may delete or overwrite existing +files. Relative paths are resolved against the server root directory. + +For QuestDB instances using Docker, `cairo.sql.copy.root` must be defined +using either the environment variable `QDB_CAIRO_SQL_COPY_ROOT` or the +`cairo.sql.copy.root` property in `server.conf`. The mounted source path and +the configured copy root must be identical. It is optional to define +`QDB_CAIRO_SQL_COPY_WORK_ROOT`. + +Example Docker command: + +```shell +docker run -p 9000:9000 \ +-v "/tmp/questdb:/var/lib/questdb" \ +-v "/tmp/questdb/my_input_root:/var/lib/questdb/questdb_import" \ +-e QDB_CAIRO_SQL_COPY_ROOT=/var/lib/questdb/questdb_import \ +questdb/questdb +``` + +Where: + +- `-v "/tmp/questdb/my_input_root:/var/lib/questdb/questdb_import"`: Defining a + source CSV file location to be `/tmp/questdb/my_input_root` on local machine + and mounting it to `/var/lib/questdb/questdb_import` in the container. +- `-e QDB_CAIRO_SQL_COPY_ROOT=/var/lib/questdb/questdb_import`: Defining the + copy root directory to be `/var/lib/questdb/questdb_import`. + +It is important that the two paths are identical +(`/var/lib/questdb/questdb_import` in the example). + +### cairo.sql.copy.work.root + +- **Default**: `null` +- **Reloadable**: no + +Temporary import file directory. Defaults to `root_directory/tmp` if not set +explicitly. + +## Export + +Settings for `COPY TO` (Parquet export): + +### cairo.parquet.export.bloom.filter.fpp + +- **Default**: `0.01` +- **Reloadable**: no + +Default bloom filter false positive probability (FPP) for Parquet exports via +`COPY TO` and REST `/exp`. Lower values produce larger but more accurate +filters. Range: 0.0 to 1.0. + +### cairo.parquet.export.statistics.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables or disables generation of column statistics (min/max, null count) in +Parquet exports. + +### cairo.sql.copy.export.root + +- **Default**: `export` +- **Reloadable**: no + +Root directory for Parquet exports via `COPY TO` SQL. This path must not +overlap with other directories (e.g. `db`, `conf`) of the running instance, +otherwise export may delete or overwrite existing files. Relative paths are +resolved against the server root directory. + +Parquet export is also impacted by the general Parquet encoding parameters in +the [Cairo engine](/docs/configuration/cairo-engine/#parquet-encoding) +configuration. diff --git a/documentation/configuration/database-replication.md b/documentation/configuration/database-replication.md new file mode 100644 index 000000000..4fcccc240 --- /dev/null +++ b/documentation/configuration/database-replication.md @@ -0,0 +1,326 @@ +--- +title: Replication +description: Configuration settings for database replication in QuestDB Enterprise. +--- + +:::note + +Replication is [Enterprise](/enterprise/) only. + +::: + +Replication enables high availability clusters by streaming WAL segments to an +object store, where replica instances pick them up. These settings control +the primary's upload behavior, replica polling, object store request handling, +and WAL cleanup. + +For setup instructions, see the +[replication operations](/docs/high-availability/setup/) guide. + +For an overview of the concept, see the +[replication concept](/docs/high-availability/overview/) page. + +For a tuning guide, see the +[replication tuning guide](/docs/high-availability/tuning/). + +## General + +### replication.object.store + +- **Default**: none +- **Reloadable**: no + +A configuration string for connecting to an object store. The format is +`scheme::key1=value;key2=value2;…`. Ignored if replication is disabled. + +### replication.role + +- **Default**: `none` +- **Reloadable**: no + +Defaults to `none` for stand-alone instances. To enable replication, set to +one of: `primary`, `replica`. + +### replication.summary.interval + +- **Default**: `1m` +- **Reloadable**: no + +Frequency for printing replication progress summary in the logs. + +## Primary settings + +### cairo.wal.segment.rollover.size + +- **Default**: `2097152` +- **Reloadable**: no + +The size of the WAL segment before it is rolled over. Default is 2 MiB. +Defaults to `0` unless `replication.role=primary` is set. + +### cairo.writer.command.queue.capacity + +- **Default**: `32` +- **Reloadable**: no + +Maximum writer ALTER TABLE and replication command capacity. Shared between +all tables. + +### replication.primary.checksum + +- **Default**: service-dependent +- **Reloadable**: no + +Whether a checksum should be calculated for each uploaded artifact. Required +for some object stores. Options: `never`, `always`. + +### replication.primary.compression.level + +- **Default**: `1` +- **Reloadable**: no + +Zstd compression level. Valid values are from 1 to 22. + +### replication.primary.compression.threads + +- **Default**: calculated (half the number of CPU cores) +- **Reloadable**: no + +Maximum number of threads used to perform file compression operations before +uploading to the object store. + +### replication.primary.sequencer.part.txn.count + +- **Default**: `5000` +- **Reloadable**: no + +Sets the transaction chunking size for each compressed batch. Smaller is +better for constrained networks but more costly. + +### replication.primary.throttle.window.duration + +- **Default**: `10000` +- **Reloadable**: no + +The millisecond duration of the sliding window used to process replication +batches. + +### replication.primary.upload.truncated + +- **Default**: `true` +- **Reloadable**: no + +Skip trailing, empty column data inside a WAL column file. + +## Replica settings + +### replication.replica.poll.interval + +- **Default**: `1000` +- **Reloadable**: no + +Millisecond polling rate for a replica instance to check for new changes. + +## Request settings + +### replication.requests.base.timeout + +- **Default**: `10s` +- **Reloadable**: no + +Replication upload/download request timeout. + +### replication.requests.buffer.size + +- **Default**: `32768` +- **Reloadable**: no + +Buffer size used for object-storage downloads. + +### replication.requests.max.batch.size.fast + +- **Default**: `64` +- **Reloadable**: no + +Number of parallel requests allowed during the fast process (non-resource +constrained). + +### replication.requests.max.batch.size.slow + +- **Default**: `2` +- **Reloadable**: no + +Number of parallel requests allowed during the slow process (error/resource +constrained path). + +### replication.requests.max.concurrent + +- **Default**: `0` +- **Reloadable**: no + +Limit on the number of concurrent object store requests. `0` means unlimited. + +### replication.requests.min.throughput + +- **Default**: `262144` +- **Reloadable**: no + +Expected minimum network speed for replication transfers. Used to expand the +timeout and account for network delays. + +### replication.requests.retry.attempts + +- **Default**: `3` +- **Reloadable**: no + +Maximum number of times to retry a failed object store request before logging +an error and reattempting later after a delay. + +### replication.requests.retry.interval + +- **Default**: `200` +- **Reloadable**: no + +How long to wait in milliseconds before retrying a failed operation. + +## Metrics + +### replication.metrics.dropped.table.poll.count + +- **Default**: `10` +- **Reloadable**: no + +How many scrapes of the Prometheus metrics endpoint before dropped tables +will no longer appear. + +### replication.metrics.per.table + +- **Default**: `true` +- **Reloadable**: no + +Enable per-table replication metrics on the Prometheus metrics endpoint. + +## WAL cleaner + +The WAL cleaner removes obsolete WAL data from object storage after it has +been consumed by all replicas and retained for the configured backup window. + +### replication.primary.cleaner.backup.window.count + +- **Default**: `backup.cleanup.keep.latest.n` or `5` +- **Reloadable**: no + +Minimum complete backups/checkpoints per instance before cleanup starts. +Defaults to `backup.cleanup.keep.latest.n` if backups are enabled, otherwise +`5`. + +### replication.primary.cleaner.checkpoint.source + +- **Default**: `true` +- **Reloadable**: no + +Use checkpoint history as a cleanup trigger source. + +### replication.primary.cleaner.delete.concurrency + +- **Default**: 4 to 12 (auto) +- **Reloadable**: no + +Concurrent deletion tasks. Derived from `replication.requests.max.concurrent`. +Range: 4 to 32. + +### replication.primary.cleaner.dropped.table.cooloff + +- **Default**: `1h` +- **Reloadable**: no + +Wait time after `DROP TABLE` before removing the table's data from object +storage. Guards against clock skew. + +### replication.primary.cleaner.enabled + +- **Default**: `true` +- **Reloadable**: no + +Master switch for the WAL cleaner. + +### replication.primary.cleaner.interval + +- **Default**: `10m` +- **Reloadable**: no + +Time between cleanup cycles. Range: 1s to 24h. + +### replication.primary.cleaner.max.requests.per.second + +- **Default**: service-dependent +- **Reloadable**: no + +Rate limit for object store delete requests. Set to `0` for unlimited. +Range: 0 to 10000. + +### replication.primary.cleaner.progress.write.interval + +- **Default**: `5s` +- **Reloadable**: no + +How often progress is persisted during a cleanup cycle. Lower values mean +less re-work after a crash but more writes. Range: 100ms to 60s. + +### replication.primary.cleaner.retry.attempts + +- **Default**: `20` +- **Reloadable**: no + +Retries for transient object store failures during cleanup. Range: 0 to 100. + +### replication.primary.cleaner.retry.interval + +- **Default**: `2s` +- **Reloadable**: no + +Delay between cleanup retries. Range: 0 to 5m. + +## Checkpoint history + +### checkpoint.history.enabled + +- **Default**: `true` (when replication is enabled) +- **Reloadable**: no + +Enable the checkpoint history tracker. Requires replication. + +### checkpoint.history.keep.count + +- **Default**: `100` +- **Reloadable**: no + +Maximum checkpoint records retained per instance. + +### checkpoint.history.long.retry.interval + +- **Default**: `1m` +- **Reloadable**: no + +Retry interval for syncing checkpoint history to the object store after burst +retries fail. + +## Native I/O threads + +### native.async.io.threads + +- **Default**: CPU count +- **Reloadable**: no + +The number of async (network) I/O threads used for replication and cold +storage. The default should be appropriate for most use cases. + +### native.max.blocking.threads + +- **Default**: CPU count x 4 +- **Reloadable**: no + +Maximum number of threads for parallel blocking disk I/O read/write +operations for replication. These threads are ephemeral: spawned as needed +and shut down after a short idle period. They are not CPU-bound, hence the +relatively large count. The default should be appropriate for most use cases. diff --git a/documentation/configuration/http-min-server.md b/documentation/configuration/http-min-server.md new file mode 100644 index 000000000..30772f28d --- /dev/null +++ b/documentation/configuration/http-min-server.md @@ -0,0 +1,70 @@ +--- +title: Minimal HTTP server +description: Configuration settings for the minimal HTTP server in QuestDB. +--- + +The minimal HTTP server provides the health check and Prometheus metrics +endpoints, running on a separate port (default 9003) from the main HTTP server. +This lightweight server remains responsive even when the main server is under +heavy load. + +## http.min.bind.to + +- **Default**: `0.0.0.0:9003` +- **Reloadable**: no + +IPv4 address and port of the server. `0.0.0.0` binds to all network +interfaces, otherwise the IP address must be one of the existing network +adapters. + +## http.min.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable or disable the minimal HTTP server. + +## http.min.net.connection.hint + +- **Default**: `false` +- **Reloadable**: no + +Windows-specific flag to overcome OS limitations on TCP backlog size. + +## http.min.net.connection.limit + +- **Default**: `4` +- **Reloadable**: no + +Active connection limit. + +## http.min.net.connection.timeout + +- **Default**: `300000` +- **Reloadable**: no + +Idle connection timeout in milliseconds. + +## http.min.worker.affinity + +- **Default**: none +- **Reloadable**: no + +Core number to pin the worker thread to. + +## http.min.worker.count + +- **Default**: auto +- **Reloadable**: no + +By default, the minimal HTTP server uses the shared thread pool for CPU core +counts of 16 and below, and a dedicated thread for counts above 16. When `0`, +the server uses the shared pool. Do not set the pool size to more than `1`. + +## http.min.worker.haltOnError + +- **Default**: `false` +- **Reloadable**: no + +Flag that indicates if the worker thread must stop when an unexpected error +occurs. diff --git a/documentation/configuration/http-server.md b/documentation/configuration/http-server.md new file mode 100644 index 000000000..65b14ef09 --- /dev/null +++ b/documentation/configuration/http-server.md @@ -0,0 +1,555 @@ +--- +title: HTTP server +description: Configuration settings for the Web Console and REST API in QuestDB. +--- + +The HTTP server hosts the [Web Console](/docs/getting-started/web-console/overview/), +the REST API, and ILP-over-HTTP ingestion, all on the same port (default 9000). +These settings control networking, authentication, query result formatting, +text import parsing, and context paths. + +## Server + +### http.bind.to + +- **Default**: `0.0.0.0:9000` +- **Reloadable**: no + +IP address and port of the HTTP server. `0.0.0.0` binds to all network +interfaces. You can specify the IP address of any individual network interface. + +### http.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable or disable the HTTP server. + +### http.frozen.clock + +- **Default**: `false` +- **Reloadable**: no + +Sets the clock to always return zero. Used for internal testing. + +### http.server.keep.alive + +- **Default**: `true` +- **Reloadable**: no + +If set to `false`, the server will disconnect the client after completion of +each request. + +### http.version + +- **Default**: `HTTP/1.1` +- **Reloadable**: no + +Protocol version. Other supported value is `HTTP/1.0`. + +### http.worker.affinity + +- **Default**: none +- **Reloadable**: no + +Comma-separated list of CPU core indexes. The number of items must equal the +worker count. + +### http.worker.count + +- **Default**: `0` +- **Reloadable**: no + +Number of threads in the private HTTP worker pool. When `0`, the HTTP server +uses the shared worker pool. Values above `0` enable a private pool. + +### http.worker.haltOnError + +- **Default**: `false` +- **Reloadable**: no + +Flag that indicates if the worker thread must stop when an unexpected error +occurs. Changing the default value is strongly discouraged. + +## Authentication + +### http.password + +- **Default**: N/A +- **Reloadable**: no + +Password for HTTP Basic Authentication in QuestDB Open Source. QuestDB +Enterprise supports more advanced authentication via RBAC. + +### http.user + +- **Default**: N/A +- **Reloadable**: no + +Username for HTTP Basic Authentication in QuestDB Open Source. QuestDB +Enterprise supports more advanced authentication via RBAC. + +## Connections + +### http.connection.pool.initial.capacity + +- **Default**: `4` +- **Reloadable**: no + +Initial size of the pool of reusable objects that hold connection state. +Should be configured to the maximum realistic load to avoid runtime resizes. + +### http.connection.string.pool.capacity + +- **Default**: `128` +- **Reloadable**: no + +Initial size of the string pool shared by the HTTP header and multipart +content parsers. + +### http.ilp.connection.limit + +- **Default**: none +- **Reloadable**: no + +Soft limit for simultaneous ILP connections. When breached, new connections +are rejected but existing connections remain open as long as +`http.net.connection.limit` is not exceeded. + +### http.keep-alive.max + +- **Default**: `10000` +- **Reloadable**: no + +See `http.keep-alive.timeout`. Must be `0` when `http.version` is set to +`HTTP/1.0`. + +### http.keep-alive.timeout + +- **Default**: `5` +- **Reloadable**: no + +Used together with `http.keep-alive.max` to set the value of the HTTP +`Keep-Alive` response header, instructing the browser to keep the TCP +connection open. Must be `0` when `http.version` is set to `HTTP/1.0`. + +### http.net.bind.to + +- **Default**: `0.0.0.0:9000` +- **Reloadable**: no + +IP address and port of the HTTP server. + +### http.net.connection.hint + +- **Default**: `false` +- **Reloadable**: no + +Windows-specific flag to overcome OS limitations on TCP backlog size. + +### http.net.connection.limit + +- **Default**: `64` +- **Reloadable**: no + +Maximum number of simultaneous TCP connections to the HTTP server. Controls +server memory consumption. + +### http.net.connection.queue.timeout + +- **Default**: `5000` +- **Reloadable**: no + +Time in milliseconds a connection can wait in the listen backlog queue before +it is refused. Connections are aggressively removed from the backlog until the +active connection limit is breached. + +### http.net.connection.rcvbuf + +- **Default**: `2M` +- **Reloadable**: no + +Maximum receive buffer size on each TCP socket. If set to `-1`, the socket +receive buffer size remains unchanged from OS defaults. + +### http.net.connection.sndbuf + +- **Default**: `2M` +- **Reloadable**: no + +Maximum send buffer size on each TCP socket. If set to `-1`, the socket send +buffer size remains unchanged from OS defaults. + +### http.net.connection.timeout + +- **Default**: `300000` +- **Reloadable**: no + +TCP connection idle timeout in milliseconds. The connection is closed by the +HTTP server when this timeout lapses. + +### http.query.connection.limit + +- **Default**: none +- **Reloadable**: no + +Soft limit for simultaneous HTTP query connections. When breached, new +connections are rejected but existing connections remain open as long as +`http.net.connection.limit` is not exceeded. + +## Buffers + +### http.allow.deflate.before.send + +- **Default**: `false` +- **Reloadable**: no + +Flag that indicates if Gzip compression of outgoing data is allowed. + +### http.multipart.header.buffer.size + +- **Default**: `512` +- **Reloadable**: yes + +Buffer size in bytes used by the HTTP multipart content parser. + +### http.multipart.idle.spin.count + +- **Default**: `10000` +- **Reloadable**: no + +How long the code accumulates incoming data chunks for column and delimiter +analysis. + +### http.receive.buffer.size + +- **Default**: `1M` +- **Reloadable**: yes + +Size of the receive buffer. + +### http.request.header.buffer.size + +- **Default**: `64K` +- **Reloadable**: yes + +Size of the internal buffer allocated for HTTP request headers. The value is +rounded up to the nearest power of 2. When HTTP requests contain headers that +exceed the buffer size, the server will disconnect the client with an HTTP +error in the server log. + +### http.send.buffer.size + +- **Default**: `2M` +- **Reloadable**: yes + +Size of the internal send buffer. Larger buffers result in fewer I/O +interruptions at the expense of memory usage per connection. 2 MB is the +optimal value for most workloads. + +## Security + +### circuit.breaker.buffer.size + +- **Default**: `32` +- **Reloadable**: no + +Size of buffer to read from the HTTP connection. If this buffer returns zero +and the HTTP client is no longer sending data, SQL processing will be +terminated. + +### circuit.breaker.throttle + +- **Default**: `2000000` +- **Reloadable**: no + +Number of internal iterations (such as loops over data) before checking if +the HTTP connection is still open. + +### http.pessimistic.health.check.enabled + +- **Default**: `false` +- **Reloadable**: no + +When enabled, the health check returns HTTP 500 for any unhandled errors +since the server started. + +### http.security.interrupt.on.closed.connection + +- **Default**: `true` +- **Reloadable**: no + +Enables termination of SQL processing if the HTTP connection is closed. The +connection is checked after `circuit.breaker.throttle` iterations. The +mechanism also reads from the input stream and discards data, since some HTTP +clients send keep-alive data between requests. `circuit.breaker.buffer.size` +controls the buffer size for this. + +### http.security.max.response.rows + +- **Default**: `2^63-1` +- **Reloadable**: no + +Limit the number of response rows over HTTP. + +### http.security.readonly + +- **Default**: `false` +- **Reloadable**: no + +Forces HTTP read-only mode when `true`, disabling commands which modify data +or data structure (e.g. INSERT, UPDATE, CREATE TABLE). + +## Query cache + +### http.query.cache.block.count + +- **Default**: `4` +- **Reloadable**: no + +Number of blocks for the query cache. + +### http.query.cache.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable or disable the query cache. Cache capacity is +`number_of_blocks * number_of_rows`. + +### http.query.cache.row.count + +- **Default**: `16` +- **Reloadable**: no + +Number of rows for the query cache. + +## JSON query output + +### http.json.query.connection.check.frequency + +- **Default**: `1000000` +- **Reloadable**: no + +The value to throttle checking if the client socket has been disconnected. +Changing the default value is strongly discouraged. + +### http.json.query.double.scale + +- **Default**: `12` +- **Reloadable**: no + +The scale of string representation of `DOUBLE` values. + +### http.json.query.float.scale + +- **Default**: `4` +- **Reloadable**: no + +The scale of string representation of `FLOAT` values. + +## Text import parsing + +These settings control the heuristic text parser used for CSV import via the +`/imp` endpoint. + +### http.text.analysis.max.lines + +- **Default**: `1000` +- **Reloadable**: no + +Number of lines to read on CSV import for heuristics which determine column +names and types. Lower values detect schemas quicker but possibly with less +accuracy. 1000 is the maximum. + +### http.text.date.adapter.pool.capacity + +- **Default**: `16` +- **Reloadable**: no + +Size of the date adapter pool. Should be set to the anticipated maximum +number of `DATE` fields a text input can have. + +### http.text.json.cache.limit + +- **Default**: `16384` +- **Reloadable**: no + +JSON parser cache limit. The cache is used to compose JSON elements broken up +by TCP. This value limits the maximum length of individual tags or tag values. + +### http.text.json.cache.size + +- **Default**: `8192` +- **Reloadable**: no + +Initial size of the JSON parser cache. Must not exceed +`http.text.json.cache.limit`. Should be set to avoid runtime resizes. + +### http.text.lexer.string.pool.capacity + +- **Default**: `64` +- **Reloadable**: no + +Initial capacity of the string pool which wraps `STRING` column types in text +input. Should correspond to the maximum anticipated number of STRING columns. + +### http.text.max.required.delimiter.stddev + +- **Default**: `0.1222d` +- **Reloadable**: no + +Maximum standard deviation for the algorithm that calculates text file +delimiters. When the parser cannot recognise a delimiter, it logs the +calculated and maximum standard deviation for the delimiter candidate. + +### http.text.max.required.line.length.stddev + +- **Default**: `0.8` +- **Reloadable**: no + +Maximum standard deviation for the algorithm that classifies input as text +or binary. Values above this threshold cause input to be considered binary. + +### http.text.metadata.string.pool.capacity + +- **Default**: `128` +- **Reloadable**: no + +Initial size of the pool for objects that wrap individual elements of metadata +JSON, such as column names, date pattern strings, and locale values. + +### http.text.roll.buffer.limit + +- **Default**: `4M` +- **Reloadable**: no + +The limit of the text roll buffer. See `http.text.roll.buffer.size`. + +### http.text.roll.buffer.size + +- **Default**: `1024` +- **Reloadable**: no + +The roll buffer holds a copy of a line that has been broken up by TCP. Should +be set to the maximum length of a text line in the input. + +### http.text.timestamp.adapter.pool.capacity + +- **Default**: `64` +- **Reloadable**: no + +Size of the timestamp adapter pool. Should be set to the anticipated maximum +number of `TIMESTAMP` fields a text input can have. + +### http.text.utf8.sink.size + +- **Default**: `4096` +- **Reloadable**: no + +Initial size of the UTF-8 adapter sink. Should correspond to the maximum +individual field value length in text input. + +## Context paths + +### http.context.execute + +- **Default**: `/exec` +- **Reloadable**: no + +Context path for the SQL execution service. + +### http.context.export + +- **Default**: `/exp` +- **Reloadable**: no + +Context path for the SQL result CSV export service. + +### http.context.ilp + +- **Default**: `/write,/api/v2/write` +- **Reloadable**: no + +Context paths for the InfluxDB Line Protocol (ILP) HTTP services. Not used +by the Web Console. + +### http.context.ilp.ping + +- **Default**: `/ping` +- **Reloadable**: no + +Context path for the ILP ping endpoint. + +### http.context.import + +- **Default**: `/imp` +- **Reloadable**: no + +Context path for the file import service. + +### http.context.settings + +- **Default**: `/settings` +- **Reloadable**: no + +Context path for the service which provides server-side settings to the Web +Console. + +### http.context.table.status + +- **Default**: `/chk` +- **Reloadable**: no + +Context path for the table status service used by the Import UI in the Web +Console. + +### http.context.warnings + +- **Default**: `/warnings` +- **Reloadable**: no + +Context path for the Web Console warnings service. + +### http.context.web.console + +- **Default**: `/` +- **Reloadable**: no + +Context path for the Web Console. If other REST services remain on the +default context paths, they will move to the same context path as the Web +Console. ILP HTTP services are not affected. When default context paths are +changed, QuestDB creates copies of services on the Web Console paths so that +both the Web Console and custom services remain operational. + +## Redirects + +### http.redirect.1 + +- **Default**: `/ -> /index.html` +- **Reloadable**: no + +Redirect configuration. Format is `source -> destination`. + +### http.redirect.count + +- **Default**: `1` +- **Reloadable**: no + +Number of HTTP redirects. All redirects are 301 (Moved Permanently). + +## Static content + +### http.static.index.file.name + +- **Default**: `index.html` +- **Reloadable**: no + +Name of the index file for the Web Console. + +### http.static.public.directory + +- **Default**: `public` +- **Reloadable**: no + +The name of the directory for the public web site. diff --git a/documentation/configuration/iam.md b/documentation/configuration/iam.md new file mode 100644 index 000000000..7829c0498 --- /dev/null +++ b/documentation/configuration/iam.md @@ -0,0 +1,88 @@ +--- +title: Identity and Access Management (IAM) +sidebar_label: Identity and Access Management +description: Configuration settings for Identity and Access Management in QuestDB Enterprise. +--- + +:::note + +Identity and Access Management is available within +[QuestDB Enterprise](/enterprise/). + +::: + +Identity and Access Management (IAM) controls authentication and authorization +for all QuestDB interfaces. These settings cover the built-in admin user, +password hashing, and REST token behavior. + +For a full explanation of IAM, see the +[Identity and Access Management (IAM) documentation](/docs/security/rbac). + +### acl.admin.password + +- **Default**: `quest` +- **Reloadable**: yes + +The password of the built-in admin user. + +### acl.admin.user + +- **Default**: `admin` +- **Reloadable**: no + +Name of the built-in admin user. + +### acl.admin.user.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables or disables the built-in admin user. + +### acl.basic.auth.realm.enabled + +- **Default**: `false` +- **Reloadable**: no + +When enabled, the browser's basic auth popup window is used instead of the +Web Console's login screen. Present for backwards compatibility only. + +### acl.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables or disables Identity and Access Management. + +### acl.entity.name.max.length + +- **Default**: `255` +- **Reloadable**: no + +Maximum length of user, group, and service account names. + +### acl.password.hash.iteration.count + +- **Default**: `100000` +- **Reloadable**: no + +Number of hash iterations used in password hashing. QuestDB Enterprise never +stores passwords in plain text. Higher values are safer but slower. This +should almost never be changed. + +### acl.rest.token.refresh.threshold + +- **Default**: `10` +- **Reloadable**: no + +When a REST token is created in REFRESH mode, its TTL is extended on every +successful authentication, unless the last authentication was within this +threshold (in seconds). Removes unnecessary overhead of continuously +refreshing tokens that are used frequently. + +### line.tcp.acl.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables or disables authentication for the ILP over TCP endpoint only. diff --git a/documentation/configuration/ingestion.md b/documentation/configuration/ingestion.md new file mode 100644 index 000000000..d0b3fabcb --- /dev/null +++ b/documentation/configuration/ingestion.md @@ -0,0 +1,378 @@ +--- +title: Ingestion (ILP/HTTP) +description: Configuration settings for InfluxDB Line Protocol ingestion in QuestDB. +--- + +These settings control data ingestion via InfluxDB Line Protocol (ILP). ILP +over HTTP is the preferred ingestion method. ILP over TCP is available for +legacy workloads. UDP is deprecated. + +For general HTTP server settings (port, buffers, connections), see the +[HTTP server](/docs/configuration/http-server/) configuration. + +## General + +### line.auto.create.new.columns + +- **Default**: `true` +- **Reloadable**: no + +When enabled, automatically creates new columns when they appear in the +ingested data. When disabled, messages with new columns will be rejected. + +### line.auto.create.new.tables + +- **Default**: `true` +- **Reloadable**: no + +When enabled, automatically creates new tables when they appear in the +ingested data. When disabled, messages for non-existent tables will be +rejected. + +### line.default.partition.by + +- **Default**: `DAY` +- **Reloadable**: no + +Table partition strategy for tables created automatically by ILP. Possible +values are: `HOUR`, `DAY`, `WEEK`, `MONTH`, and `YEAR`. + +### line.log.message.on.error + +- **Default**: `true` +- **Reloadable**: no + +Controls whether malformed ILP messages are printed to the server log when +errors occur. + +## HTTP + +ILP over HTTP is the preferred way of ingesting data. + +### line.http.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable ILP over HTTP. Default port is 9000. Enabled by default in open source +versions, defaults to false and must be explicitly enabled for Enterprise. + +### line.http.ping.version + +- **Default**: `v2.2.2` +- **Reloadable**: no + +Version information for the ping response of ILP over HTTP. + +## TCP + +### line.tcp.auth.db.path + +- **Default**: none +- **Reloadable**: no + +Path to the authentication database file. + +### line.tcp.commit.interval.default + +- **Default**: `1000` +- **Reloadable**: no + +Default commit interval in milliseconds. + +### line.tcp.commit.interval.fraction + +- **Default**: `0.5` +- **Reloadable**: no + +Commit lag fraction. Used to calculate the commit interval for a table: +`commit_interval = commit_lag * fraction`. The calculated interval defines +how long uncommitted data remains uncommitted. + +### line.tcp.connection.pool.capacity + +- **Default**: `64` +- **Reloadable**: no + +Maximum number of pooled connections for this interface. + +### line.tcp.disconnect.on.error + +- **Default**: `true` +- **Reloadable**: no + +Disconnect TCP sockets that send malformed messages. + +### line.tcp.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable or disable ILP over TCP. + +### line.tcp.io.worker.affinity + +- **Default**: none +- **Reloadable**: no + +Comma-separated list of CPU core indexes to pin I/O worker threads to. +Core indexes are 0-based. + +### line.tcp.io.worker.count + +- **Default**: `0` +- **Reloadable**: no + +Number of dedicated I/O worker threads for parsing TCP input. When `0`, +uses the shared worker pool. + +### line.tcp.io.worker.sleep.threshold + +- **Default**: `1000` +- **Reloadable**: no + +Number of consecutive idle loop iterations before the I/O worker goes to +sleep. + +### line.tcp.io.worker.yield.threshold + +- **Default**: `10` +- **Reloadable**: no + +Number of consecutive idle loop iterations before the I/O worker thread +yields. + +### line.tcp.maintenance.job.interval + +- **Default**: `1000` +- **Reloadable**: no + +Maximum time in milliseconds between maintenance jobs committing uncommitted +data on inactive tables. + +### line.tcp.max.measurement.size + +- **Default**: `32768` +- **Reloadable**: no + +Maximum size of any measurement. + +### line.tcp.min.idle.ms.before.writer.release + +- **Default**: `500` +- **Reloadable**: no + +Minimum idle time in milliseconds before a table writer is released. + +### line.tcp.msg.buffer.size + +- **Default**: `32768` +- **Reloadable**: no + +Size of the buffer read from the queue. Maximum size of a write request, +regardless of the number of measurements. + +### line.tcp.net.bind.to + +- **Default**: `0.0.0.0:9009` +- **Reloadable**: no + +IP address and port for the TCP listener. By default, listens on all network +interfaces. + +### line.tcp.net.connection.hint + +- **Default**: `false` +- **Reloadable**: no + +Windows-specific flag to overcome OS limitations on TCP backlog size. + +### line.tcp.net.connection.limit + +- **Default**: `256` +- **Reloadable**: yes + +Maximum number of simultaneous connections. Controls server memory +consumption. + +### line.tcp.net.connection.queue.timeout + +- **Default**: `5000` +- **Reloadable**: no + +Time in milliseconds a connection can wait in the listen backlog queue before +it is refused. Connections are aggressively removed from the backlog until +the active connection limit is breached. + +### line.tcp.net.connection.rcvbuf + +- **Default**: `-1` +- **Reloadable**: no + +Maximum receive buffer size on each TCP socket. If set to `-1`, the socket +receive buffer remains unchanged from OS defaults. + +### line.tcp.net.connection.timeout + +- **Default**: `300000` +- **Reloadable**: no + +Connection idle timeout in milliseconds. Connections are closed by the server +when this timeout lapses. + +### line.tcp.timestamp + +- **Default**: `n` +- **Reloadable**: no + +Input timestamp resolution. Possible values are `n`, `u`, `ms`, `s`, and `h`. + +### line.tcp.writer.halt.on.error + +- **Default**: `false` +- **Reloadable**: no + +Flag that indicates if the writer thread must stop when an unexpected error +occurs. + +### line.tcp.writer.queue.capacity + +- **Default**: `128` +- **Reloadable**: no + +Size of the queue between I/O jobs and writer jobs. Each queue entry +represents a measurement. + +### line.tcp.writer.worker.affinity + +- **Default**: none +- **Reloadable**: no + +Comma-separated list of CPU core indexes to pin writer worker threads to. +Core indexes are 0-based. + +### line.tcp.writer.worker.count + +- **Default**: `0` +- **Reloadable**: no + +Number of dedicated I/O worker threads for writing data to tables. When `0`, +uses the shared worker pool. + +### line.tcp.writer.worker.sleep.threshold + +- **Default**: `1000` +- **Reloadable**: no + +Number of consecutive idle loop iterations before the writer worker goes to +sleep. + +### line.tcp.writer.worker.yield.threshold + +- **Default**: `10` +- **Reloadable**: no + +Number of consecutive idle loop iterations before the writer worker thread +yields. + +## UDP (deprecated) + +:::note + +The UDP receiver is deprecated since QuestDB version 6.5.2. We recommend +[ILP over HTTP](/docs/ingestion/ilp/overview/) instead. + +::: + +### line.udp.bind.to + +- **Default**: `0.0.0.0:9009` +- **Reloadable**: no + +IP address and port for the UDP listener. By default, listens on all network +interfaces. + +### line.udp.commit.mode + +- **Default**: `nosync` +- **Reloadable**: no + +Commit durability. Available values are `nosync`, `sync`, and `async`. + +### line.udp.commit.rate + +- **Default**: `1000000` +- **Reloadable**: no + +For packet bursts, the number of continuously received messages after which +the receiver will force a commit. The receiver commits regardless of this +parameter when there are no messages. + +### line.udp.enabled + +- **Default**: `false` +- **Reloadable**: no + +Enable or disable the UDP receiver. + +### line.udp.join + +- **Default**: `232.1.2.3` +- **Reloadable**: no + +Multicast address the receiver joins. Ignored when the receiver is in +unicast mode. + +### line.udp.msg.buffer.size + +- **Default**: `2048` +- **Reloadable**: no + +Buffer used to receive a single message. Should be roughly equal to your MTU +size. + +### line.udp.msg.count + +- **Default**: `10000` +- **Reloadable**: no + +Linux only. Maximum number of messages to receive at once via the +`recvmmsg()` system call. + +### line.udp.own.thread + +- **Default**: `false` +- **Reloadable**: no + +When `true`, the UDP receiver uses its own thread with busy spin for +performance. When `false`, the receiver uses shared worker threads. + +### line.udp.own.thread.affinity + +- **Default**: `-1` +- **Reloadable**: no + +CPU core to pin the receiver thread to. `-1` means no affinity (OS +schedules the thread). Only valid when `line.udp.own.thread` is `true`. + +### line.udp.receive.buffer.size + +- **Default**: `8388608` +- **Reloadable**: no + +UDP socket buffer size. Larger buffers help reduce message loss during +bursts. + +### line.udp.timestamp + +- **Default**: `n` +- **Reloadable**: no + +Input timestamp resolution. Possible values are `n`, `u`, `ms`, `s`, and `h`. + +### line.udp.unicast + +- **Default**: `false` +- **Reloadable**: no + +When `true`, UDP uses unicast. Otherwise multicast. diff --git a/documentation/configuration/logging-metrics.md b/documentation/configuration/logging-metrics.md new file mode 100644 index 000000000..b67cb22d7 --- /dev/null +++ b/documentation/configuration/logging-metrics.md @@ -0,0 +1,27 @@ +--- +title: Logging & Metrics +description: Configuration settings for logging and metrics in QuestDB. +--- + +These settings control log output format and timezone. Additional log +configuration is available in `log.conf`. + +For more information and details of Prometheus metrics, see the +[Logging & Metrics](/docs/operations/logging-metrics/) documentation. + +## log.level.verbose + +- **Default**: `false` +- **Reloadable**: no + +Converts short-hand log level indicators (`E`, `C`, `I`) into long-hand +(`ERROR`, `CRITICAL`, `INFO`). + +## log.timezone + +- **Default**: `UTC` +- **Reloadable**: no + +Sets the timezone for log timestamps. Can be a timezone ID such as +`Antarctica/McMurdo`, `SystemDefault` to use the system timezone, or the +default UTC with `Z` suffix. diff --git a/documentation/configuration/materialized-views.md b/documentation/configuration/materialized-views.md new file mode 100644 index 000000000..49f73b12c --- /dev/null +++ b/documentation/configuration/materialized-views.md @@ -0,0 +1,46 @@ +--- +title: Materialized views +description: Configuration settings for materialized views in QuestDB. +--- + +These settings control materialized view SQL support and the background refresh +job. Materialized views can use dedicated worker threads or share the server's +common pool. + +## cairo.mat.view.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables or disables SQL support and the refresh job for materialized views. + +## cairo.mat.view.parallel.sql.enabled + +- **Default**: `true` +- **Reloadable**: no + +When disabled, SQL executed by the materialized view refresh job always runs +single-threaded. + +## mat.view.refresh.worker.affinity + +- **Default**: equal to the CPU core count +- **Reloadable**: no + +Comma-separated list of numerical CPU core indexes. + +## mat.view.refresh.worker.count + +- **Default**: `0` +- **Reloadable**: no + +Number of dedicated worker threads assigned to refresh materialized views. +When `0`, uses the shared worker pool. + +## mat.view.refresh.worker.haltOnError + +- **Default**: `false` +- **Reloadable**: no + +Flag that indicates if the worker thread must stop when an unexpected error +occurs. diff --git a/documentation/configuration/oidc.md b/documentation/configuration/oidc.md new file mode 100644 index 000000000..e0c5932bd --- /dev/null +++ b/documentation/configuration/oidc.md @@ -0,0 +1,232 @@ +--- +title: OpenID Connect (OIDC) +description: Configuration settings for OpenID Connect integration in QuestDB Enterprise. +--- + +:::note + +OpenID Connect is [Enterprise](/enterprise/) only. + +::: + +OpenID Connect (OIDC) support is part of QuestDB's Identity and Access +Management. The database can be integrated with any OAuth2/OIDC Identity +Provider (IdP). + +For detailed information about OIDC, see the +[OpenID Connect (OIDC) integration guide](/docs/security/oidc). + +## General + +### acl.oidc.audience + +- **Default**: none (defaults to the client ID) +- **Reloadable**: no + +OAuth2 audience as set on the tokens issued by the OIDC Provider. Defaults +to the client ID if not set. + +### acl.oidc.client.id + +- **Default**: none +- **Reloadable**: no + +Client name assigned to QuestDB in the OIDC server. Required when OIDC is +enabled. + +### acl.oidc.configuration.url + +- **Default**: none +- **Reloadable**: no + +URL where the OpenID Provider's configuration information can be loaded in +JSON format. Should always end with `/.well-known/openid-configuration`. + +### acl.oidc.enabled + +- **Default**: `false` +- **Reloadable**: no + +Enables or disables OIDC authentication. When enabled, several other +configuration options must also be set. + +### acl.oidc.host + +- **Default**: none +- **Reloadable**: no + +OIDC provider hostname. Required when OIDC is enabled, unless the OIDC +configuration URL is set. + +### acl.oidc.http.timeout + +- **Default**: `30000` +- **Reloadable**: no + +OIDC provider HTTP request timeout in milliseconds. + +### acl.oidc.port + +- **Default**: `443` +- **Reloadable**: no + +OIDC provider port number. + +### acl.oidc.redirect.uri + +- **Default**: none +- **Reloadable**: no + +The redirect URI tells the OIDC server where to redirect the user after +successful authentication. If not set, the Web Console defaults it to the +location where it was loaded from (`window.location.href`). + +### acl.oidc.scope + +- **Default**: `openid` +- **Reloadable**: no + +The OIDC server asks consent for the scopes listed in this property. The +scope `openid` is mandatory and must always be included. + +## Authentication flows + +### acl.oidc.pg.token.as.password.enabled + +- **Default**: `false` +- **Reloadable**: no + +When enabled, the PGWire endpoint supports OIDC authentication. The OAuth2 +token should be sent in the password field, while the username field should +contain the string `_sso`, or left empty if that is an option. + +### acl.oidc.pkce.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables or disables PKCE for the Authorization Code Flow. This should always +be enabled in production. The Web Console is not fully secure without it. + +### acl.oidc.ropc.flow.enabled + +- **Default**: `false` +- **Reloadable**: no + +Enables or disables the Resource Owner Password Credentials flow. When +enabled, this flow must also be configured in the OIDC Provider. + +## Endpoints + +### acl.oidc.authorization.endpoint + +- **Default**: `/as/authorization.oauth2` +- **Reloadable**: no + +OIDC Authorization Endpoint. The default value should work for the Ping +Identity Platform. + +### acl.oidc.public.keys.endpoint + +- **Default**: `/pf/JWKS` +- **Reloadable**: no + +JSON Web Key Set (JWKS) Endpoint. Provides the list of public keys used to +decode and validate ID tokens issued by the OIDC Provider. The default value +should work for the Ping Identity Platform. + +### acl.oidc.token.endpoint + +- **Default**: `/as/token.oauth2` +- **Reloadable**: no + +OIDC Token Endpoint. The default value should work for the Ping Identity +Platform. + +### acl.oidc.userinfo.endpoint + +- **Default**: `/idp/userinfo.openid` +- **Reloadable**: no + +OIDC User Info Endpoint. Used to retrieve additional user information +containing group memberships. The default value should work for the Ping +Identity Platform. + +## TLS + +These settings control TLS between QuestDB and the OIDC provider. For general +TLS encryption across QuestDB interfaces, see the +[TLS configuration](/docs/configuration/tls/). + +### acl.oidc.tls.enabled + +- **Default**: `true` +- **Reloadable**: no + +Whether the OIDC provider requires a secure connection. If the OpenID +Provider endpoints do not require TLS, this can be set to `false`. This is +unlikely in production. + +### acl.oidc.tls.keystore.password + +- **Default**: none +- **Reloadable**: no + +Keystore password. Required if a keystore file is configured and is password +protected. + +### acl.oidc.tls.keystore.path + +- **Default**: none +- **Reloadable**: no + +Path to a keystore file containing trusted Certificate Authorities. Used when +validating the certificate of the OIDC provider. Not required if the +provider's certificate is signed by a public CA. + +### acl.oidc.tls.validation.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables or disables TLS certificate validation. Disable this if working with +self-signed certificates. Validation is strongly recommended in production. +QuestDB checks that the certificate is valid and issued for the server to +which it connects. + +## User and group claims + +### acl.oidc.cache.ttl + +- **Default**: `30000` +- **Reloadable**: no + +User info cache entry TTL in milliseconds. QuestDB caches user info responses +for each valid access token. This setting controls how often the access token +is validated and user info refreshed. + +### acl.oidc.groups.claim + +- **Default**: `groups` +- **Reloadable**: no + +The name of the custom claim in the user information that contains the +group memberships of the user. + +### acl.oidc.groups.encoded.in.token + +- **Default**: `false` +- **Reloadable**: no + +When `true`, QuestDB looks for group memberships in the ID token instead of +calling the User Info endpoint. Set to `true` if the OIDC Provider encodes +group memberships directly into the token. + +### acl.oidc.sub.claim + +- **Default**: `sub` +- **Reloadable**: no + +The name of the claim in the user information that contains the user's name. +Could be a username, full name, or email. Displayed in the Web Console and +logged for audit purposes. diff --git a/documentation/configuration/overview.md b/documentation/configuration/overview.md index a21a9081a..5d2aa018a 100644 --- a/documentation/configuration/overview.md +++ b/documentation/configuration/overview.md @@ -3,33 +3,18 @@ title: Configuration description: Server configuration keys reference documentation. --- -import { ConfigTable } from "@theme/ConfigTable" -import sharedWorkerConfig from "./configuration-utils/\_shared-worker.config.json" -import httpConfig from "./configuration-utils/\_http.config.json" -import cairoConfig from "./configuration-utils/\_cairo.config.json" -import parallelSqlConfig from "./configuration-utils/\_parallel-sql.config.json" -import walConfig from "./configuration-utils/\_wal.config.json" -import csvImportConfig from "./configuration-utils/\_csv-import.config.json" -import parquetExportConfig from "./configuration-utils/\_parquet-export.config.json" -import postgresConfig from "./configuration-utils/\_postgres.config.json" -import tcpConfig from "./configuration-utils/\_tcp.config.json" -import udpConfig from "./configuration-utils/\_udp.config.json" -import replicationConfig from "./configuration-utils/\_replication.config.json" -import iamConfig from "./configuration-utils/\_iam.config.json" -import oidcConfig from "./configuration-utils/\_oidc.config.json" -import logConfig from "./configuration-utils/\_log.config.json" -import matViewConfig from "./configuration-utils/\_mat-view.config.json" -import storagePolicyConfig from "./configuration-utils/\_storage-policy.config.json" -import configValidationConfig from "./configuration-utils/\_config-validation.config.json" -import telemetryConfig from "./configuration-utils/\_telemetry.config.json" +import Tabs from "@theme/Tabs" +import TabItem from "@theme/TabItem" This page describes methods for configuring QuestDB server settings. -Configuration can be set either: +Configuration can be set using: -- In the `server.conf` configuration file available in the +- The `server.conf` configuration file available in the [root directory](/docs/concepts/deep-dive/root-directory-structure/) -- Using environment variables +- Environment variables +- [Command-line options](#command-line-options) for startup behavior such as + root directory, service tags, and web console redeployment When a key is absent from both the config file and the environment variables, the default value is used. @@ -217,298 +202,344 @@ All reloadable properties can be also queried from the server: (SHOW PARAMETERS) WHERE reloadable = true; ``` -## Keys and default values +## Command-line options -This section lists the configuration keys available to QuestDB by topic or -subsystem. Parameters for specifying buffer and memory page sizes are provided -in the format `n`, where `` can be one of the following: +QuestDB may be started, stopped and passed configuration options from the +command line. On Windows, the QuestDB server can also start an +[interactive session](#interactive-session-windows). -- `m` for **MB** -- `k` for **kB** +### Options -For example: +The following sections describe the options that may be passed to QuestDB when +starting the server from the command line. -```ini title="Setting maximum send buffer size to 2MB per TCP socket" -http.net.connection.sndbuf=2m + + + + + + + + +```shell +./questdb.sh [start|stop|status] [-d dir] [-f] [-n] [-t tag] ``` -### Shared worker + -QuestDB uses three specialized worker pools to handle different workloads: -- **Network pool**: handles HTTP, PostgreSQL, and ILP server I/O -- **Query pool**: executes parallel query operations (filters, group-by) -- **Write pool**: manages WAL apply jobs, table writes, materialized view refresh, and housekeeping tasks + - +```shell +questdb [start|stop|status] [-d dir] [-f] [-n] [-t tag] +``` + + -### HTTP server + + +```shell +questdb.exe [start|stop|status|install|remove] \ + [-d dir] [-f] [-j JAVA_HOME] [-t tag] +``` -This section describes configuration settings for the -[Web Console](/docs/getting-started/web-console/overview/) and the REST API available by default on port -`9000`. For details on the use of this component, refer to the -[web console documentation](/docs/getting-started/web-console/overview/) page. + - + -### Cairo engine +#### Start -This section describes configuration settings for the Cairo SQL engine in -QuestDB. +`start` - starts QuestDB as a service. - +| Option | Description | +| ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `-d` | Expects a `dir` directory value which is a folder that will be used as QuestDB's root directory. For more information and the default values, see the [default root](#default-root-directory) section below. | +| `-t` | Expects a `tag` string value which will be as a tag for the service. This option allows users to run several QuestDB services and manage them separately. If this option is omitted, the default tag will be `questdb`. | +| `-f` | Force re-deploying the [Web Console](/docs/getting-started/web-console/overview/). Without this option, the [Web Console](/docs/getting-started/web-console/overview/) is cached and deployed only when missing. | +| `-n` | Do not respond to the HUP signal. This keeps QuestDB alive after you close the terminal window where you started it. | +| `-j` | **Windows only!** This option allows to specify a path to `JAVA_HOME`. | -### WAL table configurations +:::note -The following WAL tables settings on parallel threads are configurable for -applying WAL data to the table storage: +- When running multiple QuestDB services, a tag must be used to disambiguate + between services for `start` and `stop` commands. There will be conflicting + ports and root directories if only the tag flag is specified when starting + multiple services. Each new service should have its own config file or should + be started with separate port and root directory options. - +- When running QuestDB as Windows service you can check status in both: + - Windows Event Viewer - look for events with "QuestDB" source in Windows Logs + | Application . + - service log file - `$dataDir\log\service-%Y-%m-%dT%H-%M-%S.txt` (default is + `C:\Windows\System32\qdbroot\log\service-%Y-%m-%dT%H-%M-%S.txt` ) -### COPY settings +::: -#### Import + -This section describes configuration settings for using `COPY` to import large -CSV files, or export parquet files. -Settings for `COPY FROM` (import): + - + -**CSV import configuration for Docker** + -For QuestDB instances using Docker: +```shell +./questdb.sh start [-d dir] [-f] [-n] [-t tag] +``` -- `cairo.sql.copy.root` must be defined using one of the following settings: - - The environment variable `QDB_CAIRO_SQL_COPY_ROOT`. - - The `cairo.sql.copy.root` in `server.conf`. -- The path for the source CSV file is mounted. -- The source CSV file path and the path defined by `QDB_CAIRO_SQL_COPY_ROOT` are - identical. -- It is optional to define `QDB_CAIRO_SQL_COPY_WORK_ROOT`. + -The following is an example command to start a QuestDB instance on Docker, in -order to import a CSV file: + ```shell -docker run -p 9000:9000 \ --v "/tmp/questdb:/var/lib/questdb" \ --v "/tmp/questdb/my_input_root:/var/lib/questdb/questdb_import" \ --e QDB_CAIRO_SQL_COPY_ROOT=/var/lib/questdb/questdb_import \ -questdb/questdb +questdb start [-d dir] [-f] [-n] [-t tag] ``` -Where: + -- `-v "/tmp/questdb/my_input_root:/var/lib/questdb/questdb_import"`: Defining a - source CSV file location to be `/tmp/questdb/my_input_root` on local machine - and mounting it to `/var/lib/questdb/questdb_import` in the container. -- `-e QDB_CAIRO_SQL_COPY_ROOT=/var/lib/questdb/questdb_import`: Defining the - copy root directory to be `/var/lib/questdb/questdb_import`. + -It is important that the two path are identical -(`/var/lib/questdb/questdb_import` in the example). +```shell +questdb.exe start [-d dir] [-f] [-j JAVA_HOME] [-t tag] +``` + -#### Export + - +##### Default root directory -Parquet export is also generally impacted by query execution and parquet conversion parameters. +By default, QuestDB's [root directory](/docs/concepts/deep-dive/root-directory-structure/) +will be the following: -If not overridden, the following default setting will be used. + - + -### Parallel SQL execution + -This section describes settings that can affect the level of parallelism during -SQL execution, and therefore can also have an impact on performance. + - +```shell +$HOME/.questdb +``` -### Postgres wire protocol + -This section describes configuration settings for client connections using -PostgresSQL wire protocol. + - +Path on Macs with Apple Silicon (M1 or M2) chip: -### InfluxDB Line Protocol (ILP) +```shell +/opt/homebrew/var/questdb +``` -This section describes ingestion settings for incoming messages using InfluxDB -Line Protocol. +Path on Macs with Intel chip: -| Property | Default | Description | -| ---------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| line.default.partition.by | DAY | Table partition strategy to be used with tables that are created automatically by InfluxDB Line Protocol. Possible values are: `HOUR`, `DAY`, `WEEK`, `MONTH`, and `YEAR`. | -| line.auto.create.new.columns | true | When enabled, automatically creates new columns when they appear in the ingested data. When disabled, messages with new columns will be rejected. | -| line.auto.create.new.tables | true | When enabled, automatically creates new tables when they appear in the ingested data. When disabled, messages for non-existent tables will be rejected. | -| line.log.message.on.error | true | Controls whether malformed ILP messages are printed to the server log when errors occur. | +```shell +/usr/local/var/questdb +``` -#### HTTP specific settings + -ILP over HTTP is the preferred way of ingesting data. + -| Property | Default | Description | -| ---------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -| line.http.enabled | true | Enable ILP over HTTP. Default port is 9000. Enabled by default within open source versions, defaults to false and must be enabled for Enterprise. | -| line.http.ping.version | v2.2.2 | Version information for the ping response of ILP over HTTP. | -| HTTP properties | Various | See [HTTP settings](/docs/configuration/overview/#http-server) for general HTTP configuration. ILP over HTTP inherits from HTTP settings. | +```shell +C:\Windows\System32\qdbroot +``` -#### TCP specific settings + - + -#### UDP specific settings +#### Stop -:::note +`stop` - stops a service. -The UDP receiver is deprecated since QuestDB version 6.5.2. We recommend ILP -over HTTP instead, or less frequently -[ILP over TCP](/docs/ingestion/ilp/overview/). +| Option | Description | +| ------ | ------------------------------------------------------------------------------------------------------------------ | +| `-t` | Expects a `tag` string value which to stop a service by tag. If this is omitted, the default tag will be `questdb` | -::: + - + -### Database replication + -:::note + -Replication is [Enterprise](/enterprise/) only. +```shell +./questdb.sh stop +``` -::: + -Replication enables high availability clusters. + -For setup instructions, see the -[replication operations](/docs/high-availability/setup/) guide. +```shell +questdb stop +``` -For an overview of the concept, see the -[replication concept](/docs/high-availability/overview/) page. + -For a tuning guide see, the -[replication tuning guide](/docs/high-availability/tuning/). + - +```shell +questdb.exe stop +``` -### Identity and Access Management (IAM) + -:::note + -Identity and Access Management is available within -[QuestDB Enterprise](/enterprise/). +#### Status -::: +`status` - shows the status for a service. -Identity and Access Management (IAM) ensures that data can be accessed only by -authorized users. The below configuration properties relate to various -authentication and authorization features. +| Option | Description | +| ------ | -------------------------------------------------------------------------------------------------------------- | +| `-t` | Expects a `tag` string value which to stop a service by tag. If this is omitted, the default will be `questdb` | -For a full explanation of IAM, see the -[Identity and Access Management (IAM) documentation](/docs/security/rbac). + - + -### OpenID Connect (OIDC) + -:::note + -OpenID Connect is [Enterprise](/enterprise/) only. +```shell +./questdb.sh status +``` -::: + -OpenID Connect (OIDC) support is part of QuestDB's Identity and Access -Management. The database can be integrated with any OAuth2/OIDC Identity -Provider (IdP). + -For detailed information about OIDC, see the -[OpenID Connect (OIDC) integration guide](/docs/security/oidc). +```shell +questdb status +``` - + -### Config Validation + -The database startup phase checks for configuration issues, such as invalid or -deprecated settings. Issues may be classified as advisories or errors. Advisory -issues are [logged](/docs/concepts/deep-dive/root-directory-structure/#log-directory) -without causing the database to stop its startup sequence: These are usually -setting deprecation warnings. Configuration errors can optionally cause the -database to fail its startup. +```shell +questdb.exe status +``` - + -_We recommended enabling strict validation._ + -### Telemetry +#### Install (Windows) -QuestDB sends anonymous telemetry data with information about usage which helps -us improve the product over time. We do not collect any personally-identifying -information, and we do not share any of this data with third parties. +`install` - installs the Windows QuestDB service. The service will start +automatically at startup. - +```shell +questdb.exe install +``` -## Materialized views +#### Remove (Windows) -:::info +`remove` - removes the Windows QuestDB service. It will no longer start at +startup. -Materialized View support is now generally available (GA) and ready for production use. +```shell +questdb.exe remove +``` -If you are using versions earlier than `8.3.1`, we suggest you upgrade at your earliest convenience. +### Interactive session (Windows) -::: +You can start QuestDB interactively by running `questdb.exe`. This will launch +QuestDB interactively in the active `Shell` window. QuestDB will be stopped when +the Shell is closed. -The following settings are available in `server.conf`: +#### Default root directory - +When started interactively, QuestDB's root directory defaults to the `current` +directory. -## Storage policy +#### Stop -:::note +To stop, press Ctrl+C in the terminal or close it +directly. -Storage policy is [Enterprise](/enterprise/) only. +## Config validation -::: +The database startup phase checks for configuration issues, such as invalid or +deprecated settings. Issues may be classified as advisories or errors. Advisory +issues are [logged](/docs/concepts/deep-dive/root-directory-structure/#log-directory) +without causing the database to stop its startup sequence: These are usually +setting deprecation warnings. Configuration errors can optionally cause the +database to fail its startup. + +### config.validation.strict -Storage policies automate partition lifecycle management, including local -deletion and cold storage offloading. +- **Default**: `false` +- **Reloadable**: no -For details, see the -[storage policy concept](/docs/concepts/storage-policy/) page. +When enabled, startup fails if there are configuration errors. - +_We recommend enabling strict validation._ -## Logging & Metrics +## Configuration reference -The following settings are available in `server.conf`: +The full configuration reference is divided by subsystem. Each page lists +every available property with its default value, whether it can be reloaded +at runtime, and a description. - +Parameters for specifying buffer and memory page sizes use the format +`n`, where `` can be: + +- `m` for **MB** +- `k` for **kB** + +For example: + +```ini title="Setting maximum send buffer size to 2MB per TCP socket" +http.net.connection.sndbuf=2m +``` -Further settings are available in `log.conf`. For more information, and details -of our Prometheus metrics, please visit the -[Logging & Metrics](/docs/operations/logging-metrics/) documentation. +| Section | Description                                                                                  | Enterprise only | +|---------|-------------|:----------:| +| [Cairo engine](/docs/configuration/cairo-engine/) | SQL engine settings | | +| [COPY settings](/docs/configuration/copy-settings/) | CSV import and Parquet export | | +| [HTTP server](/docs/configuration/http-server/) | Web Console and REST API | | +| [IAM](/docs/configuration/iam/) | Identity and Access Management | ✓ | +| [Ingestion (ILP/HTTP)](/docs/configuration/ingestion/) | InfluxDB Line Protocol settings | | +| [Logging & Metrics](/docs/configuration/logging-metrics/) | Log levels and metrics | | +| [Materialized views](/docs/configuration/materialized-views/) | Materialized view refresh settings | | +| [Minimal HTTP server](/docs/configuration/http-min-server/) | Health check and metrics endpoint | | +| [OpenID Connect (OIDC)](/docs/configuration/oidc/) | OIDC integration | ✓ | +| [Parallel SQL execution](/docs/configuration/parallel-sql-execution/) | Query parallelism settings | | +| [Postgres wire protocol](/docs/configuration/postgres-wire-protocol/) | PostgreSQL wire protocol connections | | +| [Replication](/docs/configuration/database-replication/) | High availability cluster replication | ✓ | +| [Shared workers](/docs/configuration/shared-workers/) | Worker thread pools | | +| [Storage policy](/docs/configuration/storage-policy/) | Partition lifecycle management | ✓ | +| [Telemetry](/docs/configuration/telemetry/) | Anonymous usage telemetry | | +| [TLS encryption](/docs/configuration/tls/) | TLS settings for all interfaces | ✓ | +| [WAL table configurations](/docs/configuration/wal/) | Write-Ahead Log settings | | diff --git a/documentation/configuration/parallel-sql-execution.md b/documentation/configuration/parallel-sql-execution.md new file mode 100644 index 000000000..5fdeda744 --- /dev/null +++ b/documentation/configuration/parallel-sql-execution.md @@ -0,0 +1,57 @@ +--- +title: Parallel SQL execution +description: Configuration settings for parallel SQL execution in QuestDB. +--- + +These settings control the level of parallelism during SQL execution, affecting +both filter performance and memory usage. JIT compilation of filter expressions +requires parallel filtering to be enabled. + +## cairo.page.frame.column.list.capacity + +- **Default**: `16` +- **Reloadable**: no + +Column list capacity for each slot of the reduce queue. Used by JIT-compiled +filter functions. Larger values reduce memory allocation rate but increase +minimum RSS size. + +## cairo.page.frame.reduce.queue.capacity + +- **Default**: `64` +- **Reloadable**: no + +Reduce queue capacity for data processing. Should be large enough to supply +tasks for worker threads in the shared worker pool. + +## cairo.page.frame.rowid.list.capacity + +- **Default**: `256` +- **Reloadable**: no + +Row ID list initial capacity for each slot of the reduce queue. Larger values +reduce memory allocation rate but increase minimum RSS size. + +## cairo.page.frame.shard.count + +- **Default**: `4` +- **Reloadable**: no + +Number of shards for both dispatch and reduce queues. Shards reduce queue +contention between SQL statements executed concurrently. + +## cairo.sql.parallel.filter.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable or disable parallel SQL filter execution. JIT compilation only takes +place when this setting is enabled. + +## cairo.sql.parallel.filter.pretouch.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable column pre-touch as part of parallel SQL filter execution, to improve +query performance for large tables. diff --git a/documentation/configuration/postgres-wire-protocol.md b/documentation/configuration/postgres-wire-protocol.md new file mode 100644 index 000000000..129f91507 --- /dev/null +++ b/documentation/configuration/postgres-wire-protocol.md @@ -0,0 +1,298 @@ +--- +title: Postgres wire protocol +description: Configuration settings for PostgreSQL wire protocol connections in QuestDB. +--- + +These settings control client connections using the PostgreSQL wire protocol, +including networking, authentication, query caching, and worker threads. + +## Server + +### pg.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable or disable the Postgres wire protocol interface. + +### pg.worker.affinity + +- **Default**: none +- **Reloadable**: no + +Comma-separated list of CPU core indexes to pin worker threads to. Example: +`pg.worker.affinity=1,2,3`. + +### pg.worker.count + +- **Default**: `0` +- **Reloadable**: no + +Number of dedicated worker threads for PostgreSQL wire protocol queries. +When `0`, uses the shared worker pool. + +### pg.daemon.pool + +- **Default**: `true` +- **Reloadable**: no + +Whether to run all PostgreSQL wire protocol worker threads in daemon mode. + +### pg.halt.on.error + +- **Default**: `false` +- **Reloadable**: no + +Whether ingestion should stop upon internal error. + +## Authentication + +### pg.password + +- **Default**: `quest` +- **Reloadable**: yes + +Postgres database password. + +### pg.readonly.password + +- **Default**: `quest` +- **Reloadable**: yes + +Postgres database read-only user password. + +### pg.readonly.user + +- **Default**: `user` +- **Reloadable**: yes + +Postgres database read-only user username. + +### pg.readonly.user.enabled + +- **Default**: `false` +- **Reloadable**: yes + +Enable or disable the Postgres database read-only user account. When enabled, +this additional user can open read-only connections to the database. + +### pg.user + +- **Default**: `admin` +- **Reloadable**: yes + +Postgres database username. + +## Connections + +### pg.connection.pool.capacity + +- **Default**: `64` +- **Reloadable**: no + +Maximum number of pooled connections for this interface. + +### pg.net.bind.to + +- **Default**: `0.0.0.0:8812` +- **Reloadable**: no + +IP address and port for the Postgres wire protocol server. `0.0.0.0` binds +to all network interfaces. + +### pg.net.connection.hint + +- **Default**: `false` +- **Reloadable**: no + +Windows-specific flag to overcome OS limitations on TCP backlog size. + +### pg.net.connection.limit + +- **Default**: `64` +- **Reloadable**: yes + +Maximum number of simultaneous Postgres connections. Controls server memory +consumption. + +### pg.net.connection.queue.timeout + +- **Default**: `300000` +- **Reloadable**: no + +Time in milliseconds a connection can wait in the listen backlog queue before +it is refused. Connections are aggressively removed from the backlog until the +active connection limit is breached. + +### pg.net.connection.rcvbuf + +- **Default**: `-1` +- **Reloadable**: no + +Maximum send buffer size on each TCP socket. If set to `-1`, the socket +buffer remains unchanged from OS defaults. + +### pg.net.connection.sndbuf + +- **Default**: `-1` +- **Reloadable**: no + +Maximum receive buffer size on each TCP socket. If set to `-1`, the socket +buffer remains unchanged from OS defaults. + +### pg.net.connection.timeout + +- **Default**: `300000` +- **Reloadable**: no + +Connection idle timeout in milliseconds. Connections are closed by the server +when this timeout lapses. + +## Buffers + +### pg.binary.param.count.capacity + +- **Default**: `2` +- **Reloadable**: no + +Initial capacity for the pool used for binary bind variables. + +### pg.max.blob.size.on.query + +- **Default**: `512k` +- **Reloadable**: no + +For binary values, clients receive an error when requesting blob sizes above +this value. + +### pg.recv.buffer.size + +- **Default**: `1M` +- **Reloadable**: yes + +Size of the buffer for receiving data. + +### pg.send.buffer.size + +- **Default**: `1M` +- **Reloadable**: yes + +Size of the buffer for sending data. + +## Security + +### pg.security.readonly + +- **Default**: `false` +- **Reloadable**: no + +Forces PostgreSQL wire protocol read-only mode when `true`, disabling +commands which modify data or data structure (e.g. INSERT, UPDATE, CREATE +TABLE). + +## Query cache + +### pg.insert.cache.block.count + +- **Default**: `8` +- **Reloadable**: no + +Number of blocks to cache INSERT query execution plans. + +### pg.insert.cache.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable or disable the INSERT query cache. Cache capacity is +`number_of_blocks * number_of_rows`. + +### pg.insert.cache.row.count + +- **Default**: `8` +- **Reloadable**: no + +Number of rows to cache for INSERT query execution plans. + +### pg.named.statement.limit + +- **Default**: `64` +- **Reloadable**: yes + +Size of the named statement pool. + +### pg.select.cache.block.count + +- **Default**: `16` +- **Reloadable**: no + +Number of blocks to cache SELECT query execution plans. + +### pg.select.cache.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable or disable the SELECT query cache. Cache capacity is +`number_of_blocks * number_of_rows`. + +### pg.select.cache.row.count + +- **Default**: `16` +- **Reloadable**: no + +Number of rows to cache for SELECT query execution plans. + +### pg.update.cache.block.count + +- **Default**: `8` +- **Reloadable**: no + +Number of blocks to cache UPDATE query execution plans. + +### pg.update.cache.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable or disable the UPDATE query cache. Cache capacity is +`number_of_blocks * number_of_rows`. + +### pg.update.cache.row.count + +- **Default**: `8` +- **Reloadable**: no + +Number of rows to cache for UPDATE query execution plans. + +## Internal pools + +### pg.character.store.capacity + +- **Default**: `4096` +- **Reloadable**: no + +Size of the CharacterStore. + +### pg.character.store.pool.capacity + +- **Default**: `64` +- **Reloadable**: no + +Size of the CharacterStore pool capacity. + +## Locale + +### pg.date.locale + +- **Default**: `en` +- **Reloadable**: no + +The locale to handle date types. + +### pg.timestamp.locale + +- **Default**: `en` +- **Reloadable**: no + +The locale to handle timestamp types. diff --git a/documentation/configuration/shared-workers.md b/documentation/configuration/shared-workers.md new file mode 100644 index 000000000..e585b5765 --- /dev/null +++ b/documentation/configuration/shared-workers.md @@ -0,0 +1,91 @@ +--- +title: Shared workers +description: Configuration settings for QuestDB worker thread pools. +--- + +QuestDB distributes work across three specialized thread pools. Each pool can +be sized independently, letting you balance CPU resources between network I/O, +query execution, and write operations based on your workload profile. + +- **Network pool**: handles HTTP, PostgreSQL, and ILP server I/O +- **Query pool**: executes parallel query operations (filters, group-by) +- **Write pool**: manages WAL apply jobs, table writes, materialized view + refresh, and housekeeping tasks + +All three pools use the same default sizing formula, which scales with the +number of available CPUs: + +| CPU count | Default threads per pool | +|-----------|--------------------------| +| > 32 | `max(2, CPU count - 2)` | +| > 16 | `max(2, CPU count - 1)` | +| ≤ 16 | `max(2, CPU count)` | + +On a typical 8-core machine, each pool defaults to 8 threads. On a 64-core +server, each pool gets 62 threads. Adjust these values when you need to shift +CPU capacity toward a specific workload, for example reducing the network pool +and increasing the query pool on a read-heavy system. + +## shared.network.worker.affinity + +- **Default**: none +- **Reloadable**: no + +Comma-delimited list of CPU ids, one per thread specified in +`shared.network.worker.count`. By default, threads have no CPU affinity. +Pinning threads to specific cores can reduce context-switch overhead on +NUMA systems. + +## shared.network.worker.count + +- **Default**: see table above +- **Reloadable**: no + +Number of worker threads for the network pool, which handles HTTP, PostgreSQL, +and ILP server I/O. Increasing this value raises network I/O parallelism at +the expense of CPU resources available to queries and writes. + +## shared.query.worker.affinity + +- **Default**: none +- **Reloadable**: no + +Comma-delimited list of CPU ids, one per thread specified in +`shared.query.worker.count`. By default, threads have no CPU affinity. + +## shared.query.worker.count + +- **Default**: see table above +- **Reloadable**: no + +Number of worker threads for the query pool, which executes parallel query +operations such as filters and group-by. Increasing this value raises query +parallelism at the expense of CPU resources available to network I/O and +writes. + +## shared.worker.haltOnError + +- **Default**: `false` +- **Reloadable**: no + +When enabled, a worker thread stops if it encounters an unexpected error. +Intended for debugging. In production, leave this disabled so that transient +errors do not reduce the size of a thread pool. + +## shared.write.worker.affinity + +- **Default**: none +- **Reloadable**: no + +Comma-delimited list of CPU ids, one per thread specified in +`shared.write.worker.count`. By default, threads have no CPU affinity. + +## shared.write.worker.count + +- **Default**: see table above +- **Reloadable**: no + +Number of worker threads for the write pool, which manages WAL apply jobs, +table writes, materialized view refresh, and housekeeping tasks. Increasing +this value raises write parallelism at the expense of CPU resources available +to network I/O and queries. diff --git a/documentation/configuration/storage-policy.md b/documentation/configuration/storage-policy.md new file mode 100644 index 000000000..e97d19bd6 --- /dev/null +++ b/documentation/configuration/storage-policy.md @@ -0,0 +1,67 @@ +--- +title: Storage policy +description: Configuration settings for storage policies in QuestDB Enterprise. +--- + +:::note + +Storage policy is [Enterprise](/enterprise/) only. + +::: + +Storage policies automate partition lifecycle management, including local +deletion and cold storage offloading. These settings control the scan interval, +retry behavior, and worker threads for the storage policy engine. + +For details, see the +[storage policy concept](/docs/concepts/storage-policy/) page. + +## storage.policy.check.interval + +- **Default**: `15m` +- **Reloadable**: no + +How often QuestDB scans for partitions to process. + +## storage.policy.max.reschedule.count + +- **Default**: `20` +- **Reloadable**: no + +Maximum number of retries before abandoning a storage policy task. + +## storage.policy.retry.interval + +- **Default**: `1m` +- **Reloadable**: no + +Retry interval for failed storage policy tasks. + +## storage.policy.worker.affinity + +- **Default**: `-1` +- **Reloadable**: no + +CPU affinity for each storage policy worker thread (comma-separated list). +`-1` means no affinity. + +## storage.policy.worker.count + +- **Default**: `2` +- **Reloadable**: no + +Number of storage policy worker threads. Setting to `0` disables the feature. + +## storage.policy.worker.sleep.timeout + +- **Default**: `100ms` +- **Reloadable**: no + +Sleep duration when a storage policy worker has no tasks to process. + +## storage.policy.writer.wait.timeout + +- **Default**: `30s` +- **Reloadable**: no + +Timeout for acquiring the table writer during storage policy operations. diff --git a/documentation/configuration/telemetry.md b/documentation/configuration/telemetry.md new file mode 100644 index 000000000..537fea52c --- /dev/null +++ b/documentation/configuration/telemetry.md @@ -0,0 +1,31 @@ +--- +title: Telemetry +description: Configuration settings for anonymous telemetry in QuestDB. +--- + +QuestDB sends anonymous telemetry data with information about usage which helps +us improve the product over time. We do not collect any personally-identifying +information, and we do not share any of this data with third parties. + +## telemetry.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enable or disable anonymous usage metrics collection. + +## telemetry.hide.tables + +- **Default**: `false` +- **Reloadable**: no + +Hides telemetry tables from `select * from tables()` output. When enabled, +telemetry tables will not be visible in the Web Console table view. + +## telemetry.queue.capacity + +- **Default**: `512` +- **Reloadable**: no + +Capacity of the internal telemetry queue, which is the gateway of all +telemetry events. This queue capacity does not require tweaking. diff --git a/documentation/configuration/tls.md b/documentation/configuration/tls.md new file mode 100644 index 000000000..d9fe63f56 --- /dev/null +++ b/documentation/configuration/tls.md @@ -0,0 +1,116 @@ +--- +title: TLS encryption +description: Configuration settings for TLS encryption across QuestDB interfaces. +--- + +:::note + +TLS encryption is [Enterprise](/enterprise/) only. + +::: + +TLS can be enabled globally for all QuestDB interfaces, or selectively for +individual endpoints. Per-endpoint settings override the global configuration. + +For OIDC-specific TLS settings +(keystore, certificate validation), see the +[OIDC configuration](/docs/configuration/oidc/#tls). + +## Global + +### tls.cert.path + +- **Default**: none +- **Reloadable**: no + +Path to the certificate used for TLS encryption globally. The certificate +should be DER-encoded and saved in PEM format. + +### tls.enabled + +- **Default**: `false` +- **Reloadable**: no + +Enables or disables TLS encryption globally for all QuestDB interfaces. + +### tls.private.key.path + +- **Default**: none +- **Reloadable**: no + +Path to the private key used for TLS encryption globally. + +## HTTP server + +These settings override the global TLS settings for the HTTP server only. + +### http.tls.cert.path + +- **Default**: none +- **Reloadable**: no + +Path to the certificate used for TLS encryption for the HTTP server only. +The certificate should be DER-encoded and saved in PEM format. + +### http.tls.enabled + +- **Default**: `false` +- **Reloadable**: no + +Enables or disables TLS encryption for the HTTP server only. + +### http.tls.private.key.path + +- **Default**: none +- **Reloadable**: no + +Path to the private key used for TLS encryption for the HTTP server only. + +## Minimal HTTP server + +### http.min.tls.cert.path + +- **Default**: none +- **Reloadable**: no + +Path to the certificate used for TLS encryption for the minimal HTTP server +only. The certificate should be DER-encoded and saved in PEM format. + +### http.min.tls.enabled + +- **Default**: `false` +- **Reloadable**: no + +Enables or disables TLS encryption for the minimal HTTP server only. + +### http.min.tls.private.key.path + +- **Default**: none +- **Reloadable**: no + +Path to the private key used for TLS encryption for the minimal HTTP server +only. + +## ILP over TCP + +### line.tcp.tls.cert.path + +- **Default**: none +- **Reloadable**: no + +Path to the certificate used for TLS encryption for ILP over TCP only. The +certificate should be DER-encoded and saved in PEM format. + +### line.tcp.tls.enabled + +- **Default**: `false` +- **Reloadable**: no + +Enables or disables TLS encryption for ILP over TCP only. + +### line.tcp.tls.private.key.path + +- **Default**: none +- **Reloadable**: no + +Path to the private key used for TLS encryption for ILP over TCP only. diff --git a/documentation/configuration/wal.md b/documentation/configuration/wal.md new file mode 100644 index 000000000..61c8d0749 --- /dev/null +++ b/documentation/configuration/wal.md @@ -0,0 +1,75 @@ +--- +title: WAL table configurations +description: Configuration settings for WAL tables in QuestDB. +--- + +These settings control the Write-Ahead Log (WAL) subsystem, including parallel +apply threads, segment rollover, commit squashing, and cleanup of applied WAL +files. + +## cairo.wal.apply.parallel.sql.enabled + +- **Default**: `true` +- **Reloadable**: no + +When disabled, SQL executed by the WAL apply job always runs single-threaded. + +## cairo.wal.max.lag.txn.count + +- **Default**: `20` +- **Reloadable**: no + +Maximum number of transactions that can be kept invisible when writing to a +WAL table. Once reached, a full commit occurs. If not set, defaults to the +rounded value of `cairo.wal.squash.uncommitted.rows.multiplier`. + +## cairo.wal.purge.interval + +- **Default**: `30000` +- **Reloadable**: no + +Period in milliseconds of how often WAL-applied files are cleaned up from +disk. + +## cairo.wal.segment.rollover.row.count + +- **Default**: `200000` +- **Reloadable**: no + +Number of rows written to the same WAL segment before starting a new segment. +Triggers in conjunction with `cairo.wal.segment.rollover.size` (whichever +threshold is reached first). + +## cairo.wal.squash.uncommitted.rows.multiplier + +- **Default**: `20.0` +- **Reloadable**: no + +Multiplier applied to `cairo.max.uncommitted.rows` to calculate the limit of +rows that can be kept invisible when writing to a WAL table under heavy load, +when multiple transactions are being applied. This reduces the number of +out-of-order (O3) commits by squashing multiple commits together. Setting it +too low can increase O3 commit frequency and decrease throughput. Setting it +too high may cause excessive memory usage and increase latency. + +## wal.apply.worker.affinity + +- **Default**: equal to the CPU core count +- **Reloadable**: no + +Comma-separated list of CPU core indexes. + +## wal.apply.worker.count + +- **Default**: equal to the CPU core count +- **Reloadable**: no + +Number of dedicated worker threads assigned to handle WAL table data. + +## wal.apply.worker.haltOnError + +- **Default**: `false` +- **Reloadable**: no + +Flag that indicates if the worker thread must stop when an unexpected error +occurs. diff --git a/documentation/getting-started/capacity-planning.md b/documentation/getting-started/capacity-planning.md index dba569149..0257762a5 100644 --- a/documentation/getting-started/capacity-planning.md +++ b/documentation/getting-started/capacity-planning.md @@ -182,7 +182,7 @@ for other database processes to use. ### CPU cores By default, QuestDB tries to use all available CPU cores. -[The guide on shared worker configuration](/docs/configuration/overview/#shared-worker) +[The guide on shared worker configuration](/docs/configuration/shared-workers/) explains how to change the default settings. Assuming that the disk is not bottlenecked on IOPS, the throughput of read-only queries scales proportionally with the number of available cores. As a result, a machine with more cores will diff --git a/documentation/getting-started/quick-start-utils/_options-not-windows.partial.mdx b/documentation/getting-started/quick-start-utils/_options-not-windows.partial.mdx index 2578bc18d..585e0fbba 100644 --- a/documentation/getting-started/quick-start-utils/_options-not-windows.partial.mdx +++ b/documentation/getting-started/quick-start-utils/_options-not-windows.partial.mdx @@ -1,6 +1,6 @@ | Option | Description | | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `-d` | Expects a `dir` directory value which is a folder that will be used as QuestDB's root directory. For more information and the default values, see the [default root](/docs/configuration/command-line-options/#default-root-directory-1) section below. | +| `-d` | Expects a `dir` directory value which is a folder that will be used as QuestDB's root directory. For more information and the default values, see the [default root](/docs/configuration/overview/#default-root-directory) section below. | | `-t` | Expects a `tag` string value which will be as a tag for the service. This option allows users to run several QuestDB services and manage them separately. If this option is omitted, the default tag will be `questdb`. | | `-f` | Force re-deploying the [Web Console](/docs/getting-started/web-console/overview/). Without this option, the [Web Console](/docs/getting-started/web-console/overview/) is cached and deployed only when missing. | | `-n` | Do not respond to the HUP signal. This keeps QuestDB alive after you close the terminal window where you started it. | diff --git a/documentation/getting-started/quick-start-utils/_options-windows.partial.mdx b/documentation/getting-started/quick-start-utils/_options-windows.partial.mdx index 6641a704c..12d194e7e 100644 --- a/documentation/getting-started/quick-start-utils/_options-windows.partial.mdx +++ b/documentation/getting-started/quick-start-utils/_options-windows.partial.mdx @@ -2,7 +2,7 @@ | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `install` | Installs the Windows QuestDB service. The service will start automatically at startup. | | `remove` | Removes the Windows QuestDB service. It will no longer start at startup. | -| `-d` | Expects a `dir` directory value which is a folder that will be used as QuestDB's root directory. For more information and the default values, see the [default root](/docs/configuration/command-line-options/#default-root-directory-1) section below. | +| `-d` | Expects a `dir` directory value which is a folder that will be used as QuestDB's root directory. For more information and the default values, see the [default root](/docs/configuration/overview/#default-root-directory) section below. | | `-t` | Expects a `tag` string value which will be as a tag for the service. This option allows users to run several QuestDB services and manage them separately. If this option is omitted, the default tag will be `questdb`. | | `-f` | Force re-deploying the [Web Console](/docs/getting-started/web-console/overview/). Without this option, the [Web Console](/docs/getting-started/web-console/overview/) is cached and deployed only when missing. | | `-j` | **Windows only!** This option allows to specify a path to `JAVA_HOME`. | diff --git a/documentation/high-availability/setup.md b/documentation/high-availability/setup.md index e279346e6..7ec89a12b 100644 --- a/documentation/high-availability/setup.md +++ b/documentation/high-availability/setup.md @@ -6,11 +6,8 @@ description: storage, primary and replica nodes. --- -import { ConfigTable } from "@theme/ConfigTable" import { EnterpriseNote } from "@site/src/components/EnterpriseNote" -import replicationConfig from "../configuration/configuration-utils/\_replication.config.json" - This guide covers setting up primary-replica replication. @@ -185,7 +182,8 @@ export QDB_REPLICATION_OBJECT_STORE="azblob::..." ``` ::: - +For the full list of replication properties, see the +[Replication configuration](/docs/configuration/database-replication/) reference. For tuning options, see the [Tuning guide](/docs/high-availability/tuning/). diff --git a/documentation/ingestion/ilp/advanced-settings.md b/documentation/ingestion/ilp/advanced-settings.md index f02333696..0381fe575 100644 --- a/documentation/ingestion/ilp/advanced-settings.md +++ b/documentation/ingestion/ilp/advanced-settings.md @@ -386,5 +386,5 @@ things: ``` Refer to - [InfluxDB Line Protocol's configuration](/docs/configuration/overview/#influxdb-line-protocol-ilp) + [InfluxDB Line Protocol's configuration](/docs/configuration/ingestion/) documentation for more on these configuration settings. diff --git a/documentation/ingestion/ilp/overview.md b/documentation/ingestion/ilp/overview.md index e327b1bf2..a0db5c1dd 100644 --- a/documentation/ingestion/ilp/overview.md +++ b/documentation/ingestion/ilp/overview.md @@ -65,7 +65,7 @@ line.http.enabled=true ## Server-Side Configuration The HTTP receiver configuration can be completely customized using -[QuestDB configuration keys for ILP](/docs/configuration/overview/#influxdb-line-protocol-ilp). +[QuestDB configuration keys for ILP](/docs/configuration/ingestion/). Configure the thread pools, buffer and queue sizes, receiver IP address and port, load balancing, and more. diff --git a/documentation/ingestion/import-csv.md b/documentation/ingestion/import-csv.md index bb6216456..fad81f382 100644 --- a/documentation/ingestion/import-csv.md +++ b/documentation/ingestion/import-csv.md @@ -130,7 +130,7 @@ csvstack *.csv > singleFile.csv #### Configure `COPY` -- Enable `COPY` and [configure](/docs/configuration/overview/#copy-settings) the `COPY` +- Enable `COPY` and [configure](/docs/configuration/copy-settings/) the `COPY` directories to suit your server. - `cairo.sql.copy.root` must be set for `COPY` to work. diff --git a/documentation/ingestion/message-brokers/telegraf.md b/documentation/ingestion/message-brokers/telegraf.md index cd85acaf4..b05bf5d63 100644 --- a/documentation/ingestion/message-brokers/telegraf.md +++ b/documentation/ingestion/message-brokers/telegraf.md @@ -78,7 +78,7 @@ the inputs in more detail: ## Writing to QuestDB over HTTP QuestDB expects Influx Line Protocol messages over HTTP on port `9000`. To change -the default port, see the [HTTP server configuration](/docs/configuration/overview/#http-server) +the default port, see the [HTTP server configuration](/docs/configuration/http-server/) section of the server configuration page. Create a new file named `questdb.conf` in one of the locations Telegraf can diff --git a/documentation/operations/_http-minimal.config.json b/documentation/operations/_http-minimal.config.json deleted file mode 100644 index 7ab7d787c..000000000 --- a/documentation/operations/_http-minimal.config.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "http.min.enabled": { - "default": "true", - "description": "Enable or disable Minimal HTTP server." - }, - "http.min.bind.to": { - "default": "0.0.0.0:9003", - "description": "IPv4 address and port of the server. `0` means it will bind to all network interfaces, otherwise the IP address must be one of the existing network adapters." - }, - "http.min.net.connection.limit": { - "default": "4", - "description": "Active connection limit." - }, - "http.min.net.connection.timeout": { - "default": "300000", - "description": "Idle connection timeout in milliseconds." - }, - "http.min.net.connection.hint": { - "default": "false", - "description": "Windows specific flag to overcome OS limitations on TCP backlog size." - }, - "http.min.worker.count": { - "default": "", - "description": "By default, minimal HTTP server uses shared thread pool for CPU core count 16 and below. It will use dedicated thread for core count above 16. When `0`, the server will use the shared pool. Do not set pool size to more than `1`." - }, - "http.min.worker.affinity": { - "default": "", - "description": "Core number to pin thread to." - }, - "http.min.worker.haltOnError": { - "default": "false", - "description": "Flag that indicates if the worker thread must stop when an unexpected error occurs." - } -} diff --git a/documentation/operations/logging-metrics.md b/documentation/operations/logging-metrics.md index f22ba39bc..c37aa9f95 100644 --- a/documentation/operations/logging-metrics.md +++ b/documentation/operations/logging-metrics.md @@ -3,8 +3,6 @@ title: Logging and metrics description: Configure and understand QuestDB logging and metrics, including log levels, configuration options, and Prometheus integration. --- -import { ConfigTable } from "@theme/ConfigTable" -import httpMinimalConfig from "./_http-minimal.config.json" This page outlines logging in QuestDB. It covers how to configure logs via `log.conf` and expose metrics via Prometheus. @@ -202,9 +200,8 @@ and the server will respond with identical results, e.g.: curl -v http://127.0.0.1:9003/status ``` -The following configuration options can be set in your `server.conf`: - - +For configuration options, see the +[Minimal HTTP server](/docs/configuration/http-min-server/) reference. :::warning diff --git a/documentation/query/rest-api.md b/documentation/query/rest-api.md index 5ba3d3b5c..3d3ee40e0 100644 --- a/documentation/query/rest-api.md +++ b/documentation/query/rest-api.md @@ -112,7 +112,7 @@ The `/exec` entrypoint takes a SQL query and returns results as JSON. We can use this for quick SQL inserts too, but note that there's no support for parameterized queries that are necessary to avoid SQL injection issues. Prefer -[InfluxDB Line Protocol](/docs/configuration/overview/#influxdb-line-protocol-ilp) if you +[InfluxDB Line Protocol](/docs/configuration/ingestion/) if you need high-performance inserts. diff --git a/documentation/sidebars.js b/documentation/sidebars.js index bf99c48ad..ea45c1353 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -583,7 +583,23 @@ module.exports = { type: "doc", label: "Overview", }, - "configuration/command-line-options", + "configuration/cairo-engine", + "configuration/copy-settings", + "configuration/http-server", + "configuration/iam", + "configuration/ingestion", + "configuration/logging-metrics", + "configuration/materialized-views", + "configuration/http-min-server", + "configuration/oidc", + "configuration/parallel-sql-execution", + "configuration/postgres-wire-protocol", + "configuration/database-replication", + "configuration/shared-workers", + "configuration/storage-policy", + "configuration/telemetry", + "configuration/tls", + "configuration/wal", ], }, diff --git a/documentation/tutorials/influxdb-migration.md b/documentation/tutorials/influxdb-migration.md index c97fc0d27..1d087d406 100644 --- a/documentation/tutorials/influxdb-migration.md +++ b/documentation/tutorials/influxdb-migration.md @@ -250,7 +250,7 @@ Now that you know the basics of InfluxDB Line Protocol, you may want to dig deeper into customization options for ILP or HTTP, learn more about the protocol, or compare QuestDB with Influx: -- [ILP `server.conf` configuration](/docs/configuration/overview/#influxdb-line-protocol-ilp) +- [ILP `server.conf` configuration](/docs/configuration/ingestion/) - [ILP API Overview](/docs/ingestion/ilp/overview/) - [Comparing TimescaleDB and QuestDB performance and architecture blog](/blog/timescaledb-vs-questdb-comparison/) - [Deep dive of QuestDB vs InfluxDB internals](/blog/2024/02/26/questdb-versus-influxdb/)