From c994b67a8ddc37a2bbe506d757c574d9fcb13f39 Mon Sep 17 00:00:00 2001 From: "Theodor S. Midtlien" Date: Wed, 20 May 2026 12:38:09 +0200 Subject: [PATCH 1/6] Add docs for logging env vars --- src/pages/client/environment-variables.mdx | 8 ++++++++ src/pages/help/troubleshooting-client.mdx | 24 ++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/src/pages/client/environment-variables.mdx b/src/pages/client/environment-variables.mdx index 99269285f..3dbd34ae8 100644 --- a/src/pages/client/environment-variables.mdx +++ b/src/pages/client/environment-variables.mdx @@ -112,6 +112,14 @@ These control the exponential backoff when the client loses its connection to th |---|---|---| | `NB_MANAGEMENT_GRPC_MAX_MSG_SIZE` | All | Maximum size (in bytes) of a single gRPC message from the management server. Increase this when the network map (peers, firewall rules, routes, DNS config) exceeds the default 4 MB gRPC limit. | +## Logging + +| Variable | Platform | Description | +|---|---|---| +| `NB_LOG_FORMAT` | All | Override the log output format. Accepted values: `json` (one JSON object per line) or `syslog` (compact format used when logging to syslog). Anything else falls back to the default human-readable text formatter. The `syslog` value is also selected automatically when the log destination is set to `syslog`. | +| `NB_LOG_MAX_SIZE_MB` | All | Maximum size in megabytes of a single log file before NetBird's built-in rotation triggers (default: `15`). Only effective when the built-in rotation is active — see `NB_LOG_DISABLE_ROTATION` and [Log rotation](/help/troubleshooting-client#log-rotation). | +| `NB_LOG_DISABLE_ROTATION` | All | Set to `true` to disable NetBird's built-in log rotation. NetBird will write to the configured log file directly and leave rotation to an external tool (e.g. system `logrotate` on Linux). On Linux, NetBird also auto-disables its own rotation when it detects a `logrotate` config that mentions `netbird` — this variable lets you force the same behavior on other platforms or when the auto-detection misses your config. | + ## Other | Variable | Platform | Description | diff --git a/src/pages/help/troubleshooting-client.mdx b/src/pages/help/troubleshooting-client.mdx index edc596672..924d2078d 100644 --- a/src/pages/help/troubleshooting-client.mdx +++ b/src/pages/help/troubleshooting-client.mdx @@ -94,6 +94,30 @@ By default, client logs are located in the `/var/log/netbird/client.log` file on You can analyze the logs to identify the root cause of the problem. If you need help, open a [github issue](https://github.com/netbirdio/netbird/issues/new/choose) and attach the logs. +### Log rotation + +NetBird rotates `client.log` itself by default — old files are written next to the active log as gzipped archives (e.g. `client-2025-04-01T12-00-00.000.log.gz`). The maximum size of the active file is controlled by [`NB_LOG_MAX_SIZE_MB`](/client/environment-variables#logging) (default: 15 MB). + +Running NetBird's built-in rotation **and** the `logrotate` service for Linux on the same file causes problems: `logrotate` renames the file out from under the daemon, and the daemon keeps writing to the old file, so new entries land in the rotated file instead of `client.log` or corruption happens. + +To avoid this, NetBird scans `/etc/logrotate.conf` and `/etc/logrotate.d/*` on Linux startup. If any non-comment line mentions `netbird`, the client logs a warning and disables its own rotation, letting `logrotate` own the file. You will see a line like this in `client.log`: + +```text +log rotation conflict detected in: "/etc/logrotate.d/netbird", rotation is disabled +``` + +When this happens, rotated files are produced by `logrotate` and will follow your `logrotate` config, typically `client.log.1`, `client.log.2.gz`, `client.log.3.gz`, etc. The `debug bundle` command looks for gzipped archives first and only falls back to uncompressed `.log.` files when no `.gz` files are present, so a `logrotate` config that uses `delaycompress` will leave the most recent rotation as an uncompressed `client.log.1` that the bundle currently skips. If you want every rotated log in the bundle, configure `logrotate` to either compress every rotation (`compress` and no `delaycompress`) or leave them all uncompressed (`nocompress`). + +If you want to force this behavior on a non-Linux platform, or your `logrotate` config does not match the auto-detection (for example it lives outside `/etc/logrotate.d/` or does not contain the literal string `netbird`), set [`NB_LOG_DISABLE_ROTATION=true`](/client/environment-variables#logging) on the daemon. When the env flag is set, NetBird logs: + +```text +log rotation is disabled by env flag +``` + + + If you disable built-in rotation without configuring an external rotator, `client.log` will grow without bound. Make sure something is rotating the file before turning rotation off. + + ### Debug bundle A debug archive containing the recent logs and the status at the time of execution can be generated with the following From da9dfcaf1895efaee2077079be8aa47eeb772a63 Mon Sep 17 00:00:00 2001 From: "Theodor S. Midtlien" Date: Wed, 20 May 2026 14:35:37 +0200 Subject: [PATCH 2/6] Correct corruption --- src/pages/help/troubleshooting-client.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/pages/help/troubleshooting-client.mdx b/src/pages/help/troubleshooting-client.mdx index 924d2078d..803cee248 100644 --- a/src/pages/help/troubleshooting-client.mdx +++ b/src/pages/help/troubleshooting-client.mdx @@ -96,9 +96,9 @@ a [github issue](https://github.com/netbirdio/netbird/issues/new/choose) and att ### Log rotation -NetBird rotates `client.log` itself by default — old files are written next to the active log as gzipped archives (e.g. `client-2025-04-01T12-00-00.000.log.gz`). The maximum size of the active file is controlled by [`NB_LOG_MAX_SIZE_MB`](/client/environment-variables#logging) (default: 15 MB). +NetBird rotates `client.log` itself by default, old files are written next to the active log as gzipped archives (e.g. `client-2025-04-01T12-00-00.000.log.gz`). The maximum size of the active file is controlled by [`NB_LOG_MAX_SIZE_MB`](/client/environment-variables#logging) (default: 15 MB). -Running NetBird's built-in rotation **and** the `logrotate` service for Linux on the same file causes problems: `logrotate` renames the file out from under the daemon, and the daemon keeps writing to the old file, so new entries land in the rotated file instead of `client.log` or corruption happens. +Running NetBird's built-in rotation **and** the `logrotate` service for Linux on the same file causes problems: `logrotate` renames the file out from under the daemon, and the daemon keeps writing to the old file. This results in `client.log` being corrupted with zero bytes and some logging lost. To avoid this, NetBird scans `/etc/logrotate.conf` and `/etc/logrotate.d/*` on Linux startup. If any non-comment line mentions `netbird`, the client logs a warning and disables its own rotation, letting `logrotate` own the file. You will see a line like this in `client.log`: From 1051d5beec2758c09a54d5bda4a3ba0000ce9393 Mon Sep 17 00:00:00 2001 From: "Theodor S. Midtlien" Date: Wed, 20 May 2026 16:12:09 +0200 Subject: [PATCH 3/6] Change note to warning --- src/pages/help/troubleshooting-client.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/pages/help/troubleshooting-client.mdx b/src/pages/help/troubleshooting-client.mdx index 803cee248..3f24de310 100644 --- a/src/pages/help/troubleshooting-client.mdx +++ b/src/pages/help/troubleshooting-client.mdx @@ -114,9 +114,9 @@ If you want to force this behavior on a non-Linux platform, or your `logrotate` log rotation is disabled by env flag ``` - + If you disable built-in rotation without configuring an external rotator, `client.log` will grow without bound. Make sure something is rotating the file before turning rotation off. - + ### Debug bundle From 270a159c15c086e3bc3fa496eae71080bb32ac71 Mon Sep 17 00:00:00 2001 From: "Theodor S. Midtlien" Date: Wed, 20 May 2026 16:40:06 +0200 Subject: [PATCH 4/6] Add note about copytruncate --- src/pages/help/troubleshooting-client.mdx | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/pages/help/troubleshooting-client.mdx b/src/pages/help/troubleshooting-client.mdx index 3f24de310..7833ecbe9 100644 --- a/src/pages/help/troubleshooting-client.mdx +++ b/src/pages/help/troubleshooting-client.mdx @@ -106,7 +106,13 @@ To avoid this, NetBird scans `/etc/logrotate.conf` and `/etc/logrotate.d/*` on L log rotation conflict detected in: "/etc/logrotate.d/netbird", rotation is disabled ``` -When this happens, rotated files are produced by `logrotate` and will follow your `logrotate` config, typically `client.log.1`, `client.log.2.gz`, `client.log.3.gz`, etc. The `debug bundle` command looks for gzipped archives first and only falls back to uncompressed `.log.` files when no `.gz` files are present, so a `logrotate` config that uses `delaycompress` will leave the most recent rotation as an uncompressed `client.log.1` that the bundle currently skips. If you want every rotated log in the bundle, configure `logrotate` to either compress every rotation (`compress` and no `delaycompress`) or leave them all uncompressed (`nocompress`). +When this happens, rotated files are produced by `logrotate` and will follow your `logrotate` config, typically `client.log.1`, `client.log.2.gz`, `client.log.3.gz`, etc. The `debug bundle` command looks for gzipped archives first and only falls back to uncompressed `.log.` files when no `.gz` files are present, so a `logrotate` config that uses `delaycompress` will leave the most recent rotation as an uncompressed `client.log.1` that the bundle currently skips. + + +To use `logrotate`, we require `copytruncate` to be set for the config file, otherwise the daemon needs to be restarted for the new log file to be opened after rotation. + +If you want every rotated log in the bundle, configure `logrotate` to either compress every rotation (`compress` and no `delaycompress`) or leave them all uncompressed (`nocompress`). + If you want to force this behavior on a non-Linux platform, or your `logrotate` config does not match the auto-detection (for example it lives outside `/etc/logrotate.d/` or does not contain the literal string `netbird`), set [`NB_LOG_DISABLE_ROTATION=true`](/client/environment-variables#logging) on the daemon. When the env flag is set, NetBird logs: From 8959cbaad780265f371dce9eb2e37218463bc613 Mon Sep 17 00:00:00 2001 From: "Theodor S. Midtlien" Date: Thu, 21 May 2026 19:10:06 +0200 Subject: [PATCH 5/6] Simplify and correct behavior --- src/pages/client/environment-variables.mdx | 4 ++-- src/pages/help/troubleshooting-client.mdx | 12 ++++-------- 2 files changed, 6 insertions(+), 10 deletions(-) diff --git a/src/pages/client/environment-variables.mdx b/src/pages/client/environment-variables.mdx index 3dbd34ae8..4ce0a7834 100644 --- a/src/pages/client/environment-variables.mdx +++ b/src/pages/client/environment-variables.mdx @@ -117,8 +117,8 @@ These control the exponential backoff when the client loses its connection to th | Variable | Platform | Description | |---|---|---| | `NB_LOG_FORMAT` | All | Override the log output format. Accepted values: `json` (one JSON object per line) or `syslog` (compact format used when logging to syslog). Anything else falls back to the default human-readable text formatter. The `syslog` value is also selected automatically when the log destination is set to `syslog`. | -| `NB_LOG_MAX_SIZE_MB` | All | Maximum size in megabytes of a single log file before NetBird's built-in rotation triggers (default: `15`). Only effective when the built-in rotation is active — see `NB_LOG_DISABLE_ROTATION` and [Log rotation](/help/troubleshooting-client#log-rotation). | -| `NB_LOG_DISABLE_ROTATION` | All | Set to `true` to disable NetBird's built-in log rotation. NetBird will write to the configured log file directly and leave rotation to an external tool (e.g. system `logrotate` on Linux). On Linux, NetBird also auto-disables its own rotation when it detects a `logrotate` config that mentions `netbird` — this variable lets you force the same behavior on other platforms or when the auto-detection misses your config. | +| `NB_LOG_MAX_SIZE_MB` | All | Maximum size in megabytes of a single log file before NetBird's built-in rotation triggers (default: `15`). Only effective when the built-in rotation is active, see `NB_LOG_DISABLE_ROTATION` and [Log rotation](/help/troubleshooting-client#log-rotation). | +| `NB_LOG_DISABLE_ROTATION` | All | Set to `true` to disable NetBird's built-in log rotation. NetBird will write to the configured log file directly and leave rotation to an external tool (e.g. system `logrotate` on Linux). On Linux, NetBird also auto-disables its own rotation when it detects a `logrotate` config that mentions `netbird`, this variable lets you force the same behavior on other platforms or when the auto-detection misses your config. Also see [Log rotation troubleshooting](/help/troubleshooting-client#log-rotation). | ## Other diff --git a/src/pages/help/troubleshooting-client.mdx b/src/pages/help/troubleshooting-client.mdx index 7833ecbe9..19acfdfa9 100644 --- a/src/pages/help/troubleshooting-client.mdx +++ b/src/pages/help/troubleshooting-client.mdx @@ -98,23 +98,19 @@ a [github issue](https://github.com/netbirdio/netbird/issues/new/choose) and att NetBird rotates `client.log` itself by default, old files are written next to the active log as gzipped archives (e.g. `client-2025-04-01T12-00-00.000.log.gz`). The maximum size of the active file is controlled by [`NB_LOG_MAX_SIZE_MB`](/client/environment-variables#logging) (default: 15 MB). -Running NetBird's built-in rotation **and** the `logrotate` service for Linux on the same file causes problems: `logrotate` renames the file out from under the daemon, and the daemon keeps writing to the old file. This results in `client.log` being corrupted with zero bytes and some logging lost. - -To avoid this, NetBird scans `/etc/logrotate.conf` and `/etc/logrotate.d/*` on Linux startup. If any non-comment line mentions `netbird`, the client logs a warning and disables its own rotation, letting `logrotate` own the file. You will see a line like this in `client.log`: +Running NetBird's built-in rotation **and** the `logrotate` service for Linux on the same file causes problems where the log file might become corrupted and logs lost. To avoid this, NetBird scans `/etc/logrotate.conf` and `/etc/logrotate.d/*` on Linux startup. If any non-comment line mentions `netbird`, the client logs a warning and disables its own rotation, letting `logrotate` own the file. You will see a line like this in `netbird.err` or the console: ```text log rotation conflict detected in: "/etc/logrotate.d/netbird", rotation is disabled ``` -When this happens, rotated files are produced by `logrotate` and will follow your `logrotate` config, typically `client.log.1`, `client.log.2.gz`, `client.log.3.gz`, etc. The `debug bundle` command looks for gzipped archives first and only falls back to uncompressed `.log.` files when no `.gz` files are present, so a `logrotate` config that uses `delaycompress` will leave the most recent rotation as an uncompressed `client.log.1` that the bundle currently skips. - -To use `logrotate`, we require `copytruncate` to be set for the config file, otherwise the daemon needs to be restarted for the new log file to be opened after rotation. +To use `logrotate`, we require `copytruncate` and not `create` to be set in the config file, otherwise the daemon needs to be restarted for the new log file to be opened after rotation. -If you want every rotated log in the bundle, configure `logrotate` to either compress every rotation (`compress` and no `delaycompress`) or leave them all uncompressed (`nocompress`). +Netbird supports `compress`, `delaycompress` and `nocompress`. -If you want to force this behavior on a non-Linux platform, or your `logrotate` config does not match the auto-detection (for example it lives outside `/etc/logrotate.d/` or does not contain the literal string `netbird`), set [`NB_LOG_DISABLE_ROTATION=true`](/client/environment-variables#logging) on the daemon. When the env flag is set, NetBird logs: +If you want to force disable log rotation (for example if your `logrotate` config lives outside `/etc/logrotate.d/` or does not contain the literal string `netbird`), set [`NB_LOG_DISABLE_ROTATION=true`](/client/environment-variables#logging) on the daemon. When the env flag is set, `netbird.err` or the console will show: ```text log rotation is disabled by env flag From 5cf62469a8095dcd445d46af063a35c1de44c3b9 Mon Sep 17 00:00:00 2001 From: "Theodor S. Midtlien" Date: Fri, 22 May 2026 13:23:13 +0200 Subject: [PATCH 6/6] Add BSD docs --- src/pages/help/troubleshooting-client.mdx | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/src/pages/help/troubleshooting-client.mdx b/src/pages/help/troubleshooting-client.mdx index 19acfdfa9..356ee47d0 100644 --- a/src/pages/help/troubleshooting-client.mdx +++ b/src/pages/help/troubleshooting-client.mdx @@ -110,7 +110,20 @@ To use `logrotate`, we require `copytruncate` and not `create` to be set in the Netbird supports `compress`, `delaycompress` and `nocompress`. -If you want to force disable log rotation (for example if your `logrotate` config lives outside `/etc/logrotate.d/` or does not contain the literal string `netbird`), set [`NB_LOG_DISABLE_ROTATION=true`](/client/environment-variables#logging) on the daemon. When the env flag is set, `netbird.err` or the console will show: +On macOS and BSD systems, the equivalent of `logrotate` is `newsyslog`, configured via `/etc/newsyslog.conf` or files in `/etc/newsyslog.d/`. NetBird does not auto-detect `newsyslog` configurations, so if you want `newsyslog` to manage `client.log` you must set [`NB_LOG_DISABLE_ROTATION=true`](/client/environment-variables#logging) on the daemon to disable the built-in rotation. An example `/etc/newsyslog.d/netbird.conf` entry: + +```text +# logfilename [owner:group] mode count size when flags +/var/log/netbird/client.log root:wheel 600 10 15360 * Z +``` + +The `Z` flag tells `newsyslog` to gzip rotated files, which the debug bundle can read. + + +`newsyslog` rotates by renaming the active file and creating a new one (analogous to `logrotate`'s default `create` mode, not `copytruncate`). After rotation, the netbird daemon still holds the file descriptor of the renamed file and will keep writing to it until restarted. Restart the daemon after `newsyslog` runs, or write logs to syslog instead and let `newsyslog` manage the syslog file. + + +If you want to force disable log rotation (for example if your `logrotate` config lives outside `/etc/logrotate.d/` or does not contain the literal string `netbird`, or you are using `newsyslog` or another external rotator), set [`NB_LOG_DISABLE_ROTATION=true`](/client/environment-variables#logging) on the daemon. When the env flag is set, `netbird.err` or the console will show: ```text log rotation is disabled by env flag