diff --git a/candidates.md b/candidates.md new file mode 100644 index 00000000..ad460c9f --- /dev/null +++ b/candidates.md @@ -0,0 +1,861 @@ +# Command Candidates + +## Table Of Contents + +- [Decision Lens](#decision-lens) +- [Entry Format](#entry-format) +- [Accepted Candidates](#accepted-candidates) + - [`stat`](#stat) + - [`lsof`](#lsof) + - [`journalctl`](#journalctl) + - [`systemctl`](#systemctl) + - [`free`](#free) + - [`ps` memory fields and sorting](#ps-memory-fields-and-sorting) + - [`pmap`](#pmap) + - [`vmstat`](#vmstat) + - [`uptime`](#uptime) +- [Rejected / Deferred Candidates](#rejected--deferred-candidates) + - [`top`](#top) + - [`strace`](#strace) + - [`perf`](#perf) + - [`dmesg`](#dmesg) + - [`pgrep`](#pgrep) + - [`kill`](#kill) + - [`crontab`](#crontab) + - [`flock`](#flock) + - [`nice`](#nice) + - [`cleanup`](#cleanup) + - [`rm`](#rm) + - [`systemd-tmpfiles`](#systemd-tmpfiles) + - [`coredumpctl`](#coredumpctl) + - [`sysctl`](#sysctl) + - [`tee`](#tee) + - [`tune2fs`](#tune2fs) +- [Scenario Gap Report](#scenario-gap-report) + +## Decision Lens + +LLM-based agents are the main users of rshell. Candidate decisions should optimize rshell's usefulness for AI agents resolving investigations: commands should be easy for agents to choose correctly, produce bounded and explainable output, expose enough host context to diagnose issues, and keep remediation actions explicit, auditable, and constrained by rshell's safety model. + +## Entry Format + +This is a concise decision record for generally useful investigation commands. Each entry should focus on the candidate decision only: accept, defer, or reject. Do not evaluate the implementation design here; implementation details, parser behavior, platform-specific mechanics, and code-level boundaries should be handled in a later implementation plan. Use 🟢 for reasons to support it and 🔴 for reasons to reject, defer, or narrow the scope, so the decision is easy to scan. + +Each entry should include: type, decision, evidence, existing coverage, minimum subset, target syntax, and fit/scope rationale. Mention implementation constraints only when they materially affect the accept/defer/reject decision. Put accepted or planned work under "Accepted Candidates" and rejected or deferred work under "Rejected / Deferred Candidates". Rejected alternatives should usually live inside the related candidate entry; add standalone rejected entries only when the command is likely to be proposed again. + +## Accepted Candidates + +### `stat` + +Type: new builtin + +Decision: add narrow builtin + +Evidence: the inode exhaustion runbook uses `stat -f /var/spool/` to confirm total and free inodes for the filesystem backing a specific path. + +Already covered? Partially covered by `df -i` / `df -ih`, but `df` currently does not accept `FILE` operands. There is no direct way to ask "which filesystem backs this path, and how many inodes are free there?" + +Minimum subset: `stat -f PATH...` + +Target syntax: +- `stat -f /var/spool/` + +🟢 Fit: useful for path-targeted filesystem inode investigation. + +🔴 Scope: do not start with a full GNU/BSD `stat` implementation. + +Implementation boundary: user-supplied paths must go through `AllowedPaths`; unlike `df` mount enumeration, these paths are operator input rather than hardcoded kernel pseudo-files. + +### `lsof` + +Type: new investigation builtin + +Decision: add narrow builtin for deleted-open file diagnostics + +Evidence: the unrotated / unbounded log runbook uses `lsof | grep deleted | grep log` during investigation and verification to find deleted-but-open log files whose directory entries are gone but whose disk blocks remain allocated until the owning process releases the file descriptor. + +Already covered? Not covered. `df` can show that disk space is still consumed, while `du`, `find`, and `ls` cannot see an unlinked file. `ss` is not a substitute because it reports socket state rather than open regular files, and this shell intentionally rejects `ss -p` process disclosure. + +Minimum subset: deleted-open regular file diagnostics only. Exact syntax and implementation design are deferred. + +Target syntax: +- Deferred: `lsof` workflow for deleted-open files, equivalent to the runbook's `lsof | grep deleted | grep -i log` + +🟢 Fit: closes a real disk-space investigation gap where existing filesystem commands cannot identify the process holding reclaimed-looking space. + +🟢 Fit: read-only, bounded process/file-descriptor metadata is agent-friendly when scoped to deleted-open files instead of a full host-wide open-file inventory. + +🔴 Scope: do not add full `lsof`. General FD listing, socket inspection, argv disclosure, network modes, and mutation-oriented behavior are outside this candidate. + +🔴 Visibility: even the narrow diagnostic shape exposes process-owned file-descriptor metadata and paths that may be outside `AllowedPaths`; that host-visibility trade-off must be documented when implementation is designed. + +### `journalctl` + +Type: new Linux/systemd investigation builtin with narrow remediation-mode support + +Decision: add bounded journal inspection plus remediation-gated `--vacuum-size=SIZE` + +Evidence: the remediation scenarios use `journalctl` across core dump investigation, crash-loop diagnosis, database retention cleanup history, unrotated log remediation, cron storm correlation, and OOM-kill checks. The unrotated / unbounded log runbook also uses `journalctl --disk-usage` and `journalctl --vacuum-size=500M` to recover space consumed by journald itself. + +Already covered? Partially covered for adjacent signals. `df` and `du` can show that `/var/log` or journal directories are consuming disk, `cat` / `grep` can inspect syslog-style text logs and journald config files when paths are allowed, and `dmesg` can cover some kernel messages. Current rshell does not provide structured journald reads, service-unit filtering, reliable boot/kernel journal access, or a safe equivalent to `journalctl --vacuum-size`. `logrotate` does not manage journald binary journal files, `truncate` is unsafe for those files, and raw deletion is intentionally not exposed. + +Minimum subset: +- Read-only investigation: `journalctl --disk-usage`, `journalctl -u UNIT -n N --no-pager`, `journalctl -u UNIT --since TIME`, `journalctl -k --since TIME` +- Remediation mode only: `journalctl --vacuum-size=SIZE` + +Target syntax: +- `journalctl -u cron --since "2 hours ago" | grep -E "CMD|session"` +- `journalctl -u -n 30 --no-pager` +- `journalctl -k --since "6 hours ago" | grep -i "killed process\|oom"` +- `journalctl --disk-usage` +- `journalctl --vacuum-size=500M` + +🟢 Fit: closes repeated investigation gaps where agents need recent unit, kernel, crash, cron, OOM, or maintenance-job evidence from journald rather than only filesystem logs. + +🟢 Fit: bounded journal reads with explicit unit, time, and tail filters are more agent-friendly than unbounded log scraping. + +🟢 Fit: `--vacuum-size=SIZE` maps directly to the journald disk-recovery use case and is safer than exposing raw deletion of journal files because journald chooses old archived data to remove. + +🔴 Scope: do not add full `journalctl` compatibility initially. Defer live follow (`-f`), cursor/export modes, JSON/output-format variants, boot selection, catalog output, arbitrary field queries, `--directory`, `--file`, `--root`, `--image`, and remote journal sources. + +🔴 Scope: defer unscoped whole-journal reads such as `journalctl --since TIME` initially. Unit-scoped reads provide the repeated scenario value without granting arbitrary application-log visibility across the host. + +🔴 Remediation scope: reject `--vacuum-time`, `--vacuum-files`, arbitrary journal-file deletion, and journald configuration edits initially. Time-based and file-count vacuuming are easier for agents to misuse because they are less directly tied to the disk-headroom target than size-based vacuuming. + +🔴 Visibility: journal reads can expose service logs, executable paths, usernames, hostnames, kernel messages, and application error content that may not be reachable through `AllowedPaths`. + +Configuration requirement: unit-scoped journal reads require `AllowedCommands` for `rshell:journalctl` and an exact unit name present in either `interp.AllowedSystemdUnitRead([]string{...})` or `interp.AllowedSystemdUnitWrite([]string{...})`. For example, `interp.AllowedSystemdUnitRead([]string{"mysql.service", "cron.service", "logrotate.timer"})` allows bounded `journalctl -u mysql.service ...` and `journalctl -u cron.service ...`; `interp.AllowedSystemdUnitWrite([]string{"mysql.service"})` also allows bounded unit journal reads for `mysql.service` because write permission implies read permission. These unit lists do not authorize unscoped whole-journal reads. + +Implementation boundary: do not invoke the host `journalctl` binary. Keep Linux/systemd support explicit and fail clearly elsewhere. Treat journal visibility as a deliberate host metadata boundary like `ps`, `ss`, `ip route`, `df`, and planned `vmstat`; document that unit journal reads are controlled by `AllowedSystemdUnitRead` / `AllowedSystemdUnitWrite`, not by `AllowedPaths`. For reads, require bounded output through a supported `--since`, `-n`, or equivalent cap, reject live streaming initially, cap line lengths and total rows, and respect context cancellation. For `--vacuum-size=SIZE`, require remediation mode, enforce a conservative minimum retained size, report before/after disk usage, document that historical journal entries are deleted, and avoid accepting user-controlled journal source paths until there is a separate design for their sandbox semantics. + +### `systemctl` + +Type: new Linux/systemd investigation builtin with narrow service-control remediation support + +Decision: add bounded systemd unit inspection plus remediation-gated `start`, `stop`, `restart`, and `reload` for explicitly allowed units. + +Evidence: the remediation scenarios use `systemctl` to inspect crash loops, failed units, restart counts, maintenance timers, logrotate timers, kdump/crash service state, and post-remediation service health. They also use service control to stop crash loops or runaway services, restart leaking services, reload database services, release deleted log file descriptors, restart services after configuration changes, and start services after a fix is deployed. + +Already covered? Not covered. `ps` can show process state, `journalctl` can show service logs, and filesystem commands can inspect unit/config files when paths are allowed, but current rshell has no safe systemd unit-state view, no timer listing, no failed-unit discovery, no restart-count query, and no managed-service remediation path. Raw `kill` is deferred and is not a substitute for systemd-managed services because it can trigger auto-restart behavior and bypass unit lifecycle semantics. + +Minimum subset: +- Read-only investigation: `systemctl status UNIT`, `systemctl show UNIT --property=NRestarts`, `systemctl --state=failed`, `systemctl list-timers` +- Remediation mode only, for explicitly allowlisted units: `systemctl start UNIT`, `systemctl stop UNIT`, `systemctl restart UNIT`, `systemctl reload UNIT` + +Target syntax: +- `systemctl --state=failed` +- `systemctl status ` +- `systemctl status logrotate.timer` +- `systemctl show --property=NRestarts` +- `systemctl list-timers | grep logrotate` +- `systemctl restart ` +- `systemctl stop ` +- `systemctl start ` +- `systemctl reload mysql` + +🟢 Fit: closes a repeated investigation gap for discovering failed units, checking timer health, confirming restart loops, and verifying service state after remediation. + +🟢 Fit: service-level `start` / `stop` / `restart` / `reload` is safer for systemd-managed processes than raw PID signalling because it uses the unit lifecycle and can suppress or apply manager restart semantics intentionally. + +🟢 Fit: a narrow remediation subset maps directly to the scenario evidence without accepting persistent unit-policy changes. + +🔴 Scope: do not add full `systemctl` compatibility initially. Defer `mask`, `unmask`, `enable`, `disable`, `edit`, `daemon-reload`, `daemon-reexec`, arbitrary `show`, `list-units`, job management, environment changes, unit file writes, and user-manager / remote-machine modes. + +🔴 Remediation scope: service-control verbs can cause downtime, dropped in-flight requests, session loss, or log collection gaps. They must require remediation mode and a trusted unit allowlist rather than allowing any script-supplied unit name. + +🔴 Visibility: read-only unit and timer listings expose host service names, unit state, process identifiers, result codes, timestamps, and scheduling metadata that may not be reachable through `AllowedPaths`. + +Configuration requirement: unit-scoped systemd operations require `AllowedCommands` for `rshell:systemctl` plus exact unit allowlists. `interp.AllowedSystemdUnitRead([]string{...})` permits read-only unit operations such as `systemctl status UNIT` and `systemctl show UNIT --property=NRestarts`; `interp.AllowedSystemdUnitWrite([]string{...})` permits the accepted service-control verbs (`start`, `stop`, `restart`, `reload`) for those exact units only, implies read permission for the same units, and still requires remediation mode. Unit names must be exact strings such as `mysql.service`, `cron.service`, or `logrotate.timer`; do not support globs, regexes, aliases, or script-controlled broad matches initially. + +Implementation boundary: keep Linux/systemd support explicit and fail clearly elsewhere. Initial `status` output should be metadata-only: no journal tail, no process tree, and no argv disclosure. Treat systemd unit visibility as a deliberate host metadata boundary like `ps`, `ss`, `ip route`, `df`, `journalctl`, and planned `vmstat`; document that unit reads and writes are controlled by `AllowedSystemdUnitRead` / `AllowedSystemdUnitWrite`, not by `AllowedPaths`. Keep output bounded, cap list rows, respect context cancellation, and do not implement this as a raw wrapper around the full host `systemctl` command. + +### `free` + +Type: new builtin + +Decision: add narrow read-only investigation builtin + +Evidence: the memory leak runbook uses `free -h` to confirm host-level memory pressure before narrowing to a leaking process, and uses `free -h` again during verification to confirm that available memory recovered after remediation. + +Already covered? Not covered by a simple host-memory snapshot. `ps` identifies high-memory processes, and `vmstat` explains whether host pressure is turning into swap, I/O wait, CPU contention, or queue growth, but neither replaces the quick total/used/free/available/swap view that agents need at the start and end of a memory investigation. + +Minimum subset: +- `free` +- `free -h` + +Target syntax: +- `free -h` + +🟢 Fit: bounded, read-only, familiar output for confirming whether the host is under memory pressure. + +🟢 Fit: complements `ps` memory fields and `vmstat`; it gives the first-pass host snapshot, while those commands explain process ownership and pressure dynamics. + +🔴 Scope: do not treat `free` as a remediation command or as the primary time-series pressure tool. Repeated sampling and trend interpretation belong with `vmstat` or higher-level telemetry. + +### `ps` memory fields and sorting + +Type: existing builtin enhancement + +Decision: add narrow enhancement + +Evidence: common host-pressure investigation needs deterministic process memory sorting for agents. + +Already covered? Partially covered by `ps`, but current `ps` omits RSS, VSZ, `%MEM`, custom columns, and sorting. + +Minimum subset: +- `ps -e -o pid,ppid,comm,rss,vsz,pmem --sort=-rss` +- `ps -e -o pid,ppid,comm,rss,vsz,pmem --sort=-pmem` + +Target syntax: +- `ps -e -o pid,ppid,comm,rss,vsz,pmem --sort=-rss | head` +- `ps -e -o pid,ppid,comm,rss,vsz,pmem --sort=-pmem | head` + +🟢 Fit: single-shot, bounded output is better for LLM agents than live terminal UI output. + +🟢 Fit: builds on the existing `ps` investigation builtin and follows familiar `ps -o` / GNU `--sort` syntax. + +🔴 Scope: do not expose full argv fields such as `args` or `command`; preserve the current process-name-only privacy boundary. + +Implementation boundary: keep supported `-o` fields explicit and small. Prefer piping to `head` for top-N output instead of inventing a non-standard `--limit` flag. + +### `pmap` + +Type: new investigation builtin + +Decision: add narrow read-only builtin for per-process memory-map diagnostics. + +Evidence: the memory leak runbook uses `pmap -x | sort -k3 -rn | head -30` after `smaps_rollup` to identify the largest mappings inside a leaking process. This helps agents distinguish large anonymous heap regions from file-backed mappings without dumping the full per-VMA `smaps` file. + +Already covered? Partially covered when operators expose procfs through `AllowedPaths`: agents can read `/proc//maps`, `/proc//smaps`, or `/proc//smaps_rollup` directly. Raw proc files are noisy, large, and easy for agents to parse incorrectly. A narrow `pmap` gives a bounded, familiar table over the same operator-authorized proc data. + +Minimum subset: +- `pmap -x PID` + +Target syntax: +- `pmap -x | sort -k3 -rn | head -30` + +🟢 Fit: closes a memory-leak investigation gap between aggregate process memory (`ps`, `/proc//status`, `smaps_rollup`) and raw, verbose `/proc//smaps` output. + +🟢 Fit: output is read-only, single-process, table-shaped, and naturally bounded by pipelines such as `sort` and `head`, making it easier for agents to rank the mappings that matter. + +🟢 Fit: when procfs is already exposed through `AllowedPaths`, `pmap` adds safer ergonomics and output normalization rather than new raw host authority. + +🔴 Visibility: `pmap` exposes memory-map metadata such as address ranges, permissions, anonymous mapping labels, and mapped file paths. This can reveal deployment paths or runtime layout for the selected process. + +🔴 Scope: do not add broad procps compatibility, all-process modes, argv/env disclosure, raw memory reads, or any behavior that writes files or invokes the host `pmap` binary. + +🔴 Platform: Linux/procfs first. Portable macOS/Windows support is deferred and should not block the initial candidate. + +Implementation boundary: do not make `pmap` a new unconditional `/proc` visibility bypass. Read procfs through the configured proc root only when that root is exposed by `AllowedPaths`; if procfs is not allowed, fail rather than silently expanding host visibility. Cap line lengths and mapping count, support one PID per invocation initially, respect context cancellation, and never read `/proc//cmdline`, `/proc//environ`, or process memory contents. + +### `vmstat` + +Type: new builtin + +Decision: add broad GNU/procps-compatible investigation builtin; prioritize implementation after `free` and `ps` memory fields. + +Evidence: the memory leak runbook uses `vmstat -s` to confirm host-level memory counters and `vmstat 2 10` to observe pressure over time. The cron storm runbook uses `vmstat 1 30` to check whether the run queue is staying above CPU capacity and whether the load is CPU-bound or I/O-bound. + +Already covered? Not covered as a single host-pressure view. Existing or planned tools cover separate slices: +- `free` should be the simpler host-memory snapshot command, but it does not show runnable queues, blocked tasks, CPU split, I/O wait, or whether pressure is changing over repeated samples. +- `ps` identifies expensive processes by RSS or CPU, but it does not explain whether the host itself is saturated, paging, I/O-bound, or just running one hot process. +- `uptime` reports load averages, but not why load is high; `vmstat` adds the `r` and `b` queues plus CPU `us` / `sy` / `wa` / `st` breakdown. +- `top` is familiar but deferred because live terminal output is less deterministic for agents; `vmstat DELAY COUNT` gives a compact bounded time series. +- `df`, `du`, and `find` cover storage capacity and file growth, not runtime pressure from CPU scheduling, swap churn, or block I/O. + +Coverage added: one bounded table correlating scheduler pressure (`r`, `b`), memory and swap (`free`, `buff`, `cache`, `si`, `so`), block I/O (`bi`, `bo`), interrupts/context switches (`in`, `cs`), and CPU state (`us`, `sy`, `id`, `wa`, `st`). That helps agents distinguish CPU-bound cron storms, disk-heavy jobs causing I/O wait, memory leaks causing paging, and general host saturation before choosing a more specific follow-up command. + +Minimum subset: broad read-only GNU/procps `vmstat` compatibility for kernel-state visibility, including the default report, `-s`, and bounded `DELAY COUNT` sampling. + +Target syntax: +- `vmstat -s` +- `vmstat 2 10` +- GNU/procps-compatible read-only display modes and formatting controls as the implementation grows. + +🟢 Fit: complements `free` and `ps` by showing whether host memory pressure is turning into swap, I/O wait, CPU contention, or runnable/blocked queue growth. + +🟢 Fit: single-shot output and count-bounded sampling are agent-friendly when unbounded live monitoring is rejected. + +🟢 Fit: familiar Linux runbook command with strong diagnostic value for memory leaks and broader host-pressure investigations. + +🔴 Scope: do not invoke the host `vmstat` binary, do not mutate kernel or filesystem state, and do not turn user operands such as device or partition names into file paths. + +🔴 Platform: Linux/procfs first. Portable macOS/Windows support is deferred and should not block the initial candidate. + +Implementation boundary: read only hardcoded kernel-state sources through the configured `ProcPath` or equivalent internal kernel readers. These reads intentionally bypass `AllowedPaths`, like `ss`, `ip route`, and `df`, because the opened paths are not derived from script input; document the operator-visibility trade-off in `README.md`, `SHELL_FEATURES.md`, and `AGENTS.md` when implemented. Treat device or partition operands as filters over kernel-reported entries, not filesystem paths to open. Reject unbounded sampling such as `vmstat 2`; require positive `DELAY` and `COUNT`, enforce an internal total-duration cap below the shell timeout, and respect context cancellation between samples. + +### `uptime` + +Type: new builtin + +Decision: add narrow read-only investigation builtin; prioritize after `vmstat` and `ps` CPU/process sorting. + +Evidence: the cron storm runbook uses `uptime` to check whether load average remains above CPU capacity during investigation and whether load has recovered during verification. The signal is also generally useful across host-pressure investigations as a quick "is this host broadly under pressure?" context check. + +Already covered? Partially covered by `vmstat`, which is more diagnostic because it explains scheduler pressure, blocked tasks, CPU split, I/O wait, and swap behavior. `uptime` is still useful as a cheaper single-shot summary and verification command, but it should not be the primary root-cause tool. + +Minimum subset: `uptime` + +Target syntax: +- `uptime` + +🟢 Fit: bounded, read-only, familiar output that gives agents quick host context before or after deeper pressure investigation. + +🟢 Fit: complements `vmstat` and `ps` by providing a compact load-average snapshot for triage and post-remediation verification. + +🔴 Scope: load average alone is easy to overinterpret; documentation and examples should steer agents to `vmstat` and `ps` when they need to explain why load is high. + +🔴 Priority: lower value than `vmstat` and `ps` CPU/process sorting because it does not identify the cause of pressure. + +Implementation boundary: do not invoke the host `uptime` binary. Keep the builtin read-only and focused on host context; defer GNU/procps formatting details and optional fields to the implementation design. + +## Rejected / Deferred Candidates + +### `top` + +Type: new builtin + +Decision: do not add initially + +Evidence: process pressure investigations often ask for "top processes", but agents need deterministic, bounded output that works in non-interactive scripts. + +Already covered? Partially covered by `ps`. The accepted `ps` memory fields and sorting enhancement covers the agent-friendly workflow without introducing live terminal UI behavior. + +Minimum subset: none initially. + +Target syntax: +- Rejected: `top` +- Preferred alternative: `ps -e -o pid,ppid,comm,rss,vsz,pmem --sort=-rss | head` + +🟢 Fit: familiar command name for CPU and memory pressure investigation. + +🔴 Agent fit: interactive/live terminal output is harder for LLM agents to consume reliably than single-shot, bounded tables. + +🔴 Scope: adding enough `top` compatibility to match user expectations would overlap heavily with `ps`, require terminal-oriented behavior, and increase maintenance cost without improving the core agent workflow. + +Implementation boundary: defer `top` unless users specifically need its syntax. Prefer deterministic `ps` sorting for top-N process investigations. + +### `strace` + +Type: new privileged Linux process-observation builtin candidate + +Decision: do not add initially; defer until rshell has an explicit privileged process-observation policy. + +Evidence: the runaway process / infinite loop runbook uses `strace -p -c -f -e trace=all -- sleep 5` to sample syscall activity after identifying a single spinning PID. The output helps distinguish a pure userspace CPU loop from a lock spin, I/O retry loop, or a process that is actually waiting in `epoll_wait` / `select`. + +Already covered? Partially covered by safer triage signals. `ps`, `/proc//wchan`, `/proc//stat`, `vmstat`, and `uptime` can identify a sustained CPU consumer and distinguish userspace CPU from broader host pressure. The accepted `systemctl` candidate gives a safer remediation path for systemd-managed services. These do not provide syscall-frequency evidence, but the current scenario uses that evidence for deeper diagnosis rather than the minimum incident triage path. + +Minimum subset: none initially. Deferred possible subset: PID-targeted, attach-only, duration-bounded, output-capped syscall summary. + +Target syntax: +- Rejected initially: `strace -p -c -f -e trace=all -- sleep 5` +- Rejected initially: `strace -p -e trace=all` +- Deferred possible subset: a narrow attach-only syscall summary for one validated PID and a short fixed duration + +🟢 Fit: useful when operators need to preserve diagnostic evidence before terminating a runaway process and need to distinguish pure CPU spin from syscall-heavy retry, lock, or I/O behavior. + +🔴 Scenario weight: appears in one scenario as a deeper root-cause diagnostic, so it should not displace lower-risk CPU investigation work. + +🔴 Privilege / reliability: attaching to another process generally requires `CAP_SYS_PTRACE` or matching ownership plus permissive ptrace settings, and is commonly blocked in containers or hardened hosts. + +🔴 Agent fit: real `strace` output is noisy, potentially unbounded, and easy for agents to over-collect unless rshell defines duration limits, syscall filters, row caps, and output shaping. + +🔴 Visibility / perturbation: ptrace can expose syscall arguments, file paths, network endpoints, signals, timing, and application data fragments that are not constrained by `AllowedPaths`; attaching can also perturb or slow the target process. + +🔴 Scope: do not add command-launch tracing, output-file modes, arbitrary `-e` expressions, full syscall argument dumps, unbounded live tracing, or broad process-tree tracing initially. + +Implementation boundary: do not invoke the host `strace` binary. Do not add ptrace/process attachment until a privileged process-observation design defines target validation, PID identity checks, ownership and capability behavior, output redaction and caps, context cancellation, and how rshell should prevent PID reuse or wrong-process attachment. + +### `perf` + +Type: new privileged Linux investigation builtin candidate + +Decision: do not add initially; defer until rshell has repeated profiling evidence or an explicit privileged profiling mode. + +Evidence: the runaway process / infinite loop runbook uses `perf top -p ` only as an optional deep CPU profiling step after identifying the spinning process with `top` / `ps`, checking `/proc//wchan`, and sampling syscall activity with `strace`. + +Already covered? Partially covered by safer triage signals. `ps` process sorting, `/proc//stat`, `/proc//wchan`, `vmstat`, and `uptime` can identify a single sustained CPU consumer and distinguish userspace CPU spin from broader host pressure. They do not identify the hot function inside the process, but that is developer root-cause evidence rather than required incident triage/remediation evidence. + +Minimum subset: none initially. + +Target syntax: +- Rejected initially: `perf top -p ` +- Rejected initially: `perf top -p -d 5` + +🟢 Fit: useful when operators need function-level CPU profiling before terminating a runaway process. + +🔴 Scenario weight: appears in only one scenario and only as optional deep profiling, so it should not displace lower-risk CPU investigation work. + +🔴 Privilege / reliability: practical use often requires `CAP_PERFMON`, `CAP_SYS_ADMIN`, or permissive `perf_event_paranoid` settings, and is commonly blocked in containers. + +🔴 Agent fit: `perf top` is a live profiler rather than a naturally bounded, deterministic table. Making it agent-friendly would require rshell-specific duration limits, sampling limits, and output shaping. + +🔴 Visibility: perf sampling can expose kernel/user symbols, mapped library paths, addresses, and execution hotspots that are not constrained by `AllowedPaths`. + +🔴 Scope: full `perf` compatibility would pull in system-wide profiling, recording files, tracepoints, probes, call graphs, event selection, and kernel-version-specific behavior. That is much broader than the current runbook need. + +Implementation boundary: do not invoke the host `perf` binary. Do not add `perf_event_open` access until rshell has an explicit privileged profiling design. If revisited, start Linux-only, PID-targeted, read-only, duration-bounded, and output-capped; reject system-wide profiling, record/report file generation, tracepoints, kprobes/uprobes, eBPF, call graphs, and arbitrary event selection initially. + +### `dmesg` + +Type: new investigation builtin candidate + +Decision: do not add initially; prefer accepted `journalctl -k` for kernel-message investigation and revisit only if non-systemd / journald-unavailable hosts become an explicit target. + +Evidence: the core dump, slow leak, unbounded cache, and GC-pressure scenarios use `dmesg | grep -i "oom..."` or `dmesg | grep -i "killed process"` to check whether the kernel OOM killer terminated a process. + +Already covered? Mostly covered by the accepted `journalctl` candidate. `journalctl -k --since TIME` provides the same kernel-message investigation path for systemd/journald hosts, with stronger time bounding and more agent-friendly filtering. `dmesg` would add value mainly on minimal or non-systemd Linux hosts where journald is unavailable. + +Minimum subset: none initially. Deferred possible subset: read-only kernel-ring-buffer display only, with bounded output. + +Target syntax: +- Rejected initially: `dmesg | grep -i "out of memory\|oom_kill\|killed process"` +- Preferred alternative: `journalctl -k --since "6 hours ago" | grep -i "killed process\|oom"` +- Deferred possible subset, if non-systemd support becomes necessary: `dmesg` + +🟢 Fit: familiar Linux investigation command for confirming OOM kills, kernel panics, driver errors, and other kernel-originated signals. + +🟢 Fit: can cover Linux hosts without systemd/journald if rshell later decides that environment is important. + +🔴 Redundancy: the planned `journalctl -k` path already covers the current scenario evidence while also supporting time filters and journal metadata. + +🔴 Visibility: kernel logs can expose device names, host configuration, process names, usernames, addresses, driver messages, and application-adjacent error content that may not be reachable through `AllowedPaths`. + +🔴 Privilege / reliability: modern Linux deployments commonly restrict kernel-ring-buffer reads to privileged users. `dmesg` output is also volatile and may lose older OOM evidence, while journald can retain timestamped kernel messages across a longer incident window. + +🔴 Scope: do not add `dmesg` just because runbooks mention it as a familiar alias for kernel logs; adding both `journalctl -k` and `dmesg` creates overlapping host-log visibility boundaries. + +Implementation boundary: do not invoke the host `dmesg` binary. If revisited, make Linux-only support explicit, reject ring-buffer mutation such as clear/read-clear modes, reject live follow initially, cap output, respect context cancellation, and document that kernel-log reads intentionally bypass `AllowedPaths` because they expose host kernel state rather than user-selected filesystem paths. + +### `pgrep` + +Type: new investigation builtin candidate + +Decision: do not add initially; defer until rshell has a clear argv-disclosure policy or repeated evidence that agents need comm-only PID lookup semantics. + +Evidence: the cron storm runbook uses `pgrep -a -f ` to find overlapping cron job instances by matching the full command line and printing PID plus argv. + +Already covered? The runbook-compatible behavior is not covered because current `ps` intentionally exposes only process comm/executable names and does not read argv. A safe comm-only subset is mostly covered by `ps -e` plus `grep`, and the accepted `ps` custom-column enhancement should make the preferred workflow more explicit with `ps -e -o pid,ppid,comm,etime`. + +Minimum subset: none initially. + +Target syntax: +- Rejected initially: `pgrep -a -f ` +- Deferred safe subset, if needed later: `pgrep PATTERN`, `pgrep -x PATTERN`, `pgrep -l PATTERN` +- Preferred alternative: `ps -e -o pid,ppid,comm,etime | grep ` + +🟢 Fit: familiar, bounded PID lookup command with useful scripting exit codes. + +🟢 Fit: a comm-only implementation could reuse the existing process provider without introducing new host read surfaces. + +🔴 Runbook gap: the valuable cron-storm form is `-f`, because cron jobs often appear as `sh`, `bash`, `python`, or another interpreter in the comm field; matching the script path requires argv. + +🔴 Safety: supporting `-f` or `-a` as expected would require reading and optionally printing `/proc//cmdline`, which breaks the current `ps` privacy boundary that avoids exposing command-line secrets. + +🔴 Scope: a safe comm-only `pgrep` adds convenience and exit-code semantics but little new diagnostic power over `ps`, so it should not displace higher-value `ps` field and sorting work. + +Implementation boundary: do not invoke the host `pgrep` binary, do not read `/proc//cmdline`, and do not support full-command matching or argv output unless rshell explicitly adopts and documents an argv-disclosure mode. If comm-only `pgrep` is revisited, match only against `procinfo.ProcInfo.Cmd`, keep supported flags explicit and small, cap output by the existing process-list cap, and preserve standard no-match exit semantics. + +### `kill` + +Type: new remediation command candidate + +Decision: defer until rshell has a broader process-control / service-control design. + +Evidence: the cron storm runbook uses `kill ` for immediate relief when overlapping job instances are safe to stop, and uses `pkill -f ` when the job is safe to stop entirely. The runaway process runbook uses `kill `, `kill -9 `, and `kill -9 -$PGID` for unmanaged processes, but explicitly prefers `systemctl stop ` for systemd-managed services and warns that abrupt termination can drop in-flight work, leave on-disk state inconsistent, or destroy the primary diagnostic artifact. + +Already covered? Investigation is partially covered by existing or accepted read-only commands such as `ps`, `vmstat`, `uptime`, and `journalctl`. Immediate process termination is not covered. For managed services, the safer remediation path points toward a service-control design rather than raw PID signalling. For cron storms, the broad `pkill -f` form depends on argv matching, which conflicts with the current `ps` / `pgrep` privacy boundary that avoids reading `/proc//cmdline`. + +Minimum subset: none initially. Deferred possible subset: remediation-only, PID-only `kill PID...` sending default `SIGTERM`, after rshell has an explicit process-control policy. + +Target syntax: +- Deferred: `kill ` +- Deferred initially: `kill -9 ` +- Deferred initially: `kill -9 -` +- Rejected as part of this candidate: `pkill -f ` + +🟢 Fit: closes a real immediate-relief gap for unmanaged runaway processes and excess overlapping cron jobs. + +🟢 Fit: a future PID-only default-`SIGTERM` subset could be explicit, bounded, and auditable when paired with remediation mode and process identity checks. + +🔴 Safety: process signalling can cause downtime, lost in-flight work, inconsistent on-disk state, and loss of diagnostic evidence. `SIGKILL`, process-group kills, and broad pattern kills have much higher blast radius than default `SIGTERM` to a specific PID. + +🔴 Scope: `AllowedPaths` cannot constrain process targets, so adding `kill` creates a new host-mutation boundary around PID reuse, target ownership, process identity validation, service auto-restart behavior, audit output, and OS permission failures. + +🔴 Agent fit: the most convenient cron-storm form, `pkill -f`, requires argv matching and pattern-based termination. That is easy for agents to overmatch and would require a separate argv-disclosure decision. + +Implementation boundary: do not invoke the host `kill` or `pkill` binaries. Do not add raw process signalling until a process-control / service-control design defines target validation, signal allowlist, remediation-mode gating, audit/reporting behavior, and how rshell should steer agents toward service-level controls when the target is systemd-managed. + +### `crontab` + +Type: new investigation builtin candidate + +Decision: do not add initially; defer read-only `crontab -l`, reject persistent cron edits. + +Evidence: the cron storm runbook uses `crontab -l` to correlate CPU spikes with per-user cron schedules. Remediation examples require editing schedules to add `flock`, add `nice`, or stagger jobs, but those are durable host mutations rather than simple command inspection. + +Already covered? System cron inspection is mostly covered by existing file-reading commands when paths are allowed: `cat /etc/crontab`, `cat /etc/cron.d/*`, and `ls /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/`. The unique gap is current-user crontab spool visibility via `crontab -l`. + +Minimum subset: none initially. Deferred possible subset: `crontab -l` for the current user only. + +Target syntax: +- Deferred: `crontab -l` +- Rejected: `crontab -e` +- Rejected: `crontab FILE` +- Rejected: `crontab -r` +- Rejected initially: `crontab -l -u USER` + +🟢 Fit: useful for correlating recurring CPU spikes with per-user cron schedules. + +🔴 Incremental value: after excluding writes, most safe cron inspection is already possible through `cat`, `ls`, and `grep` over allowed cron files. + +🔴 Safety: persistent edits can silently change future host behavior, disable unrelated jobs, or install recurring command execution. This is higher risk than existing remediation builtins such as `truncate` and `logrotate`. + +🔴 Scope: user crontab storage and daemon reload behavior vary across platforms and distributions. `-u USER` also introduces identity and privilege semantics. + +Implementation boundary: do not invoke the host `crontab` binary, do not edit or install crontabs, do not remove crontabs, and do not support `-u` until rshell has a clear user-identity policy. If revisited, start with read-only current-user listing, bounded output, and explicit documentation that durable cron remediation needs a separate design with dry-run diff, backup, strict parser, rollback/audit story, and remediation-mode gating. + +### `flock` + +Type: new remediation command candidate + +Decision: do not add; high risk for the runbook remediation shape + +Evidence: the cron storm runbook uses `flock -n /var/lock/.lock ` and `flock -w 60 /var/lock/.lock ` to prevent overlapping future cron runs. The important runbook path is wrapping the real scheduled job, for example `/path/to/job.sh`, not merely testing a lock with an inert builtin. + +Already covered? Investigation is covered by accepted or planned read-only commands such as `ps`, `uptime`, and `vmstat`. Immediate relief is a separate process-control problem (`kill` / `pkill`). Durable prevention through cron editing and job wrapping is not currently covered by rshell's remediation model. + +Minimum subset: none initially. + +Target syntax: +- Rejected: `flock -n /var/lock/myjob.lock /path/to/job.sh` +- Rejected: `flock -w 60 /var/lock/myjob.lock /path/to/job.sh` +- Lower-risk but insufficient for the runbook: `flock -n /var/lock/myjob.lock echo "got lock"` + +🟢 Fit: directly addresses the overlapping-instance failure mode described in the cron storm runbook. + +🔴 Agent fit: a successful command would hide a durable scheduling change behind a runtime wrapper. The script text alone does not prove that future cron launches are protected unless rshell also safely edits or validates the cron entry. + +🔴 Scope: the runbook needs the external job-command form, which would require executing arbitrary host job scripts while holding a lock. Restricting execution to rshell builtins would only support verification, not the actual remediation. + +🔴 Safety: implementing full `flock` semantics introduces lock-file creation/opening, advisory locking, blocking or wait-time behavior, nested command execution, file-descriptor locking forms, and `-c` shell-string execution. The external command and `-c` forms would bypass or greatly complicate rshell's existing command safety boundary. + +🔴 Platform: Unix advisory locks and Windows file locking have materially different semantics, so a portable implementation would need careful platform-specific behavior. + +Implementation boundary: do not invoke the host `flock` binary, do not add `-c` shell-string execution, do not support fd-oriented locking forms, and do not wrap arbitrary external job scripts. If this is revisited, prefer a separate, explicit cron-remediation design that can show the persistent schedule edit, require `AllowedPaths` `:rw` for any lock file or cron file touched, cap wait durations, respect context cancellation, and execute only commands permitted by the existing rshell command policy. + +### `nice` + +Type: new remediation command candidate + +Decision: do not add; only revisit if rshell gains explicit process-launch control. + +Evidence: the cron storm runbook uses `nice -n 15 /path/to/job.sh` as a durable prevention step by editing future cron entries so legitimate jobs run with lower CPU scheduling priority. + +Already covered? Investigation is covered by accepted or planned read-only commands such as `ps`, `uptime`, and `vmstat`. Durable cron remediation is intentionally not covered today; `crontab` persistent edits are deferred and `flock` job wrapping is rejected. If rshell later adds cron editing, it can write or validate cron text that invokes the host's `/usr/bin/nice`; that still does not require an rshell `nice` builtin because cron will launch the host command later, outside rshell. `nice` also does not provide immediate relief for already-running cron storms; existing hot processes would require a separate process-control primitive such as `renice`, `kill`, or cgroup/systemd CPU controls. + +Minimum subset: none initially. + +Target syntax: +- Rejected: `nice -n 15 /path/to/job.sh` +- Rejected initially: `nice COMMAND [ARG]...` +- Not an rshell builtin target: cron entry text such as `* * * * * /usr/bin/nice -n 15 /path/to/job.sh` +- Lower-risk but low-value subset: `nice` wrapping only other rshell builtins, if rshell ever supports priority-aware builtin process launch + +🟢 Fit: useful operator guidance, or future cron-remediation template text, for reducing the CPU scheduling priority of legitimate cron work without disabling the job. + +🔴 Agent fit: the valuable runbook form is a host command wrapper applied to future cron launches. If rshell edits the cron entry, the remediation artifact is the persistent schedule diff, not an rshell `nice` invocation. + +🔴 Scope: implementing `nice` inside rshell is only useful if rshell itself launches or wraps processes. Wrapping `/path/to/job.sh` would require executing arbitrary external job scripts or adding process-launch control semantics, which is outside rshell's builtin-only safety model. + +🔴 Remediation value: `nice` changes scheduling priority but does not cap CPU, prevent overlapping instances, stop fan-out, reduce memory use, or throttle disk I/O. It is weaker than `flock` for overlap prevention and weaker than cgroup/systemd controls for hard CPU limits. + +Implementation boundary: do not add an rshell `nice` builtin merely to support cron editing. A future cron-remediation design may emit, validate, or diff cron entries containing `/usr/bin/nice`, but should leave execution to the host cron daemon. Only revisit an rshell `nice` builtin if rshell adopts explicit process-launch control; in that case, do not invoke the host `nice` binary, do not wrap arbitrary external commands without a broader launch policy, and explain when softer scheduling priority is sufficient versus when hard CPU limits or overlap prevention are required. + +### `cleanup` + +Type: new remediation builtin candidate + +Decision: defer dedicated deletion-oriented cleanup primitive pending a separate deletion-safety design. + +Evidence: disk remediation runbooks need deletion behavior that is more structured than raw `rm`: keep at least one recent core dump while deleting older dumps, delete stale rotated logs without touching active logs, delete temp/session files older than a safe TTL, and recover inodes from large numbers of stale small files. + +Already covered? Partially covered for byte recovery by `truncate` and `logrotate`, which can reclaim space from explicit active files through `AllowedPaths` `:rw`. Not covered for inode recovery or stale-file deletion: raw `rm` is not exposed, `find -delete` is blocked, and `systemd-tmpfiles` / `coredumpctl clean` are rejected as broad host-policy engines rather than rshell-native cleanup primitives. + +Minimum subset: deferred. If accepted later, start with regular-file cleanup only; recursive directory deletion should require a separate decision. + +Target syntax: +- Deferred: exact syntax is not accepted yet. +- Possible future shape: `cleanup --dry-run --path /var/lib/systemd/coredump --type f --keep-newest 1` +- Possible future shape: `cleanup --dry-run --path /var/log/ --type f --name "*.log.[0-9]*"` +- Possible future shape: `cleanup --dry-run --path /tmp --max-depth 2 --type f --mtime +1` + +🟢 Fit: deletion is a real remediation gap for disk and inode incidents where truncation cannot recover the relevant resource. + +🟢 Fit: an rshell-native cleanup command can make destructive remediation explicit and auditable instead of inheriting silent POSIX `rm` behavior. + +🟢 Fit: regular-file cleanup maps directly to the highest-confidence runbook cases: core dumps, rotated logs, temp files, and stale session files. + +🔴 Safety: deletion is irreversible and can destroy forensic evidence, active session state, undelivered work, or files still needed by running processes. + +🔴 Scope: exact predicates, dry-run output, deletion limits, traversal behavior, symlink handling, and partial-failure semantics materially affect safety and should be designed before accepting implementation. + +🔴 Scope: recursive directory deletion is not part of the initial candidate; build-artifact directory cleanup (`node_modules`, `__pycache__`) has a larger blast radius and should be evaluated separately. + +Implementation boundary: do not implement `cleanup` as an alias or wrapper around host `rm`, `find`, `systemd-tmpfiles`, or `coredumpctl`. If revisited, require remediation mode, `AllowedPaths` `:rw`, dry-run/reporting, explicit path operands, regular-file defaults, no final symlink following, traversal and count limits, and context cancellation. + +### `rm` + +Type: new remediation command candidate + +Decision: do not add raw `rm`; defer deletion-oriented remediation to a dedicated future design. + +Evidence: disk remediation runbooks use deletion to recover space or inodes: the core-dump flood runbook uses `rm -f /var/crash/*` and deletion of all but the most recent systemd core dump; the unrotated / unbounded log runbook uses `rm -f` for old rotated log files; the inode exhaustion runbook uses `find ... -delete`, `xargs rm -f`, and `rm -rf` examples for stale small files or build artifacts. + +Already covered? Investigation is mostly covered by `df`, `du`, `find`, `ls`, `sort`, `head`, and planned `stat`. Remediation is intentionally only partially covered: `truncate` and `logrotate` recover byte space from explicit active files through `AllowedPaths` `:rw`, but they do not remove stale files and do not recover inodes consumed by many small files. Raw deletion is currently not exposed: `rm` is rejected as unknown and `find -delete` is blocked for sandbox safety. The separate deferred `cleanup` candidate is the preferred direction for any future deletion support. + +Minimum subset: none for raw `rm`. The deferred `cleanup` candidate may start with explicit regular-file cleanup only; recursive directory deletion is deferred. + +Target syntax: +- Rejected: `rm -f /var/crash/*` +- Rejected: `rm -f /var/log//*.log.[0-9]*` +- Rejected: `rm -rf /app/**/__pycache__` +- Deferred separate candidate: an rshell-native cleanup helper with explicit path operands, predicates, dry-run/reporting, and deletion limits. + +🟢 Fit: deletion is a real remediation gap for stale core dumps, old rotated logs, and inode exhaustion where truncation is insufficient. + +🟢 Fit: a future cleanup primitive could make deletion explicit, auditable, and constrained while still covering high-value disk runbook cases. + +🔴 Safety: `rm` is irreversible, can destroy forensic evidence, and has high blast radius when combined with globs or recursive flags. + +🔴 Agent fit: the command name carries broad POSIX expectations (`rm -rf`, silent success with `-f`, recursive tree deletion) that are stronger than the narrow remediation behavior rshell should expose. + +🔴 Scope: even a narrow `rm -f FILE...` subset would either surprise users by rejecting common forms or pressure rshell toward broader deletion semantics. It also does not encode runbook safeguards such as keeping the newest dump, deleting only stale files older than a TTL, or reporting exactly what would be removed. + +Implementation boundary: do not invoke the host `rm` binary, do not add an `rm` alias for a safer cleanup helper, and do not enable `find -delete` as part of this candidate. If deletion is revisited, evaluate it as a separate remediation design with explicit operands, dry-run output, `AllowedPaths` `:rw` enforcement, regular-file defaults, no final symlink following, traversal and count limits, context cancellation, and a separate decision before recursive directory deletion. + +### `systemd-tmpfiles` + +Type: new builtin / remediation command candidate + +Decision: do not add raw builtin; defer any deletion-oriented cleanup primitive + +Evidence: the temp-file and build-artifact disk-space runbook lists `systemd-tmpfiles --clean` as remediation, plus a permanent-fix path that writes `/etc/tmpfiles.d/tmp-cleanup.conf` and then applies it with `systemd-tmpfiles --clean`. + +Already covered? Investigation is mostly covered by `df`, `du`, `find`, `ls`, `sort`, and `head`. Remediation is intentionally only partially covered: `truncate` and `logrotate` recover space from explicit file operands through `AllowedPaths` `:rw`, while recursive deletion is not exposed and `find -delete` is blocked for sandbox safety. + +Minimum subset: none for a raw `systemd-tmpfiles` builtin. + +Target syntax: +- Rejected: `systemd-tmpfiles --clean` +- Rejected: `systemd-tmpfiles --dry-run --clean` + +🟢 Fit: this is a familiar Linux remediation command and matches operator runbooks for scheduled `/tmp` cleanup. + +🔴 Agent fit: the command text does not reveal the cleanup plan. With no explicit config operand, behavior is driven by host tmpfiles configuration, so an LLM agent cannot infer the deletion set from the script it produced. + +🔴 Scope: `systemd-tmpfiles` is a broad system-policy engine, not a narrow temp cleanup command. It can create, remove, clean, write, adjust modes/ownership, use globs, and apply age rules from config. Reimplementing enough `tmpfiles.d` semantics to be compatible would be large and high-risk; wrapping the host binary would bypass rshell's builtin safety model. + +🔴 Platform: Linux/systemd-only. This does not match rshell's preference for portable builtins unless a platform-specific command has unusually strong investigation value. + +Implementation boundary: do not invoke the host `systemd-tmpfiles` binary from a builtin, do not parse tmpfiles.d config, and do not add recursive deletion as part of this candidate. If rshell later supports deletion for remediation, prefer a separate rshell-native cleanup helper with explicit path operands, explicit age/size predicates, dry-run output, context cancellation, traversal limits, and `AllowedPaths` `:rw` enforcement. + +### `coredumpctl` + +Type: new investigation builtin candidate + +Decision: do not add initially; defer read-only `list` / `info`, reject dump extraction, debugger execution, and cleanup/remediation behavior. + +Evidence: the core-dump flood runbook uses `coredumpctl list` to identify recent crashes by timestamp, PID, executable, signal, and corefile state, and uses `coredumpctl info` to inspect the most recent dump during root-cause analysis. The same runbook also lists cleanup/remediation forms, but those cross into deletion and disk-recovery behavior rather than bounded investigation. + +Already covered? Disk-flood triage is mostly covered by existing commands: `find /var/crash /var/lib/systemd/coredump -mmin -10 -ls` confirms recent dump creation, `ls -lhtr ... | tail` shows newest dump files, `du -sh` measures dump-directory impact, `df -h` checks remaining capacity, and `cat /proc/sys/kernel/core_pattern` verifies where dumps are written. The missing value is systemd-journal crash metadata: executable path, PID, UID/GID, signal, and whether the corefile is present, missing, truncated, or journal-only. + +Minimum subset: none initially. Deferred possible subset: `coredumpctl list` and `coredumpctl info` only, with bounded row/time filters. + +Target syntax: +- Deferred: `coredumpctl list` +- Deferred: `coredumpctl list -n 20` +- Deferred: `coredumpctl info` +- Rejected: `coredumpctl dump` +- Rejected: `coredumpctl debug` +- Rejected: `coredumpctl gdb` +- Rejected: `coredumpctl --output=FILE dump` +- Rejected: any cleanup/deletion behavior such as `coredumpctl clean --disk-free 5G` +- Preferred disk-triage alternatives: `find /var/crash /var/lib/systemd/coredump -mmin -10 -ls`, `du -sh /var/crash/ /var/lib/systemd/coredump/`, `df -h /var/crash`, `cat /proc/sys/kernel/core_pattern` + +🟢 Fit: the read-only `list` / `info` shape is familiar, bounded, and useful when an agent must identify which executable is crash-looping and what signal produced the core dump. + +🔴 Incremental value: for the disk-space alert itself, existing file and filesystem commands already confirm active dump growth, estimate fill rate, measure reclaimed headroom, and verify dump destinations. + +🔴 Scope: `dump` can emit very large binary core files, `--output` writes files, and `debug` / `gdb` execute an external debugger. These do not fit rshell's bounded, builtin-only investigation model. + +🔴 Safety: cleanup/remediation belongs in a separate rshell-native cleanup design with explicit paths, dry-run output, `AllowedPaths` `:rw` enforcement, and deletion/traversal limits. Do not treat `coredumpctl` as the cleanup primitive. + +🔴 Platform: Linux/systemd-journal-only. Unlike `/proc`-backed commands, rshell does not yet have an explicit systemd journal metadata access boundary. + +Implementation boundary: do not invoke the host `coredumpctl` binary, do not read or emit raw core payloads, do not create output files, and do not spawn debuggers. If revisited, first define a journald/systemd metadata boundary: data source, field allowlist, whether journal reads bypass `AllowedPaths` or require explicit allowed journal paths, platform behavior, output caps, and privacy expectations for executable paths and user/group identifiers. + +### `sysctl` + +Type: new kernel-parameter investigation/remediation builtin candidate + +Decision: do not add initially; reject generic `sysctl` reads/writes and defer any narrow `kernel.core_pattern` remediation until rshell has a broader core-dump remediation design. + +Evidence: the core-dump flood runbook uses `sysctl -w kernel.core_pattern='|/bin/false'` to temporarily stop future dump generation while a crash loop is being fixed, then uses another `sysctl -w kernel.core_pattern=...` command to restore the original dump handler. The command reference mentions `sysctl` only for Core Dump Flood remediation. + +Already covered? Investigation is covered by `cat /proc/sys/kernel/core_pattern` when that proc path is allowed. The safer immediate remediation is to stop the crash loop with `systemctl stop ` before cleaning dumps. Durable prevention is better expressed as explicit service/systemd configuration changes such as `LimitCORE=0`, `DefaultLimitCORE=0`, or coredump storage limits. The remaining gap is only a temporary, host-wide `kernel.core_pattern` write. + +Minimum subset: none initially. + +Target syntax: +- Rejected: `sysctl -a` +- Rejected: `sysctl KEY` +- Rejected: `sysctl -w KEY=VALUE` +- Rejected: `sysctl -p` +- Rejected: `sysctl --system` +- Deferred possible subset, only if core-dump flood remains a priority gap: a remediation-mode, allowlisted way to set `kernel.core_pattern` for the crash-flood workflow. + +🟢 Fit: temporarily redirecting core dumps can stop future dump files while preserving the current running system and avoiding additional service disruption. + +🔴 Scenario weight: current evidence is a single remediation step in one scenario, not a repeated investigation or remediation pattern. + +🔴 Safety: `kernel.core_pattern` is host-wide. Changing it can suppress forensic artifacts for unrelated crashes, and restoring the wrong value can permanently alter crash handling. + +🔴 Agent fit: generic `sysctl` is an unbounded key/value interface over kernel state. Agents can easily choose plausible but dangerous keys that are unrelated to the scenario. + +🔴 Scope: full `sysctl` compatibility would include arbitrary reads, arbitrary writes, config-file loading via `-p` / `--system`, platform-specific namespaces, and persistent host-policy interaction. That is much broader than the runbook need. + +Implementation boundary: do not invoke the host `sysctl` binary and do not expose arbitrary `/proc/sys` reads or writes through a builtin. If revisited, avoid a generic `sysctl` surface; design a narrow core-dump remediation primitive or an allowlisted `kernel.core_pattern` operation with remediation-mode gating, explicit before/after reporting, original-value guidance, Linux-only behavior, and documentation that the write bypasses `AllowedPaths` because it mutates host kernel state rather than a user-selected filesystem path. + +### `tee` + +Type: new remediation-capable pipeline builtin candidate + +Decision: do not add initially; defer until scenarios show a need for write-through pipeline capture. + +Evidence: no current remediation scenario uses `tee`, and no current scenario requires preserving stdout while also writing the same stream to a file. + +Already covered? Mostly covered by remediation-mode `>` / `>>` for file writes, plus `truncate` and `logrotate` for explicit log remediation. Not covered: copying a pipeline stream to a file while continuing to pass the stream downstream. + +Minimum subset: none initially. + +Target syntax: +- Rejected initially: `cmd | tee FILE` +- Rejected initially: `cmd | tee -a FILE` +- Preferred alternative when stdout preservation is unnecessary: `cmd > FILE` or `cmd >> FILE` + +🟢 Fit: familiar command for capturing diagnostic output while continuing a pipeline. + +🔴 Evidence: no scenario demand under this repo's candidate decision lens. + +🔴 Safety: `tee` creates a second file-write surface outside shell redirection syntax, so it must duplicate remediation-mode gating, `AllowedPaths` `:rw` checks, no-symlink write handling, streaming limits, partial-failure semantics, and cancellation behavior. + +🔴 Scope: even narrow `tee [-a] FILE...` has multi-file writes, binary streaming, broken-pipe behavior, infinite-input handling, and cross-platform write semantics. + +Implementation boundary: do not invoke the host `tee` binary. If revisited, require remediation mode for file operands, enforce `AllowedPaths` `:rw` and no-symlink write semantics exactly like file redirections, reject unsupported GNU extensions, stream with bounded buffers, and respect context cancellation. + +### `tune2fs` + +Type: new Linux/ext filesystem command candidate + +Decision: do not add initially; prefer path- and mount-based inode diagnostics through `df` and planned `stat`. + +Evidence: the inode exhaustion runbook uses `tune2fs -l /dev/sda1 | grep -i inode` to inspect total and free inode counts for an ext filesystem. This is the only scenario evidence for `tune2fs`. + +Already covered? Mostly covered by `df -i` / `df -ih` for mount-level inode usage and the accepted `stat -f PATH...` candidate for path-targeted total/free inode checks. The larger remaining gap in the same runbook is `du --inodes`, not ext superblock inspection. + +Minimum subset: none initially. + +Target syntax: +- Rejected: `tune2fs -l /dev/sda1 | grep -i inode` +- Preferred alternatives: `df -i`, `df -ih`, `stat -f /var/spool/` + +🟢 Fit: read-only `-l` can expose ext2/3/4 superblock fields, including inode counts, in a familiar operator format. + +🔴 Scope: `tune2fs` is primarily a filesystem tuning command with many mutating flags. Supporting only `-l` is surprising, while supporting broader compatibility would create a high-risk filesystem administration surface. + +🔴 Safety: the useful operand is a user-selected block device such as `/dev/sda1`. Exposing direct device reads would either require expanding `AllowedPaths` semantics to block devices or creating a new host-device visibility bypass unlike hardcoded kernel-state readers such as `df`, `ss`, and `ip route`. + +🔴 Value: the incident answer is duplicated by `df -i` plus planned `stat -f`, and the command is Linux/ext-specific rather than a general filesystem diagnostic. + +Implementation boundary: do not invoke the host `tune2fs` binary and do not add a generic block-device superblock parser. If revisited, restrict the design to read-only Linux ext2/3/4 metadata, reject all mutating flags, define explicit block-device sandbox semantics, and document why the `df` / `stat -f` path is insufficient. + +## Scenario Gap Report + +This report compares `candidates.md` with the remediation runbooks under `scenarios/` and calls out important commands or command families that are used by the scenarios but do not currently have candidate entries. It intentionally does not override the decisions above; it is a triage list for future candidate records. + +### High-Impact Gaps + +#### `docker` command family + +Scenario evidence: Docker disk scenarios repeatedly depend on `docker system df`, `docker system prune`, `docker buildx du`, `docker buildx prune`, `docker images`, `docker ps`, and `docker volume prune`. + +Impact: high. These commands are central to diagnosing and remediating Docker image, build-cache, stopped-container, and orphaned-volume disk pressure. The scenarios distinguish Docker daemon-reported reclaimable space from raw `du` output, and in the overlay2 case `docker system df` is the authoritative view. + +Coverage gap: not covered by current candidates. Existing `df`, `du`, `find`, and `ls` can show that `/var/lib/docker` is large, but they cannot safely identify reclaimable Docker objects or perform daemon-aware pruning. + +Candidate direction: add a dedicated `docker` family candidate, likely with read-only investigation first (`docker system df`, `docker images`, `docker ps`, `docker volume ls`, `docker buildx du`) and remediation-mode-only prune operations if the safety model can support Docker-daemon authority. + +#### `psql` and PostgreSQL remediation queries + +Scenario evidence: database growth, orphaned PostgreSQL replication slot, and long-running database transaction scenarios use `psql` for database introspection and remediation, including `pg_replication_slots`, `pg_stat_activity`, `pg_cancel_backend`, `pg_terminate_backend`, `pg_drop_replication_slot`, `VACUUM`, `REINDEX`, `ALTER SYSTEM`, and `pg_reload_conf()`. + +Impact: high. These workflows can stop disk growth, unblock VACUUM, release WAL retention, and terminate harmful sessions, but the remediation actions can also roll back in-flight work, invalidate replicas or CDC consumers, lock tables, or alter persistent database policy. + +Coverage gap: not covered by current candidates. Filesystem commands can show WAL or database directories growing, but they cannot identify the database object or session causing the growth, and they cannot perform database-native cleanup. + +Candidate direction: add a `psql` / PostgreSQL candidate even if the decision is to reject or defer most behavior. A useful candidate record should separate read-only monitoring queries from high-risk remediation queries and should explicitly address credential handling, query allowlisting, output bounds, and whether rshell should ever execute database-mutating SQL. + +#### `mysql` + +Scenario evidence: the database-growth scenario uses `mysql` to inspect and purge binary logs, including `SHOW BINARY LOGS`, `SHOW VARIABLES LIKE ...`, `PURGE BINARY LOGS`, and possible `SET GLOBAL binlog_expire_logs_seconds`. + +Impact: medium-high. MySQL binary log accumulation can fill the database volume, and `PURGE BINARY LOGS` can reclaim space without service downtime when used correctly. Misuse can break replication or remove logs needed for recovery. + +Coverage gap: not covered by current candidates. Existing filesystem commands can show `/var/lib/mysql` growth, but they cannot distinguish table data from binary logs or apply database-native retention changes. + +Candidate direction: add a `mysql` candidate or include it in a broader database-client candidate. As with PostgreSQL, separate read-only introspection from remediation and document why arbitrary SQL execution is likely outside the initial rshell scope. + +#### Process `/proc` introspection + +Scenario evidence: memory and CPU scenarios directly use `/proc//status`, `/proc//smaps_rollup`, `/proc//stat`, `/proc//wchan`, and `/proc//fd` counts. Existing accepted candidates partially cover this through `ps` memory fields and `pmap`, but not all scenario needs. + +Impact: high for investigation, low for remediation. These reads are key for distinguishing memory leaks, GC pressure, thread growth, kernel wait state, and file-descriptor growth. They are also naturally bounded when narrowed to one PID. + +Coverage gap: partially covered, but not explicitly tracked as a candidate. Raw `cat` and `ls` can read these paths only when procfs paths are allowed, and raw `/proc` formats are easy for agents to parse incorrectly. `pmap` covers memory maps, but not `wchan`, selected status fields, fd counts, or CPU tick sampling from `stat`. + +Candidate direction: add a narrow process-introspection candidate or explicitly document that these remain path-based reads under `AllowedPaths`. A candidate should avoid argv/env disclosure unless rshell deliberately changes its process privacy boundary. + +### Lower-Priority Gaps + +#### `sleep` and `watch` + +Scenario evidence: several runbooks use `sleep` in polling loops and `watch` for repeated visual checks of RSS, socket counts, WAL size, and core-dump creation. + +Impact: medium. Repeated measurement is useful for verification and trend confirmation, but it can often be replaced by bounded commands such as `vmstat DELAY COUNT` or explicit short loops. + +Candidate direction: consider a bounded `sleep` builtin if shell loops are intended to be first-class. Defer or reject `watch` unless rshell wants a general repeated-command surface with strict interval, count, timeout, and output caps. + +#### Package and build cache tools: `npm`, `pip`, `gradle`, `go` + +Scenario evidence: the temp-file and build-artifact scenario uses `npm cache verify`, `npm cache clean --force`, `pip cache info`, `pip cache purge`, `gradle --stop`, and `go clean -modcache`. + +Impact: medium. These commands can reclaim build-host cache space, but they are ecosystem-specific and usually affect future build speed rather than immediate production service health. `gradle --stop` can kill in-progress builds. + +Candidate direction: defer initially. Prefer generic filesystem investigation plus a future rshell-native cleanup primitive for stale cache directories before adding package-manager-specific CLIs. + +#### Mail queue commands: `postqueue` and `postsuper` + +Scenario evidence: the inode-exhaustion scenario uses `postqueue -p` to inspect queue depth and `postsuper -d ALL deferred` to delete deferred mail. + +Impact: narrow but high-risk. Mail queue deletion can recover inodes, but `postsuper -d ALL deferred` irreversibly drops undelivered mail. + +Candidate direction: reject initially or add an explicit deferred/rejected candidate if mail-queue inode exhaustion becomes a recurring priority. Do not expose broad mail-queue mutation without a dedicated policy and confirmation model. + +#### Destructive filesystem administration: `mkfs.ext4` + +Scenario evidence: inode exhaustion mentions `mkfs.ext4 -i 4096 /dev/sdb1` as a long-term filesystem reformatting option. + +Impact: very high risk, low rshell fit. Reformatting a filesystem requires outage planning, device targeting, backups, and human approval. + +Candidate direction: explicitly reject if it is likely to be proposed. It should remain outside rshell remediation scope. diff --git a/docs/system-service-journal-config-research.md b/docs/system-service-journal-config-research.md new file mode 100644 index 00000000..75b766ec --- /dev/null +++ b/docs/system-service-journal-config-research.md @@ -0,0 +1,243 @@ +# System Service And Journal Config Research + +## Context + +`candidates.md` accepts narrow `systemctl` and `journalctl` builtins because the +remediation scenarios repeatedly need service state, service control, journald +logs, kernel journal evidence, and journald disk recovery. + +The key design problem is that these commands expose or mutate host state that +is not governed by `AllowedPaths`: + +- `AllowedCommands` controls whether a builtin may run at all. +- `AllowedPaths` controls user-supplied filesystem access. +- `ModeRemediation` controls whether write-capable behavior is available. +- None of the existing knobs can express "this service may be restarted" or + "journald vacuum is allowed". + +The research goal was to find the smallest additional config surface that keeps +rshell useful for scenario-driven AI agents without silently expanding host +visibility or host mutation. + +## Scenario Evidence + +The scenarios require both service-scoped operations and a small set of +unscoped read-only discovery operations. + +| Need | Scenario evidence | Config implication | +| --- | --- | --- | +| Inspect and control a known service | `systemctl status `, `restart`, `stop`, `start`, `reload` appear across crash-loop, runaway CPU, memory leak, cache growth, GC pressure, Docker, and log scenarios. | Needs exact service policy plus verb-level control. | +| Read logs for a known service | `journalctl -u -n ...` and `journalctl -u --since ...` appear in crash-loop, cron storm, and core dump workflows. | Service read policy should authorize unit-scoped journal reads. | +| Discover failed units | `systemctl --state=failed` is used to identify crash-looping units. | This is unscoped read-only host discovery. | +| Discover maintenance timers | `systemctl list-timers | grep ...` is used for logrotate and database retention checks. | This is unscoped read-only host discovery. | +| Check journald disk use | `journalctl --disk-usage` is used in the unbounded log scenario. | This is unscoped read-only journal metadata. | +| Check kernel/OOM evidence | `journalctl -k --since ...` is used for OOM and kernel-kill investigation. | This is unscoped read-only kernel journal access. | +| Vacuum journald | `journalctl --vacuum-size=500M` is used for disk recovery. | This is unscoped mutation and needs explicit opt-in plus remediation mode. | + +## Agreed Policy Model + +The design should add a system-service policy boundary, but keep the config +minimal. + +`AllowedCommands` remains the first gate. If `rshell:systemctl` or +`rshell:journalctl` is not allowed, none of the corresponding command behavior +is reachable. + +The new service config should use generic "system service" names rather than +public API names tied to systemd. The v1 implementation can still be explicitly +Linux/systemd-backed. + +Proposed API shape: + +```go +interp.AllowedCommands([]string{ + "rshell:systemctl", + "rshell:journalctl", +}) + +interp.AllowedSystemServiceRead([]string{ + "mysql.service", + "cron.service", + "logrotate.timer", +}) + +interp.AllowedSystemServiceControl([]interp.SystemServiceControlGrant{ + { + Service: "mysql.service", + Verbs: []interp.SystemServiceVerb{ + interp.SystemServiceRestart, + interp.SystemServiceReload, + }, + }, +}) + +interp.AllowSystemJournalVacuum() +interp.WithMode(interp.ModeRemediation) +``` + +Suggested public types: + +```go +type SystemServiceVerb string + +const ( + SystemServiceStart SystemServiceVerb = "start" + SystemServiceStop SystemServiceVerb = "stop" + SystemServiceRestart SystemServiceVerb = "restart" + SystemServiceReload SystemServiceVerb = "reload" +) + +type SystemServiceControlGrant struct { + Service string + Verbs []SystemServiceVerb +} +``` + +## Authorization Semantics + +| Operation | Required config | +| --- | --- | +| `systemctl --state=failed` | `AllowedCommands("rshell:systemctl")` | +| `systemctl list-timers` | `AllowedCommands("rshell:systemctl")` | +| `systemctl status UNIT` | `AllowedCommands("rshell:systemctl")` plus `AllowedSystemServiceRead(UNIT)` or a control grant for `UNIT` | +| `systemctl show UNIT --property=NRestarts` | `AllowedCommands("rshell:systemctl")` plus `AllowedSystemServiceRead(UNIT)` or a control grant for `UNIT` | +| `systemctl start/stop/restart/reload UNIT` | `AllowedCommands("rshell:systemctl")` plus `AllowedSystemServiceControl` for the exact verb and `UNIT`, plus remediation mode | +| `journalctl --disk-usage` | `AllowedCommands("rshell:journalctl")` | +| `journalctl -k --since TIME` | `AllowedCommands("rshell:journalctl")`; output must be bounded | +| `journalctl -u UNIT -n N --no-pager` | `AllowedCommands("rshell:journalctl")` plus `AllowedSystemServiceRead(UNIT)` or a control grant for `UNIT` | +| `journalctl -u UNIT --since TIME` | `AllowedCommands("rshell:journalctl")` plus `AllowedSystemServiceRead(UNIT)` or a control grant for `UNIT`; output must be bounded | +| `journalctl --vacuum-size=SIZE` | `AllowedCommands("rshell:journalctl")` plus `AllowSystemJournalVacuum()` plus remediation mode | + +Control grants imply read access for the same service. This supports pre-checks +and post-checks without forcing callers to duplicate service names in both read +and control config. + +## Minimal Config Decision + +Do not add separate config for narrow unscoped read-only discovery in v1. + +Once an operator explicitly allows `rshell:systemctl`, the command may expose +only the narrow unscoped read-only surfaces needed by scenarios: + +- failed unit discovery +- timer discovery + +Once an operator explicitly allows `rshell:journalctl`, the command may expose +only the narrow unscoped read-only journal surfaces needed by scenarios: + +- journal disk usage +- bounded kernel journal reads + +This mirrors existing rshell precedent: commands such as `df`, `ss`, `ip route`, +and `ps` expose bounded host metadata once the command itself is allowed. They +do not require an additional "host observability" config. + +Do require explicit config for unscoped mutation. `journalctl --vacuum-size=SIZE` +deletes host-wide historical journal data, so command allowlisting and +remediation mode are not enough on their own. + +## Service Identity Rules + +Configured service identities should be exact and canonical. Do not support +globs, regexes, broad prefixes, or script-controlled matching. + +The command parser may apply limited systemctl-compatible normalization before +policy checks. For example, a bare `mysql` command operand can normalize to +`mysql.service`, while explicit unit names such as `logrotate.timer` remain +exact. + +Examples: + +```go +interp.AllowedSystemServiceRead([]string{"mysql.service"}) +``` + +can authorize: + +```sh +systemctl status mysql +systemctl status mysql.service +journalctl -u mysql -n 50 --no-pager +journalctl -u mysql.service -n 50 --no-pager +``` + +but it should not authorize arbitrary aliases, globs, or unrelated units. + +## Deferred Scope + +Do not include full `systemctl` or `journalctl` compatibility in v1. + +Deferred `systemctl` examples: + +- `mask` +- `unmask` +- `enable` +- `disable` +- `edit` +- `daemon-reload` +- `daemon-reexec` +- arbitrary `show` +- arbitrary `list-units` +- job management +- environment changes +- unit file writes +- user-manager or remote-machine modes + +Deferred `journalctl` examples: + +- live follow +- cursor/export modes +- JSON and other output-format variants +- boot selection +- catalog output +- arbitrary field queries +- unscoped whole-journal reads such as `journalctl --since TIME` +- `--directory` +- `--file` +- `--root` +- `--image` +- remote journal sources +- `--vacuum-time` +- `--vacuum-files` + +`daemon-reload` and `daemon-reexec` are intentionally deferred even though +scenarios mention them. They are unscoped systemd manager mutations and mostly +make sense after unit/config-file edits that rshell is not modeling safely in +this v1 policy. + +## Implementation Recommendation + +The research recommendation is to avoid invoking host `systemctl` or +`journalctl` binaries as raw wrappers. Use native/internal systemd and journal +access paths or tightly scoped internal adapters. + +Rationale: + +- The command grammar should be the supported rshell subset, not the full host + binary surface. +- Argument validation, output bounds, and context cancellation are easier to + enforce internally. +- Wrapper behavior would depend on host systemd/journalctl versions. +- A raw wrapper risks accidentally accepting flags outside the intended safety + model. + +This remains an implementation recommendation rather than a separately agreed +policy decision. + +## Summary + +The minimal viable config addition is: + +- `AllowedSystemServiceRead([]string{...})` +- `AllowedSystemServiceControl([]SystemServiceControlGrant{...})` +- `AllowSystemJournalVacuum()` + +Everything else should continue to flow through existing gates: + +- `AllowedCommands` gates command availability. +- `AllowedPaths` gates user-supplied filesystem paths. +- `ModeRemediation` gates mutations. + +This keeps service control explicit, preserves a small API surface, supports the +scenario-backed unscoped read-only discovery commands, and avoids granting +unscoped destructive journal cleanup without deliberate opt-in. diff --git a/scenarios/cpu/cpu-diagnosing-and-remediating-a-cron-job-storm.md b/scenarios/cpu/cpu-diagnosing-and-remediating-a-cron-job-storm.md new file mode 100644 index 00000000..25856279 --- /dev/null +++ b/scenarios/cpu/cpu-diagnosing-and-remediating-a-cron-job-storm.md @@ -0,0 +1,276 @@ +# CPU - Cron Storm + +**Signal:** `system.cpu.user` or `system.load.1` spikes at predictable, recurring intervals aligned with a cron schedule +**IssueType:** `cpu_usage` +**Metric (typical):** `system.cpu.user`, `system.load.1`, `system.load.5` + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`top`| investigation| `top -b -n 1 \| head -30` +`ps`| investigation| `ps aux --sort=-%cpu \| head -20` · `ps -eo pid,ppid,cmd,etime \| grep ` +`crontab`| investigation| `crontab -l` · `cat /etc/cron.d/*` · `cat /etc/crontab` +`journalctl`| investigation| `journalctl -u cron --since "1 hour ago"` · `journalctl -u crond` +`grep`| investigation| `grep CRON /var/log/syslog \| tail -50` +`pgrep`| investigation| `pgrep -a -f ` +`uptime`| investigation| `uptime` +`vmstat`| investigation| `vmstat 1 15` +`kill`| remediation| `kill ` · `pkill -f ` +`flock`| remediation| `flock -n /var/lock/.lock ` +`nice`| remediation| `nice -n 15 /path/to/job.sh` + +* * * + +## What Happens + +Multiple cron jobs fire simultaneously and compete for CPU. A single job can also trigger a storm if it fans out parallel child processes, or if previous instances have not finished before the next run starts, causing overlap accumulation over successive intervals. + +Common causes: + + * All jobs scheduled at `0 * * * *` or `0 0 * * *` without schedule staggering + + * A job that takes longer than its run interval spawns a new instance while the previous one is still running; over time N overlapping instances pin the host + + * A job that itself parallelises heavily (`xargs -P`, `parallel`, or spawning many background subshells) + + * A configuration management tool (Puppet, Chef, Ansible) triggered by cron across many hosts simultaneously, causing a fleet-wide CPU spike + + * A maintenance cron (log rotation, database vacuuming, cache invalidation) that is heavier than expected and was not sized against the host's available CPU headroom + + + + +* * * + +## Detection + +Detected via `system.cpu.user` or `system.load.1` breaching the monitor threshold. The defining characteristic is periodicity: the spike recurs at fixed intervals aligned with a cron schedule (every hour, every 15 minutes, every midnight). This distinguishes it from a runaway loop (continuous, single-PID) or GC pressure (correlated with memory). + +**Correlated signals to check:** + + * `system.load.1` spikes sharply and decays within minutes (duration approximates job runtime) + + * `system.cpu.iowait` elevated if the job is disk-heavy (log rotation, backup, database maintenance) + + * Application request error rate spiking at the same time: cron-induced CPU saturation can delay application thread scheduling + + * Multiple hosts spiking simultaneously: suggests a fleet-wide scheduled job (Puppet runs, centralized cron) + + + + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Commands must run on the host +Access to crontab files for the running user(s) and `/etc/cron.d/`| Needed to identify scheduled jobs and their timing +Knowledge of the approximate time the spike occurs| Allows correlation of the spike time to cron schedule entries + +### Steps + + 1. **Confirm the CPU spike is periodic** + + + + + + # In Datadog: view system.cpu.user over 24 h with 1 h granularity + # Look for peaks at regular intervals (every hour, every 15 min, midnight, etc.) + # Note the exact minute-of-hour when spikes occur + + 2. **Identify which processes are consuming CPU during the spike** + + + + + + # During or just after a spike: + ps aux --sort=-%cpu | head -20 + # Note: process name, PID, CPU%, elapsed time (TIME column) + + top -b -n 1 | head -30 + + 3. **Correlate spike time to cron schedule entries** + + + + + + # System-wide cron jobs + cat /etc/crontab + cat /etc/cron.d/* + ls -la /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ + + # Per-user crontab + crontab -l + # Run as other users if needed: sudo crontab -l -u + + 4. **Check cron execution logs** + + + + + + # systemd-based cron (Debian/Ubuntu) + journalctl -u cron --since "2 hours ago" | grep -E "CMD|session" + + # syslog-based + grep CRON /var/log/syslog | tail -100 + # Look for: (user) CMD (/path/to/script) entries matching the spike times + + 5. **Check for overlapping job instances** + + + + + + pgrep -a -f + # Multiple PIDs with different start times = overlapping instances + + ps -eo pid,ppid,cmd,etime | grep + # ETIME shows elapsed time; overlapping instances have different values + + 6. **Check load and CPU over a short window** + + + + + + vmstat 1 30 + # r column (run queue): if consistently > number of CPU cores, system is CPU-saturated + uptime + # Load average above CPU core count indicates saturation + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +Access to edit crontab files| Required to reschedule or disable jobs +Understand the job's purpose before killing it| Some jobs (backups, log rotation) must complete; killing mid-run may leave state inconsistent +Coordinate schedule changes with team if the job is shared infrastructure| Rescheduling a job used by multiple teams requires alignment + +### Immediate Relief + +**Kill excess overlapping instances (keep the oldest running instance):** + + + # List all instances sorted by elapsed time + ps -eo pid,etime,cmd | grep | sort -k2 + # Kill all but the longest-running (oldest) instance + kill + +**If the job is safe to stop entirely:** + + + pkill -f + +### Prevent Overlapping Runs + +**Add**`flock` to enforce a single running instance: + + + # -n exits immediately if the lock is already held (no queuing) + * * * * * /usr/bin/flock -n /var/lock/myjob.lock /path/to/job.sh + + # Or wait up to 60 s before giving up + * * * * * /usr/bin/flock -w 60 /var/lock/myjob.lock /path/to/job.sh + +### Stagger Schedules + +**Offset jobs that fire at round intervals:** + + + # Before: all fire at the top of the hour + 0 * * * * /path/to/job-a.sh + 0 * * * * /path/to/job-b.sh + 0 * * * * /path/to/job-c.sh + + # After: spread across the hour + 2 * * * * /path/to/job-a.sh + 17 * * * * /path/to/job-b.sh + 34 * * * * /path/to/job-c.sh + +**Use**`RandomizedDelaySec` (systemd timers) for fleet-wide staggering: + + + # /etc/systemd/system/myjob.timer + [Timer] + OnCalendar=hourly + RandomizedDelaySec=300 # up to 5-minute random offset per host + +### Reduce Job Resource Usage + +**Apply CPU scheduling limits to cap blast radius:** + + + # Lower CPU scheduling priority + * * * * * nice -n 15 /path/to/job.sh + + # Or throttle via cgroup using systemd-run + * * * * * systemd-run --scope --slice=background.slice --property=CPUQuota=25% /path/to/job.sh + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +Inspecting `ps`, crontab files, logs| None| Read-only +`kill` on a single overlapping job instance| None for the application| The job instance is terminated; verify it is not mid-write to a critical file before killing +`pkill -f `| None for the application; job work is lost| Kills all matching instances; confirm the job is safe to abort +Editing crontab to stagger schedules| None immediately| Only affects future runs +Adding `flock` to cron entry| None immediately| Only affects future runs; prevents overlap going forward +Adding `nice` or `CPUQuota`| None| Throttles future runs; does not stop current jobs +Converting to systemd timer with `RandomizedDelaySec`| None| Requires creating `.service` and `.timer` units; the delay applies from the next scheduled run + +**Killing a mid-run backup or maintenance job is the highest-risk action.** Confirm the job is idempotent or can be safely re-run before terminating it. + +* * * + +## Verification + + + # Confirm CPU has recovered + top -b -n 1 | head -5 + uptime # load average should be below CPU core count + + # Confirm no overlapping instances remain + pgrep -a -f + + # Confirm flock is effective: attempt to acquire the lock while the job is running + flock -n /var/lock/myjob.lock echo "got lock" || echo "lock held — only one instance running" + +In Datadog, verify: + + * `system.cpu.user` no longer spikes at the previously problematic schedule times + + * `system.load.1` stays below CPU core count throughout the schedule window + + * If staggered: the spike is smaller and offset from the original schedule time + + + + +* * * + +## Related Scenarios + + * If the CPU spike is not periodic but still caused by many short-lived processes, check for a process supervisor restarting a crashing service in a tight loop (`systemctl status ` showing many restarts). + + * If the spike occurs on all hosts simultaneously, the job is triggered from a central scheduler (Rundeck, Airflow, CI pipeline) rather than local cron; the staggering fix must be applied at the scheduler level. + + * If `system.cpu.iowait` is elevated alongside CPU, the job is disk-bound; address disk throughput (see disk space scenarios) in addition to CPU scheduling. + + diff --git a/scenarios/cpu/cpu-diagnosing-and-remediating-a-runaway-process-or-infinite-loop.md b/scenarios/cpu/cpu-diagnosing-and-remediating-a-runaway-process-or-infinite-loop.md new file mode 100644 index 00000000..ccdb7e43 --- /dev/null +++ b/scenarios/cpu/cpu-diagnosing-and-remediating-a-runaway-process-or-infinite-loop.md @@ -0,0 +1,279 @@ +# CPU - Runaway Process / Infinite Loop + +**Signal:** `system.cpu.user` sustained high (80–100%) attributable to a single PID; does not resolve on its own +**IssueType:** `cpu_usage` +**Metric (typical):** `system.cpu.user`, `system.load.1` + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`top`| investigation| `top -o %CPU` · `top -p ` +`ps`| investigation| `ps aux --sort=-%cpu \| head -10` · `ps -p -o pid,ppid,cmd,pcpu,etime` +`/proc//wchan`| investigation| `cat /proc//wchan` +`/proc//stat`| investigation| `awk '{print "utime=" $14 " stime=" $15}' /proc//stat` +`strace`| investigation| `strace -p -c -f` · `strace -p -e trace=all` +`perf`| investigation| `perf top -p ` +`uptime`| investigation| `uptime` +`vmstat`| investigation| `vmstat 1 10` +`kill`| remediation| `kill ` · `kill -9 ` +`systemctl`| remediation| `systemctl stop ` · `systemctl status ` + +* * * + +## What Happens + +A process enters a loop that does not terminate and continuously consumes one CPU core (or more if multi-threaded). Unlike a cron storm (many short-lived processes sharing CPU), this is a single long-lived process pinned at high CPU indefinitely. Unlike GC pressure (CPU and memory rise together), CPU is high but memory is typically stable. + +Common causes: + + * A logic bug where a loop termination condition is never met (off-by-one error, concurrent write invalidating the exit condition, expected event never firing) + + * Tight retry logic with no backoff that loops on a transient error (e.g., retrying a failed network call in a `while true` without sleep) + + * Recursive processing of a cyclic data structure (a graph or linked list with a cycle that the traversal code does not detect) + + * A race condition where a thread spins checking a flag that is never set by another thread + + * A misconfigured event loop that re-fires immediately instead of waiting for I/O + + + + +* * * + +## Detection + +Detected via `system.cpu.user` sustained above the monitor threshold without decaying. The distinguishing characteristic is a single PID consistently at or near 100% CPU for minutes or hours — a flat, continuous, indefinite plateau. Unlike a cron storm (spikes and decays) or GC pressure (bursty with memory correlation), this does not self-resolve. + +**Correlated signals to check:** + + * `system.load.1` elevated but not at extreme multiples of core count (one runaway thread saturates one core, not all) + + * Application request latency rising: if the runaway process is the application itself, all threads compete with the spinning thread for CPU time + + * Application monitor going to NO_DATA or ALERT if the event loop is blocked by the spinning thread + + * No corresponding `system.mem.used` rise: if memory is also climbing, consider GC pressure instead + + + + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Commands must run on the host +Ability to run `top`, `ps`, `strace`| Standard tools; `strace` requires root or `CAP_SYS_PTRACE` +`perf` installed if deep CPU profiling is needed| Optional; available via `linux-tools-common` / `perf` package + +### Steps + + 1. **Identify the spinning PID** + + + + + + top -o %CPU + # Look for a single process with CPU% near 100 that does not decay + # Note: on a multi-core host, 100% = one full core; 400% = four cores + + ps aux --sort=-%cpu | head -10 + # Confirms PID, username, exact command, and how long it has been running (TIME column) + + 2. **Confirm it is a single long-running process** + + + + + + ps -p -o pid,ppid,cmd,pcpu,etime + # ETIME: a process running for > 5 minutes at 100% CPU is a runaway + # Short ETIME that repeats = process is spawning repeatedly (cron storm pattern instead) + + 3. **Check what the process is doing at the kernel level** + + + + + + # What kernel function is the process sleeping in (if any)? + cat /proc//wchan + # Common values: + # 0 or "running": in userspace — tight CPU loop with no kernel wait + # futex_wait_queue_me: waiting on a mutex (possible spin or deadlock) + # do_select / ep_poll: waiting for I/O (not a CPU loop) + + # Syscall frequency: high = active I/O; near-zero = pure userspace CPU loop + strace -p -c -f -e trace=all -- sleep 5 + + 4. **Sample the hot code path (requires**`perf`) + + + + + + perf top -p -d 5 + # Shows which functions are consuming CPU cycles + # The top symbol usually identifies the looping function + + 5. **Check CPU time breakdown (userspace vs kernel)** + + + + + + # Fields 14 (utime) and 15 (stime) in clock ticks + awk '{print "utime=" $14 " stime=" $15}' /proc//stat + # Take two samples 10 s apart; if utime grows fast but stime is flat: pure userspace loop + + 6. **Check thread count and state** + + + + + + ls /proc//task | wc -l + cat /proc//status | grep Threads + # If many threads all in 'R' state: multi-threaded spin + + 7. **Determine if the process is systemd-managed** + + + + + + systemctl status 2>/dev/null + # If Restart=always, killing the process immediately respawns it; use systemctl stop instead + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +Confirm the PID belongs to the expected process before killing| Killing the wrong process causes unplanned downtime +Check `systemctl status` to understand restart behavior| If `Restart=always`, a `kill` immediately respawns the process; use `systemctl stop` instead +Collect `strace` or `perf` output before killing if root cause analysis is needed| The spinning process is the primary diagnostic artifact; killing it destroys the evidence +Note whether the process holds critical state| An abrupt kill may leave data in an inconsistent state + +### Immediate Relief + +**For a systemd-managed service:** + + + # Stop the service cleanly (suppresses auto-restart) + systemctl stop + + # Verify it stopped + systemctl status + + # Restart when ready + systemctl start + +**For an unmanaged process:** + + + # Graceful termination first + kill + + # Confirm the process is gone + sleep 3 && ps -p + + # If SIGTERM is ignored (common in tight loops): + kill -9 # SIGKILL — cannot be caught or ignored + +**For a multi-threaded runaway:** + + + # Kill the entire process group + PGID=$(ps -o pgid= -p | tr -d ' ') + kill -9 -$PGID + +### Prevent Recurrence + +**Add a CPU quota to cap blast radius while the fix is developed:** + + + # /etc/systemd/system/.service.d/cpu-limit.conf + [Service] + CPUQuota=200% # Max 2 full cores; adjust to expected peak + + + systemctl daemon-reload + systemctl restart + +With `CPUQuota` set, a runaway loop is throttled automatically without killing the process, preserving it for diagnosis. + +**Fix the underlying loop condition.** OS-level data to hand to the developer: + +Observation| Likely code path +---|--- +Near-zero syscalls, `wchan = 0`| Pure CPU loop; look for `while(true)` or missing termination condition +High `futex` syscall rate| Spin on a lock; look for a mutex that is never released +High `read`/`write` syscall rate| I/O retry loop; look for retry logic without backoff + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +`top`, `ps`, `strace`, `perf top`| None| Read-only; `strace` attaches to the process but does not stop it +`systemctl stop `| **Service goes down**| Clean shutdown; auto-restart suppressed; requires explicit `systemctl start` to restore +`kill ` (SIGTERM)| **Process terminates**| Graceful if the signal is handled; in-flight requests may be dropped +`kill -9 ` (SIGKILL)| **Abrupt process termination**| No cleanup; in-flight requests lost; on-disk state may be inconsistent +`kill -9 -`| **Entire process group terminated**| All child processes killed simultaneously +Adding `CPUQuota` \+ `daemon-reload`| None| Takes effect on next service start +Adding `CPUQuota` \+ restart| **Brief service interruption**| Same as restart + +`systemctl stop` is always preferred over `kill -9` for systemd-managed services: it runs `ExecStop`, flushes in-flight work, and suppresses auto-restart until you are ready. + +* * * + +## Verification + + + # Confirm CPU has recovered + top -b -n 1 | head -10 + # The spinning PID should no longer appear near the top + + uptime + # Load average should drop below CPU core count within 1-2 minutes + + # If service was restarted, confirm it is healthy + systemctl status + +In Datadog, verify: + + * `system.cpu.user` drops to baseline after the process is terminated + + * `system.load.1` recovers within 1–2 minutes + + * The application service monitor returns to OK state + + * CPU does not immediately climb back to 100% — if it does, the service restarted with the same bug still present; disable auto-restart and investigate before re-enabling + + + + +* * * + +## Related Scenarios + + * If CPU immediately returns to 100% after a restart, disable auto-restart (`systemctl edit ` → `Restart=no`) and set `CPUQuota` before re-enabling. + + * If `strace` shows the process is mostly waiting in `epoll_wait` or `select` (I/O wait, not a CPU spin), the high CPU reading may be misleading; check `system.cpu.iowait` and focus on the I/O subsystem instead. + + * If multiple PIDs are each near 100%, see the Cron Storm scenario. + + diff --git a/scenarios/cpu/cpu-gc-pressure-and-runtime-collector-thrashing.md b/scenarios/cpu/cpu-gc-pressure-and-runtime-collector-thrashing.md new file mode 100644 index 00000000..e5cd50cc --- /dev/null +++ b/scenarios/cpu/cpu-gc-pressure-and-runtime-collector-thrashing.md @@ -0,0 +1,320 @@ +# CPU - GC Pressure (Runtime Collector Thrashing) + +**Signal:** `system.cpu.user` elevated in a long-running process; rises over time or stays persistently high; correlated with `system.mem.used` sawtooth +**IssueType:** `cpu_usage` +**Metric (typical):** `system.cpu.user`, `system.mem.used` + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`top`| investigation| `top -p ` · `top -o %CPU` +`ps`| investigation| `ps -p -o pid,cmd,pcpu,pmem,etime` +`/proc//stat`| investigation| `awk '{print "utime=" $14 " stime=" $15}' /proc//stat` +`/proc//status`| investigation| `cat /proc//status \| grep -E 'VmRSS\|VmSize'` +`vmstat`| investigation| `vmstat 1 30` +`find`| investigation| `find /var/log /tmp /opt -name "*.log" -path "*gc*" 2>/dev/null` +`grep`| investigation| `cat /proc//cmdline \| tr '\\0' ' '` +`dmesg`| investigation| `dmesg \| grep -i "oom\|killed process"` +`systemctl`| remediation| `systemctl restart ` + +* * * + +## What Happens + +A runtime's garbage collector is spending an increasing fraction of CPU time trying to reclaim memory. At OS level, the process's CPU consumption rises without a corresponding increase in application throughput. The memory metric simultaneously shows a rising sawtooth: the heap fills, the collector runs (consuming CPU), reclaims some memory, and the cycle restarts — but each cycle the heap fills faster or to a higher floor than before. + +As heap pressure increases, the collector runs more frequently and for longer. At the extreme ("GC thrashing"), the process spends more CPU time collecting than running application code, and throughput collapses despite high CPU usage. + +Common causes: + + * Heap size is too small for the current workload — the configured max heap is frequently hit and the collector is forced to run at high frequency + + * Allocation rate has increased (traffic spike, batch job, data ingestion surge) without a corresponding heap headroom increase + + * A memory leak is slowly reducing available heap space, forcing more frequent collections (GC pressure is masking an underlying leak) + + * Short-lived but large objects allocated at high rate, overwhelming the young generation collector + + * Objects surviving collection longer than expected (large retained object graphs, long-lived request contexts) + + + + +* * * + +## Detection + +Detected via `system.cpu.user` sustained above the monitor threshold in a long-running process, correlated with `system.mem.used` showing a sawtooth pattern. The distinguishing characteristics from other CPU scenarios: + + * CPU and memory metrics move together: when memory rises toward the heap limit, CPU spikes as the collector runs; when the collection completes, CPU briefly drops and memory drops, then the cycle repeats + + * The CPU spike is in the process running the runtime (JVM, Node.js, Python), not in a separate system process + + * `system.cpu.sys` remains low (GC runs in userspace; kernel involvement is minimal) + + * The pattern is bursty and periodic, unlike a runaway loop (which is flat and continuous) + + + + +**Correlated signals to check:** + + * `system.mem.used` sawtooth with a rising floor confirms heap pressure alongside CPU + + * Application request latency rising: GC pause times add directly to P99/P999 latency + + * Application error spans: long GC pauses cause downstream timeouts that cascade into errors + + * `system.cpu.user` spikes sync with the sawtooth troughs in `system.mem.used` (each trough is a collection event) + + + + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Commands must run on the host +Know which process is the runtime (JVM, Python, Node.js)| CPU attribution requires identifying the right PID +GC log file location if the runtime is configured to write one| Provides timestamps and pause durations not visible at OS level + +### Steps + + 1. **Identify the high-CPU process and confirm it is a runtime** + + + + + + ps aux --sort=-%cpu | head -10 + # Look for: java, python, node, ruby, dotnet + # Confirm it is a long-running process (TIME column shows cumulative CPU; ELAPSED shows wall clock age) + + ps -p -o pid,cmd,pcpu,pmem,etime + + 2. **Confirm CPU correlates with memory sawtooth** + + + + + + # Sample CPU and RSS together every 10 seconds + PID= + while true; do + cpu=$(ps -p $PID -o pcpu= | tr -d ' ') + rss=$(awk '/VmRSS/{print $2}' /proc/$PID/status) + echo "$(date +%T) CPU=${cpu}% RSS=${rss} kB" + sleep 10 + done + # GC signature: CPU spikes exactly when RSS drops (collection event) + # Runaway loop signature: CPU is high but RSS is flat + + 3. **Check CPU time breakdown (userspace vs kernel)** + + + + + + # Read utime (field 14) and stime (field 15) from /proc//stat + # Take two samples 10 seconds apart and compare delta + awk '{print "utime=" $14 " stime=" $15}' /proc//stat + sleep 10 + awk '{print "utime=" $14 " stime=" $15}' /proc//stat + # If utime delta >> stime delta: userspace (GC / application), not kernel + # High stime would suggest I/O or syscall pressure instead + + 4. **Check the runtime's startup flags for heap configuration** + + + + + + # JVM: look for -Xmx (max heap), -Xms (initial heap), -Xlog:gc (GC logging) + cat /proc//cmdline | tr '\0' ' ' + + # Node.js: look for --max-old-space-size + cat /proc//cmdline | tr '\0' ' ' | grep -o 'max-old-space[^ ]*' + + # Python: no heap flag; check if process is running with tracemalloc or objgraph + + 5. **Look for GC log files on disk** + + + + + + # JVM writes GC logs to a path configured via -Xlog:gc:file= or -Xloggc= + find /var/log /tmp /opt /home -name "*.log" 2>/dev/null | xargs grep -l "GC\|Heap\|Pause" 2>/dev/null | head -10 + + # Check cmdline for explicit GC log path + cat /proc//cmdline | tr '\0' ' + ' | grep -iE "gc.*log|log.*gc|Xlog" + + 6. **Measure how much of host RAM the process is consuming** + + + + + + cat /proc//status | grep -E 'VmRSS|VmSize|VmSwap' + free -h + # If VmRSS is > 60-70% of total RAM, the runtime is competing with the OS page cache + # and other processes for physical memory + + 7. **Check for OOM kills (GC pressure can precede OOM if heap exhausts before the next collection)** + + + + + + dmesg | grep -i "out of memory\|killed process" + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +Know the current heap / memory configuration of the runtime| Increasing heap without knowing the current setting risks over-allocating +Confirm the host has available RAM before increasing heap| Giving the runtime more heap than the host can provide triggers OOM kills for other processes +Coordinate a service restart with the owning team| Increasing heap or changing GC flags requires a restart to take effect + +### Immediate Relief + +**Restart the process to temporarily relieve GC pressure:** + + + systemctl restart + systemctl status + +Effective if GC pressure is caused by heap fragmentation or accumulated long-lived objects from the current session. If the root cause is heap too small for current traffic, pressure returns after restart within minutes. + +### Increase Available Heap (Runtime-Specific) + +First confirm the host has enough free RAM: + + + free -h + # "available" must exceed the additional heap headroom you plan to allocate + +**JVM — increase max heap:** + + + # Find current setting + cat /proc//cmdline | tr '\0' ' + ' | grep -i xmx + + # Edit the service start script or systemd unit Environment= line, e.g.: + # -Xmx2g → -Xmx4g + # Then restart: + systemctl restart + +**Node.js — increase V8 heap:** + + + # Find current setting + cat /proc//cmdline | tr '\0' ' + ' | grep max-old-space + + # Edit start command to add or increase, e.g.: + # node --max-old-space-size=4096 app.js + +**Python — no heap size flag; GC CPU pressure usually indicates a reference cycle accumulation:** + + + # Python GC collects reference cycles; if CPU is high from Python GC, + # the root cause is likely object retention — treat as a Slow Application Leak + +### Set a Memory Limit to Bound Blast Radius + + + # /etc/systemd/system/.service.d/memory-limit.conf + [Service] + MemoryMax=6G # set above expected heap + runtime overhead, below host RAM + MemorySwapMax=0 + Restart=on-failure + RestartSec=10s + + + systemctl daemon-reload && systemctl restart + +### Fix the Underlying Cause + +OS-level observations guide the correct application-side fix: + +OS observation| Likely cause| Application fix +---|---|--- +RSS plateaus + high CPU (no RSS growth)| Heap too small for current traffic| Increase `-Xmx` or equivalent +RSS rising + high CPU (sawtooth floor climbing)| Memory leak forcing frequent GC| Fix the leak; GC pressure is a symptom +CPU spikes only during traffic surges| Allocation rate exceeds collector throughput under load| Increase heap; review hot allocation paths +CPU high at night / off-peak| Scheduled job or batch triggering bulk allocation| Investigate cron or maintenance job heap usage + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +Polling `/proc`, `ps`, `vmstat`| None| Read-only +`systemctl restart `| **Brief service interruption**| In-flight requests dropped; in-memory state lost +Increasing heap + restart| **Brief service interruption**| Same as restart; new heap size takes effect on next JVM/Node.js start +Setting `MemoryMax` \+ `daemon-reload`| None| Takes effect on next start +Setting `MemoryMax` \+ restart| **Brief service interruption**| Same as restart +**Increasing heap beyond available host RAM**| **Multiple processes OOM-killed**| Allocating more heap than the host has RAM causes cascading OOM kills across the host; always verify `free -h` first + +**Increasing heap beyond available host RAM is the primary risk.** It causes OOM kills across the entire host, not just the target service. + +* * * + +## Verification + + + # Confirm CPU has recovered + ps -p -o pid,pcpu,pmem + # %CPU should be at normal baseline + + # Confirm the sawtooth floor is no longer rising + PID= + while true; do + rss=$(awk '/VmRSS/{print $2}' /proc/$PID/status) + cpu=$(ps -p $PID -o pcpu= | tr -d ' ') + echo "$(date +%T) RSS=${rss} kB CPU=${cpu}%" + sleep 15 + done + + # Confirm no OOM kills + dmesg | grep -i "oom" | tail -5 + +In Datadog, verify: + + * `system.cpu.user` drops to baseline and remains stable + + * `system.mem.used` sawtooth floor is no longer rising (indicates the heap increase or leak fix is working) + + * Application request latency returns to baseline — GC pauses add directly to tail latency, so P99 recovery confirms the fix + + * If heap was increased: `system.mem.used` settles at a new, higher, stable plateau rather than continuing to climb + + + + +* * * + +## Related Scenarios + + * If the sawtooth floor keeps rising even after a heap increase, the root cause is a memory leak, not insufficient heap; treat as a Slow Application Leak. + + * If CPU is high but `system.mem.used` shows no sawtooth (RSS is flat), this is not GC pressure; see the Runaway Process / Infinite Loop scenario. + + * GC pressure that only occurs during traffic surges points to allocation rate exceeding collector throughput under load; consider reducing allocation rate at the application level rather than continually increasing heap. + + diff --git a/scenarios/disk/disk-core-dump-flood-from-crash-loop.md b/scenarios/disk/disk-core-dump-flood-from-crash-loop.md new file mode 100644 index 00000000..9a8ef1c9 --- /dev/null +++ b/scenarios/disk/disk-core-dump-flood-from-crash-loop.md @@ -0,0 +1,303 @@ +# Disk - Core Dump Flood from Crash Loop + +**Signal:** Rapid, continuous growth in `/var/crash` or `/var/lib/systemd/coredump/`; `system.disk.in_use` rising by GBs per minute; service monitor simultaneously in ALERT or NO_DATA +**IssueType:** `disk_usage` +**Device (typical):** Root partition or a dedicated `/var/crash` volume + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`systemctl`| both| `systemctl status ` · `systemctl stop ` · `systemctl daemon-reexec` +`journalctl`| investigation| `journalctl -u -n 50 --no-pager` · `journalctl -u -f` +`coredumpctl`| both| `coredumpctl list` · `coredumpctl clean --disk-free 5G` +`sysctl`| remediation| `sysctl -w kernel.core_pattern=\|/bin/false` +`ls`| investigation| `ls -lhtr /var/crash/ \| tail -20` · `ls -lhtr /var/lib/systemd/coredump/ \| tail -20` +`du`| investigation| `du -sh /var/crash/ /var/lib/systemd/coredump/` +`rm`| remediation| `rm -f /var/crash/*` +`find`| investigation| `find /var/crash /var/lib/systemd/coredump -mmin -10 -ls` +`cat`| investigation| `cat /proc/sys/kernel/core_pattern` + +* * * + +## What Happens + +This scenario combines two simultaneous failures: a service is crash-looping AND the resulting dump files are filling the disk. Neither resolves on its own. + +**The crash loop mechanism** : a service supervisor (systemd `Restart=on-failure`, Kubernetes `restartPolicy: Always`, or a custom watchdog) detects the crash and restarts the service. The service crashes again immediately, often within seconds, and the supervisor restarts it again. This continues indefinitely unless the supervisor hits a restart limit or is manually stopped. + +**The dump flood mechanism** : each crash writes a new core dump. If the process has a 1 GB RSS footprint and crashes every 60 seconds, `/var/crash` fills at roughly 1 GB per minute. With a 50 GB `/var/crash` partition, the disk is full in under an hour. + +**Why it is worse than a simple crash** : once the disk fills, the service cannot be recovered even if the bug is fixed — the patch deployment fails (`no space left on device`), and any process on the host that tries to write (log files, metrics, other services) may also fail. A single crashing service can take down an otherwise healthy host. + +**Common causes** : + + * A bad deploy introduced a crash bug that triggers immediately on startup (the service never reaches a healthy state, so the supervisor keeps restarting it) + + * An OOM condition on every start: the service is allocated insufficient memory and the OOM killer terminates it on every boot + + * A configuration error that causes a panic on init (missing required env var, malformed config file, missing secret mount) + + * An external dependency that the service cannot handle being absent (it crashes rather than degrading gracefully) + + * `Restart=always` with no `StartLimitBurst` or `StartLimitIntervalSec` cap, or a crash that resets the restart counter on each cycle + + + + +* * * + +## Detection + +The platform detects this via `system.disk.in_use` threshold. The distinguishing feature versus a single crash is the **growth rate** : a crash loop produces a staircase pattern on `system.disk.in_use` where each step corresponds to one dump. If the steps are spaced seconds to minutes apart, a loop is active. + +**Look for simultaneous signals** : + + * Service monitor in ALERT or NO_DATA at the same time as the disk alert — confirms the crash is active, not historical + + * `process.run_time.sum` or `process.cpu.normalized.pct` showing a process repeatedly appearing and disappearing + + * High restart count in `kubernetes.containers.restarts` (Kubernetes) or systemd `NRestarts` counter + + + + +**Estimate time-to-full from the crash rate** : + + + # Estimate: crash interval ≈ time between most-recent dump files + ls -lhtr /var/lib/systemd/coredump/ | tail -5 + # If 5 dumps appeared in the last 2 minutes: crash rate ≈ 30s/crash + # If process RSS is ~2 GB: disk fills at ~4 GB/min + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| All commands run locally +Root or `admin` group access| Dump directories require elevated access +Know the service name| Needed to find the crashing process and stop the supervisor + +### Steps + + 1. **Confirm a loop is active (not historical)** + + + + + + # Check if new dumps are still appearing + watch -n5 'ls -lhtr /var/lib/systemd/coredump/ | tail -5' + # If the listing changes every few seconds, the loop is live + + # Or count dumps created in the last 10 minutes + find /var/crash /var/lib/systemd/coredump -mmin -10 -ls | wc -l + + 2. **Identify the crashing service** + + + + + + # systemd-coredump: shows crash history with timestamp, PID, executable + coredumpctl list | tail -20 + + # Check which service has been restarting + systemctl --state=failed + journalctl -xe --since "10 minutes ago" | grep -i "start request\|failed\|crash\|signal" + + 3. **Check crash rate and restart count** + + + + + + systemctl status + # Look for: "Main PID", "Active: activating (start) ...", "NRestarts: 47" + # A high NRestarts or rapidly changing status confirms a loop + + journalctl -u -n 30 --no-pager + # Look for alternating "Started" / "Failed" / "Scheduled restart" lines + + 4. **Check current disk headroom** + + + + + + du -sh /var/crash/ /var/lib/systemd/coredump/ + df -h /var/crash # or df -h / if crash dir is on root + # Estimate minutes until full: remaining_space / (dump_size * crash_rate) + + 5. **Verify where dumps are being written** + + + + + + cat /proc/sys/kernel/core_pattern + # Common values: + # |/usr/lib/systemd/systemd-coredump ... → goes to systemd-coredump + # /var/crash/core.%e.%p.%t → goes to /var/crash + # /dev/null → dumps disabled (nothing to clean) + +* * * + +## Remediation + +**Order matters** : stop the loop before cleaning. If you delete dumps while the service is still crash-looping, new dumps refill the space within minutes. + +### Preconditions + +Precondition| Rationale +---|--- +Notify the owning team before stopping the service| The service is already down, but stopping the supervisor ends any in-progress recovery attempts and may affect SLA calculations or on-call escalation +Preserve at least one dump for analysis| One dump is sufficient to diagnose the crash; keep the most recent before deleting the rest +Coordinate if multiple services share the partition| Disk pressure may be affecting other services; communicate before taking actions that change host-level config + +### Step 1: Stop the crash loop + + + # Stop the service and prevent automatic restart + systemctl stop + + # If systemd keeps restarting it despite stop, mask it temporarily + systemctl mask + # Undo later with: systemctl unmask + +For Kubernetes: scale the deployment to 0 replicas, or cordon the node and drain to stop further scheduling there. + +### Step 2: Disable dump generation (prevents new dumps if the loop resumes) + +**Immediate, non-persistent (survives until reboot only):** + + + # Redirect all core dumps to /dev/null + sysctl -w kernel.core_pattern='|/bin/false' + # Verify: + cat /proc/sys/kernel/core_pattern + +**Persistent, per-service via systemd:** + + + # Override file: /etc/systemd/system/.d/no-coredump.conf + [Service] + LimitCORE=0 + +Apply with `systemctl daemon-reload && systemctl restart ` (after the loop is fixed). + +**Persistent, system-wide via systemd:** + + + # /etc/systemd/system.conf + DefaultLimitCORE=0 + +Apply with `systemctl daemon-reexec` (does not restart running services). + +### Step 3: Recover disk space + + + # Keep the most recent dump for analysis; delete the rest + ls -t /var/lib/systemd/coredump/ | tail -n +2 | xargs -I{} rm /var/lib/systemd/coredump/{} + + # Or use coredumpctl to leave a target free amount + coredumpctl clean --disk-free 5G + + # Or delete everything if analysis is waived / already done + rm -f /var/crash/* + rm -f /var/lib/systemd/coredump/* + +### Step 4: Fix the underlying crash + +This is the only permanent resolution. One dump is enough to identify the crash type: + + + # Get the backtrace from the most recent dump + coredumpctl info # most recent entry + # Look for: signal received, executable, backtrace lines + +Crash type| Next step +---|--- +SIGSEGV / SIGABRT with backtrace| Identify the faulting frame; correlate with recent deploy; roll back or hotfix +OOM kill (signal 9 from kernel)| The process is being killed by the OOM killer on every start — raise its memory limit or reduce footprint before restarting +SIGTERM / clean exit + supervisor restart| The service is exiting cleanly but the supervisor treats any non-zero exit as a crash — check exit codes and supervisor restart policy +Panic on init (config / dependency)| Service cannot start due to a missing config value or unreachable dependency — fix config, restore the dependency, or add a startup health check + +### Step 5: Re-enable dumps and restart + +Once the fix is deployed: + + + # Re-enable dump generation (revert sysctl) + sysctl -w kernel.core_pattern="|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" + # or whatever the original pattern was (check /etc/sysctl.d/ for the configured value) + + # Unmask and start the service + systemctl unmask + systemctl start + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +Inspecting dumps, `coredumpctl list`| None| Read-only +`systemctl stop `| **Service remains down** (it was already down)| Ends the crash loop; no additional disruption beyond what the crash already caused +`systemctl mask `| **Prevents recovery until unmasked**| Use only if `stop` is insufficient to halt the loop; remember to unmask before deploying the fix +`sysctl -w kernel.core_pattern=...`| None| Host-wide but affects only future crashes; no impact on running processes +`rm -f /var/crash/*`| None| Post-mortem artifacts only; deleting dumps has no runtime effect +`coredumpctl clean`| None| Same as above +`systemctl daemon-reexec`| None for running services| Re-executes the systemd manager binary; does not restart managed services +**Fixing the crash and restarting the service**| Service comes back up — brief restart gap| This is the desired outcome + +**The primary risk is unmasking or restarting the service before the fix is deployed.** If the service is restarted while still crash-prone, the loop resumes and the disk refills. Confirm the fix is in place before step 5. + +* * * + +## Verification + + + # Confirm no new dumps are appearing + watch -n10 'ls -lhtr /var/lib/systemd/coredump/ | tail -5' + # Should be static + + # Confirm disk recovered + df -h /var/crash # or df -h / + + # Confirm service is stable + systemctl status + journalctl -u -n 20 --no-pager + # Should show "Active: active (running)" with no recent restart events + + # Confirm NRestarts is no longer climbing + systemctl show --property=NRestarts + +In Datadog, verify: + + * `system.disk.in_use` levels off and then drops as the dump directory is cleaned + + * Service monitor returns to OK state and stays there (no flapping) + + * No new `no space left on device` errors in application or system logs + + + + +* * * + +## Related Scenarios + + * If the crash was triggered by OOM, the root cause is memory pressure — see Memory: Slow Application Leak or Memory: Unbounded Cache / Session Growth for investigation of the memory growth that preceded the kill. + + * If other services on the same host began failing at the same time (writes rejected, log files not rotating), the disk filled enough to affect them — treat as a host-level incident, not just a single-service issue. + + * For a single crash (no loop), see [Disk Space Issues Due to Core Dumps and Crash Management]() for detailed forensic analysis steps and dump retention configuration. + + diff --git a/scenarios/disk/disk-docker-image-and-build-cache-bloat.md b/scenarios/disk/disk-docker-image-and-build-cache-bloat.md new file mode 100644 index 00000000..1551be35 --- /dev/null +++ b/scenarios/disk/disk-docker-image-and-build-cache-bloat.md @@ -0,0 +1,263 @@ +# Disk - Docker Image and Build Cache Bloat + +**Signal:** `system.disk.in_use` rising steadily on the partition hosting `/var/lib/docker`; `docker system df` shows large RECLAIMABLE on Images or Build Cache +**IssueType:** `disk_usage` +**Device (typical):** Root partition or a dedicated Docker data volume (`/dev/sda1`, `/dev/nvme0n1p1`) + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`docker system`| both| `docker system df` · `docker system df -v` · `docker system prune -f` · `docker system prune -a -f` +`docker images`| investigation| `docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}\t{{.CreatedSince}}"` · `docker images -f dangling=true` +`docker buildx`| both| `docker buildx du` · `docker buildx prune -f` +`docker ps`| investigation| `docker ps -a --filter status=exited` +`du`| investigation| `du -sh /var/lib/docker/` · `du -sh /var/lib/docker/buildkit/` +`df`| investigation| `df -h /var/lib/docker` + +* * * + +## What Happens + +On any host that builds or pulls Docker images (CI runners, build agents, deploy nodes), Docker's storage directory grows continuously unless cleanup runs regularly: + + * **Image accumulation** : every service deploy pushes a new image tag; the old tag is untagged (becoming "dangling") but the layers remain on disk. A host that deploys a service multiple times per day accumulates dozens of image versions within weeks. + + * **Build cache growth** : BuildKit retains every intermediate layer from every `docker build` invocation. This cache is invisible to `docker images` — it only appears in `docker system df` and `docker buildx du`. On active build hosts it routinely reaches 10-30 GB with no running containers involved. + + * **Stopped container filesystems** : containers that exited (from CI jobs, one-off tasks, or crashes) retain their writable layer on disk until explicitly removed. + + + + +The failure mode is distinctive: disk fills enough to block new image pulls or builds, but **running containers continue operating normally** until they need to write to disk themselves. This means the first observable symptom is deployment or CI pipeline failures, not a service crash. + +Common host types affected: + + * CI runners that build Docker images + + * Kubernetes nodes with aggressive pull-always policies across many services + + * Hosts where services are deployed by updating a container (pull new image, stop old container, start new) + + * Developer workstations used for image building over months + + + + +* * * + +## Detection + +Detected via `system.disk.in_use` monitor. The signal shape is a **slow, steady rise over days or weeks** that accelerates on hosts with frequent builds or deploys. Unlike core dumps (sudden spike) or log rotation failures (traffic-correlated rise), this grows at roughly a constant rate tied to deployment frequency. + +**First failure mode before disk full:** + +The partition hosting `/var/lib/docker` typically hits trouble around 85-90% — `docker pull` and `docker build` begin failing with `no space left on device` while the rest of the host (running containers, application logs) is still healthy. Alert on 80% to leave room for the cleanup operation itself. + +**Correlated signals:** + + * CI pipeline failures or deploy jobs failing with `no space left on device` + + * `docker pull` errors in service logs around the time of a deployment + + * `system.disk.in_use` rising on the Docker volume without a corresponding rise on other volumes + + + + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Commands must run locally +Member of `docker` group or root| All `docker` commands require daemon access +Docker daemon is running| `docker system df` and `docker buildx du` require the daemon + +### Steps + + 1. **Confirm Docker is the consumer** + + + + + + df -h /var/lib/docker # or: df -h / if Docker is on root + du -sh /var/lib/docker/ + + 2. **Get the full breakdown** + + + + + + docker system df + # Output shows RECLAIMABLE per category: + # TYPE TOTAL ACTIVE SIZE RECLAIMABLE + # Images 47 12 18.4GB 14.2GB (77%) + # Containers 23 3 1.1GB 1.0GB (94%) + # Local Volumes 11 4 3.2GB 1.8GB (56%) + # Build Cache - - 4.7GB 4.7GB + +Focus on Images and Build Cache — these are the two categories most likely to be large on a host that builds or deploys containers. If Build Cache alone is many GB, this is a build-agent host with no cache eviction configured. + + 3. **Identify which images are accumulating** + + + + + + # All images sorted by size (largest first) + docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}\t{{.CreatedSince}}" \ + | sort -k3 -h -r | head -20 + + # Dangling images only (untagged; safe to remove) + docker images -f dangling=true + + # Count of images per repository to spot accumulation + docker images --format "{{.Repository}}" | sort | uniq -c | sort -rn | head -10 + + 4. **Check build cache breakdown** + + + + + + docker buildx du 2>/dev/null || docker builder du + # Shows cache entries with size and last-used time + + 5. **Check stopped containers** + + + + + + docker ps -a --filter status=exited --format "table {{.Names}}\t{{.Status}}\t{{.Size}}\t{{.Image}}" + # Large size values here mean the container wrote significant data before exiting + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +No deployment in-flight| `docker system prune -a` during an active image pull can remove intermediate layers mid-transfer, causing the pull to fail and the deployment to retry. Check with the deploy system before running +Confirm named volumes are not needed| `docker system prune` without `--volumes` is safe; only add `--volumes` after confirming orphaned volumes hold no state you need + +### Immediate Space Recovery + +**Step 1 — safe prune (stopped containers, dangling images, unused networks, build cache; does NOT remove tagged images in use by any container):** + + + docker system prune -f + +**Step 2 — if more space is needed: remove tagged images not referenced by any running or stopped container:** + + + docker system prune -a -f + +This is safe if the host can re-pull images on next deploy. Avoid during an active rolling deployment. + +**Build cache only (zero runtime risk):** + + + docker buildx prune -f + # or + docker builder prune -f + +**Verbose output to see what was actually freed:** + + + docker system prune -a -f 2>&1 | tail -5 + # "Total reclaimed space: 14.3GB" + +### Prevent Recurrence + +**Option 1 — scheduled prune cron job (recommended for CI/build hosts):** + + + # /etc/cron.d/docker-cleanup + # Prune nightly; keep images used in the last 48 hours + 0 3 * * * root docker system prune -a -f --filter "until=48h" >> /var/log/docker-cleanup.log 2>&1 + +**Option 2 — limit BuildKit cache size in the daemon (Docker 23+):** + + + // /etc/docker/daemon.json — add or merge: + { + "builder": { + "gc": { + "enabled": true, + "defaultKeepStorage": "20GB" + } + } + } + +Then reload: `systemctl restart docker` (note: this restarts all containers; see Service Impact). + +**Option 3 — add a cleanup step to every CI pipeline:** + + + # GitLab CI example + cleanup: + stage: .post + script: + - docker system prune -f + when: always + +For comprehensive guidance on daemon-level configuration and edge cases (overlay2 storage driver, devicemapper, Docker-in-Kubernetes), see [Disk Space Management for Docker: Investigating and Remediating Container Layer Usage](). + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +`docker system df`| None| Read-only query to the daemon +`docker system prune -f`| None| Only removes stopped containers and dangling images; running containers and their images are untouched +`docker system prune -a -f`| None for running containers| Removes unused tagged images too; safe unless a deployment is actively pulling an image mid-transfer +`docker buildx prune -f`| None| Build cache only; zero runtime impact +`systemctl restart docker`| **Full host container outage**| Stops ALL running containers; required only if changing `daemon.json`. Plan a maintenance window and coordinate with container owners. + +**The deployment-failure-before-crash pattern is the key distinguishing feature of this scenario.** When `/var/lib/docker` fills up, `docker pull` and `docker build` fail immediately. Running containers continue until they need to write to disk (logs, tmp files). The host is not "down" but all new deployments and CI jobs to it will fail. + +* * * + +## Verification + + + # Confirm space reclaimed + docker system df # RECLAIMABLE should be near 0 on Images and Build Cache + df -h /var/lib/docker # usage should be below threshold + + # Confirm next deploy can pull + docker pull /: # should succeed without error + +In Datadog, verify: + + * `system.disk.in_use` on the Docker volume drops below the alert threshold within a few minutes of the prune completing + + * Subsequent CI pipeline runs or deploy jobs succeed without `no space left on device` errors + + + + +* * * + +## Related Scenarios + + * If stopped containers are large contributors (large size in `docker ps -a`), investigate whether services are writing to container-local paths instead of mounted volumes — data written inside a container accumulates in its writable layer; see the Temp Files & Build Artifacts scenario for general writable-layer cleanup patterns. + + * If the disk is already full and `docker system prune` itself fails with `no space left on device`, you need to free space outside of Docker first (delete log files, clear `/tmp`) before the Docker daemon can complete its cleanup. + + * For daemon-level configuration, overlay2 edge cases, and devicemapper storage driver details, see [Disk Space Management for Docker: Investigating and Remediating Container Layer Usage](). + + diff --git a/scenarios/disk/disk-inode-exhaustion-from-small-file-accumulation.md b/scenarios/disk/disk-inode-exhaustion-from-small-file-accumulation.md new file mode 100644 index 00000000..3ab9dc6c --- /dev/null +++ b/scenarios/disk/disk-inode-exhaustion-from-small-file-accumulation.md @@ -0,0 +1,313 @@ +# Disk - Inode Exhaustion from Small File Accumulation + +**Signal:** `system.fs.inodes.in_use` at or near 1.0 per device (Datadog Agent); `system.filesystem.inodes.usage{state=used}` approaching max (OpenTelemetry host metrics); confirmed on the host with `df -i` showing IUse% at 100% while `df -h` shows significant free space; processes failing with `No space left on device` or `ENOSPC` despite `system.disk.in_use` appearing normal +**IssueType:** `disk_usage` +**Device (typical):** Root partition, `/tmp`, `/var`, or any partition hosting a workload that creates many small files + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`df`| investigation| `df -i` · `df -ih` +`find`| both| `find /tmp -maxdepth 3 -type f \| wc -l` · `find / -xdev -maxdepth 6 -printf '%h +| sort | uniq -c | sort -rn | head -20` || | +`ls`| investigation| `ls \| wc -l` · `ls -la /tmp \| wc -l` +`du`| investigation| `du --inodes -d 3 /var \| sort -rn \| head -20` +`stat`| investigation| `stat -f /var/spool/` +`rm`| remediation| `find /tmp -maxdepth 1 -type f -mtime +1 -delete` +`tune2fs`| investigation| `tune2fs -l /dev/sda1 \| grep -i inode` + +* * * + +## What Happens + +Every file on a Unix filesystem, regardless of its size, occupies exactly one **inode** — a metadata entry that records the file's owner, permissions, timestamps, and pointer to its data blocks. The number of inodes on a filesystem is fixed at creation time (ext4 allocates roughly one inode per 16 KB of storage by default). Once all inodes are consumed, no new files can be created, even if gigabytes of data blocks are still free. + +This is the core confusion: the error is identical to a full disk (`No space left on device` / `ENOSPC`), but `system.disk.in_use` and byte-level dashboards look completely normal. + +**Why it is hard to notice in advance** : `system.disk.in_use` tracks bytes, not inodes. `system.fs.inodes.in_use` (Datadog) and `system.filesystem.inodes.usage` (OTel) are the correct metrics, but they are not included in most default disk monitors. Inode exhaustion can accumulate invisibly for weeks — the inode counter climbs toward 1.0 while byte usage stays flat — and only becomes visible the moment the last inode is consumed. + +**Common workloads that exhaust inodes** : + +Workload| Mechanism +---|--- +PHP session files| Each HTTP session creates one file in `/var/lib/php/sessions/`; sessions expire slowly; a busy site accumulates millions +Mail queue| Each queued message is a file; a stuck queue or spam run fills inodes quickly +Build artifacts| `node_modules/` trees, Java `.class` files, Python `__pycache__`/`.pyc`, Gradle caches — each package or class is a separate file +Container overlay layers| Each container layer creates many small metadata files under `/var/lib/docker/overlay2/` +Log rotation fragments| Aggressive rotation with short `rotate` counts splits one log stream into thousands of small files +Metrics/cache files| Prometheus TSDB, StatsD, or similar tools writing one file per metric series +`/tmp` temp files from crashed processes| Processes that crash after creating temp files but before deleting them leave orphaned files indefinitely + +* * * + +## Detection + +The platform may not alert on this scenario unless inode monitoring is explicitly configured. If you receive an alert, it likely came from one of: + + * A monitor on `system.fs.inodes.in_use` (Datadog) or `system.filesystem.inodes.usage` (OTel) — if configured + + * Application error log monitors catching repeated `ENOSPC` or `Too many open files` errors + + * An on-call alert from a failed deployment or failed write operation + + + + +**First indication is usually an application error, not a disk alert.** Common symptoms: + + * Web servers returning 500 errors; application logs showing `failed to create temp file`, `cannot create socket`, or `ENOSPC` + + * `cron` jobs silently failing (cron cannot write its lock file) + + * Package managers (`apt`, `yum`) failing to install or update packages + + * New files cannot be created but existing files can still be read and written to + + * `df -h` looks normal; engineer spends time investigating "disk full" on what appears to be a healthy host + + + + +**Confirming inode exhaustion** : + + + df -i + # Filesystem Inodes IUsed IFree IUse% Mounted on + # /dev/sda1 3932160 3932160 0 100% / + # /dev/sdb1 1048576 800000 248576 76% /data + # + # IUse% at 100% on / confirms the issue + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| All commands run locally +Root or sudo access| Some directories require elevated access to read file counts +Time: finding the culprit directory can take minutes| `find` across large filesystems is slow; use `-xdev` to stay on one partition and `-maxdepth` to limit scope; start from likely suspects (`/tmp`, `/var`, application data dirs) before doing a full scan + +### Steps + + 1. **Confirm inode exhaustion and identify the affected partition** + + + + + + df -i + # IUse% at or near 100% on one or more partitions + + df -ih # same output with human-readable inode counts + + 2. **Find the directory with the most files — fast approach (start with suspects)** + + + + + + # Check common culprits first — each takes seconds + ls /tmp | wc -l + ls /var/spool/mail/ | wc -l + ls /var/lib/php/sessions/ 2>/dev/null | wc -l + ls /var/spool/postfix/deferred/ 2>/dev/null | wc -l + find /var/tmp -maxdepth 2 -type f | wc -l + + 3. **Inode usage by directory (du --inodes, fast and safe)** + + + + + + # Shows which directories consume the most inodes; -d 3 limits depth + du --inodes -d 3 /var 2>/dev/null | sort -rn | head -20 + du --inodes -d 3 /tmp 2>/dev/null | sort -rn | head -20 + + 4. **Full filesystem scan (slower; use when suspects above yield nothing)** + + + + + + # Print the parent directory of every file, count, sort — stays on one filesystem with -xdev + find / -xdev -maxdepth 8 -printf '%h + ' 2>/dev/null \ + | sort | uniq -c | sort -rn | head -20 + # Output: count directory — the top line is the culprit + + 5. **Inspect the culprit directory** + + + + + + # Once you know the directory (e.g. /var/lib/php/sessions): + ls /var/lib/php/sessions | wc -l # total file count + ls -lt /var/lib/php/sessions | head # most recently modified + ls -lut /var/lib/php/sessions | tail # oldest (least recently accessed) + + # Check the age distribution + find /var/lib/php/sessions -mtime +1 | wc -l # older than 1 day + find /var/lib/php/sessions -mtime +7 | wc -l # older than 7 days + + 6. **Check filesystem inode limits** + + + + + + # How many inodes does the filesystem have total? + tune2fs -l /dev/sda1 | grep -i inode + # Inode count: 3932160 + # Free inodes: 0 + + stat -f /var/spool/ + # Inodes: 3932160 Free Inodes: 0 + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +Identify the file type before deleting| Session files, mail queue files, and cache files may contain active user data or undelivered messages; confirm with the owning team before bulk deletion +Do not delete files that are open by running processes| Files with active file descriptors should be truncated (not deleted) if the process must keep running; use `lsof` to check before bulk `rm` +Test deletion on a small batch first| On a directory with millions of files, `rm -rf *` can itself fail (argument list too long); use `find ... -delete` instead + +### Immediate Relief + +**Session files (PHP, Ruby, etc.) — safe to delete files older than the session TTL:** + + + # Delete PHP session files not accessed in more than 24 hours + find /var/lib/php/sessions -maxdepth 1 -type f -atime +1 -delete + + # If even find is slow (millions of files), use xargs in batches: + find /var/lib/php/sessions -maxdepth 1 -type f -atime +1 \ + | xargs -P4 -n1000 rm -f + +**Temp files — files older than 1 day in**`/tmp` are generally safe: + + + find /tmp -maxdepth 2 -type f -mtime +1 -delete + find /var/tmp -maxdepth 2 -type f -mtime +1 -delete + +**Build artifacts (node_modules, .pyc, .class) — check with the owning team:** + + + # Example: delete Python bytecode cache + find /app -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null + + # Example: delete node_modules from old build directories + find /builds -maxdepth 3 -name 'node_modules' -type d -mtime +7 -exec rm -rf {} + + +**Mail queue (postfix) — only after confirming messages are safe to drop:** + + + # List queue depth + postqueue -p | tail -1 + + # Flush or delete deferred messages (coordinate with the mail team) + postsuper -d ALL deferred + +### Prevent Recurrence + +**Monitor inodes explicitly** — add a Datadog monitor on `system.fs.inodes.in_use`: + + + # Metric: system.fs.inodes.in_use + # Alert threshold: > 0.80 (80%) + # Group by: host, device + # This gives early warning before the 1.0 hard wall is hit + +For OTel pipelines: alert on `system.filesystem.inodes.usage{state=used}` as a fraction of `system.filesystem.inodes.usage` total. + +**Configure application session / temp file TTLs:** + + + # PHP: set session.gc_maxlifetime and session.gc_probability in php.ini + # Example: expire sessions after 1440 seconds (24 minutes), run GC on 1% of requests + session.gc_maxlifetime = 1440 + session.gc_probability = 1 + session.gc_divisor = 100 + +**Use**`systemd-tmpfiles` for automatic cleanup of temp directories: + + + # /etc/tmpfiles.d/app-sessions.conf + # Delete files in /var/lib/php/sessions older than 1 day + D /var/lib/php/sessions 0700 www-data www-data 1d + +**Long-term: reformat the partition with more inodes** (requires downtime): + + + # Only feasible if the filesystem can be unmounted and reformatted + # Create ext4 with one inode per 4 KB instead of the default 16 KB: + mkfs.ext4 -i 4096 /dev/sdb1 + # Warning: this reduces maximum file storage capacity; only appropriate + # for partitions dedicated to small-file workloads + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +`df -i`, `find ... \| wc -l`, `du --inodes`| None| Read-only investigation +Deleting old session files| **Active sessions are terminated**| Users with sessions in the deleted files are logged out; acceptable if files are older than the TTL, but confirm the TTL before deleting +Deleting temp files| Minimal| Only affects processes that opened the file and expected it to persist — most temp files are safe; use `lsof` on suspect files if unsure +Deleting build artifacts| None for running services| Only affects future builds; running services do not read `node_modules` or `.class` files at runtime after startup +`postsuper -d ALL deferred`| **Undelivered mail is lost**| Irreversible; only do this with explicit authorization from the mail system owner +Reformatting the partition| **Full outage for the service using that partition**| Requires unmounting; schedule a maintenance window + +**The primary operational risk is deleting active session files.** If session files are younger than the application's session TTL, logged-in users will be unexpectedly logged out. When in doubt, delete only files older than 2x the TTL, not all files. + +* * * + +## Verification + + + # Confirm inodes are freed + df -i + # IUse% on the affected partition should be below 100% + + # Confirm file creation now works + touch /tmp/inode-test-$$ && rm /tmp/inode-test-$$ + # Should complete without error + + # Confirm the triggering application has recovered + # (restart if it cached the ENOSPC error) + systemctl restart # if needed + +In Datadog, verify: + + * `system.fs.inodes.in_use` on the affected device drops below 0.80 + + * Application error rate returns to baseline (no more `ENOSPC` errors in logs) + + * If a monitor on `system.fs.inodes.in_use` does not yet exist, create one now at 80% — this scenario gives no warning from `system.disk.in_use` alone + + + + +* * * + +## Related Scenarios + + * If the inode-consuming workload is session files from a PHP or web application, the session accumulation rate is traffic-driven — a sudden traffic spike can also trigger this; monitor session TTL and GC settings proactively. + + * If the culprit is build artifacts under a build directory, co-locate with the Temp Files & Build Artifacts scenario for a broader cleanup of build-related disk consumers. + + * If `/var/lib/docker/overlay2/` is consuming inodes, it is the same Docker storage that causes byte-level disk bloat — refer to Disk: Docker Image and Build Cache Bloat for remediation steps. + + * After resolving the immediate crisis, add `system.fs.inodes.in_use` monitoring. This scenario is Critical precisely because it is invisible until the last inode is consumed. + + diff --git a/scenarios/disk/disk-orphaned-postgresql-replication-slot.md b/scenarios/disk/disk-orphaned-postgresql-replication-slot.md new file mode 100644 index 00000000..3d5eb666 --- /dev/null +++ b/scenarios/disk/disk-orphaned-postgresql-replication-slot.md @@ -0,0 +1,279 @@ +# Disk - Orphaned PostgreSQL Replication Slot + +**Signal:** `system.disk.in_use` rising continuously on the PostgreSQL data volume; `pg_replication_slots` shows one or more slots with `active = false`; WAL directory growing without bound +**IssueType:** `disk_usage` +**Device (typical):** Dedicated PostgreSQL data volume or root partition hosting `$PGDATA` + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`psql`| both| `SELECT ... FROM pg_replication_slots` · `SELECT pg_drop_replication_slot('name')` · `SELECT ... FROM pg_stat_replication` +`du`| investigation| `du -sh $PGDATA/pg_wal/` +`ls`| investigation| `ls $PGDATA/pg_wal/ \| wc -l` +`df`| investigation| `df -h $PGDATA` + +* * * + +## What Happens + +A PostgreSQL **replication slot** is a guarantee: the primary will retain all WAL segments produced since the slot's `restart_lsn` until the slot's consumer confirms it has processed them. Slots are used by physical streaming replicas, logical replication subscribers, and CDC tools (Debezium, pglogical, etc.). + +When the consumer of a slot goes away without calling `pg_drop_replication_slot()` — a replica decommissioned without cleanup, a CDC pipeline that failed and was abandoned, a logical subscriber that was deleted at the application layer but not in the database — the slot remains. It is now **orphaned** : `active = false`, no connection associated, and `restart_lsn` frozen at the point when the consumer last confirmed. + +PostgreSQL keeps every WAL segment since that frozen `restart_lsn` on disk, indefinitely. Unlike a long-running transaction (which will eventually commit or roll back), an orphaned slot has no natural expiry. On a busy primary, this can mean gigabytes of WAL accumulate per hour with no ceiling. The primary will eventually halt all writes when the data volume fills. + +**Why it is hard to catch early** : the `system.disk.in_use` rise looks identical to normal database growth. The only distinguishing signal is correlating WAL directory size with `pg_replication_slots` — which requires database-level visibility that standard host monitoring does not provide by default. + +**Common causes** : + + * A replica host was decommissioned (terminated, rebuilt) without running `pg_drop_replication_slot()` on the primary first + + * A Debezium, pglogical, or other CDC connector was undeployed or failed permanently; the database slot was never cleaned up + + * A logical replication subscriber table was dropped but the subscription was not, leaving the publisher-side slot inactive + + * A standby was promoted and the old primary's slot for it was never removed + + * A developer created a slot for testing (`pg_create_logical_replication_slot`) and forgot to drop it + + + + +* * * + +## Detection + +Detected via `system.disk.in_use` on the PostgreSQL data volume. The signal shape is a **slow, steady rise** — indistinguishable from normal database growth until correlated with WAL directory size and slot state. + +**Correlated signals that narrow it to a slot problem** : + + * `du -sh $PGDATA/pg_wal/` is large and growing while database table sizes (`pg_database_size()`) are stable + + * `pg_replication_slots` has rows with `active = false` + + * `postgresql.replication.delay` shows zero or no connected replicas while a slot still exists + + + + +**Estimate time-to-disk-full from WAL growth rate** : + + + # Run twice, 60 seconds apart; compute delta + du -sh $PGDATA/pg_wal/ + # If WAL grows ~500 MB/min on a write-heavy primary: hours, not days, until full + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Required to check WAL directory size +`psql` access with at least `pg_monitor` role| Sufficient to query `pg_replication_slots` and `pg_stat_replication`; `pg_drop_replication_slot` requires superuser +Know the value of `$PGDATA`| Needed to find the WAL directory; retrieve with `SHOW data_directory;` if unknown + +### Steps + + 1. **Check WAL directory size and headroom** + + + + + + sudo -u postgres psql -c "SHOW data_directory;" + + du -sh $PGDATA/pg_wal/ + ls $PGDATA/pg_wal/ | wc -l # each segment is typically 16 MB + df -h $PGDATA + + 2. **List all replication slots and their WAL retention** + + + + + + SELECT slot_name, + slot_type, + active, + active_pid, + restart_lsn, + confirmed_flush_lsn, + pg_size_pretty( + pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) + ) AS wal_retained + FROM pg_replication_slots + ORDER BY active, restart_lsn; + +Focus on rows where `active = false`. The `wal_retained` column shows exactly how much WAL is being held on disk for each inactive slot. A value of many GB from an inactive slot is the smoking gun. + + 3. **Confirm no active replica is using the slot** + + + + + + -- Active streaming connections (connected replicas/subscribers) + SELECT application_name, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, + sync_state + FROM pg_stat_replication; + + -- Cross-reference: if a slot appears in pg_replication_slots with active=false + -- and does NOT appear in pg_stat_replication, it has no active consumer + + 4. **Check when the slot last advanced (proxy: WAL lag age)** + + + + + + -- How far behind is the slot's restart_lsn from the current WAL position? + -- This tells you approximately how long the slot has been idle + SELECT slot_name, + active, + pg_size_pretty( + pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) + ) AS wal_lag, + pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS wal_lag_bytes + FROM pg_replication_slots + WHERE NOT active + ORDER BY wal_lag_bytes DESC; + + 5. **Check for**`max_slot_wal_keep_size` setting (PostgreSQL 13+) + + + + + + SHOW max_slot_wal_keep_size; + -- '-1' means unlimited — no cap on WAL retained per slot + -- A positive value (e.g. '10GB') means PostgreSQL will invalidate the slot + -- rather than retain more WAL than the limit, protecting disk at the cost + -- of forcing the slot's consumer to rebuild from scratch + +* * * + +## Remediation + +**Before dropping a slot: confirm it is truly orphaned.** Dropping a slot for a replica that is temporarily disconnected (network partition, restart) causes that replica to fall irrecoverably behind — it will need a full base backup to resync. The cost of a false positive is high. + +### Preconditions + +Precondition| Rationale +---|--- +Confirm with the DBA or infra team that the slot's consumer no longer exists| The slot name often identifies the consumer (`debezium`, `replica_us_east`, etc.); verify the named system is decommissioned, not just temporarily offline +Superuser or `pg_drop_replication_slot` privilege| Required to drop a slot; `pg_monitor` alone is not sufficient +Notify the owning team before dropping| If the slot was for a replica, dropping it makes that replica's WAL stream permanently invalid; the replica owner must know to rebuild + +### Drop the orphaned slot + + + -- Verify one more time before dropping + SELECT slot_name, active, wal_retained + FROM ( + SELECT slot_name, active, + pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_retained + FROM pg_replication_slots + ) s + WHERE slot_name = ''; + + -- Drop (irreversible) + SELECT pg_drop_replication_slot(''); + + -- Confirm it is gone + SELECT slot_name FROM pg_replication_slots; + +### WAL reclaim after drop + +WAL recycling is automatic once the slot is dropped and the next checkpoint completes. The WAL directory will shrink over the next few minutes as PostgreSQL recycles segments it no longer needs to retain. + + + # Poll to confirm WAL is shrinking + while true; do + echo "$(date +%T) WAL=$(du -sh $PGDATA/pg_wal/ | cut -f1)" + sleep 30 + done + +### Prevent recurrence + +**Set**`max_slot_wal_keep_size` (PostgreSQL 13+) — caps WAL retained per slot; the slot is invalidated rather than filling the disk: + + + -- Apply globally via postgresql.conf + ALTER SYSTEM SET max_slot_wal_keep_size = '10GB'; + SELECT pg_reload_conf(); + + -- Verify + SHOW max_slot_wal_keep_size; + +When a slot is invalidated by this limit, PostgreSQL sets `pg_replication_slots.conflicting = true`; the consumer must perform a full resync. This is a loud failure (the consumer breaks) rather than a silent disk-fill. Prefer loud failures. + +**Add a slot monitoring query as a Datadog custom check or alerting rule** : + + + -- Alert if any inactive slot is retaining more than 5 GB of WAL + SELECT slot_name, + pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_retained + FROM pg_replication_slots + WHERE NOT active + AND pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) > 5 * 1024^3; + +**Operational hygiene** : add slot cleanup to any replica decommission checklist. The slot must be dropped on the primary before the replica host is terminated. + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +Querying `pg_replication_slots`, `pg_stat_replication`| None| Read-only +`pg_drop_replication_slot()`| None for the primary| The primary continues running normally; WAL recycling resumes automatically +`pg_drop_replication_slot()`| **Replica/consumer is permanently invalidated**| Any replica or subscriber that was using this slot — even if temporarily offline — can no longer catch up; it must be rebuilt from a base backup. This is irreversible. +`ALTER SYSTEM SET max_slot_wal_keep_size + pg_reload_conf()`| None| Applies immediately to slot WAL retention limits; no restart required + +`pg_drop_replication_slot` is the single high-risk action in this runbook. The disk impact is zero-risk (WAL is post-consumer data); the replica impact is severe and irreversible if the consumer is not actually orphaned. Always confirm consumer state before dropping. + +* * * + +## Verification + + + -- Confirm the slot is gone + SELECT slot_name FROM pg_replication_slots; + -- Should not include the dropped slot + + -- Confirm WAL is shrinking + -- du -sh $PGDATA/pg_wal/ (run twice 60 s apart) + + -- Confirm no remaining inactive slots with large WAL retention + SELECT slot_name, active, + pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_retained + FROM pg_replication_slots + WHERE NOT active; + -- Should return 0 rows (or rows with negligible retained WAL) + +In Datadog, verify: + + * `system.disk.in_use` on the PostgreSQL data volume levels off and begins to drop as WAL is recycled + + * No new ALERT from the disk monitor within 30 minutes + + + + +* * * + +## Related Scenarios + + * If WAL growth was accompanied by table bloat (dead tuples not reclaimed by VACUUM), a long-running transaction may also have been blocking VACUUM during the period the slot was inactive; see Memory: Long-Running DB Transaction for investigation steps. + + * If the WAL directory filled the disk before the slot was dropped, PostgreSQL may have paused writes; free space on the volume (or drop the slot) before the database can resume accepting writes. + + * If `max_slot_wal_keep_size` invalidated the slot before you could investigate (PostgreSQL 13+, positive limit set), the slot's `conflicting` flag is true and `restart_lsn` is null; the consumer must rebuild from a base backup regardless. + + diff --git a/scenarios/disk/disk-space-issues-due-to-core-dumps-and-crash-management.md b/scenarios/disk/disk-space-issues-due-to-core-dumps-and-crash-management.md new file mode 100644 index 00000000..7b7942c5 --- /dev/null +++ b/scenarios/disk/disk-space-issues-due-to-core-dumps-and-crash-management.md @@ -0,0 +1,291 @@ +# Disk Space - Core Dumps + +**Signal:** `system.disk.in_use` high; typically a sudden spike rather than a gradual rise +**IssueType:** `disk_usage` +**Device (typical):** Root partition or a dedicated `/var/crash` volume + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`ls`| investigation| `ls -lh /var/crash/` · `ls -lh /var/lib/systemd/coredump/` +`find`| investigation| `find / -name "core.*" 2>/dev/null \| xargs ls -lh` +`du`| investigation| `du -sh /var/crash/ /var/lib/systemd/coredump/` +`cat`| investigation| `cat /proc/sys/kernel/core_pattern` +`coredumpctl`| both| `coredumpctl list` · `coredumpctl info ` · `coredumpctl clean --disk-free 1G` +`dmesg`| investigation| `dmesg \| grep -i "oom_kill"` +`journalctl`| investigation| `journalctl -k \| grep -i "killed process"` +`systemctl`| both| `systemctl status kdump` · `systemctl daemon-reexec` +`rm`| remediation| `rm -f /var/crash/*` · `rm -f /var/lib/systemd/coredump/*` + +* * * + +## What Happens + +When a process crashes, the kernel or a crash reporter writes a core dump (a snapshot of the process memory at the time of failure) to disk. A single dump can range from hundreds of MB to tens of GB depending on the process's memory footprint. Repeated crashes from the same service fill the disk rapidly and can themselves trigger further crashes (processes that try to write to a full disk also fail). + +Common triggers: + + * An application has a memory bug (segfault, stack overflow) and crashes in a loop + + * An OOM killer event caused a process to be killed, and the crash reporter wrote a dump before clean exit + + * Kernel crash dump (`kdump`) was triggered by a kernel panic and wrote a vmcore file + + * The crash dump destination (`/var/crash`, `core_pattern`) is on the same partition as application data or logs + + * Core dump file size limits were never set (`ulimit -c unlimited` in a startup script) + + * `systemd-coredump` is enabled and accumulating dumps in `/var/lib/systemd/coredump/` + + + + +* * * + +## Detection + +The platform detects this via `system.disk.in_use` breaching the monitor threshold. Because crashes produce sudden large files, the metric often shows a sharp step-function rise rather than a gradual trend. This distinguishes it from log accumulation (which trends upward) and database growth (which is slow and steady). + +**Correlated signals to check:** + + * Elevated error span or error log counts at the time of the disk spike (the crashing application was generating errors before it crashed) + + * Application monitor going to ALERT or NO_DATA around the same time (the process died) + + * `system.mem.page_faults` spikes preceding the crash event + + + + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Required to inspect the filesystem +Root or `adm` group access| Core dumps under `/var/crash` and `/var/lib/systemd/coredump/` typically require root to read +`find`, `du`, `ls`, `dmesg` available on the host| Standard on all Linux distributions +Kernel crash dump analysis tools (`crash`, `kdump`) if a kernel panic is suspected| Needed to inspect vmcore files + +### Steps + + 1. **Locate core dump files** + + + + + + # Common locations + ls -lh /var/crash/ + ls -lh /var/lib/systemd/coredump/ + ls -lh /tmp/core* /tmp/*.core 2>/dev/null + + # Search the whole filesystem for core files (slow on large disks) + find / -name "core" -o -name "core.*" -o -name "*.core" 2>/dev/null | xargs ls -lh 2>/dev/null + + # Check kernel core_pattern to know where dumps land + cat /proc/sys/kernel/core_pattern + + + 2. **Identify the crashing process** + + + + + + # systemd-coredump provides structured metadata + coredumpctl list + # Shows: TIME, PID, UID, GID, SIG, COREFILE, EXE + + coredumpctl info + # Shows: signal received, backtrace (if symbols available), executable path + + + 3. **Check for OOM kills** + + + + + + dmesg | grep -i "out of memory" + dmesg | grep -i "oom_kill" + # Look for: "Out of memory: Kill process () score " + + journalctl -k --since "1 hour ago" | grep -i "killed process" + + + 4. **Check for kernel panics** + + + + + + ls -lh /var/crash/ + # vmcore files from kdump are typically > 1 GB + + # Check if kdump service is active + systemctl status kdump 2>/dev/null || systemctl status crash 2>/dev/null + + + 5. **Check crash frequency** + + + + + + # How many times has this process crashed recently? + coredumpctl list | grep + + # Or check application process supervisor logs + journalctl -u --since "24 hours ago" | grep -i "exit\|crash\|signal\|killed" + + + 6. **Estimate total disk consumed by dumps** + + + + + + du -sh /var/crash/ /var/lib/systemd/coredump/ /tmp/core* 2>/dev/null + + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +Core dumps must have been analyzed (or explicitly waived) before deletion| Dumps are the primary forensic artifact for diagnosing the crash; deleting them before analysis destroys the evidence +Root access| Required to delete files in `/var/crash` and coredump directories +The underlying crash must be addressed separately| Disk space is recovered by deleting dumps, but the crash will recur unless the root cause is fixed +If the crashing service owns critical data, coordinate with the owning team| Some services need controlled restart; do not assume it is safe to restart any service + +### Immediate Space Recovery + +**Delete analyzed or waived core dump files:** + + + # Delete all files in crash directory + rm -f /var/crash/* + + # Delete systemd-coredump archives + rm -f /var/lib/systemd/coredump/* + + # Or vacuum via coredumpctl (keeps N most recent) + coredumpctl clean --disk-free 1G + # Removes oldest dumps until 1 GB is free + + +**Delete kernel vmcore files (kdump):** + + + # Only after analysis or explicit waiver from the kernel/SRE team + rm -f /var/crash/vmcore* + + +### Prevent Accumulation + +**Disable core dumps for a service (prevents new dumps while the crash root cause is being fixed):** + + + # /etc/security/limits.conf or /etc/security/limits.d/.conf + soft core 0 + hard core 0 + + +**Disable core dumps globally via systemd:** + + + # /etc/systemd/system.conf or /etc/systemd/user.conf + DefaultLimitCORE=0 + + +Apply with `systemctl daemon-reexec` (does not restart running services). + +**Cap systemd-coredump storage:** + + + # /etc/systemd/coredump.conf + [Coredump] + Storage=external + Compress=yes + ProcessSizeMax=2G + ExternalSizeMax=2G + MaxUse=10G + KeepFree=5G + + +Apply with `systemctl restart systemd-coredump.socket`. + +### Fix the Underlying Crash + +This is the only permanent solution. Steps depend on the crash type: + +Crash type| Next step +---|--- +Segfault / SIGSEGV| Analyze the backtrace from `coredumpctl info`; identify the faulting code path; deploy a fix +OOM kill| Increase memory limits for the container/service, or fix a memory leak; check `system.mem.used` trend +Recurring application panic| Review application error logs just before the crash; correlate with a recent deploy +Kernel panic (vmcore)| Escalate to kernel/infra team; analyze with `crash` tool + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +Deleting core dump files| None| Dumps are post-mortem artifacts; the processes that wrote them are already dead +Deleting vmcore (kernel crash dump)| None| The kernel has already recovered or the host has already rebooted +`coredumpctl clean`| None| Only removes old dump archives +Setting `DefaultLimitCORE=0` \+ `systemctl daemon-reexec`| None for running services| `daemon-reexec` re-executes the systemd manager but does not restart managed services; the limit applies to newly started services only +Restarting `systemd-coredump.socket`| None| Only the dump handler restarts; application services continue +**Fixing the crashing application**| **Service restart required**| Deploying a bug fix typically requires restarting the affected service; coordinate for a rolling restart or maintenance window +**OOM fix via memory limit increase**| **Service restart required**| Memory limits (cgroup, systemd `MemoryMax`, Kubernetes resource limits) require a service or pod restart to take effect +**Host reboot** (only if kernel panic fix requires it)| **Full host outage**| All services on the host go down; schedule a maintenance window and ensure workloads can failover + +**The crash itself is the primary service disruption.** By the time remediation starts, the affected process has already died. Disk cleanup has no additional service impact. The service restart to deploy a fix is the only planned disruption. + +* * * + +## Verification + + + # Confirm dump files are removed + ls -lh /var/crash/ /var/lib/systemd/coredump/ + + # Confirm disk space recovered + df -h + + # Confirm the crashing service is healthy + systemctl status + coredumpctl list --since "1 hour ago" # should show no new entries + + +In Datadog, verify: + + * `system.disk.in_use` drops below the warning threshold + + * The application's error span/log count returns to baseline + + * The service monitor returns to OK state + + + + +* * * + +## Related Scenarios + + * If the crash was caused by OOM, memory pressure will also show up in the health score via `mem_used_ratio`; check `system.mem.used` and `system.mem.total`. + + * If the host is filling up repeatedly from new dumps despite setting limits, the crash loop is ongoing and the service itself needs to be stopped until the fix is deployed. + + diff --git a/scenarios/disk/disk-space-issues-due-to-unrotated-or-unbounded-log-files.md b/scenarios/disk/disk-space-issues-due-to-unrotated-or-unbounded-log-files.md new file mode 100644 index 00000000..4dc1e606 --- /dev/null +++ b/scenarios/disk/disk-space-issues-due-to-unrotated-or-unbounded-log-files.md @@ -0,0 +1,318 @@ +# Disk Space - Unrotated / Unbounded Logs + +**Signal:** `system.disk.in_use` rising steadily; log directory growth rate correlates with application activity +**IssueType:** `disk_usage` +**Device (typical):** Root partition or a dedicated `/var/log` volume + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`df`| investigation| `df -h` +`du`| investigation| `du -h --max-depth=2 /var/log \| sort -rh \| head -20` +`ls`| investigation| `ls -lhS /var/log//` +`find`| investigation| `find /var/log -name "*.log" -size +500M` · `find /var/log -mtime -1 -name "*.log"` +`logrotate`| both| `logrotate -d /etc/logrotate.d/` · `logrotate -f /etc/logrotate.conf` +`cat`| investigation| `cat /etc/logrotate.d/` +`lsof`| investigation| `lsof \| grep deleted \| grep log` +`journalctl`| both| `journalctl --disk-usage` · `journalctl --vacuum-size=500M` · `journalctl --vacuum-time=7d` +`systemctl`| investigation| `systemctl status logrotate.timer` · `systemctl list-timers \| grep logrotate` +`truncate`| remediation| `truncate -s 0 /var/log//.log` +`rm`| remediation| `rm -f /var/log//*.log.` + +* * * + +## What Happens + +Log files grow as applications write to them. Without rotation or retention limits, a log file grows indefinitely for the lifetime of the service. Unlike core dumps (sudden large files) or database growth (slow data accumulation), log growth is typically steady and proportional to application activity — accelerating during high-traffic periods or error storms and decelerating overnight. + +Common causes: + + * A service has no logrotate configuration — logs have never been rotated since the service was deployed + + * logrotate is configured but the `rotate` count is too high or `maxsize` / `size` thresholds are never triggered + + * A log storm: an application bug or error loop writes error lines at high rate, growing the log file orders of magnitude faster than normal + + * The logrotate timer or cron job is broken or was never enabled (`systemctl status logrotate.timer`) + + * Old rotated log files (`*.log.1`, `*.log.2.gz`) are never deleted because the `rotate N` count is set too high or missing + + * A process holds the deleted log file open after rotation: the inode stays allocated and disk space is not reclaimed until the process is restarted or sends SIGHUP to re-open the file + + * `journald` is configured with no size limit and accumulates logs from all systemd services indefinitely + + + + +* * * + +## Detection + +The platform detects this via `system.disk.in_use` breaching the monitor threshold. The defining characteristic is a steady, linear rise over hours or days — proportional to application log output. This distinguishes it from core dumps (sudden step-function spike) and database growth (very slow, data-volume-driven). + +**Correlated signals to check:** + + * `system.disk.in_use` growth rate matches known application traffic patterns (higher during business hours, flat overnight) — confirms log growth rather than another source + + * Application error span count elevated: a log storm is often preceded or accompanied by a spike in error logs + + * Check `system.disk.in_use` on `/var/log` specifically if it is a separate volume + + * `journalctl --disk-usage` growing beyond expected bounds on systemd hosts + + + + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Commands must run on the host +Read access to `/var/log` and logrotate config directories| Required to inspect log sizes and rotation config +Ability to run `lsof`| Requires root or equivalent to list all open file descriptors + +### Steps + + 1. **Identify which log directory is consuming the most space** + + + + + + du -h --max-depth=2 /var/log 2>/dev/null | sort -rh | head -20 + # Focus on directories with unexpected sizes or that are growing + + 2. **Find the largest individual log files** + + + + + + find /var/log -name "*.log" -o -name "*.log.*" 2>/dev/null | xargs ls -lhS 2>/dev/null | head -20 + # Large uncompressed rotated files (*.log.1, *.log.2) indicate rotation without cleanup + # A single huge *.log file indicates no rotation at all + + 3. **Check how fast the active log file is growing** + + + + + + # Sample size twice, 60 seconds apart + LOG=/var/log//.log + size1=$(stat -c%s "$LOG" 2>/dev/null); sleep 60; size2=$(stat -c%s "$LOG" 2>/dev/null) + echo "Growth: $((size2 - size1)) bytes/min" + # High growth rate = active log storm; low rate = accumulation over time without rotation + + 4. **Check the logrotate configuration for the service** + + + + + + cat /etc/logrotate.d/ + # Key fields to verify: + # rotate N — how many old files to keep (missing = keep forever) + # size / maxsize — trigger rotation at this file size (missing = only time-based) + # daily/weekly — rotation frequency + # compress — whether old files are gzip'd + # postrotate — script to signal the process to re-open log files + + # Dry-run logrotate to see what it would do + logrotate -d /etc/logrotate.d/ + + 5. **Check if the logrotate timer is running** + + + + + + systemctl status logrotate.timer + systemctl list-timers | grep logrotate + # Confirm last trigger time and next scheduled run + # If inactive or failed: logrotate has not run recently + + 6. **Check for deleted-but-open log files (space not reclaimed after rotation)** + + + + + + lsof | grep deleted | grep -i log + # Output: COMMAND PID USER FD SIZE NAME (deleted) + # If the active log path shows as "(deleted)", the process is still writing to the old inode + # Space is not reclaimed until the process is restarted or re-opens the file + + 7. **Check journald disk usage** + + + + + + journalctl --disk-usage + # Shows total space used by the systemd journal + # Compare against /etc/systemd/journald.conf SystemMaxUse= setting + cat /etc/systemd/journald.conf | grep -E "SystemMaxUse|RuntimeMaxUse|MaxRetentionSec" + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +Confirm the log files are not needed for an active incident or audit| Deleting or truncating logs destroys forensic evidence; check with the owning team if an incident is open +Identify whether a log storm is ongoing before truncating| If the application is actively in an error loop, truncating the log recovers space temporarily but the file regrows immediately; fix the root cause first +Confirm the process handles SIGHUP for log re-open (if using copytruncate-less rotation)| Some processes require SIGHUP or a restart to re-open the log file after rotation + +### Immediate Space Recovery + +**Delete old rotated log files:** + + + # List rotated files for the service first + ls -lh /var/log//*.log.* /var/log//*.gz 2>/dev/null + + # Delete all rotated (non-active) log files + rm -f /var/log//*.log.[0-9]* + rm -f /var/log//*.log.*.gz + +**Truncate the active log file (preserves the file and inode; process keeps writing):** + + + # Safe: truncate in-place — the process's file descriptor stays valid + truncate -s 0 /var/log//.log + + # Alternative using shell redirection + > /var/log//.log + +**Force a logrotate run immediately:** + + + logrotate -f /etc/logrotate.d/ + # -f forces rotation even if the size/time threshold has not been met + +**Reclaim space from deleted-but-open log files:** + + + # Signal the process to re-open its log file (if it supports SIGHUP) + kill -HUP + + # Verify the deleted file entry is gone + lsof | grep deleted | grep -i log + + # If SIGHUP is not supported, a service restart is required + systemctl restart + +**Vacuum journald:** + + + # Keep only the last 500 MB + journalctl --vacuum-size=500M + + # Or keep only the last 7 days + journalctl --vacuum-time=7d + +### Fix the Underlying Configuration + +**Add or fix a logrotate config for the service:** + + + # /etc/logrotate.d/ + /var/log//*.log { + daily + rotate 7 + compress + delaycompress + missingok + notifempty + maxsize 500M + postrotate + systemctl reload 2>/dev/null || true + endscript + } + +Key settings: + +Setting| Purpose +---|--- +`rotate 7`| Keep at most 7 old log files; delete older ones automatically +`maxsize 500M`| Rotate immediately if the file exceeds 500 MB, regardless of schedule +`compress` / `delaycompress`| Gzip old logs; delay one cycle so the previous file is not compressed while still potentially in use +`postrotate`| Reload or signal the service to re-open the new log file after rotation + +**Cap journald size permanently:** + + + # /etc/systemd/journald.conf + [Journal] + SystemMaxUse=2G + MaxRetentionSec=1month + +Apply with: + + + systemctl restart systemd-journald + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +`du`, `find`, `ls`, `lsof`, `logrotate -d`| None| Read-only inspection +`rm` on old rotated log files| None| Rotated files are no longer written to by the application +`truncate -s 0` on the active log file| None| The process's file descriptor remains valid; it continues writing from offset 0 +`logrotate -f`| None for most services| The postrotate script may send SIGHUP; confirm the service handles it gracefully +`kill -HUP `| None for most services| Causes the process to re-open log files; may briefly flush buffers +`journalctl --vacuum-*`| None| Only removes old journal data; no running processes affected +`systemctl restart systemd-journald`| **Brief gap in log collection**| Journal entries from all services are buffered and may be lost during the restart window +`systemctl restart ` (to release deleted fd)| **Brief service interruption**| Required only if the process does not support SIGHUP for log re-open + +**Truncating an active log file is always safe.** The process keeps its file descriptor; writes continue to the now-empty file. No restart is needed and no data is lost from the running process. + +* * * + +## Verification + + + # Confirm log directory has shrunk + du -sh /var/log// + + # Confirm disk space recovered + df -h + + # Confirm no deleted-but-open log files remain + lsof | grep deleted | grep -i log + + # Confirm logrotate timer is active and will run on schedule + systemctl list-timers | grep logrotate + +In Datadog, verify: + + * `system.disk.in_use` drops after cleanup and the growth rate returns to a sustainable slope (or flat if the storm is resolved) + + * If a log storm was the cause: application error span count returns to baseline, confirming the underlying issue has been fixed + + + + +* * * + +## Related Scenarios + + * If the log file is growing at an abnormal rate (gigabytes per hour rather than per day), an application is in an error loop; fix the root cause before truncating — the file will regrow immediately otherwise. + + * If `lsof | grep deleted` shows large deleted-but-open files and a service restart is not immediately possible, the space cannot be reclaimed until the process releases the file handle; quantify the size and plan a maintenance window. + + * If `/var/log` is on the root partition and is nearly full, other writes (application temp files, package manager operations) will also start failing; prioritize recovery before the partition reaches 100%. + + diff --git a/scenarios/disk/disk-space-management-addressing-database-growth-and-retention-issues.md b/scenarios/disk/disk-space-management-addressing-database-growth-and-retention-issues.md new file mode 100644 index 00000000..0038a07e --- /dev/null +++ b/scenarios/disk/disk-space-management-addressing-database-growth-and-retention-issues.md @@ -0,0 +1,331 @@ +# Disk Space - Database Growth + +**Signal:** `system.disk.in_use` high on the database data partition +**IssueType:** `disk_usage` +**Device (typical):** `/dev/sdb1`, `/dev/nvme1n1`, or a dedicated mount like `/var/lib/postgresql` or `/var/lib/mysql` + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`du`| investigation| `du -sh /var/lib/postgresql/` · `du -sh /var/lib/mysql/` +`psql`| both| `psql -U postgres -c "SELECT relname, pg_size_pretty(...) FROM pg_stat_user_tables ..."` · `VACUUM ANALYZE ` · `REINDEX INDEX CONCURRENTLY ` +`mysql`| both| `mysql -e "SHOW BINARY LOGS;"` · `PURGE BINARY LOGS BEFORE NOW() - INTERVAL 7 DAY;` +`grep`| investigation| `grep -i "cleanup\|purge" /etc/cron.d/*` +`systemctl`| investigation| `systemctl list-timers \| grep -i vacuum` +`journalctl`| investigation| `journalctl --since "7 days ago" \| grep -i "vacuum\|retention"` + +* * * + +## What Happens + +Database storage grows when data is inserted faster than it is removed, or when the database engine retains internal overhead that is not automatically reclaimed. Unlike log files, database growth is typically slow and steady unless a specific process has gone wrong. Root causes include: + + * Missing or misconfigured data retention policies (no scheduled DELETE or PURGE jobs) + + * A retention cleanup job that stopped running silently (cron failure, job crash) + + * PostgreSQL table bloat: rows are logically deleted but the dead tuples are not yet vacuumed; the physical file stays large + + * PostgreSQL index bloat: indexes grow from updates/deletes but never shrink without a REINDEX + + * MySQL binary log accumulation: replication logs not purged after the retention period + + * Uncontrolled growth of a time-series or event table with no partitioning or TTL + + * A runaway batch import or backfill job that inserted far more data than expected + + + + +* * * + +## Detection + +The platform detects this via `system.disk.in_use` on the database volume. Growth is typically gradual; the metric trends upward over days or weeks before breaching the threshold. This distinguishes it from core dumps (sudden spike) or log storms (rapid rise during an incident). + +**Correlated signals to check:** + + * APM traces showing slow queries on the affected database (table bloat increases sequential scan cost) + + * Application error spans mentioning `disk full`, `no space left on device`, or `could not write to file` + + * Database-specific metrics in Datadog if the database integration is configured: + + * `postgresql.table.bloat` (if the Postgres integration is running) + + * `mysql.replication.slave_running` going to 0 (replication can fail when bin logs fill the disk) + + + + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Required to inspect the filesystem and run database clients +Database client available on the host (`psql`, `mysql`, `redis-cli`)| Needed for introspection queries +Read-only database credentials at minimum| Introspection queries require a database connection +Knowledge of which database engine and version is running| Commands differ significantly between PostgreSQL, MySQL, and others + +### Steps + + 1. **Confirm the database directory is the consumer** + + + + + + # PostgreSQL default + du -sh /var/lib/postgresql/ + + # MySQL/MariaDB default + du -sh /var/lib/mysql/ + + # Check per-database directory sizes + du -sh /var/lib/postgresql//main/base/*/ + + + 2. **PostgreSQL: find the largest tables and databases** + + + + + + -- Connect: psql -U postgres + + -- Total size per database + SELECT datname, pg_size_pretty(pg_database_size(datname)) AS size + FROM pg_database + ORDER BY pg_database_size(datname) DESC; + + -- Largest tables in current database (including indexes and TOAST) + SELECT relname, + pg_size_pretty(pg_total_relation_size(relid)) AS total, + pg_size_pretty(pg_relation_size(relid)) AS table_only, + pg_size_pretty(pg_total_relation_size(relid) + - pg_relation_size(relid)) AS indexes_and_toast + FROM pg_stat_user_tables + ORDER BY pg_total_relation_size(relid) DESC + LIMIT 20; + + + 3. **PostgreSQL: measure dead tuple bloat** + + + + + + -- Tables with the most dead tuples (candidates for VACUUM) + SELECT relname, + n_dead_tup, + n_live_tup, + round(n_dead_tup::numeric / NULLIF(n_live_tup + n_dead_tup, 0) * 100, 1) AS dead_pct, + last_autovacuum, + last_vacuum + FROM pg_stat_user_tables + WHERE n_dead_tup > 10000 + ORDER BY n_dead_tup DESC + LIMIT 20; + + + 4. **PostgreSQL: check if autovacuum is keeping up** + + + + + + -- Tables where autovacuum is behind + SELECT relname, last_autovacuum, autovacuum_count, n_dead_tup + FROM pg_stat_user_tables + WHERE last_autovacuum < NOW() - INTERVAL '1 day' + OR last_autovacuum IS NULL + ORDER BY n_dead_tup DESC + LIMIT 10; + + + 5. **MySQL/MariaDB: find the largest tables** + + + + + + -- Connect: mysql -u root -p + + SELECT table_schema, + table_name, + ROUND((data_length + index_length) / 1024 / 1024, 1) AS mb + FROM information_schema.tables + ORDER BY data_length + index_length DESC + LIMIT 20; + + + 6. **MySQL: check binary log accumulation** + + + + + + SHOW BINARY LOGS; + -- Lists all retained bin log files and their sizes + -- If the list is long and sizes are large, purge policy is missing or broken + + SHOW VARIABLES LIKE 'expire_logs_days'; + SHOW VARIABLES LIKE 'binlog_expire_logs_seconds'; + -- A value of 0 means binary logs are never automatically purged + + + 7. **Check for failed retention cleanup jobs** + + + + + + # Check cron job history + grep -i "cleanup\|purge\|retain\|delete" /var/spool/cron/* /etc/cron.d/* 2>/dev/null + + # Check systemd timer status for database maintenance jobs + systemctl list-timers | grep -i "vacuum\|cleanup\|purge\|pg\|mysql" + + # Check recent job execution + journalctl --since "7 days ago" | grep -i "vacuum\|cleanup\|retention" + + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +Write credentials for the database| Required for VACUUM, REINDEX, DELETE, PURGE commands +Data deletion must be reviewed against retention and compliance requirements| Deleting records prematurely may violate regulatory obligations (GDPR, HIPAA, SOX); confirm with the data owner +For large DELETE operations: confirm application write load is low or can be quiesced| Large bulk deletes compete with application writes for lock access and can cause cascading slowdowns +For REINDEX CONCURRENT: sufficient disk space must exist for the new index to be built alongside the old one| REINDEX CONCURRENT temporarily doubles index storage + +### Immediate Space Recovery + +**PostgreSQL: run VACUUM to reclaim dead tuples** + + + -- Standard VACUUM (runs online, does not lock the table) + VACUUM ANALYZE ; + + -- VACUUM FULL (reclaims space to OS; requires exclusive lock - causes downtime on that table) + -- Use only in a maintenance window + VACUUM FULL ; + + +**PostgreSQL: rebuild a bloated index** + + + -- Online rebuild (no table lock; takes longer) + REINDEX INDEX CONCURRENTLY ; + + -- All indexes on a table (online) + REINDEX TABLE CONCURRENTLY ; + + +**MySQL: purge binary logs** + + + -- Remove bin logs older than 7 days + PURGE BINARY LOGS BEFORE NOW() - INTERVAL 7 DAY; + + -- Or remove everything up to a specific log file + PURGE BINARY LOGS TO 'mysql-bin.001234'; + + +**Delete old data (application-specific):** + + + -- Example: delete records older than 90 days in batches to avoid lock contention + DELETE FROM events + WHERE created_at < NOW() - INTERVAL '90 days' + LIMIT 10000; + -- Repeat in a loop with a short sleep between batches + + +### Permanent Fix + +**MySQL: set binary log expiry:** + + + -- In /etc/mysql/mysql.conf.d/mysqld.cnf + [mysqld] + binlog_expire_logs_seconds = 604800 -- 7 days + + +Apply with `systemctl reload mysql` (if supported) or a scheduled restart. + +**PostgreSQL: tune autovacuum aggressiveness:** + + + -- Per-table setting for a high-churn table + ALTER TABLE + SET (autovacuum_vacuum_scale_factor = 0.01, + autovacuum_analyze_scale_factor = 0.005); + + +**Add or fix the data retention job:** + + + # Example cron entry for a nightly delete job + 0 2 * * * postgres psql -d -c "DELETE FROM events WHERE created_at < NOW() - INTERVAL '90 days';" >> /var/log/db-cleanup.log 2>&1 + + +**PostgreSQL: implement table partitioning** (for very high-volume tables): partition by time and drop old partitions instead of row-by-row DELETE. Dropping a partition is nearly instantaneous and produces no dead tuples. + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +`VACUUM ANALYZE`| None| Online operation; table remains readable and writable throughout +`REINDEX INDEX CONCURRENTLY`| None| Online; takes longer than standard REINDEX but does not lock the table +Batch DELETE (small batches with sleep)| Minimal; brief row-level locks per batch| Application may see slightly slower query response on the affected table during batch runs; use off-peak hours for large datasets +`PURGE BINARY LOGS` (MySQL)| None| Only removes archived replication logs; no active sessions affected +`VACUUM FULL`| **Table-level exclusive lock**| The table is inaccessible for reads and writes for the duration (seconds to hours depending on table size); schedule a maintenance window +Standard `REINDEX` (without CONCURRENTLY)| **Table-level lock**| Same as VACUUM FULL; block all queries on the table; use CONCURRENTLY variant instead +`systemctl reload mysql` (applying `binlog_expire_logs_seconds`)| None if reload is supported; otherwise restart required| Check if `mysql -e "SET GLOBAL binlog_expire_logs_seconds=604800;"` can apply it without a restart +Adding table partitioning to an existing table| **Major migration**| Requires locking and rewriting the table; plan as a dedicated migration with downtime or use `pg_partman` \+ logical replication for zero-downtime migration +Fixing a failed retention job (editing cron)| None immediately| Only affects future scheduled runs + +**The two high-impact operations to avoid in production without a maintenance window are**`VACUUM FULL`**and non-concurrent**`REINDEX`**.** For all other cleanup steps, the database and its dependent services remain fully available. + +* * * + +## Verification + + + # Confirm disk usage has dropped + df -h + + # PostgreSQL: confirm dead tuples have been cleared + psql -U postgres -c "SELECT relname, n_dead_tup FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 10;" + + # MySQL: confirm binary logs have been purged + mysql -u root -p -e "SHOW BINARY LOGS;" + + # In Datadog + # avg:system.disk.in_use{host:,device:} + # Should drop below warning threshold within 1-5 minutes of space being reclaimed + + +* * * + +## Related Scenarios + + * If the disk fills completely before VACUUM can run, PostgreSQL may stop accepting writes (`could not write to file` errors). In this case, immediate space recovery (moving or deleting old binary logs, removing tmp files on the same partition) must happen first to allow VACUUM to proceed. + + * Slow queries surfaced in APM may persist even after dead tuples are removed if indexes were also bloated; follow with `REINDEX CONCURRENTLY`. + + diff --git a/scenarios/disk/disk-space-management-cleaning-temp-files-and-build-artifacts.md b/scenarios/disk/disk-space-management-cleaning-temp-files-and-build-artifacts.md new file mode 100644 index 00000000..9aae8fee --- /dev/null +++ b/scenarios/disk/disk-space-management-cleaning-temp-files-and-build-artifacts.md @@ -0,0 +1,295 @@ +# Disk Space - Temp Files and Build Artifacts + +**Signal:** `system.disk.in_use` high on root partition or a partition hosting build workdirs +**IssueType:** `disk_usage` +**Device (typical):** Root partition; occasionally a dedicated `/build` or `/workspace` volume + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`du`| investigation| `du -h --max-depth=2 / \| sort -rh \| head -30` · `du -sh ~/.npm ~/.cache/pip ~/.m2 ~/.gradle/caches` +`find`| both| `find /tmp /var/tmp -size +100M` · `find /tmp -mindepth 1 -mtime +1 -exec rm -rf {} +` +`lsof`| investigation| `lsof /tmp` +`npm`| both| `npm cache verify` · `npm cache clean --force` +`pip`| both| `pip cache info` · `pip cache purge` +`gradle`| remediation| `gradle --stop` +`go`| remediation| `go clean -modcache` +`rm`| remediation| `rm -rf ~/.m2/repository` · `rm -rf ~/.gradle/caches` +`systemd-tmpfiles`| remediation| `systemd-tmpfiles --clean` + +* * * + +## What Happens + +Build systems, package managers, and runtime tools accumulate files in temporary and cache directories that are not automatically cleaned up. Unlike log accumulation (which is steady) or database growth (which is slow), this pattern often shows step-function spikes during CI job runs followed by a plateau when the job ends but artifacts are not cleaned. + +Common sources: + + * **CI/CD pipelines** that clone repos, compile code, and produce artifacts (binaries, containers, test outputs) without a cleanup step + + * **Package manager caches** that download dependencies to local cache directories and never invalidate them: `~/.npm`, `~/.cache/pip`, `~/.m2/repository`, `~/.gradle/caches` + + * **/tmp not cleaned** : Long-running hosts accumulate temporary files from processes that crash or do not clean up after themselves; systemd-tmpfiles defaults clean `/tmp` on reboot, not on a schedule + + * **/var/tmp** : Unlike `/tmp`, this directory survives reboots and is rarely cleaned automatically + + * **Application working directories** that write intermediate files (extracted archives, decompressed data, temporary uploads) without a cleanup phase + + + + +* * * + +## Detection + +The platform detects this via `system.disk.in_use` breaching the threshold. The shape of the metric is the key diagnostic clue: + + * Spikes that correlate with CI job timing (check if the spike aligns with your pipeline schedule) suggest build artifact accumulation + + * Slow, continuous growth suggests package cache accumulation + + * Sudden spikes at random times may indicate an application writing large temp files without cleanup + + + + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Commands must run on the host +Read access to home directories and build workspaces| Package caches live in user home directories; access is needed to inspect them +Knowledge of which CI system and build tool the host runs| Commands differ between npm, pip, Maven, Gradle, Bazel, etc. + +### Steps + + 1. **Identify the largest consumers** + + + + + + du -h --max-depth=2 / 2>/dev/null | sort -rh | head -30 + # Focus on: /tmp, /var/tmp, /home, /root, /build, /workspace, /runner + + + 2. **Check /tmp and /var/tmp** + + + + + + du -sh /tmp /var/tmp + find /tmp /var/tmp -size +100M -exec ls -lh {} \; 2>/dev/null + find /tmp /var/tmp -mtime +1 -exec ls -lh {} \; 2>/dev/null | head -30 + # Files older than 1 day that are still in /tmp are often safe to remove + + + 3. **Check package manager caches** + + + + + + # npm (Node.js) + du -sh ~/.npm 2>/dev/null + npm cache verify 2>/dev/null + + # pip (Python) + du -sh ~/.cache/pip 2>/dev/null + pip cache info 2>/dev/null + + # Maven (Java) + du -sh ~/.m2/repository 2>/dev/null + + # Gradle (Java/Kotlin) + du -sh ~/.gradle/caches 2>/dev/null + + # Cargo (Rust) + du -sh ~/.cargo/registry 2>/dev/null + + # Go module cache + du -sh $(go env GOPATH)/pkg/mod 2>/dev/null + + + 4. **Check CI/CD workspace directories** + + + + + + # GitLab Runner + du -sh /home/gitlab-runner/builds/ 2>/dev/null + + # GitHub Actions + du -sh /home/runner/_work/ 2>/dev/null + du -sh /opt/hostedtoolcache/ 2>/dev/null + + # Jenkins + du -sh /var/lib/jenkins/workspace/ 2>/dev/null + + + 5. **Check for large application temp files** + + + + + + find /tmp /var/tmp /opt /srv -name "*.tmp" -o -name "*.part" -o -name "*.download" 2>/dev/null | xargs ls -lh 2>/dev/null | sort -k5 -rh | head -20 + + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +Confirm no running process is actively using a file before deleting it| Deleting a file that a process has open does not reclaim space until the process closes or exits; the inode stays allocated +Check with `lsof` before removing large files from /tmp| Some processes rely on specific temp file paths +CI pipeline config access if adding a cleanup step| Requires repo access and a CI pipeline edit + +### Immediate Space Recovery + +**Clean /tmp (check for active users first):** + + + # Check if any process has files in /tmp open + lsof /tmp 2>/dev/null | head -20 + + # Delete files older than 1 day (safe on most hosts) + find /tmp -mindepth 1 -mtime +1 -exec rm -rf {} + 2>/dev/null + + # Or delete everything that is not held open by a process + find /tmp -mindepth 1 ! -exec fuser -s {} \; -exec rm -rf {} + 2>/dev/null + + +**Clean package manager caches:** + + + npm cache clean --force + + pip cache purge + + # Maven: delete the entire local repository (will be re-downloaded on next build) + rm -rf ~/.m2/repository + + # Gradle + gradle --stop && rm -rf ~/.gradle/caches + + # Go + go clean -modcache + + +**Clean CI workspace directories:** + + + # GitLab Runner: remove old build directories + find /home/gitlab-runner/builds/ -maxdepth 2 -mindepth 2 -mtime +7 -type d -exec rm -rf {} + 2>/dev/null + + # Jenkins: clean all workspaces from the Jenkins UI, or: + find /var/lib/jenkins/workspace/ -maxdepth 1 -mindepth 1 -mtime +7 -exec rm -rf {} + 2>/dev/null + + +### Permanent Fix + +**Add a cleanup step to CI pipelines:** + + + # GitLab CI: always-run cleanup job + cleanup: + stage: .post + script: + - rm -rf build/ dist/ target/ node_modules/.cache/ + - docker system prune -f || true + when: always + + # Or configure GitLab Runner to clean the workspace before each job + # config.toml + [[runners]] + [runners.custom_build_dir] + enabled = true + [runners.cache] + [runners.cache.s3] + environment = ["GIT_CLEAN_FLAGS=-fdx"] + + +**Configure systemd-tmpfiles to clean /tmp on a schedule (not just on reboot):** + + + # /etc/tmpfiles.d/tmp-cleanup.conf + # Remove files in /tmp older than 3 days + d /tmp 1777 root root 3d + + +Apply immediately: + + + systemd-tmpfiles --clean + + +**Set up a cron job for package cache cleanup on build hosts:** + + + # /etc/cron.d/build-cache-cleanup + # Run at 2 AM daily + 0 2 * * * ci-user find /home/ci-user/.npm -mtime +14 -delete 2>/dev/null; pip cache purge 2>/dev/null + + +**For Maven and Gradle on CI:** Configure them to use a shared, version-tagged cache directory that is explicitly invalidated when dependencies change (avoid global `~/.m2` on shared build hosts). + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +Delete files from /tmp older than N days| Low risk; potential process failure| Any process relying on a specific temp file path that gets deleted will fail. Use `lsof` to confirm no open file handles before mass deletion. Random temp file names (e.g., `/tmp/tmpXXXXXX`) are always safe to delete if the process that created them is no longer running +`npm cache clean --force`| None for running services| Affects only future npm install runs, which will re-download dependencies (slower first run) +`pip cache purge`| None for running services| Same as npm; existing installed packages are not affected +`rm -rf ~/.m2/repository`| None for running services; next build is slower| Maven re-downloads all dependencies on the next build; this can take minutes on a cold cache +`gradle --stop`| **Stops the Gradle build daemon**| Any in-progress Gradle builds on that host are killed. Do not run during an active CI job +`go clean -modcache`| None for running services| Affects only future builds +Deleting CI workspace directories| **Kills in-progress CI jobs if their workspace is deleted**| Always check if a job is running on the host before deleting its workspace directory. Use the CI platform's interface to drain or cancel jobs first +Adding a cleanup step to a CI pipeline| None| Only affects future pipeline runs +Configuring systemd-tmpfiles| None at config time| `systemd-tmpfiles --clean` runs a one-shot cleanup pass; does not affect running services unless they happen to own a file being cleaned (which would only occur if the file is older than the configured age) +Deleting entire npm `node_modules` (if present)| **Application restart required**| If a running Node.js application loaded modules from the deleted `node_modules`, it will crash or behave incorrectly. Only delete `node_modules` when the application is stopped or if it is a build-time directory not used at runtime + +**Key consideration for CI hosts:** A CI runner may be executing a job at the time cleanup is triggered. Always drain or pause the runner before performing aggressive cleanup (workspace deletion, Gradle daemon stop). For cloud-based ephemeral runners, this is not an issue since the host terminates after the job. + +* * * + +## Verification + + + # Confirm space has been reclaimed + df -h + + # Confirm /tmp is clean + du -sh /tmp + + # Confirm caches have been cleared + du -sh ~/.npm ~/.cache/pip ~/.m2 ~/.gradle/caches 2>/dev/null + + # In Datadog + # avg:system.disk.in_use{host:,device:} + # Should drop below warning threshold within 1-5 minutes + + +* * * + +## Related Scenarios + + * If cleanup recovers space but the disk fills again within hours during CI runs, the pipeline cleanup step is missing or ineffective; prioritize the permanent fix. + + * If `/tmp` fills rapidly without a CI job running, an application is writing large temp files without cleanup; use `lsof` and `inotifywait` to identify the writing process in real time. + + diff --git a/scenarios/disk/disk-space-management-for-docker-investigating-and-remediating-container-layer-usage.md b/scenarios/disk/disk-space-management-for-docker-investigating-and-remediating-container-layer-usage.md new file mode 100644 index 00000000..3bda5c83 --- /dev/null +++ b/scenarios/disk/disk-space-management-for-docker-investigating-and-remediating-container-layer-usage.md @@ -0,0 +1,262 @@ +# Disk Space - Container Layers + +**Signal:** `system.disk.in_use` high on root partition or a partition mounting `/var/lib/docker` +**IssueType:** `disk_usage` +**Device (typical):** `/dev/sda1`, `/dev/nvme0n1p1`, or a dedicated Docker data volume + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`du`| investigation| `du -sh /var/lib/docker` +`docker system`| both| `docker system df` · `docker system prune -f` · `docker system prune -a -f` +`docker ps`| investigation| `docker ps -a` +`docker images`| investigation| `docker images -f dangling=true` +`docker volume`| both| `docker volume ls -f dangling=true` · `docker volume prune -f` +`docker buildx`| both| `docker buildx du` · `docker buildx prune -f` + +* * * + +## What Happens + +Docker stores image layers, container filesystems, build cache, and volumes under `/var/lib/docker`. Without periodic cleanup, this directory grows continuously as: + + * New image versions are pulled but old tags are not removed + + * Containers exit or crash and are never removed (`docker ps -a` shows many stopped containers) + + * BuildKit cache accumulates from repeated `docker build` runs + + * Named or anonymous volumes are orphaned when their containers are deleted + + * Multi-stage build intermediate layers are retained by default + + + + +On CI hosts or hosts that frequently deploy new versions, this can consume tens of GBs over weeks. + +* * * + +## Detection + +Detected via `system.disk.in_use` monitor (type `HOST_DISK_USAGE`). The device in `HostMetadata` will be whichever partition `/var/lib/docker` resides on. If Docker has its own dedicated volume, that device appears directly. + +**Secondary signal:** If the host's health score also shows elevated error spans, the disk pressure may be causing container OOM kills or failed deployments, which compounds the issue. + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Commands must run locally +Permission to run `docker` commands (member of `docker` group or root)| Docker daemon queries require this +Docker daemon is running| `docker system df` requires the daemon to be up +No active container that uses volumes under investigation| Must confirm volumes are safe to remove + +### Steps + + 1. **Confirm Docker is the consumer** + + + + + + du -sh /var/lib/docker + # If this is a significant fraction of disk usage, Docker is the culprit + + + 2. **Get a full Docker disk usage breakdown** + + + + + + docker system df + # Output: + # TYPE TOTAL ACTIVE SIZE RECLAIMABLE + # Images 47 12 18.4GB 14.2GB (77%) + # Containers 23 3 1.1GB 1.0GB (94%) + # Local Volumes 11 4 3.2GB 1.8GB (56%) + # Build Cache - - 4.7GB 4.7GB + + + 3. **Identify stale images** + + + + + + # Dangling images (untagged, no longer referenced by any container) + docker images -f dangling=true + + # All images with their age + docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}\t{{.CreatedSince}}" + + # Images not used by any running or stopped container + docker image ls --filter "dangling=false" | head -30 + + + 4. **Identify stopped containers** + + + + + + docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Size}}\t{{.Image}}" + # Containers with status "Exited" or "Created" are reclaimable + + + 5. **Identify orphaned volumes** + + + + + + docker volume ls -f dangling=true + # These volumes have no container referencing them + + + 6. **Check BuildKit cache** + + + + + + docker buildx du 2>/dev/null || docker builder du + # Shows cache size and age + + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +Docker daemon must be running| `docker system prune` requires the daemon +No critical stopped containers should be removed| Stopped containers may hold data or be intentionally paused; confirm with the owning team +Named volumes must be confirmed as disposable| Volumes may contain persistent state (databases, user uploads); check before pruning +Active deployments should not be in-flight| Pruning during a deployment can remove layers needed by a pull in progress + +### Immediate Space Recovery + +**Safe prune (removes stopped containers, dangling images, unused networks; does NOT remove volumes or images in use):** + + + docker system prune -f + + +**Remove unused images including tagged images not referenced by any container:** + + + # More aggressive: removes all images not used by a running container + docker system prune -a -f + + +**Remove orphaned volumes (only after confirming they hold no needed data):** + + + docker volume prune -f + + +**Remove BuildKit cache:** + + + docker buildx prune -f + # or + docker builder prune -f + + +**All-in-one (most aggressive; run with caution on production hosts):** + + + docker system prune -a --volumes -f + + +### Permanent Fix + +**Add a scheduled cleanup cron job:** + + + # /etc/cron.d/docker-cleanup + # Run at 3 AM daily; keep images used in the last 48 hours + 0 3 * * * root docker system prune -f --filter "until=48h" >> /var/log/docker-cleanup.log 2>&1 + + +**Or configure the Docker daemon to limit image retention (Docker 25+):** + + + // /etc/docker/daemon.json + { + "image-gc-high-threshold": 85, + "image-gc-low-threshold": 80 + } + + +Restart the daemon after changing `daemon.json`: + + + systemctl restart docker + + +**For CI hosts specifically:** Add a cleanup step at the end of each CI job or pipeline: + + + # Example: GitLab CI cleanup job + cleanup: + stage: cleanup + script: + - docker system prune -f + when: always + + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +`docker system prune -f`| None| Only removes stopped containers and dangling images; running containers are untouched +`docker system prune -a -f`| None for running containers| Removes unused tagged images too; safe unless a deployment is actively pulling an image (race condition: the layer being pulled may be pruned mid-transfer, causing the pull to fail and the deployment to retry) +`docker volume prune -f`| None| Only removes volumes with no container attached; volumes in use by running containers are skipped +`docker buildx prune -f`| None| Build cache only; zero runtime impact +Adding a cleanup cron job| None| Only affects future scheduled runs +Changing `daemon.json` \+ `systemctl restart docker`| **Full outage on the host**| Restarting the Docker daemon stops ALL running containers. This is a last-resort action; coordinate with container owners and schedule a maintenance window +Configuring `image-gc-high-threshold` in `daemon.json`| **Requires daemon restart** (see above)| Plan accordingly + +**Key risk - deployment race condition:** If a new image is being pulled for a rolling deployment at the same time as `docker system prune -a`, the prune may remove intermediate layers that the pull depends on. The pull will fail and retry, causing a brief deployment hiccup. Avoid running aggressive prune commands during active deployments. + +**Docker daemon restart is the one action that causes a full host-level container outage.** It is almost never necessary for disk cleanup alone. Only do it if changing `daemon.json` settings that cannot be applied at runtime. + +* * * + +## Verification + + + # On the host + docker system df # reclaimable should now be near zero + df -h /var/lib/docker # or df -h / if Docker is on root + + # In Datadog + # avg:system.disk.in_use{host:,device:} + # Should drop below warning threshold within 1-5 minutes + + +* * * + +## Edge Cases + + * **Overlay2 storage driver:** Disk usage reported by `du -sh /var/lib/docker` may differ from what `docker system df` reports because overlay2 uses hard links. Trust `docker system df` as the authoritative source. + + * **devicemapper storage driver (older setups):** The thin pool may report usage differently from the filesystem. Check `dmsetup status` as well. + + * **Docker running in Kubernetes (DinD):** The host may be a Kubernetes node; coordinate volume pruning with the cluster operator to avoid removing node-level cache needed by kubelet. + + diff --git a/scenarios/memory/memory-diagnosing-and-remediating-a-slow-application-leak.md b/scenarios/memory/memory-diagnosing-and-remediating-a-slow-application-leak.md new file mode 100644 index 00000000..fe711b47 --- /dev/null +++ b/scenarios/memory/memory-diagnosing-and-remediating-a-slow-application-leak.md @@ -0,0 +1,265 @@ +# Memory - Slow Application Leak + +**Signal:** `system.mem.used` rising steadily over hours or days; sawtooth pattern where the post-collection memory floor rises with each cycle +**IssueType:** `memory_usage` +**Metric (typical):** `system.mem.used`, `system.mem.pct_usable` + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`free`| investigation| `free -h` +`ps`| investigation| `ps aux --sort=-%mem \| head -20` +`top`| investigation| `top -o %MEM` · `top -b -n 1 -p ` +`/proc//status`| investigation| `cat /proc//status \| grep -E 'VmRSS\|VmSize\|VmSwap'` +`smaps_rollup`| investigation| `cat /proc//smaps_rollup` +`pmap`| investigation| `pmap -x \| sort -k3 -rn \| head -20` +`vmstat`| investigation| `vmstat -s` · `vmstat 2 10` +`ls /proc//fd`| investigation| `ls /proc//fd \| wc -l` +`dmesg`| investigation| `dmesg \| grep -i "oom\|killed process"` +`systemctl`| remediation| `systemctl restart ` + +* * * + +## What Happens + +A process allocates memory over its lifetime and fails to release it. At OS level this manifests as the process RSS (Resident Set Size) growing continuously regardless of application load. If the runtime uses a garbage collector, the metric shows a sawtooth pattern: the collector periodically reclaims reachable-but-unused memory, but leaked objects (still referenced by a long-lived structure) survive every collection cycle. The diagnostic signature is that the post-collection floor rises over time — each trough of the sawtooth is higher than the last. + +Common causes: + + * Objects held in a static map, registry, or event listener list that is never drained + + * A queue or buffer that grows on writes but is never consumed or trimmed + + * File descriptors, sockets, or database connections opened but never closed + + * Per-request allocations stored in a cache with no TTL or size limit + + * Native memory allocated outside the managed heap (JNI, FFI, direct byte buffers) that is not released + + + + +* * * + +## Detection + +The platform detects this via `system.mem.used` or `system.mem.pct_usable` breaching the monitor threshold. The defining characteristic is trend shape: a slow, continuous rise over hours or days, not a sudden spike. On hosts with GC-based runtimes the metric shows a rising sawtooth; on non-GC runtimes it trends smoothly upward. + +**Correlated signals to check:** + + * `system.mem.pct_usable` falling below 10% — at this point the kernel reclaims pages aggressively and application latency rises + + * `system.swap.used` rising — the process is being paged to disk, a late-stage indicator + + * Application error spans mentioning `OutOfMemoryError`, `cannot allocate memory`, or `malloc failed` + + * OOM kill events: `dmesg | grep -i "killed process"` shows which PID was terminated + + + + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Commands must run on the host +Read access to `/proc` filesystem| Required to inspect per-process memory maps +Ability to run `ps`, `top`, `dmesg`| Standard tools; available on all Linux distributions + +### Steps + + 1. **Confirm host-level memory pressure** + + + + + + free -h + # Check "available" column — below 500 MB on most hosts indicates pressure + + vmstat -s | grep -E "total memory|free memory|used memory" + + 2. **Identify the highest-RSS process** + + + + + + ps aux --sort=-%mem | head -20 + # Columns: %MEM (% of RAM), RSS (resident set in KB), VSZ (virtual) + # Focus on processes with large RSS that are long-running + + 3. **Track RSS growth over time** + + + + + + # Poll RSS every 30 seconds + PID= + while true; do + rss=$(awk '/VmRSS/{print $2}' /proc/$PID/status) + echo "$(date +%T) RSS=${rss} kB" + sleep 30 + done + # A leak shows consistent upward drift; a stable process fluctuates around a mean + + 4. **Inspect anonymous vs file-backed memory** + + + + + + # Summary breakdown + cat /proc//smaps_rollup + # Key fields: + # Anonymous: heap allocations not backed by a file — where leaks live + # Shared_Clean/Shared_Dirty: shared library pages (normally stable) + # Swap: how much of this process is paged out + + # Detailed mapping list — large anonymous regions that grow over time indicate heap leak + pmap -x | sort -k3 -rn | head -30 + + 5. **Check for file descriptor leaks** + + + + + + ls /proc//fd | wc -l + # Healthy long-running process: stable count + # Leaking process: count rises continuously + + cat /proc/sys/fs/file-nr + # Shows: allocated / unused / max system-wide fds + + 6. **Check for OOM kills** + + + + + + dmesg | grep -i "out of memory\|oom_kill\|killed process" + journalctl -k --since "6 hours ago" | grep -i "killed process\|oom" + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +Identify the leaking process before acting| Restarting the wrong service causes unnecessary disruption +Coordinate with the service owner if the process holds critical in-flight state| An uncoordinated restart drops in-flight requests +The underlying leak must be fixed separately| A restart recovers memory temporarily; the leak recurs on the same schedule + +### Immediate Memory Recovery + +**Restart the leaking service:** + + + systemctl restart + systemctl status # confirm it came up healthy + +**If a restart cannot happen immediately, steer the OOM killer away from critical services:** + + + # Make the leaking process the preferred OOM kill target + echo 500 > /proc//oom_score_adj + + # Protect a co-located critical process + echo -500 > /proc//oom_score_adj + # Range: -1000 (never kill) to 1000 (kill first) + +### Prevent Runaway Growth + +**Cap memory and enable auto-restart via systemd:** + + + # /etc/systemd/system/.service.d/memory-limit.conf + [Service] + MemoryMax=2G + MemorySwapMax=0 + Restart=on-failure + RestartSec=10s + +Apply with: + + + systemctl daemon-reload + systemctl restart + +With `MemoryMax` set, the kernel OOM-kills the process when the limit is breached and systemd restarts it automatically. This bounds the impact while the fix is developed. + +### Fix the Underlying Leak + +OS-level data narrows the search before handing off to application-level profiling: + +OS observation| Likely cause +---|--- +RSS grows at constant rate independent of traffic| Background thread or timer-triggered allocation +RSS grows proportionally to request rate| Per-request object not freed +FD count grows alongside RSS| Unclosed file descriptors or sockets +`smaps_rollup` Anonymous grows; file-backed stable| Heap leak (not a library or mmap issue) +`system.swap.used` rising| Leak has been ongoing long enough to exhaust RAM + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +Polling `/proc//status`, `ps`, `pmap`| None| Read-only inspection +Adjusting `oom_score_adj`| None| Changes OOM priority only; process keeps running +`systemctl restart `| **Brief service interruption**| In-flight requests are dropped; drain the load balancer first if possible +Adding `MemoryMax` \+ `systemctl daemon-reload`| None| `daemon-reload` does not restart services; limit applies on next start +Adding `MemoryMax` \+ `systemctl restart`| **Brief service interruption**| Same as restart above +**Kernel OOM kill** (uncontrolled)| **Abrupt termination**| No graceful shutdown; in-flight requests lost; on-disk state may be incomplete + +A coordinated `systemctl restart` behind a load balancer drain is always safer than waiting for a kernel OOM kill. + +* * * + +## Verification + + + # Confirm memory recovered + free -h # "available" should have increased + + # Confirm RSS is stable post-restart (not immediately climbing) + PID=$(pgrep -f ) + watch -n 10 "awk '/VmRSS/{print \$2}' /proc/$PID/status" + + # Confirm no new OOM events + dmesg | grep -i "oom" | tail -5 + +In Datadog, verify: + + * `system.mem.used` drops sharply after the restart and remains flat (no immediate re-climb) + + * `system.mem.pct_usable` recovers above the warning threshold + + * The sawtooth floor does not resume rising — if it does, the restart was mitigation only and the leak fix has not yet been deployed + + + + +* * * + +## Related Scenarios + + * If RSS climbs back to the threshold within hours, set `MemoryMax` and let the systemd restart loop contain the impact while the fix is developed. + + * If FD count is growing, treat as a descriptor leak first; both RSS and fd exhaustion will independently crash the process. + + * If the process's memory growth correlates with unique user or session counts rather than being constant over time, see the Unbounded Cache / Session Growth scenario. + + diff --git a/scenarios/memory/memory-long-running-database-transactions.md b/scenarios/memory/memory-long-running-database-transactions.md new file mode 100644 index 00000000..e161fa79 --- /dev/null +++ b/scenarios/memory/memory-long-running-database-transactions.md @@ -0,0 +1,326 @@ +# Memory - Long-Running Database Transaction + +**Signal:** `pg_stat_activity` showing transactions open for minutes or hours; WAL directory size growing; `system.disk.in_use` rising on the PostgreSQL data volume +**IssueType:** `memory_usage` +**Metric (typical):** `postgresql.queries.count`, `system.disk.in_use` (PostgreSQL data volume) + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`psql`| both| `SELECT ... FROM pg_stat_activity` · `SELECT pg_terminate_backend(pid)` · `SELECT pg_cancel_backend(pid)` +`du`| investigation| `du -sh $PGDATA/pg_wal/` +`ls`| investigation| `ls -lh $PGDATA/pg_wal/ \| wc -l` +`grep`| investigation| `grep -E "idle_in_transaction_session_timeout\|statement_timeout\|lock_timeout" /etc/postgresql/*/*/postgresql.conf` +`cat`| investigation| `cat /proc//status \| grep VmRSS` +`df`| investigation| `df -h $PGDATA` + +* * * + +## What Happens + +A PostgreSQL transaction that stays open for an extended period holds a snapshot of the database state from the moment it began. While the transaction is open: + + * **VACUUM is blocked** : PostgreSQL cannot reclaim dead tuples (rows deleted or updated after the transaction started) because the open transaction still needs to see the old versions. Dead tuples accumulate, causing table bloat — the physical table file grows even though the live row count is stable. + + * **WAL grows** : PostgreSQL must retain Write-Ahead Log segments that the open transaction may still need, preventing WAL archival or recycling. On a busy database, WAL can grow to tens of gigabytes within hours. + + * **Locks are held** : If the transaction acquired row or table locks, all other queries waiting on those locks are blocked. Blocked queries accumulate in `pg_stat_activity`, consuming connection slots and memory. + + * **Replication can lag** : On replicas, recovery is paused at the WAL position the primary is waiting to archive; replication lag grows. + + + + +Common causes: + + * An application opened a transaction and then stalled (waiting on an external API, a slow computation, or network I/O) without setting a statement or idle-in-transaction timeout + + * A client disconnected mid-transaction without rolling back — PostgreSQL holds the transaction open until it detects the TCP connection is gone (which can take minutes depending on TCP keepalive settings) + + * A batch migration or export job wrapped too many rows in a single transaction + + * An ORM framework that wraps every request in a transaction, combined with a long-running HTTP handler + + * A forgotten `BEGIN` in an interactive `psql` session left open on a developer's laptop + + + + +* * * + +## Detection + +The platform detects this via elevated `system.disk.in_use` on the PostgreSQL data volume (WAL growth) or via a PostgreSQL integration monitor on query age. The WAL growth signal is typically the first OS-level indicator; by the time it appears, the transaction has usually been open for tens of minutes or more. + +**Correlated signals to check:** + + * `system.disk.in_use` rising on the volume hosting `$PGDATA` — specifically the `pg_wal/` subdirectory + + * `postgresql.connections` at or near `max_connections` — blocked queries pile up consuming connection slots + + * Application error spans mentioning `could not obtain lock`, `deadlock detected`, or `connection pool exhausted` + + * Replication lag metric (`postgresql.replication.delay`) growing on standbys + + + + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Required to inspect WAL directory size and run OS-level commands +Read-only DB access (`pg_monitor` role or equivalent)| Sufficient for all investigation queries (`pg_stat_activity`, `pg_locks`, `SHOW ...`). Connect with: `psql -h -U -d postgres`. If no dedicated monitoring user exists, use `sudo -u postgres psql` on the host +Know the value of `$PGDATA`| Needed to find the WAL directory; default is `/var/lib/postgresql//main` on Debian/Ubuntu. Retrieve it with `SHOW data_directory;` if unknown + +### Steps + + 1. **Check WAL directory size** + + + + + + # Retrieve PGDATA if not already known + sudo -u postgres psql -c "SHOW data_directory;" + + # WAL size + du -sh $PGDATA/pg_wal/ + ls -lh $PGDATA/pg_wal/ | wc -l # number of WAL segment files; each is typically 16 MB + + # Overall data volume headroom + df -h $PGDATA + + 2. **Find long-running transactions** + + + + + + -- All transactions open longer than 5 minutes, oldest first + SELECT pid, + usename, + application_name, + client_addr, + state, + now() - xact_start AS transaction_age, + now() - query_start AS query_age, + left(query, 120) AS current_query + FROM pg_stat_activity + WHERE xact_start IS NOT NULL + AND now() - xact_start > interval '5 minutes' + ORDER BY transaction_age DESC; + + 3. **Check for idle-in-transaction sessions** + + + + + + -- Sessions in "idle in transaction" state (transaction open, no active query) + -- These are the most dangerous: they hold a snapshot indefinitely with no CPU cost + SELECT pid, usename, application_name, client_addr, + state, + now() - xact_start AS open_for, + now() - state_change AS idle_for + FROM pg_stat_activity + WHERE state = 'idle in transaction' + ORDER BY open_for DESC; + + 4. **Check what the long-running transaction is blocking** + + + + + + -- Queries waiting on locks held by another session + SELECT blocked.pid AS blocked_pid, + blocked.query AS blocked_query, + blocking.pid AS blocking_pid, + blocking.query AS blocking_query, + now() - blocked.query_start AS waiting_for + FROM pg_stat_activity AS blocked + JOIN pg_stat_activity AS blocking + ON blocking.pid = ANY(pg_blocking_pids(blocked.pid)) + WHERE cardinality(pg_blocking_pids(blocked.pid)) > 0 + ORDER BY waiting_for DESC; + + 5. **Identify the oldest transaction horizon (xmin)** + + + + + + -- The oldest transaction ID still active — determines how far back VACUUM must retain dead tuples + SELECT min(age(backend_xmin)) AS oldest_xmin_age, + min(age(xact_start::text::xid)) AS oldest_xact_age + FROM pg_stat_activity + WHERE backend_xmin IS NOT NULL; + + -- Table bloat indicator: tables with many dead tuples + SELECT relname, n_dead_tup, n_live_tup, + round(n_dead_tup::numeric / NULLIF(n_live_tup + n_dead_tup, 0) * 100, 1) AS dead_pct, + last_autovacuum + FROM pg_stat_user_tables + WHERE n_dead_tup > 10000 + ORDER BY n_dead_tup DESC + LIMIT 15; + + 6. **Check current timeout settings** + + + + + + SHOW statement_timeout; + SHOW idle_in_transaction_session_timeout; + SHOW lock_timeout; + -- An empty string or '0' means the timeout is disabled + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +Superuser or `pg_signal_backend` role for termination| `pg_cancel_backend` and `pg_terminate_backend` require superuser, ownership of the target session, or the `pg_signal_backend` role (PostgreSQL 14+). Both functions return `false` — with no error — if the caller lacks privileges, so verify your access before relying on them. Use `sudo -u postgres psql` if no elevated role is available +Superuser for timeout changes| `ALTER SYSTEM SET` and `ALTER ROLE ... SET` require superuser or `CREATEROLE`. Read-only monitoring users cannot make these changes +Confirm the transaction is not part of a critical in-flight operation| Terminating a transaction rolls it back entirely; confirm with the application owner if the transaction was performing a migration or bulk write +Use `pg_cancel_backend` before `pg_terminate_backend` for active queries| `pg_cancel_backend` sends SIGINT (interrupts the current query but leaves the connection); `pg_terminate_backend` sends SIGTERM (closes the connection and rolls back) — try the softer option first +Timeout changes apply to new sessions only| Changes via `ALTER SYSTEM` \+ `SELECT pg_reload_conf()` apply to future connections; existing long-running sessions are not immediately affected + +### Immediate Relief + +**Connect as a user with termination privileges:** + + + sudo -u postgres psql + -- or: psql -h -U -d postgres + +**Cancel the active query without closing the connection (softer; the client can retry):** + + + SELECT pg_cancel_backend(); + -- Returns true if the signal was sent, false if permission denied or pid not found + +**Terminate the session and roll back its transaction (closes the connection):** + + + SELECT pg_terminate_backend(); + +**Terminate all idle-in-transaction sessions older than 10 minutes:** + + + SELECT pg_terminate_backend(pid) + FROM pg_stat_activity + WHERE state = 'idle in transaction' + AND now() - xact_start > interval '10 minutes'; + +**After terminating the blocking transaction, run VACUUM on the most bloated tables:** + + + -- Online VACUUM (does not lock the table) + VACUUM ANALYZE ; + +### Set Timeouts to Prevent Recurrence + +**Apply globally via**`postgresql.conf` (requires superuser): + + + -- Terminate any query running longer than 30 minutes + ALTER SYSTEM SET statement_timeout = '30min'; + + -- Terminate connections idle in transaction for more than 5 minutes + ALTER SYSTEM SET idle_in_transaction_session_timeout = '5min'; + + -- Abort any statement waiting more than 30 seconds to acquire a lock + ALTER SYSTEM SET lock_timeout = '30s'; + + -- Reload config (no restart required) + SELECT pg_reload_conf(); + +**Apply to a specific role only (less disruptive rollout):** + + + ALTER ROLE SET idle_in_transaction_session_timeout = '5min'; + ALTER ROLE SET statement_timeout = '30min'; + +**Reclaim WAL space after the blocking transaction is gone:** + +WAL recycling happens automatically once the oldest active transaction advances. Confirm WAL is shrinking: + + + # Poll WAL directory size every 30 seconds + while true; do + echo "$(date +%T) WAL=$(du -sh $PGDATA/pg_wal/ | cut -f1)" + sleep 30 + done + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +Querying `pg_stat_activity`, `pg_locks`| None| Read-only; no locks taken +`pg_cancel_backend()`| **Current query fails with error**| The client receives `ERROR: canceling statement due to user request`; the connection stays open; the application can retry +`pg_terminate_backend()`| **Connection closed; transaction rolled back**| The client receives a connection reset; all work in the transaction is lost; the application must reconnect and retry +`VACUUM ANALYZE`| None| Online; table remains fully accessible +`ALTER SYSTEM SET ... + pg_reload_conf()`| None| Applies to new connections only; existing sessions are unaffected until they reconnect +`ALTER ROLE ... SET ...`| None| Same as above; takes effect on next login for that role + +`pg_terminate_backend` is the primary risk. Any uncommitted writes in the terminated transaction are lost. For a transaction that was performing a bulk migration or multi-row insert, this means the work must be redone. Always confirm the transaction's purpose before terminating. + +* * * + +## Verification + + + -- Confirm no long-running transactions remain + SELECT pid, state, now() - xact_start AS age + FROM pg_stat_activity + WHERE xact_start IS NOT NULL + AND now() - xact_start > interval '5 minutes'; + -- Should return 0 rows + + -- Confirm no idle-in-transaction sessions remain + SELECT count(*) FROM pg_stat_activity WHERE state = 'idle in transaction'; + + -- Confirm WAL is shrinking + -- du -sh $PGDATA/pg_wal/ (run twice 60 s apart; size should be stable or decreasing) + + -- Confirm timeouts are now set + SHOW idle_in_transaction_session_timeout; + SHOW statement_timeout; + +In Datadog, verify: + + * `system.disk.in_use` on the PostgreSQL data volume levels off or drops as WAL is recycled + + * `postgresql.connections` returns to normal levels as blocked queries are unblocked + + * Application error spans mentioning lock waits or pool exhaustion cease + + + + +* * * + +## Related Scenarios + + * If table bloat is severe after a long-running transaction, dead tuples do not disappear immediately after VACUUM — they are marked for reuse but the physical file does not shrink unless `VACUUM FULL` is run (which takes an exclusive lock); see the Database Growth scenario for guidance on bloat remediation. + + * If the WAL directory filled the disk completely before the transaction was terminated, PostgreSQL may have paused writes with `PANIC: could not write to file`; immediate space recovery (deleting other files on the volume) is required before the database can resume. + + * If `pg_stat_activity` shows many sessions in `idle in transaction` from the same `application_name`, the connection pool or ORM is not committing transactions promptly; the fix is application-side (`idle_in_transaction_session_timeout` provides a safety net but does not address the root cause). + + diff --git a/scenarios/memory/memory-unbounded-cache-and-session-growth.md b/scenarios/memory/memory-unbounded-cache-and-session-growth.md new file mode 100644 index 00000000..abc31dce --- /dev/null +++ b/scenarios/memory/memory-unbounded-cache-and-session-growth.md @@ -0,0 +1,253 @@ +# Memory - Unbounded Cache / Session Growth + +**Signal:** `system.mem.used` rising steadily, correlated with user/session count or uptime; no sawtooth; growth rate decelerates or plateaus when traffic drops +**IssueType:** `memory_usage` +**Metric (typical):** `system.mem.used`, `system.mem.pct_usable` + +* * * + +## Command Reference + +Tool| Stage| Examples +---|---|--- +`free`| investigation| `free -h` +`ps`| investigation| `ps aux --sort=-%mem \| head -20` +`/proc//status`| investigation| `cat /proc//status \| grep -E 'VmRSS\|VmSize'` +`smaps_rollup`| investigation| `cat /proc//smaps_rollup` +`ss`| investigation| `ss -tp \| grep ` · `ss -s` +`ls /proc//fd`| investigation| `ls /proc//fd \| wc -l` +`vmstat`| investigation| `vmstat 2 10` +`dmesg`| investigation| `dmesg \| grep -i "oom\|killed process"` +`systemctl`| remediation| `systemctl restart ` + +* * * + +## What Happens + +An in-process cache, session store, or lookup table grows without eviction. Unlike a slow leak — which is typically unintentional and grows at a constant rate — unbounded cache growth is often by design but misconfigured: a cache with no TTL or no maximum entry count, a session map that accumulates entries for every visitor but never expires them, a connection pool that grows to meet demand but never shrinks. + +At OS level the RSS of the process grows proportionally to the number of distinct cached keys or active sessions, not at a fixed rate over time. Growth slows when traffic drops and accelerates when traffic rises. The key distinguishing test vs a slow leak is whether growth rate tracks request rate or unique-object count. + +Common causes: + + * An application cache (in-memory hash map, LRU cache configured without a max size) accumulating entries without eviction + + * HTTP session storage in memory with no session timeout configured + + * A per-connection or per-request context object stored in a registry that is never deregistered after the connection closes + + * A metrics or telemetry buffer that accumulates data points faster than they are flushed + + * A pub/sub subscriber list that grows as new clients connect but never shrinks when they disconnect + + + + +* * * + +## Detection + +Detected via `system.mem.used` or `system.mem.pct_usable` breaching the monitor threshold. The signature differs from a slow leak in two ways: the growth rate tracks traffic or session count rather than being constant, and there is typically no sawtooth (no periodic reclaim, because the objects are all reachable and the GC cannot reclaim them). + +**Correlated signals to check:** + + * `system.mem.pct_usable` trending down over days rather than hours (unbounded caches often grow slower than active leaks under typical load) + + * Application request rate metrics: if memory growth decelerates when traffic drops, the growth is session- or request-driven + + * Open socket count from `ss -s`: a growing `ESTABLISHED` count that never decreases suggests session objects are accumulating alongside open connections + + * `system.mem.swap.used` rising as a late-stage indicator + + + + +* * * + +## Investigation + +### Preconditions + +Precondition| Rationale +---|--- +SSH or remote exec access to the host| Commands must run on the host +Read access to `/proc` filesystem| Required for per-process memory and fd inspection +Application metrics or logs accessible| Correlating memory growth with request rate requires application-level signal + +### Steps + + 1. **Confirm host-level memory pressure** + + + + + + free -h + vmstat -s | grep -E "total memory|free memory" + + 2. **Identify the growing process** + + + + + + ps aux --sort=-%mem | head -20 + # Note the process with highest RSS and longest uptime + + 3. **Track RSS and correlate with traffic** + + + + + + PID= + while true; do + rss=$(awk '/VmRSS/{print $2}' /proc/$PID/status) + echo "$(date +%T) RSS=${rss} kB" + sleep 60 + done + # If RSS growth accelerates during traffic spikes and slows overnight: session/cache-driven + # If growth is constant regardless of traffic: more likely a classic leak + + 4. **Check socket and connection count** + + + + + + # Count connections associated with the process + ss -tp | grep | wc -l + + # Overall socket summary + ss -s + # ESTABLISHED count: if this grows without bound, sessions are not being closed + + 5. **Check file descriptor count** + + + + + + ls /proc//fd | wc -l + # If this grows continuously alongside ESTABLISHED socket count, connection objects are retained + + 6. **Inspect anonymous memory breakdown** + + + + + + cat /proc//smaps_rollup + # Anonymous: should be the growing region for heap-held caches + # If file-backed memory is growing, check for mmap-backed stores (e.g., SQLite WAL, RocksDB) + + 7. **Check for OOM events** + + + + + + dmesg | grep -i "out of memory\|killed process" + +* * * + +## Remediation + +### Preconditions + +Precondition| Rationale +---|--- +Identify the specific process before acting| Restarting the wrong service has no effect +For session stores: confirm whether session data loss on restart is acceptable| Some services require session persistence; coordinate with the owning team +The underlying cache or session configuration must be fixed| A restart recovers memory but growth resumes immediately after + +### Immediate Memory Recovery + +**Restart the process to flush the accumulated cache or session state:** + + + systemctl restart + systemctl status + +Note: a restart flushes all in-memory sessions. For applications with user-facing session state (logged-in users, shopping carts), this causes a forced logout for all active users. Coordinate accordingly. + +### Prevent Runaway Growth + +**Set a memory limit and enable auto-restart:** + + + # /etc/systemd/system/.service.d/memory-limit.conf + [Service] + MemoryMax=4G + MemorySwapMax=0 + Restart=on-failure + RestartSec=10s + + + systemctl daemon-reload + systemctl restart + +### Fix the Underlying Configuration + +The permanent fix is application-side, but OS-level observations guide the right fix: + +OS observation| Application-side fix +---|--- +RSS growth tracks unique session count| Add session TTL and maximum concurrent session limit +RSS growth tracks request rate| Add a max-size eviction policy to the request-scoped cache +ESTABLISHED socket count growing| Fix connection lifecycle: ensure close/release is called on disconnect +FD count growing alongside RSS| Fix descriptor lifecycle: ensure file handles are closed after use +RSS grows but request rate is flat| Growth is likely tied to unique keys (e.g., per-user or per-IP caches); add TTL + +* * * + +## Service Impact During Remediation + +Action| Service disruption| Notes +---|---|--- +`ps`, `ss`, `/proc` inspection| None| Read-only +`systemctl restart `| **Brief service interruption + session loss**| In-flight requests dropped; all in-memory sessions lost; active users are logged out +Adding `MemoryMax` \+ `daemon-reload`| None| Limit applies on next start only +Adding `MemoryMax` \+ restart| **Brief service interruption + session loss**| Same as restart above +**Kernel OOM kill**| **Abrupt termination**| No graceful shutdown +Fixing cache TTL / max-size in config| None at config time| Requires a service restart to take effect + +**Session loss is the primary coordination concern** for user-facing services. For internal or stateless services a restart is low-risk. + +* * * + +## Verification + + + # Confirm memory recovered after restart + free -h + + # Confirm RSS is no longer growing proportionally to traffic + PID=$(pgrep -f ) + watch -n 30 "awk '/VmRSS/{print \$2}' /proc/$PID/status" + + # Confirm socket count is stable + watch -n 10 "ss -tp | grep | wc -l" + +In Datadog, verify: + + * `system.mem.used` drops after the restart + + * Memory growth rate is flat or logarithmic (cache filling to steady state) rather than linear + + * If a cache TTL was added: memory should plateau at a stable level proportional to active sessions within the TTL window + + + + +* * * + +## Related Scenarios + + * If growth rate is constant regardless of traffic, the issue is more likely a slow leak; see the Slow Application Leak scenario. + + * If memory recovers after a restart but re-fills within minutes, the cache fill rate is very high; a size cap is the more effective near-term control than a TTL. + + * If socket count is growing but RSS is stable, connections may be leaking at the OS level without retaining application-level session data; focus investigation on the connection lifecycle rather than the cache. + + diff --git a/scenarios/remediation-scenarios-overview-and-command-reference.md b/scenarios/remediation-scenarios-overview-and-command-reference.md new file mode 100644 index 00000000..65266091 --- /dev/null +++ b/scenarios/remediation-scenarios-overview-and-command-reference.md @@ -0,0 +1,167 @@ +# Remediation Scenarios: Overview and Command Reference + +Each scenario in this folder describes a specific host-level issue: its signal shape, how to investigate it at OS level, how to remediate it, and what service disruption to expect. Scenarios are grouped into three families based on the resource they affect and the IssueType value that the platform attaches to the alert. + +Reference: [Q3 prioritized host issues]() + +Issue Family / Type| Primary signal| Example scenarios +---|---|--- +[**Disk**]()| `system.disk.in_use` high| Core Dumps, Core Dump Flood from Crash Loop, Docker Container Layers, Docker Image/Build Cache Bloat, Inode Exhaustion, Orphaned Postgres Replication Slot, Temp Files & Build Artifacts, Database Growth, Unrotated Logs +[**Memory**]()| `system.mem.used` rising| Slow Application Leak, Unbounded Cache / Session Growth, Long-Running DB Transaction +[**CPU**]()| `system.cpu.user` high| Cron Storm, Runaway Process / Infinite Loop, GC Pressure + +The signal shape is often the fastest way to route to the right scenario before reading further: + + * **Sudden spike, then stable** -> Core Dumps (single crash, disk) + + * **Rapid staircase rise + simultaneous service monitor ALERT or NO_DATA** -> Core Dump Flood from Crash Loop (disk) + + * **ENOSPC errors but**`df -h` shows free space -> Inode Exhaustion (disk) + + * **Periodic spike** -> Cron Storm (CPU) + + * **Steady rise correlated with traffic / business hours** -> Unrotated Logs (disk) + + * **Slow, steady rise** -> Database Growth (disk), Unbounded Cache (memory) + + * **Slow, steady rise on database volume;**`pg_wal/` large but table sizes stable -> Orphaned Postgres Replication Slot (disk) + + * **Rising sawtooth** -> Slow Application Leak (memory), GC Pressure (CPU) + + * **Slow, steady rise on Docker partition, proportional to deploy/build frequency** -> Docker Image/Build Cache Bloat (disk) + + * **Episodic or sudden rise on Docker partition after container lifecycle events** -> Docker Container Layers (disk) + + * **Continuous high CPU, single PID** -> Runaway Process / Infinite Loop (CPU) + + * **Disk rising on database volume + long**`pg_stat_activity` transaction age -> Long-Running DB Transaction (memory) + + + + +* * * + +## Disk issues -- Command Reference + +Covers: Core Dumps * Core Dump Flood from Crash Loop * Docker Container Layers * Docker Image/Build Cache Bloat * Inode Exhaustion * Orphaned Postgres Replication Slot * Temp Files & Build Artifacts * Database Growth * Unrotated Logs + + _**Privilege key:** _ + + * ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png) _Root / sudo = always requires elevated privilege_ + + * ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) _Context-dependent = depends on target files/processes or docker group membership_ + + * ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) _None = safe to run as a service account_ + + + + +Tool| Scenarios| Stage| Privilege Required +---|---|---|--- +`cat`| Core Dumps * Core Dump Flood * Logs| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`coredumpctl`| Core Dumps * Core Dump Flood| both| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (view: none; delete/manage: root) +`df`| Core Dump Flood * Docker Image/Build Cache * Inode Exhaustion * Logs * Replication Slot| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`dmesg`| Core Dumps| investigation| ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png) (CAP_SYS_ADMIN on Linux 5.x+) +`docker buildx`| Docker Container Layers * Docker Image/Build Cache| both| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png)Context-dependent (docker group required; equivalent to root) +`docker images`| Docker Container Layers * Docker Image/Build Cache| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png)Context-dependent (docker group required; equivalent to root) +`docker ps`| Docker Container Layers * Docker Image/Build Cache| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png)Context-dependent (docker group required; equivalent to root) +`docker system`| Docker Container Layers * Docker Image/Build Cache| both| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png)Context-dependent (docker group required; equivalent to root) +`docker volume`| Docker Container Layers| both| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png)Context-dependent (docker group required; equivalent to root) +`du`| Core Dumps * Core Dump Flood * Docker Container Layers * Docker Image/Build Cache * Inode Exhaustion * Replication Slot * Temp Files * Database * Logs| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png)Context-dependent (root-owned directories require sudo) +`find`| Core Dumps * Core Dump Flood * Inode Exhaustion * Temp Files * Logs| both| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png)Context-dependent (root-owned directories require sudo) +`go`| Temp Files| remediation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`gradle`| Temp Files| remediation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`grep`| Database| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`journalctl`| Core Dumps * Core Dump Flood * Database * Logs| both| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png)Context-dependent (system journal requires root; user journal does not) +`logrotate`| Logs| both| ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png) (system log files are root-owned) +`ls`| Core Dumps * Core Dump Flood * Inode Exhaustion * Replication Slot * Logs| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`lsof`| Temp Files * Logs| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png)Context-dependent (full cross-process visibility requires root) +`mysql`| Database| both| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png)Context-dependent (DB admin credentials required for some operations) +`npm`| Temp Files| both| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`pip`| Temp Files| both| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`psql`| Database * Replication Slot| both| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png)Context-dependent (DB superuser required for DROP REPLICATION SLOT and pg_terminate_backend) +`rm`| Core Dumps * Core Dump Flood * Inode Exhaustion * Temp Files * Logs| remediation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (root required for system-owned files) +`stat`| Inode Exhaustion| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`systemctl`| Core Dumps * Core Dump Flood * Database * Docker Image/Build Cache * Logs| both| ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png) (service start/stop/restart requires root) +`systemd-tmpfiles`| Inode Exhaustion * Temp Files| remediation| ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png) +`sysctl`| Core Dump Flood| remediation| ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png)(kernel parameter writes require root) +`truncate`| Logs| remediation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (root required for system-owned log files) +`tune2fs`| Inode Exhaustion| investigation| ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png)(filesystem-level changes require root) + +* * * + +## Memory issues -- Command Reference + +Covers: Slow Application Leak * Unbounded Cache / Session Growth * Long-Running DB Transaction + + _**Privilege key:** _ + + * __![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png) _Root / sudo = always requires elevated privilege_ + + * ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) _Context-dependent = depends on target files/processes or docker group membership_ + + * ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) _None = safe to run as a service account_ + + + + +Tool| Scenarios| Stage| Privilege Required +---|---|---|--- +`/proc//smaps_rollup`| Slow Leak * Unbounded Cache| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (root required for processes owned by other users) +`/proc//status`| Slow Leak * Unbounded Cache * DB Transaction| investigation| Context-dependent (root required for processes owned by other users) +`cat`| DB Transaction| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`df`| DB Transaction| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`dmesg`| Slow Leak * Unbounded Cache| investigation| ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png) (CAP_SYS_ADMIN on Linux 5.x+) +`du`| DB Transaction| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (root-owned directories require sudo) +`free`| Slow Leak * Unbounded Cache| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`grep`| DB Transaction| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`ls`| DB Transaction| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`ls /proc//fd`| Slow Leak * Unbounded Cache| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (root required for processes owned by other users) +`pmap`| Slow Leak| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (root required for processes owned by other users) +`ps`| Slow Leak * Unbounded Cache| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`psql`| DB Transaction| both| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (DB superuser required for pg_terminate_backend) +`ss`| Unbounded Cache| investigation| None +`systemctl`| Slow Leak * Unbounded Cache| remediation| ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png) (service restart requires root) +`top`| Slow Leak| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`vmstat`| Slow Leak * Unbounded Cache| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) + +* * * + +## CPU issues -- Command Reference + +Covers: Cron Storm * Runaway Process / Infinite Loop * GC Pressure + + _**Privilege key:** _ + + * __![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png) _Root / sudo = always requires elevated privilege_ + + * ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) _Context-dependent = depends on target files/processes or docker group membership_ + + * ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) _None = safe to run as a service account_ + + + + +Tool| Scenarios| Stage| Privilege Required +---|---|---|--- +`/proc//stat`| Runaway Loop * GC Pressure| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (root required for processes owned by other users) +`/proc//status`| GC Pressure| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (root required for processes owned by other users) +`/proc//wchan`| Runaway Loop| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (root required for processes owned by other users) +`crontab`| Cron Storm| investigation| None (own crontab); root required to inspect other users' crontabs +`dmesg`| GC Pressure| investigation| ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png)(CAP_SYS_ADMIN on Linux 5.x+) +`find`| GC Pressure| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (root-owned directories require sudo) +`flock`| Cron Storm| remediation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`grep`| Cron Storm * GC Pressure| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`journalctl`| Cron Storm| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (system journal requires root; user journal does not) +`kill`| Cron Storm * Runaway Loop| remediation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (root required to signal another user's process) +`nice`| Cron Storm| remediation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) Context-dependent (negative nice values / priority increase require root) +`perf`| Runaway Loop| investigation| ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png) (requires CAP_SYS_ADMIN or CAP_PERFMON; often blocked in containers) +`pgrep`| Cron Storm| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`ps`| Cron Storm * Runaway Loop * GC Pressure| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`strace`| Runaway Loop| investigation| ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png) (requires CAP_SYS_PTRACE to attach to another process) +`systemctl`| Runaway Loop * GC Pressure| remediation| ![content.emoticon.crown](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/72/1f451.png) (service restart requires root) +`top`| Cron Storm * Runaway Loop * GC Pressure| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`uptime`| Cron Storm * Runaway Loop| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) +`vmstat`| Cron Storm * Runaway Loop * GC Pressure| investigation| ![\(blue star\)](https://datadoghq.atlassian.net/wiki/s/-436536684/6452/534630fb30addd8e6f6fdb1d30a20ca17b4fd5f6/_/images/icons/emoticons/star_blue.png) + +