atomk-hermes-skills/skills/devops/oom-diagnosis/SKILL.md

name, description, category, triggers

name	description	category	triggers
oom-diagnosis	Diagnose and fix OOM (Out of Memory) kills on Linux servers — memory profiling, identifying memory hogs, adjusting OOM scores, and reducing memory pressure.	devops	Process keeps getting killed / disappearing User reports OOM, out of memory, or process crashes under load free -h shows low available memory or heavy swap usage dmesg/journalctl shows oom-killer invocations

OOM Diagnosis & Memory Optimization

Systematic diagnosis and remediation when processes get killed by the Linux OOM killer.

Step 1: Assess current state

free -h                          # overall memory picture
ps aux --sort=-%mem | head -20   # top consumers by RSS

Step 2: Find OOM killer evidence

# Recent OOM events (most reliable sources first)
grep -i "oom-kill\|Killed process\|out of memory" /var/log/kern.log.1 | tail -20
zgrep -i "oom-kill\|Killed process" /var/log/kern.log.*.gz | tail -20
sudo dmesg | grep -i "oom\|killed process"
journalctl -k --no-pager | grep -i "oom\|killed"

The kernel log table with rss and oom_score_adj columns reveals which process was killed and why.

Step 3: Check OOM scores

# For each suspect process
cat /proc/<PID>/oom_score        # higher = more likely to be killed
cat /proc/<PID>/oom_score_adj    # adjustment (0=default, 200+ means target)

A process with oom_score_adj ≥ 200 is deliberately marked as a preferred victim. Common sources:

• Chromium/Playwright self-adjusts to 300 (wants to be killed to avoid data loss)
• Some daemons set it explicitly
• Hermes gateway may set it for self-protection in very-low-memory environments

Step 4: Identify common memory hogs

On a typical 2GB VPS running Hermes + web + automation:

Process	Typical RSS	Fix
warp-svc (Cloudflare)	200-300MB	sudo systemctl stop warp-svc && sudo systemctl disable warp-svc if not needed
Apache prefork workers	40-60MB each	Reduce workers in mpm_prefork.conf
Chromium/Playwright leftovers	80-150MB	Kill stale instances after use
Hermes LSP (TypeScript)	350-650MB	hermes config set lsp.enabled false + kill the spawned node/tsserver processes
Hermes gateway	120-240MB	Accept; it's load-bearing
Hermes session	150-200MB+	Accept; normal per-session cost

Step 5: Fix Apache workers (Docker)

When Apache runs inside a Docker container:

# Find and read current config
sudo docker exec <container> cat /etc/apache2/mods-enabled/mpm_prefork.conf

# Adjust: StartServers→2, MinSpareServers→2, MaxSpareServers→4, MaxRequestWorkers→10
sudo docker exec <container> bash -c "sed -i \
  -e 's/StartServers.*[0-9]/StartServers             2/' \
  -e 's/MinSpareServers.*[0-9]/MinSpareServers          2/' \
  -e 's/MaxSpareServers.*[0-9]/MaxSpareServers          4/' \
  -e 's/MaxRequestWorkers.*[0-9]/MaxRequestWorkers        10/' \
  /etc/apache2/mods-enabled/mpm_prefork.conf"

# Reload
sudo docker exec <container> apache2ctl graceful

Step 6: Adjust OOM score (last resort)

# Remove a process from the OOM victim list
echo 0 | sudo tee /proc/<PID>/oom_score_adj

# Protect a critical process entirely
echo -1000 | sudo tee /proc/<PID>/oom_score_adj

WARNING: Lowering OOM score just shifts the burden. If memory runs out, a different process dies. Only do this combined with actual memory reduction.

Step 7: Verify

free -h
ps aux --sort=-%mem | head -15

Pitfalls

• dmesg requires root on many systems; fall back to kernel logs
• OOM events may be rotated out quickly on busy systems
• Killing a process doesn't free its swapped pages immediately
• Apache in Docker has no systemctl — use docker exec <container> apache2ctl graceful
• oom_score_adj=200 on Hermes gateway is sometimes intentional self-protection
• CRITICAL: oom_score_adj inherits to child processes. If the gateway has adj=200, every spawned child (LSP, tsserver, typingsInstaller) inherits it. This can mark 600MB+ of TypeScript tooling as preferred OOM victims alongside the gateway itself. Check for pid in $(pgrep -P <gateway_pid>); do cat /proc/$pid/oom_score_adj; done — if children inherit 200, lower the gateway's adj first: echo 0 | sudo tee /proc/<gateway_pid>/oom_score_adj, then kill and restart the children.

Post-fix

After resolving the immediate OOM, check if the machine needs an upgrade. 2GB is marginal for Hermes + browser automation + web server + database. 4GB is the practical minimum for a multi-service setup.

Quick wins (biggest impact first)

1. Disable LSP if unused: hermes config set lsp.enabled false then kill spawned tsserver processes. Can free 350-650MB instantly.
2. Lower gateway oom_score_adj: echo 0 | sudo tee /proc/<gateway_pid>/oom_score_adj — prevents children inheriting the 200 target score.
3. Reduce Apache workers (see Step 5).
4. Stop warp-svc if Cloudflare WARP is not needed.