Add devops/oom-diagnosis
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
---
|
||||
name: oom-diagnosis
|
||||
description: Diagnose and fix OOM (Out of Memory) kills on Linux servers — memory profiling, identifying memory hogs, adjusting OOM scores, and reducing memory pressure.
|
||||
category: devops
|
||||
triggers:
|
||||
- Process keeps getting killed / disappearing
|
||||
- User reports OOM, out of memory, or process crashes under load
|
||||
- free -h shows low available memory or heavy swap usage
|
||||
- dmesg/journalctl shows oom-killer invocations
|
||||
---
|
||||
|
||||
# OOM Diagnosis & Memory Optimization
|
||||
|
||||
Systematic diagnosis and remediation when processes get killed by the Linux OOM killer.
|
||||
|
||||
## Step 1: Assess current state
|
||||
|
||||
```bash
|
||||
free -h # overall memory picture
|
||||
ps aux --sort=-%mem | head -20 # top consumers by RSS
|
||||
```
|
||||
|
||||
## Step 2: Find OOM killer evidence
|
||||
|
||||
```bash
|
||||
# Recent OOM events (most reliable sources first)
|
||||
grep -i "oom-kill\|Killed process\|out of memory" /var/log/kern.log.1 | tail -20
|
||||
zgrep -i "oom-kill\|Killed process" /var/log/kern.log.*.gz | tail -20
|
||||
sudo dmesg | grep -i "oom\|killed process"
|
||||
journalctl -k --no-pager | grep -i "oom\|killed"
|
||||
```
|
||||
|
||||
The kernel log table with `rss` and `oom_score_adj` columns reveals which process was killed and why.
|
||||
|
||||
## Step 3: Check OOM scores
|
||||
|
||||
```bash
|
||||
# For each suspect process
|
||||
cat /proc/<PID>/oom_score # higher = more likely to be killed
|
||||
cat /proc/<PID>/oom_score_adj # adjustment (0=default, 200+ means target)
|
||||
```
|
||||
|
||||
A process with `oom_score_adj` ≥ 200 is deliberately marked as a preferred victim. Common sources:
|
||||
- Chromium/Playwright self-adjusts to 300 (wants to be killed to avoid data loss)
|
||||
- Some daemons set it explicitly
|
||||
- Hermes gateway may set it for self-protection in very-low-memory environments
|
||||
|
||||
## Step 4: Identify common memory hogs
|
||||
|
||||
On a typical 2GB VPS running Hermes + web + automation:
|
||||
|
||||
| Process | Typical RSS | Fix |
|
||||
|---------|------------|-----|
|
||||
| warp-svc (Cloudflare) | 200-300MB | `sudo systemctl stop warp-svc && sudo systemctl disable warp-svc` if not needed |
|
||||
| Apache prefork workers | 40-60MB each | Reduce workers in mpm_prefork.conf |
|
||||
| Chromium/Playwright leftovers | 80-150MB | Kill stale instances after use |
|
||||
| Hermes LSP (TypeScript) | 350-650MB | `hermes config set lsp.enabled false` + kill the spawned node/tsserver processes |
|
||||
| Hermes gateway | 120-240MB | Accept; it's load-bearing |
|
||||
| Hermes session | 150-200MB+ | Accept; normal per-session cost |
|
||||
|
||||
## Step 5: Fix Apache workers (Docker)
|
||||
|
||||
When Apache runs inside a Docker container:
|
||||
|
||||
```bash
|
||||
# Find and read current config
|
||||
sudo docker exec <container> cat /etc/apache2/mods-enabled/mpm_prefork.conf
|
||||
|
||||
# Adjust: StartServers→2, MinSpareServers→2, MaxSpareServers→4, MaxRequestWorkers→10
|
||||
sudo docker exec <container> bash -c "sed -i \
|
||||
-e 's/StartServers.*[0-9]/StartServers 2/' \
|
||||
-e 's/MinSpareServers.*[0-9]/MinSpareServers 2/' \
|
||||
-e 's/MaxSpareServers.*[0-9]/MaxSpareServers 4/' \
|
||||
-e 's/MaxRequestWorkers.*[0-9]/MaxRequestWorkers 10/' \
|
||||
/etc/apache2/mods-enabled/mpm_prefork.conf"
|
||||
|
||||
# Reload
|
||||
sudo docker exec <container> apache2ctl graceful
|
||||
```
|
||||
|
||||
## Step 6: Adjust OOM score (last resort)
|
||||
|
||||
```bash
|
||||
# Remove a process from the OOM victim list
|
||||
echo 0 | sudo tee /proc/<PID>/oom_score_adj
|
||||
|
||||
# Protect a critical process entirely
|
||||
echo -1000 | sudo tee /proc/<PID>/oom_score_adj
|
||||
```
|
||||
|
||||
WARNING: Lowering OOM score just shifts the burden. If memory runs out, a *different* process dies. Only do this combined with actual memory reduction.
|
||||
|
||||
## Step 7: Verify
|
||||
|
||||
```bash
|
||||
free -h
|
||||
ps aux --sort=-%mem | head -15
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- `dmesg` requires root on many systems; fall back to kernel logs
|
||||
- OOM events may be rotated out quickly on busy systems
|
||||
- Killing a process doesn't free its swapped pages immediately
|
||||
- Apache in Docker has no `systemctl` — use `docker exec <container> apache2ctl graceful`
|
||||
- `oom_score_adj=200` on Hermes gateway is sometimes intentional self-protection
|
||||
- **CRITICAL: oom_score_adj inherits to child processes.** If the gateway has adj=200, every spawned child (LSP, tsserver, typingsInstaller) inherits it. This can mark 600MB+ of TypeScript tooling as preferred OOM victims alongside the gateway itself. Check `for pid in $(pgrep -P <gateway_pid>); do cat /proc/$pid/oom_score_adj; done` — if children inherit 200, lower the gateway's adj first: `echo 0 | sudo tee /proc/<gateway_pid>/oom_score_adj`, then kill and restart the children.
|
||||
|
||||
## Post-fix
|
||||
|
||||
After resolving the immediate OOM, check if the machine needs an upgrade. 2GB is marginal for Hermes + browser automation + web server + database. 4GB is the practical minimum for a multi-service setup.
|
||||
|
||||
### Quick wins (biggest impact first)
|
||||
|
||||
1. **Disable LSP if unused:** `hermes config set lsp.enabled false` then kill spawned tsserver processes. Can free 350-650MB instantly.
|
||||
2. **Lower gateway oom_score_adj:** `echo 0 | sudo tee /proc/<gateway_pid>/oom_score_adj` — prevents children inheriting the 200 target score.
|
||||
3. **Reduce Apache workers** (see Step 5).
|
||||
4. **Stop warp-svc** if Cloudflare WARP is not needed.
|
||||
Reference in New Issue
Block a user