Reading /proc for fun and profit
/proc is the part of Linux I keep going back to. Every kernel
release adds another file and almost nothing gets removed. A few
files I open regularly:
/proc/<pid>/status — the human-readable summary. VmRSS is
your actual memory usage, VmSwap is what got paged out (if you
have swap), and voluntary_ctxt_switches vs nonvoluntary_ctxt_switches
tells you whether the process is yielding or being preempted.
/proc/<pid>/io — read/write bytes broken down. read_bytes and
write_bytes are the ones that hit the block device. rchar and
wchar include reads served from page cache, which is usually less
interesting unless you’re looking for “why is this process
generating so much page-cache churn?”
/proc/<pid>/maps — every memory mapping. Combine with
/proc/<pid>/smaps (more detail, slower) when you need
per-segment RSS.
/proc/diskstats — columns 4 (sectors_read), 8 (sectors_written),
and 11 (time_in_progress_ms) at a 5-second interval give you a
reasonable proxy for “is this disk saturated.” iostat -x 5 is
mostly a presentation layer over this file.
/proc/net/sockstat — TCP/UDP/raw socket counts. If inuse
keeps growing while alloc keeps growing too, you’ve got a
socket leak somewhere.
Trick I use: watch -n 1 cat /proc/<pid>/status next to htop
gives you a clearer picture than either one alone.