Optimizing and monitoring Linux performance is crucial for system stability.
This guide covers performance monitoring, debugging tools, and system administration techniques.
Includes Brendan Gregg’s insights and basic BPFtrace examples.
uptime # Show load averages
top # Interactive process monitor
htop # Enhanced version of top
mpstat -P ALL 1 # Per-CPU utilization every second
🔹 Load Average shows 1 min, 5 min, and 15 min averages.
ps aux | grep nginx # Find process by name
pidstat -u 1 # Per-process CPU usage
kill -9 <PID> # Force kill a process
🔹 Use pidstat to track CPU usage per process in real-time.
perf & Flamegraph📌 Flamegraphs visualize CPU usage hotspots.
git clone https://github.com/brendangregg/FlameGraph.git ~/FlameGraph
cd ~/FlameGraph
chmod +x flamegraph.pl stackcollapse-perf.pl
--call-graph Optionsfp (Frame Pointer) - Uses stack frame pointers for backtrace (default, may miss some functions due to compiler optimizations).dwarf - Uses DWARF debugging info for accurate stack traces but has higher overhead.lbr (Last Branch Record) - Uses CPU hardware LBR for low-overhead call tracing (only on supported CPUs).❯ sudo perf record --call-graph lbr -p $(pgrep asd)
Error:
cpu-clock:pppH: PMU Hardware doesn't support sampling/overflow-interrupts. Try 'perf stat'
If LBR is unsupported, fallback to DWARF for stack tracing:
sudo perf record --call-graph dwarf -p $(pgrep asd)
sudo perf script | ~/FlameGraph/stackcollapse-perf.pl | ~/FlameGraph/flamegraph.pl > perf.svg
🔹 Why? Some CPUs disable LBR in virtualized environments or lack the required PMU (Performance Monitoring Unit). Using DWARF ensures accurate stack traces at the cost of slightly higher overhead.
free -m # Show memory usage in MB
vmstat 1 # Show memory, CPU, I/O usage every second
🔹 Look for high swap usage; it indicates memory pressure.
iostat -x 1 # Per-disk I/O stats
ioping -c 10 /dev/sda # Measure disk latency
🔹 High await in iostat indicates disk bottlenecks.
iftop # Real-time bandwidth usage
ss -tulpn # Show listening sockets and PIDs
🔹 Use ss instead of netstat for faster output.
ethtool -S eth0 # NIC statistics
ping -c 10 8.8.8.8 # Check latency
🔹 Check rx_missed_errors for NIC buffer overruns.
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s opened %s\n", comm, str(args->filename)); }'
🔹 Traces all file opens in real-time.
bpftrace -e 'tracepoint:sched:sched_switch { @switches[comm] = count(); }'
🔹 Identifies processes with high context switches.
bpftrace -e 'tracepoint:tcp:tcp_connect { printf("PID %d connecting to %s\n", pid, str(args->daddr)); }'
🔹 Shows TCP connection attempts in real-time.
echo 10 > /proc/sys/vm/swappiness
🔹 Lower values prioritize RAM over swap usage.
echo always > /sys/kernel/mm/transparent_hugepage/enabled
🔹 Improves performance for large memory workloads.
bpftrace for low-overhead tracing.perf and flamegraphs to visualize CPU performance.🚀 Happy Performance Tuning! 🔥