Skip to main content

Debugging

System Call Tracing​

strace - Trace System Calls​

# Basic usage
strace command          # Trace system calls of command
strace -p PID          # Trace running process
strace -c command      # Count system calls
strace -e trace=open command  # Trace specific system calls

# Advanced options
strace -f command      # Follow child processes
strace -o output.txt command  # Save output to file
strace -e trace=network command  # Trace network calls
strace -e trace=file command     # Trace file operations
strace -r command      # Show relative timestamps
strace -t command      # Show absolute timestamps

ltrace - Trace Library Calls​

ltrace command         # Trace library calls
ltrace -p PID          # Trace running process
ltrace -c command      # Count library calls
ltrace -e malloc command  # Trace specific library calls
ltrace -f command      # Follow child processes

Process Debugging​

GDB - GNU Debugger​

# Basic GDB usage
gdb program            # Start GDB with program
gdb program core       # Debug with core dump
gdb -p PID            # Attach to running process

# GDB commands
(gdb) run             # Start program
(gdb) break main      # Set breakpoint at main
(gdb) continue        # Continue execution
(gdb) next            # Next line
(gdb) step            # Step into function
(gdb) backtrace       # Show call stack
(gdb) info registers  # Show register values
(gdb) print variable  # Print variable value
(gdb) quit            # Exit GDB

Process Analysis​

# Process information
ps aux                 # All processes
ps -elf               # Long format
ps -eH                # Process hierarchy
pstree                # Process tree
pgrep process_name    # Find process by name

# Process details
cat /proc/PID/status  # Process status
cat /proc/PID/maps    # Memory mappings
cat /proc/PID/fd/     # File descriptors
lsof -p PID           # Open files by process

Core Dumps​

# Enable core dumps
ulimit -c unlimited   # Set core dump size
echo "core" > /proc/sys/kernel/core_pattern

# Analyze core dump
gdb program core      # Debug with core dump
file core             # Core dump information

Memory Debugging​

Valgrind - Memory Error Detection​

# Memory leak detection
valgrind --leak-check=full ./program
valgrind --leak-check=full --show-leak-kinds=all ./program

# Memory error detection
valgrind --tool=memcheck ./program
valgrind --track-origins=yes ./program

# Performance profiling
valgrind --tool=callgrind ./program
valgrind --tool=cachegrind ./program

Memory Analysis Tools​

# Memory usage
pmap PID              # Process memory map
cat /proc/PID/smaps   # Detailed memory info
cat /proc/PID/status | grep Vm  # Virtual memory info

# Memory debugging
mtrace                # Memory tracing (glibc)
dmalloc               # Debug malloc library

File System Debugging​

File System Analysis​

# Check file system
fsck /dev/sda1        # Check filesystem
fsck -v /dev/sda1     # Verbose check
e2fsck -f /dev/sda1   # Force check ext2/3/4

# File system debugging
debugfs /dev/sda1     # Debug ext2/3/4 filesystem
tune2fs -l /dev/sda1  # Filesystem parameters
dumpe2fs /dev/sda1    # Dump filesystem info

File Operations Debugging​

# Monitor file operations
inotifywait -m /path  # Monitor file changes
inotifywatch /path    # Watch file statistics

# File locking
fuser /path/file      # Show processes using file
lsof /path/file       # Show open file descriptors

Network Debugging​

Network Analysis​

# Network monitoring
tcpdump -i eth0       # Packet capture
tcpdump -i eth0 port 80  # Capture specific port
wireshark             # GUI network analyzer
tshark                # Command-line Wireshark

# Network connections
netstat -tulpn        # All connections with PIDs
ss -tulpn             # Modern replacement
lsof -i               # Network connections by process

Network Troubleshooting​

# DNS debugging
dig +trace domain.com # Trace DNS resolution
nslookup domain.com   # DNS lookup
host domain.com       # Simple DNS lookup

# Network connectivity
ping -c 4 host        # Test connectivity
traceroute host       # Trace network path
mtr host             # Real-time network diagnostics
telnet host port     # Test port connectivity

Performance Debugging​

CPU Performance​

# CPU profiling
perf record ./program # Record performance data
perf report          # Analyze performance data
perf top            # Real-time performance monitoring
perf stat ./program  # Performance statistics

# CPU usage analysis
top -p PID          # Monitor specific process
htop                # Enhanced process viewer

I/O Performance​

# I/O monitoring
iostat -x 1         # Extended I/O statistics
iotop              # I/O usage by process
sar -d 1 5         # Disk activity

# Block device debugging
blktrace /dev/sda   # Block layer tracing
btrace /dev/sda     # Block trace analysis

Kernel Debugging​

Kernel Messages​

# Kernel ring buffer
dmesg              # Kernel messages
dmesg -w           # Watch kernel messages
dmesg -T           # Human-readable timestamps
dmesg -l err       # Show only error messages

# Kernel logs
journalctl -k      # Kernel messages via systemd
cat /var/log/kern.log  # Kernel log file

Kernel Modules​

# Module information
lsmod              # List loaded modules
modinfo module_name  # Module information
modprobe module_name # Load module
rmmod module_name   # Remove module

# Module debugging
modprobe -v module_name  # Verbose module loading
dmesg | grep module_name # Module messages

System Calls and Kernel Tracing​

# Kernel tracing
trace-cmd record -e syscalls ./program  # Record system calls
trace-cmd report    # Analyze trace data

# Ftrace
echo function > /sys/kernel/debug/tracing/current_tracer
cat /sys/kernel/debug/tracing/trace

Application Debugging​

Dynamic Analysis​

# Runtime analysis
objdump -d program  # Disassemble program
readelf -a program  # ELF file analysis
strings program     # Extract strings from binary
nm program         # List symbols

# Library dependencies
ldd program        # Show library dependencies
ldconfig -v        # Configure library cache

Static Analysis​

# Code analysis
cppcheck source.c  # Static code analysis
splint source.c    # Secure programming lint

Log Analysis and Debugging​

System Logs​

# View logs
journalctl         # Systemd journal
journalctl -xe     # Recent entries with explanations
journalctl -f      # Follow log entries
journalctl -u service  # Service-specific logs

# Traditional logs
tail -f /var/log/syslog    # Follow system log
tail -f /var/log/messages  # System messages
grep -i error /var/log/syslog  # Search for errors

Log Analysis Tools​

# Search and filter
grep -C 5 "error" /var/log/syslog  # Context around matches
awk '/error/ {print NR, $0}' /var/log/syslog  # Line numbers
less +/error /var/log/syslog  # Search in less

# Log rotation
logrotate -d /etc/logrotate.conf  # Debug log rotation

Debugging Scripts and Tools​

Bash Debugging​

# Script debugging
bash -x script.sh   # Trace script execution
bash -v script.sh   # Verbose script execution
set -x             # Enable debugging in script
set +x             # Disable debugging in script

# Script analysis
shellcheck script.sh  # Shell script linter

Python Debugging​

# Python debugging
python -m pdb script.py  # Python debugger
python -u script.py      # Unbuffered output
python -v script.py      # Verbose import

Hardware Debugging​

Hardware Information​

# Hardware details
lshw               # Hardware information
lshw -short        # Hardware summary
lscpu              # CPU information
lspci              # PCI devices
lsusb              # USB devices

Hardware Monitoring​

# Temperature and sensors
sensors            # Hardware sensors
watch sensors      # Monitor sensors
cat /proc/acpi/thermal_zone/*/temperature

# Hardware errors
mcelog             # Machine check events
edac-util          # EDAC error reporting

Advanced Debugging Techniques​

Debugging with SystemTap​

# SystemTap (if available)
stap -e 'probe syscall.open { printf("open: %s\n", filename) }'
stap -e 'probe process.syscall { printf("PID %d: %s\n", pid(), name) }'

Debugging with eBPF​

# eBPF tools (if available)
bcc-tools          # BCC compiler collection
bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("open: %s\n", str(args->filename)) }'

Container Debugging​

# Docker debugging
docker logs container_id     # Container logs
docker exec -it container_id /bin/bash  # Access container
docker stats container_id    # Container statistics

# Kubernetes debugging
kubectl logs pod_name       # Pod logs
kubectl exec -it pod_name -- /bin/bash  # Access pod
kubectl describe pod pod_name  # Pod details

Debugging Best Practices​

Systematic Approach​

  1. Reproduce the issue - Ensure you can consistently reproduce the problem
  2. Gather information - Collect system information, logs, and process details
  3. Isolate the problem - Narrow down the scope of the issue
  4. Test hypotheses - Use debugging tools to test your theories
  5. Fix and verify - Implement fixes and verify they work

Common Debugging Workflow​

# 1. Initial investigation
ps aux | grep process_name
lsof -p PID
strace -p PID

# 2. Analyze system state
dmesg | tail -20
journalctl -xe
free -h && df -h

# 3. Deep dive debugging
gdb -p PID
valgrind --leak-check=full program
perf record -p PID

# 4. Monitor changes
watch -n 1 'ps aux | grep process_name'
tail -f /var/log/syslog

Documentation​

  • Keep detailed notes of your debugging process
  • Document the commands used and their outputs
  • Record the solution for future reference
  • Create scripts for common debugging scenarios