Skip to content

SysRq Trigger

When an Out-of-Memory (OOM) condition occurs, the Linux kernel may terminate processes to free memory, but diagnosing the root cause requires capturing diagnostic data during the event. The sysrq-trigger utility provides a way to invoke kernel-level commands, including memory dumps and OOM-related diagnostics, even when the system is unresponsive. This is critical for post-mortem analysis of OOM events.

Key SysRQ Triggers for OOM Debugging

The sysrq-trigger command sends specific keypresses to the kernel. Use it with caution, as some triggers can cause system instability. Relevant triggers for OOM analysis include:

  • m: Dumps memory statistics (e.g., total, free, used memory) to the console. This provides a snapshot of memory usage at the time of the trigger.
  • t: Triggers a full memory dump (kernel core dump) to disk. This is useful for analyzing the state of memory at the time of the OOM event.
  • o: Displays OOM killer details, such as the process ID (PID) of the victim and memory usage metrics. This helps identify which process was terminated.
  • d: Dumps memory to disk (similar to t but may vary by kernel version). Use this if t is unavailable or restricted.

Note: Some triggers (e.g., t or d) require the kernel to be configured with CONFIG_KALLSYMS and sufficient disk space. Always verify kernel documentation for version-specific behavior.

Example Usage

  1. Capture memory statistics: 

    sudo sysrq-trigger m
    This outputs memory metrics to the console, such as: 
    Memory statistics:
total: 1024MB, free: 128MB, buffers: 64MB, cached: 256MB

  2. Trigger a memory dump: 

    sudo sysrq-trigger t
    This generates a file like /var/crash/vmcore-dump-<timestamp>.dmp (location depends on kernel configuration).

  3. View OOM killer details: 

    sudo sysrq-trigger o
    Output might include: 
    OOM killer: pid 1234 (process-name), memory usage 1.5GB

Analyzing the Data

  • Memory dumps (t/d): Use tools like crash or gdb to analyze the core dump. For example: 
    crash /var/crash/vmcore-dump-<timestamp>.dmp /usr/lib/debug/vmlinux-<kernel-version>
  • OOM killer logs: Check /var/log/kern.log or use dmesg to find OOM-related messages: 
    dmesg | grep -i oom

Best Practices

  • Pre-configure sysrq: Ensure sysrq is enabled in the kernel (CONFIG_SYSRQ), and install sysrq-tools for command-line access.
  • Monitor disk space: Memory dumps can be large (GBs), so ensure sufficient storage is available.
  • Test in safe environments: Avoid using these triggers on production systems without proper safeguards.

Key takeaways

  • Use sysrq-trigger to capture memory stats, dumps, and OOM killer details during OOM events.
  • The m, t, o, and d triggers provide critical diagnostic data, though kernel version and configuration may affect behavior.
  • Analyze memory dumps with tools like crash and review logs in /var/log/kern.log for OOM-related messages.
  • Always verify kernel configuration and disk space before triggering memory dumps.