Debug Watchdog for Linux (v1.0) is a new tool to monitor a Linux system and stop processes of chosen executable binaries immediately after launched. Once stopped, it’s possible to start a gdb debugging session right from the first instruction, as if gdb –wait-for-process exec.bin command existed. It may also be useful to analyze the process parameters and environmental variables. Code is open source, under a GPL license.
Every executable binary is caught: no matter how it was launched (command line, graphical user interface, ssh daemon, etc.), with which user or privileges nor how long it lives. This effectiveness is achieved patching the system calls table in kernel space and hooking sys_execve. Values are restored when Debug Watchdog is turned off. For obvious reasons, this tool is intended for non-production environments only.
The difference with user-space approaches, such as periodically polling the list of processes and attaching a debugger, is effectiveness: it’s hard to catch short-lived processes without kernel help and do it from the first instruction almost impossible.
Debug Watchdog has the following components:
- Kernel module (C)
- Library (C)
- Test (C)
- UI (C++ / Qt)
Kernel module
The kernel module is loaded by the application when started. Only one process from user-space can communicate with the kernel module, through IOCTLs. If the process dies, a new one can take ownership. It’s important to notice that CAP_SYS_MODULE capability is required to dynamically load a kernel module from user-space (running with root will make it).
Once turned on, a hook is installed in sys_execve patching the system calls table. Every time a process executes sys_execve, the hook code performs the real function and then compares the executable binary with the watch list. If there is a match, a SIGSTOP signal is sent to the process before returning to user-space. The targeted process cannot handle or ignore this signal and will be stopped before executing its first instruction. A debugger can then be attached.
Library
The library, running in user space, is the only component that communicates with the kernel module. Main operations are: initialize or finalize the kernel module, watch or un-watch an executable binary and notify the application about a stopped process by means of a callback. Instead of continuously polling the kernel module to get the list of stopped processes, a SIGUSR1 signal is sent to notify of an update. The library has a dedicated thread to handle these notifications, with thread-safety mechanisms in place.
Test
The automated test case will load the library, initialize everything and spawn an ls executable binary to be stopped. It may be useful to understand the library API and rapidly test your setup.
UI
The graphical user interface was implemented in Qt5 and loads the library to enable the back-end functionality. gnome-terminal is required to launch gdb from the UI.
Only x86_64 architecture is currently supported. Tested on Fedora 25 (kernel 4.12.12-200).
Further information:
- GitHub (source code and compiled binaries)
- Screencast (demo)
- README
- Slides (PDF): Spanish – English
Any feedback, bug reports or contributions are welcomed 🙂
Update (2018-05-22): Debug Watchdog for Linux v1.1 released.