Powerful and Programmable Kernel Debugging with drgn

Kernel bugs that sneak past testing into production tend to be tricky, convoluted, and hard to reproduce. When investigating such bugs, you often want to explore various data structures, look for patterns, and check your assumptions. After falling victim to some particularly difficult bugs which were tedious to analyze with existing tools, we at Meta built drgn (https://github.com/osandov/drgn), a debugger with an emphasis on programmability. With drgn, you can essentially write code to debug your code. This unlocks huge potential for debugging complex bugs.

This talk will demonstrate drgn’s capabilities and how we have used it to solve real kernel issues. I will also discuss some new use cases we have found for it beyond kernel debugging, including monitoring, replacing debugfs, and userspace memory profiling. Finally, I will discuss future development plans and solicit feedback from the audience: what do you wish your debugger did?

Omar Sandoval, Meta