Introduction

WireGuard® is a Linux kernel module (built into all major distributions).

Recently, I have been studying WireGuard’s source code through debugging. This article documents the process of debugging the WireGuard kernel module source code.

Challenges

How to build a kernel with debug symbols
How to set up a debugging environment with QEMU
How to perform remote debugging with GDB
How to integrate graphical debugging in VSCode

Content

Approach

Build a Linux kernel with DEBUG symbols based on the latest WireGuard kernel module source code
Boot a virtual machine using QEMU with the kernel built in the previous step
Perform remote debugging via GDB

Prerequisites

The local development environment must meet the following requirements:

Packages required for building the kernel are installed¹
QEMU packages required for virtualization are installed
GDB packages required for debugging are installed
The development environment supports virtualization (if running inside a VM, ensure nested virtualization is enabled²)

Steps

Clone the WireGuard kernel module source code (which is essentially the kernel source with the latest WireGuard changes):

git clone git://git.zx2c4.com/wireguard-linux

Build the kernel (this takes a while – about 3 minutes on my machine):

cd wireguard-linux

# To ensure a consistent experience, all operations below are based on commit: fa41884c1c6d
git checkout fa41884c1c6d

DEBUG_KERNEL=yes ARCH=x86_64 make -C tools/testing/selftests/wireguard/qemu build -j$(nproc)

If you are interested in manually building a kernel, check out the Arch Linux Wiki. The Makefile in the source repository encapsulates all the operations needed to build the kernel. The command above directly invokes the repository’s Makefile to build a kernel with debug information.

A successful build looks like this:

Boot the debugging environment inside a VM using QEMU and the built kernel:

# bzImage is the kernel image produced by the build command above
# The startup command is somewhat complex to remain compatible with the project's test cases
# (i.e., to support debugging actual traffic send/receive)
qemu-system-x86_64 \
    -nodefaults \
    -nographic \
    -no-reboot \
    -m 1G -smp 4 \
    -monitor none \
    -serial stdio \
    -cpu host -machine microvm,accel=kvm,pit=off,pic=off,rtc=off,acpi=off \
    -S -gdb tcp::12345,server,nowait -append nokaslr \
    -chardev file,path=tools/testing/selftests/wireguard/qemu/build/x86_64/result,id=result \
    -device virtio-serial-device \
    -device virtserialport,chardev=result \
    -kernel tools/testing/selftests/wireguard/qemu/build/x86_64/kernel-debug/arch/x86_64/boot/bzImage

Start debugging with GDB:

# vmlinux is the kernel file with DEBUG symbols, produced by the build command above
gdb tools/testing/selftests/wireguard/qemu/build/x86_64/kernel-debug/vmlinux

In the GDB interactive session, connect to the remote debugging environment, set a breakpoint, and start debugging:

target remote localhost:12345
hbreak wg_mod_init
continue

The result looks like this:

See the video for a walkthrough:

I use GEF locally to enhance the GDB experience with features like syntax highlighting and automatic context display (showing the current line).

VSCode

Below are the steps for debugging the WireGuard kernel module inside VSCode³.

Install the official C/C++ extension, build the kernel (as described above), and create the task configuration and launch configuration files under the .vscode directory in the repository root.

Create .vscode/tasks.json with the following content:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "Start QEMU",
            "type": "shell",
            "command": [
                "qemu-system-x86_64",
                "-nodefaults",
                "-nographic",
                "-no-reboot",
                "-m", "1G", "-smp", "4",
                "-monitor", "none", "-serial", "stdio",
                "-cpu", "max", "-machine", "microvm,acpi=off",
                "-S", "-gdb", "tcp::1234,server,nowait", "-append", "nokaslr",
                "-chardev", "file,path=tools/testing/selftests/wireguard/qemu/build/x86_64/result,id=result",
                "-device", "virtio-serial-device", "-device", "virtserialport,chardev=result",
                "-kernel", "tools/testing/selftests/wireguard/qemu/build/x86_64/kernel-debug/arch/x86_64/boot/bzImage"
            ],
            "isBackground": true,
            "problemMatcher": [],
            "presentation": {
                "echo": true,
                "reveal": "always",
                "focus": false,
                "panel": "shared",
                "showReuseMessage": true,
                "clear": true,
                "close": true
            }
        },
    ]
}

Compared to the command-line QEMU command, the VSCode version differs slightly in the -cpu and -machine flags, with some options simplified (otherwise the VSCode graphical debugger would not work properly).

Create .vscode/launch.json with the following content:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Start WireGuard debugging",
      "type": "cppdbg",
      "request": "launch",
      "preLaunchTask": "Start QEMU",
      "program": "${workspaceFolder}/tools/testing/selftests/wireguard/qemu/build/x86_64/kernel-debug/vmlinux",
      "miDebuggerServerAddress": "localhost:1234",
      "args": [],
      "stopAtEntry": false,
      "cwd": "${workspaceFolder}",
      "environment": [],
      "externalConsole": false,
      "MIMode": "gdb",
      "miDebuggerArgs": "-n",
      "setupCommands": [
          {
              "description": "Load kernel GDB scripts",
              "text": "add-auto-load-safe-path ${workspaceFolder}/tools/testing/selftests/wireguard/qemu/build/x86_64/kernel-debug/",
              "ignoreFailures": true
          },
          {
            "description": "Enable pretty-printing for gdb",
            "text": "-enable-pretty-printing",
            "ignoreFailures": true
          },
          {
              "description": "Set Disassembly Flavor to Intel",
              "text": "-gdb-set disassembly-flavor intel",
              "ignoreFailures": true
          }
      ]
    }
  ]
}

Then start debugging.

The result looks like this:

See the video for a walkthrough:

Summary

The above covers my workflow for debugging the WireGuard kernel source code using GDB.

If you find any errors in this article, feel free to reach out via the Email on my homepage. You can also find my WeChat QR code on the About page.

References

My local development environment is Arch Linux. I referred to the dependency information from the official Arch Linux package. If you are using a different distribution, install the corresponding dependencies according to your distribution’s official documentation. ↩︎
Nested virtualization means creating (running) a virtual machine inside another virtual machine. For example, if you are debugging on a cloud server, you need to ensure the cloud server (which is essentially a VM) can also create VMs – i.e., running a VM inside a VM. ↩︎
My local VSCode runs via Remote SSH, with the actual development environment on an Arch Linux VM. The VSCode Remote-SSH extension allows developers to run VSCode on a server (where the actual code and development environment reside), while the local VSCode instance acts merely as an interactive client. ↩︎

Introduction#

Challenges#

Content#

Approach#

Prerequisites#

Steps#

VSCode#

Summary#

References#