A comprehensive guide for debugging kernel-level code, including kernel extensions, on macOS using two-machine debugging setup.
- Overview
- Prerequisites
- Quick Start
- Device Compatibility
- Two-Machine Debugging
- Debugging Workflow
- Kernel Variants
- Core Dump Server
- Troubleshooting
- Additional Resources
- Automatic Release Tracking
Use the Kernel Debug Kit (KDK) to debug kernel-level code, such as kernel extensions you create. The KDK supports debugging from a second Mac using a two-machine setup.
Performing Two-Machine Debugging: The KDK supports the debugging of kernel-level code, such as kernel extensions, from a second Mac.
- Target Device: The Mac that runs the code you want to debug
- Host Device: The Mac that runs the debugger
Before you begin, ensure you have:
- ✅ Two Mac computers (one as host, one as target)
- ✅ Compatible Ethernet connection (built-in Ethernet ports)
- ✅ Both Macs connected to the same network
- ✅ Xcode installed on the host device
- ✅ Administrator access on both machines
- ✅ The Kernel Debug Kit downloaded
- Install KDK on the host device
- Configure target device for debugging (disable SIP, set boot-args)
- Configure host device (install Xcode, Python packages)
- Trigger debugging on target (via panic or NMI)
- Connect from host using
lldbandkdp-remote
For detailed instructions, see the sections below.
- Use Mac's built-in Ethernet ports only
- Two-machine debugging supported with limitations (see note below)
Compatible Ethernet connections:
- Mac's built-in Ethernet ports
- Apple Thunderbolt display Ethernet port
- Apple Thunderbolt to Gigabit Ethernet adapter
- Third-party Thunderbolt Ethernet adapters with these chipsets:
- Broadcom C-IV
- Aquantia AQC107/113
- Intel 82574L
Note: You cannot perform two-machine debugging using:
- USB Ethernet adapters
- Wireless networking on any Mac
Both devices must be connected to the same network, but there are no other restrictions on how they connect to that network.
Apple Silicon Note: Apple silicon doesn't support active kernel debugging. You may inspect the current state of the kernel when it is halted due to a panic or NMI. However, you cannot set breakpoints, continue code execution, step into code, step over code, or step out of the current instruction.
Use the following steps to configure your Apple silicon as a target device for debugging.
Modify the security configuration of your system as follows:
- Reboot your Mac in Recovery Mode
- Launch Terminal and run the following command to disable System Integrity Protection (SIP):
csrutil disable
- If prompted to lower your system security to "permissive", enter
yto accept - Reboot your Mac
Run the ifconfig tool in Terminal to identify which Ethernet device your target device uses to connect to the network. In the following example, en1 is the Ethernet device connected to the network.
en0: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=60<TSO4,TSO6>
ether 32:00:13:er:19:e0
media: autoselect
status: inactive
en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=10b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV>
ether 40:6c:8f:5b:a2:96
inet6 fe80::426c:8fff:fe5b:a296%en2 prefixlen 64 scopeid 0x4
inet6 2620::1b07:114:426c:8fff:fe5b:a296 prefixlen 64 autoconf
inet6 2620::1b07:114:88d6:bbba:7ac9:b0a7 prefixlen 64 autoconf temporary
inet 10.128.19.135 netmask 0xfffff800 broadcast 10.128.23.255
nd6 options=1<PERFORMNUD>
media: autoselect (1000baseT <full-duplex>)
status: activeRun the nvram tool in Terminal to add the following arguments to your target device's boot-args key:
- debug=0x44 — Tells the kernel to wait for a debugger to attach to the device, when the kernel receives a non-maskable interrupt (NMI)
- kdp_match_name=enX — Set the value of this key to the Ethernet device (en0, en1, etc.) identified in Step 2
- wdt=-1 — Disables watchdog monitoring
Example:
sudo nvram boot-args="debug=0x44 kdp_match_name=en1 wdt=-1"Upon reboot, you may connect to your target device from the host.
Use the following steps to configure your target device for debugging.
Modify the security configuration of your system as follows:
- Reboot your Mac in Recovery Mode
- If your device has the Apple T2 Security Chip, set the Secure Boot policy to "Reduced Security"
- Launch Terminal and run the following command to disable System Integrity Protection (SIP):
csrutil disable
- Reboot your Mac
Run the ifconfig tool in Terminal to identify which Ethernet device your target device uses to connect to the network. In the following example, en1 is the Ethernet device connected to the network.
en0: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=60<TSO4,TSO6>
ether 32:00:13:er:19:e0
media: autoselect
status: inactive
en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=10b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV>
ether 40:6c:8f:5b:a2:96
inet6 fe80::426c:8fff:fe5b:a296%en2 prefixlen 64 scopeid 0x4
inet6 2620::1b07:114:426c:8fff:fe5b:a296 prefixlen 64 autoconf
inet6 2620::1b07:114:88d6:bbba:7ac9:b0a7 prefixlen 64 autoconf temporary
inet 10.128.19.135 netmask 0xfffff800 broadcast 10.128.23.255
nd6 options=1<PERFORMNUD>
media: autoselect (1000baseT <full-duplex>)
status: activeRun the nvram tool in Terminal to add the following arguments to your target device's boot-args key:
- debug=0x44 — Tells the kernel to wait for a debugger to attach to the device, when the kernel receives a non-maskable interrupt (NMI)
- kdp_match_name=enX — Set the value of this key to the Ethernet device (en0, en1, etc.) identified in Step 2
- wdt=-1 — Disables watchdog monitoring
Example:
sudo nvram boot-args="debug=0x44 kdp_match_name=en1 wdt=-1"Upon reboot, you may connect to your target device from the host.
Configure your host device to initiate debugging using the following steps:
Install Xcode. Then install these additional Python packages for kernel debugging:
xcrun python3 -m pip install --user --ignore-installed macholib
xcrun python3 -m pip install --user --ignore-installed futureInstall this KDK on the host device.
After you configure the target device, it halts and waits for an external debugger to attach in the following situations:
- If the target device panics, it automatically halts and waits for an external debugger to attach
- If you trigger an NMI on the target device, the device halts and waits for an external debugger to attach
Run lldb from Terminal and use the kdp-remote command to connect to the target device. That command accepts the hostname or IP address of the target device, and creates a debugger connection to it.
(lldb) kdp-remote {name_or_ip_address}The lldb tool automatically searches any spotlight-indexed directories and the /Library/Developer/KDKs/ directory on the local system for symbol information. If lldb is unable to find a kernel binary automatically, you can specify it on the command line when you launch lldb. For example:
lldb /Library/Developer/KDKs/<version>/System/Library/Kernels/kernelApple Silicon Note: Apple silicon doesn't support active kernel debugging. You may inspect the current state of the kernel when it is halted due to a panic or NMI. However, you cannot set breakpoints, continue code execution, step into code, step over code, or step out of the current instruction.
When you finish debugging the target device, quit lldb to disconnect the host from the target machine. When you no longer need to debug the target device, return the device to normal operation:
Run the nvram tool in Terminal to remove the boot-args key.
sudo nvram -d boot-argsModify the security configuration of your Mac by enabling SIP:
- Reboot your Mac in Recovery Mode
- Connect your Mac to the internet, via a cable or Wi-Fi (internet access may be required to restore full security)
- Launch Terminal from the Utilities menu and run the following command to enable System Integrity Protection (SIP):
csrutil enable - If prompted to raise the security to "full", enter
yto accept - If your device is an Intel-based Mac that has the Apple T2 Security Chip, set the Secure Boot policy to "Full Security"
- Reboot your Mac
The KDK includes several variants of the kernel binary and kernel extensions. These variants include additional assertions and error checking beyond what is present in the shipping version of the macOS kernel, and may be useful for debugging in various situations.
Choose the variant that best suits your needs:
| Variant | Description |
|---|---|
| kernel (release) | Matches the shipping kernel for users |
| kernel.development | Safe for everyday use during development, with minimal performance overhead |
| kernel.kasan | Has more significant performance and memory overhead, enables address sanitizer features that will panic the kernel upon memory safety violations |
When performing two-machine debugging, install these kernel variants on the target device.
Note: Apple silicon doesn't support installing the kernel and kernel extension variants from the KDK.
To install new kernel and kernel extension variants on the target device, perform the following steps:
Install this KDK on the target device.
Modify the security configuration of your Mac by disabling SIP and enabling permissive security:
- Reboot your Mac in Recovery Mode
- If your device has the Apple T2 Security Chip, set the Secure Boot policy to "Reduced Security"
- Launch Terminal from the Utilities menu and run the following commands to disable System Integrity Protection (SIP) and Authenticated Root protection:
csrutil disable csrutil authenticated-root disable
- Reboot your Mac
Run the mount command in Terminal to identify the devices of your system volume snapshot. Look for the device mounted at "/", which identifies the system volume disk. In the following example, this disk is /dev/disk4s5s1.
/dev/disk4s5s1 on / (apfs, sealed, local, read-only, journaled)
devfs on /dev (devfs, local, nobrowse)
/dev/disk4s4 on /System/Volumes/VM (apfs, local, noexec, journaled, noatime, nobrowse)
/dev/disk4s2 on /System/Volumes/Preboot (apfs, local, journaled, nobrowse)
/dev/disk4s1 on /System/Volumes/Data (apfs, local, journaled, nobrowse)To get the actual name of the system volume disk, remove the final "sX" from the device. In the preceding example, the name of the system volume disk is /dev/disk4s5.
Run the mount command in Terminal to mount the system volume disk to a temporary location. When running the mount command, always include the nobrowse mount option to prevent Spotlight from indexing the volume.
mkdir /Users/jappleseed/livemount
sudo mount -o nobrowse -t apfs /dev/disk4s5 /Users/jappleseed/livemountAdd the kernel variant files to the newly mounted disk. The easiest way to copy the files is to copy the entire /System directory of the KDK into the /System directory of your mounted disk, as shown in the following example:
sudo ditto /Library/Developer/KDKs/<version>/System /Users/jappleseed/livemount/SystemRun kmutil in Terminal to rebuild the kernel collections for the variants you added to your mounted disk. Run bless to authorize booting from your modified kernels.
sudo kmutil install --volume-root /Users/jappleseed/livemount --update-all
sudo bless --mount /Users/jappleseed/livemount --bootefi --create-snapshotRun the nvram tool in Terminal to add the following argument to your device's boot-args key:
- kcsuffix — Set the value of this key to either
developmentorkasan, based on which kernel variant you want to use. Omit this argument if you want the release kernel.
Example:
sudo nvram boot-args="debug=0x44 kdp_match_name=en1 wdt=-1 kcsuffix=development"Upon rebooting, the chosen kernel variant will be active. You can verify this by running the sysctl command in Terminal to check the kern.osbuildconfig property:
sysctl kern.osbuildconfig
# Output:
kern.osbuildconfig: developmentWhen you are finished debugging, perform the following steps to remove the kernel variants and restore your device security:
Run the nvram tool in Terminal to remove the boot-args key.
sudo nvram -d boot-argsRun the mount command in Terminal to identify the devices of your system volume snapshot. Look for the device mounted at "/", which identifies the system volume disk. In the following example, this disk is /dev/disk4s5s1.
/dev/disk4s5s1 on / (apfs, sealed, local, read-only, journaled)
devfs on /dev (devfs, local, nobrowse)
/dev/disk4s4 on /System/Volumes/VM (apfs, local, noexec, journaled, noatime, nobrowse)
/dev/disk4s2 on /System/Volumes/Preboot (apfs, local, journaled, nobrowse)
/dev/disk4s1 on /System/Volumes/Data (apfs, local, journaled, nobrowse)To get the actual name of the system volume disk, remove the final "sX" from the device. In the preceding example, the name of the system volume disk is /dev/disk4s5.
Run the mount command in Terminal to mount the system volume disk to a temporary location. When running the mount command, always include the nobrowse mount option to prevent Spotlight from indexing the volume.
mkdir /Users/jappleseed/livemount
sudo mount -o nobrowse -t apfs /dev/disk4s5 /Users/jappleseed/livemountRun bless to restore booting from the system sealed snapshot.
sudo bless --mount /Users/jappleseed/livemount --bootefi --last-sealed-snapshotModify the security configuration of your Mac:
- Reboot your Mac in Recovery Mode
- Connect your Mac to the internet, via a cable or Wi-Fi (internet access may be required to restore full security)
- Launch Terminal from the Utilities menu and run the following commands to restore Authenticated Root protection and System Integrity Protection (SIP):
csrutil authenticated-root enable csrutil enable
- If your device has the Apple T2 Security Chip, set the Secure Boot policy to "Full Security"
- Reboot your Mac
Verify you have restored the "release" kernel variant by running the sysctl command in Terminal to check the kern.osbuildconfig property:
sysctl kern.osbuildconfig
# Output:
kern.osbuildconfig: releaseYou can use another Mac to capture core dumps from your system automatically over the network. This may be useful for kernel panics that occur early in the boot cycle, or when the kernel fails to write the core-dump file to disk after a panic. You can use a core dump server simultaneously with two-machine debugging.
- Server Device: The Mac that receives the core-dump files
- Target Device: The Mac that generates the core-dump file
Configure the server device to receive the core dump files by performing the following steps:
Create a /var/tmp/PanicDumps directory on the server, and configure it to be writable by any user. This directory is where the server writes the core-dump files.
mkdir /var/tmp/PanicDumps
chmod 1777 /var/tmp/PanicDumpsRun the launchctl command in Terminal to load the kdumpd launch daemon. This service listens for incoming core dumps and writes them to /var/tmp/PanicDumps.
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.kdumpd.plistThe same compatibility requirements apply to the target device as for two-machine debugging. You must also configure the target device as if you're using it for two-machine debugging (see the relevant sections above for Apple Silicon or Intel-based Mac).
After performing those configuration steps, perform the following additional steps:
Run the nvram tool in Terminal to add the following arguments to your target device's boot-args key:
- debug=0xc44 — Tells the kernel to create a core dump when a panic or NMI occurs
- _panicd_ip=10.0.40.2 — Set the value of this key to the IP address of the server device
Example:
sudo nvram boot-args="debug=0xc44 kdp_match_name=en1 wdt=-1 _panicd_ip=10.0.40.2"After you reboot the target device, it sends core dumps to the server in the following situations:
- If the target device panics
- If you trigger an NMI on the target device
Note: Core dumps can be very large, so it may take some time for the core dump to transfer over the network. Additionally, ensure the server device has sufficient free disk space.
When you no longer want to gather core dumps from the target device, return that device to normal operation (see Finish Up Your Debugging Session section above).
When you no longer wish to capture core dumps on the server device, run the following to unload the kdumpd launch daemon:
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.kdumpd.plistProblem: Cannot connect debugger to target device
- Solution: Verify both devices are on the same network and Ethernet device is correctly identified
- Solution: Check that boot-args are correctly set using
nvram -p | grep boot-args - Solution: Ensure target device has been rebooted after setting boot-args
Problem: lldb cannot find kernel symbols
- Solution: Manually specify the kernel binary path when launching lldb:
lldb /Library/Developer/KDKs/<version>/System/Library/Kernels/kernel
- Solution: Ensure KDK is installed in
/Library/Developer/KDKs/
Problem: Kernel variant not loading on Intel Mac
- Solution: Verify kcsuffix boot-arg is correctly set
- Solution: Check that kernel collections were rebuilt using
kmutil - Solution: Verify the variant is active using
sysctl kern.osbuildconfig
Problem: Core dumps not being received by server
- Solution: Verify kdumpd daemon is loaded:
sudo launchctl list | grep kdumpd - Solution: Check
/var/tmp/PanicDumpsdirectory exists and is writable - Solution: Verify correct IP address is set in _panicd_ip boot-arg
- Solution: Ensure sufficient disk space on server device
If you encounter issues not covered here:
- Check Apple Developer documentation for the specific macOS version
- Review system logs on both devices for error messages
- Verify all prerequisites are met
- Ensure you're using compatible hardware and software versions
# Check current boot-args
nvram -p | grep boot-args
# Check kernel variant
sysctl kern.osbuildconfig
# Check SIP status
csrutil status
# List network interfaces
ifconfig
# View mounted volumes
mount⚠️ Always backup your data before modifying system security settings⚠️ Disabling SIP and modifying kernel settings can expose your system to security risks⚠️ Only perform these operations on development/testing machines⚠️ Remember to restore security settings when debugging is complete
This repository includes a framework for automated monitoring of new Kernel Debug Kit releases from Apple.
- Daily Checks: GitHub Actions workflow runs daily to check for new KDK releases
- Automatic Notifications: When a new release is detected, an issue is automatically created
- Version Tracking: All detected versions are tracked in
kdk_versions.json - Manual Trigger: You can manually trigger the check via GitHub Actions
To manually check for new KDK releases:
python3 scripts/check_kdk_releases.pyThis is a framework that requires customization for your environment:
- Authentication: Add Apple Developer credentials to the workflow
- Detection Logic: Implement KDK version detection in
scripts/check_kdk_releases.py - Workflow Secrets: Configure GitHub secrets for API access
See scripts/README.md for implementation guidance.
The automated check runs via .github/workflows/check-kdk-updates.yml. You can:
- Adjust the schedule (default: daily at 00:00 UTC)
- Manually trigger from the Actions tab
- Customize notification settings
- Add your detection logic to the workflow
Note: This is a framework/starter template. Full automation requires Apple Developer credentials and custom implementation. The framework provides infrastructure for tracking and notifications once you add your detection logic.
This documentation is provided by Apple Inc. For software licensing information, refer to your KDK installation and Apple Developer Program agreements.
Made with ❤️ for macOS Kernel Developers