Nvme cancel v615

[johnmeneghini]

NVMe Cancel Support

See: https://lore.kernel.org/linux-nvme/20250324102310.658007-1-mlombard@redhat.com/T/#m7ec348748b222a34b5e320482886496f1e77fe21

This patch set adds support for the NVMe Cancel command to the Linux NVMe host and target (nvmet) drivers.

When a command times out, the nvme host driver takes take advantage of the Cancel command (defined by TP 4097a) to abort the command that timed out instead of resetting the controller.

Support for the NVMe Abort command is mandated by the NVMe specs but the NVMe target controller is not required to perform any meaningful action when an NVMe Abort is received. For example, in the present Linux nvmet implementation, the NVMe Abort command does nothing at all). The NVMe Cancel command is optional, therefore if a Target Controller claims to support NVMe Cancel. it's reasonable to expect its implementation to perform a useful action.

This patch set modifies the tcp and rdma host drivers' timeout handlers to check if the target supports the Cancel command; if it does, the error handler will send an NVMe Cancel command and try to cancel the command that timed out, before resetting the
controller. If two commands time out at the same time the error handler will send a Cancel command with the 'multiple commands' flag set to attempt cancel all commands on the IO queue.

If the NVMe Cancel command isn't supported by the controller, the allocation of the Cancel command fails, and the error handler will fall back to the existing controller reset mechanism.

Error messages

The following dmesg output is seen on the host when a command times out:

nvme nvme0: I/O tag 3 (3003) type 4 opcode 0x1 (I/O Cmd) QID 5 timeout
nvme nvme0: I/O tag 4 (2004) type 4 opcode 0x1 (I/O Cmd) QID 5 timeout
nvme nvme0: I/O tag 5 (4005) type 4 opcode 0x1 (I/O Cmd) QID 5 timeout
nvme nvme0: Cancel status: 0x0 imm abrts = 1 def abrts = 0
nvme nvme0: I/O tag 6 (5006) type 4 opcode 0x1 (I/O Cmd) QID 5 timeout
nvme nvme0: Cancel status: 0x0 imm abrts = 63 def abrts = 0
nvme nvme0: I/O tag 3 (9003) type 4 opcode 0x1 (I/O Cmd) QID 6 timeout
nvme nvme0: I/O tag 4 (a004) type 4 opcode 0x1 (I/O Cmd) QID 6 timeout
nvme nvme0: I/O tag 5 (a005) type 4 opcode 0x1 (I/O Cmd) QID 6 timeout
nvme nvme0: Cancel status: 0x0 imm abrts = 1 def abrts = 0
nvme nvme0: I/O tag 73 (7049) type 4 opcode 0x1 (I/O Cmd) QID 6 timeout
nvme nvme0: Cancel status: 0x0 imm abrts = 5 def abrts = 0

The following dmesg output is seen on the nvme target when a command times out:

nvmet: CANCEL success: 4180
nvmet: CANCEL removed 1 requests
nvmet: CANCEL success: 40964
nvmet: CANCEL request not found: 20548
nvmet: CANCEL request not found: 12332
nvmet: CANCEL removed 3 requests

How to build and test.

There's a new KConfig option CONFIG_NVME_TARGET_DELAY_REQUESTS, which depends on CONFIG_NVME_TARGET_DEBUGFS. Make sure both of those are enabled.

On the target, nvmet debugfs should be in /sys/kernel/debugfs/nvmet. There will be a directory for every subsystem, and in each of those will be a directory for each controller.

As an example, on a target with 2 controllers:

cd /sys/kernel/debug/nvmet/nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db
ls 
ctrl1  ctrl2 

In each ctrl directory is a "delay" attribute file.

When a controller is created, it will read as "0 0"

cat ctrl1/delay
 0 0

The first number is a count of how many requests will be delayed.

The second is a delay time in ms.

You can write one value to set a count, or both to set a count and timeout.

Example 1

Delay 5 requests by 10 seconds

echo 5 10000 > ctrl1/delay
echo 5 10000 > ctrl2/delay

Example 2

Set the timeout to one minute, but don't start delaying yet

echo 0 60000 > ctrl1/delay
echo 0 60000 > ctrl2/delay

Example 3

Delay the next 3 requests by the current delay time on ctrl1

echo 3 > ctrl1/delay

Note that these are per-controller, so if error recovery reconnect it's a new controller and will be set back to 0 (and you will need to change directories to access the new controller)

To Do

1. add host support for fc and pci
2. add blktests
3. testing and debugging