Qubes 4.1 installer does not create sys-usb

[gonzalo-bulnes]

Parent issue: #600

Description

If you have an external keyboard or other USB device attached during installation, sys-usb will not be created when Qubes 4.1 is installed. If you are using a T490, Qubes 4.1 will not create a sys-usb VM because it thinks some internal hardware is an external USB device.

To resolve this issue, we need to repro the T490 STR below and come up with a workaround for users who end up without a sys-usb VM. By default, we are recommending that the VM be disposable. To test our workaround we'll need to verify that:

• The SecureDrop Workstation auto-attach and export workflow works on sd-devices via the client.

Once we publish a workaround in our docs, we can consider this issue resolved.

Context

• Hardware: T490

• Fresh R4.1 install with all default options:

  • I think by default I would have seen a "create sys-usb" option checked, but Qubes 4.1 seemed to think that I had a USB device attached and disabled this option (note that no usb device was attached to any of my laptop ports), see:
    
  • Compare with Qubes docs (they look like they might be a little outdated when it comes to config: https://www.qubes-os.org/doc/installation-guide/#initial-setup)

• First reported in Initial support for Qubes 4.1 #751 (comment)

STR

1. Fresh 4.1 install with all default options and no USB device attached during the configuration stage (after installation reboot)

Expected

• For the Qubes 4.1 configuration step to allow the creation of sys-usb

Actual

• Qubes 4.1 does not create sys-usb by default

Notes

I think we should report this upstream and the workaround could be something we do depending on timeline. Since each pilot admin will be manually reinstalling Qubes 4.1 anyway, I think it would be fine to provide steps like we did for the sys-net issue (that I run into each time I reinstall Qubes on my T490): https://workstation.securedrop.org/en/stable/admin/hardware.html?highlight=pci#lenovo-thinkpad-t490-with-8th-generation-intel-core-processor. It should just be a one-step instruction to create sys-usb manually: sudo qubesctl state.sls qvm.sys-usb

---

The original report below will be resolved by making sure we ask users to disconnect USB devices while running the installer (see #769 (comment)).

Context

• Fresh R4.1 install with default options + disposable sys-net non-default option
• Hardware: NUC11PAK
• First reported in Initial support for Qubes 4.1 #751 (comment)

Expected

• sys-usb is created during the Qubes OS installation
• even if that didn't happen, sys-usb gets created by SecureDrop before it starts relying on it

Actual

During the execution of the script scripts/provision_all, two Salt states fail because sys-usb is missing.

Suggestion

• Add a Salt state that asserts the existence of sys-usb before applying other state to it.
  Might be enough to require qvm.sys-usb in the states that rely on the VM?

  # Not sure of the syntax, but something along these lines should be possible.
  include: qvm
  
  some-state-that-relies-on-sys-usb:
    # ...
    require:
      - qvm: sys-usb

[gonzalo-bulnes]

@creviera Maybe you could give some context about the scenario in which your sys-usb was missing?

[sssoleileraaa]

Context

• Hardware: T490

• Fresh R4.1 install with all default options:

  • I think by default I would have seen a "create sys-usb" option checked, but Qubes 4.1 seemed to think that I had a USB device attached and disabled this option (note that no usb device was attached to any of my laptop ports), see:
    
  • Compare with Qubes docs (they look like they might be a little outdated when it comes to config: https://www.qubes-os.org/doc/installation-guide/#initial-setup)

• First reported in Initial support for Qubes 4.1 #751 (comment)

STR

1. Fresh 4.1 install with all default options and no USB device attached during the configuration stage (after installation reboot)

Expected

• For the Qubes 4.1 configuration step to allow the creation of sys-usb

Actual

• Qubes 4.1 does not create sys-usb by default

Notes

I think we should report this upstream and the workaround could be something we do depending on timeline. Since each pilot admin will be manually reinstalling Qubes 4.1 anyway, I think it would be fine to provide steps like we did for the sys-net issue (that I run into each time I reinstall Qubes on my T490): https://workstation.securedrop.org/en/stable/admin/hardware.html?highlight=pci#lenovo-thinkpad-t490-with-8th-generation-intel-core-processor. It should just be a one-step instruction to create sys-usb manually: sudo qubesctl state.sls qvm.sys-usb

[marmarek]

Hardware: NUC11PAK

In this case, I guest USB keyboard was in use - which prevents creating sys-usb by default.

• I think by default I would have seen a "create sys-usb" option checked, but Qubes 4.1 seemed to think that I had a USB device attached and disabled this option (note that no usb device was attached to any of my laptop ports), see:

This indeed looks wrong. Can you check lsusb after installation (in dom0 before creating sys-usb, or in sys-usb after creating it) - just to confirm the above? I guess some device gets miss-categorized as a disk or keyboard. One (hacky) way to find out is: grep -r ID_INPUT_KEYBOARD /run/udev/data/ (again - in dom0 or sys-usb, depending on where USB currently is), and then inspect if any matching file is about an USB device.

[sssoleileraaa]

This indeed looks wrong. Can you check lsusb after installation (in dom0 before creating sys-usb, or in sys-usb after creating it) - just to confirm the above? I guess some device gets miss-categorized as a disk or keyboard. One (hacky) way to find out is: grep -r ID_INPUT_KEYBOARD /run/udev/data/ (again - in dom0 or sys-usb, depending on where USB currently is), and then inspect if any matching file is about an USB device.

@marmarek, yes, I can check on sys-usb (see results below):

[user@sys-usb ~]$ lsusb
Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 004: ID 147e:2016 Upek Biometric Touchchip/Touchstrip Fingerprint Sensor
Bus 002 Device 003: ID 04f2:b681 Chicony Electronics Co., Ltd ThinkPad T490 Webcam
Bus 002 Device 005: ID 8087:0aaa Intel Corp. Bluetooth 9460/9560 Jefferson Peak (JfP)
Bus 002 Device 002: ID 0c27:3bfa RFIDeas, Inc pcProx Card Reader
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd QEMU USB Tablet
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

[user@sys-usb ~]$ grep -r ID_INPUT_KEYBOARD /run/udev/data/
/run/udev/data/c13:70:E:ID_INPUT_KEYBOARD=1
/run/udev/data/+input:input7:E:ID_INPUT_KEYBOARD=1
/run/udev/data/c13:66:E:ID_INPUT_KEYBOARD=1
/run/udev/data/+input:input2:E:ID_INPUT_KEYBOARD=1

[marmarek]

Can you check if any of those found files in /run/udev/data are about USB devices? Simply check their content, there should several device properties.

[gonzalo-bulnes]

Hardware: NUC11PAK

In this case, I guest USB keyboard was in use - which prevents creating sys-usb by default.

Your guess is correct @marmarek, I had a USB keyboard (and mouse) attached. Thanks for the explanation 👍

[sssoleileraaa]

@marmarek, it looks like your guess that some devices get miss-categorized as a usb disk or keyboard is correct. Looking at the 4 results above where ID_INPUT_KEYBOARD=1 you can also see a couple entries where E:ID_MODEL=USB_Keyboard:

/run/udev/data/c13:69:E:ID_INPUT_KEYBOARD=1
/run/udev/data/+input:input6:E:ID_INPUT_KEYBOARD=1
/run/udev/data/c13:66:E:ID_INPUT_KEYBOARD=1
/run/udev/data/+input:input2:E:ID_INPUT_KEYBOARD=1
S:input/by-id/usb-RFIDeas_USB_Keyboard-event-kbd
S:input/by-path/pci-0000:00:06.0-usb-0:1:1.0-event-kbd
I:4088852
E:ID_INPUT=1
E:ID_INPUT_KEY=1
E:ID_INPUT_KEYBOARD=1
E:ID_VENDOR=RFIDeas
E:ID_VENDOR_ENC=RFIDeas
E:ID_VENDOR_ID=0c27
E:ID_MODEL=USB_Keyboard
E:ID_MODEL_ENC=USB\x20Keyboard
E:ID_MODEL_ID=3bfa
E:ID_REVISION=0100
E:ID_SERIAL=RFIDeas_USB_Keyboard
E:ID_TYPE=hid
E:ID_BUS=usb
E:ID_USB_INTERFACES=:030101:
E:ID_USB_INTERFACE_NUM=00
E:ID_USB_DRIVER=usbhid
E:ID_PATH=pci-0000:00:06.0-usb-0:1:1.0
E:ID_PATH_TAG=pci-0000_00_06_0-usb-0_1_1_0
E:LIBINPUT_DEVICE_GROUP=3/c27/3bfa:usb-0000:00:06.0-1
G:power-switch
Q:power-switch
V:1
I:4003032
E:ID_INPUT=1
E:ID_INPUT_KEY=1
E:ID_INPUT_KEYBOARD=1
E:ID_VENDOR=RFIDeas
E:ID_VENDOR_ENC=RFIDeas
E:ID_VENDOR_ID=0c27
E:ID_MODEL=USB_Keyboard
E:ID_MODEL_ENC=USB\x20Keyboard
E:ID_MODEL_ID=3bfa
E:ID_REVISION=0100
E:ID_SERIAL=RFIDeas_USB_Keyboard
E:ID_TYPE=hid
E:ID_BUS=usb
E:ID_USB_INTERFACES=:030101:
E:ID_USB_INTERFACE_NUM=00
E:ID_USB_DRIVER=usbhid
E:ID_PATH=pci-0000:00:06.0-usb-0:1:1.0
E:ID_PATH_TAG=pci-0000_00_06_0-usb-0_1_1_0
E:ID_FOR_SEAT=input-pci-0000_00_06_0-usb-0_1_1_0
G:seat
Q:seat
V:1
S:input/by-path/platform-i8042-serio-0-event-kbd
I:2446649
E:ID_INPUT=1
E:ID_INPUT_KEY=1
E:ID_INPUT_KEYBOARD=1
E:ID_BUS=i8042
E:ID_SERIAL=noserial
E:ID_PATH=platform-i8042-serio-0
E:ID_PATH_TAG=platform-i8042-serio-0
E:LIBINPUT_DEVICE_GROUP=11/1/1:isa0060/serio0
G:power-switch
Q:power-switch
V:1
I:2414873
E:ID_INPUT=1
E:ID_INPUT_KEY=1
E:ID_INPUT_KEYBOARD=1
E:ID_BUS=i8042
E:ID_SERIAL=noserial
E:ID_PATH=platform-i8042-serio-0
E:ID_PATH_TAG=platform-i8042-serio-0
E:ID_FOR_SEAT=input-platform-i8042-serio-0
G:seat
Q:seat
V:1

I'm not sure the proper way to fix this? If I were to edit these incorrect entries (to what? I don't know...) and reload, would that resolve this issue?

[marmarek]

E:ID_VENDOR=RFIDeas
E:ID_VENDOR_ENC=RFIDeas
E:ID_VENDOR_ID=0c27
E:ID_MODEL=USB_Keyboard

Ugh, I guess "RFIDeas USB Keyboard" is some internal device, right? Is there any option to disable it for the installation time? Maybe some BIOS setting?

Anyway, for the current released ISO, I don't have better solution than create sys-usb manually after installation. For future ISO (maybe even including 4.1.1), I'll consider relaxing the check to allow creating sys-usb in case of USB keyboard, not hard disable it.

[sssoleileraaa]

That sounds good and like a nice future improvement! Thanks for looking into it @marmarek :) For now, we can just update our docs with a workaround before we release Qubes 4.1 support (so far it seems to only have happened on a T490).

[sssoleileraaa]

@zenmonkeykstop also tested on a T490 and did not run into this issue. I'm going to wipe and re-install Qubes 4.1 (probably not until next week) so I'll try repro'ing again.

[sssoleileraaa]

Just noting my workaround is to run sudo qubesctl state.sls qvm.sys-usb which uses the correct disposable template (for 4.1.1-rc this is fedora-26-dvm). And I also have to run sudo qubesctl state.sls qvm.usb-keyboard to fix Denied qubes.InputKeyboard from sys-usb to dom0 errors.

Reboot seems to be necessary for thumb drives to be show up in the Devices menu under the USB Devices section and to be able to attach drives to VMs.

[gonzalo-bulnes]

Guessing you meant fedora-36-dvm @creviera ?

[marmarek]

For future ISO (maybe even including 4.1.1), I'll consider relaxing the check to allow creating sys-usb in case of USB keyboard, not hard disable it.

FYI, this will not make into 4.1.1, the failure mode (creating sys-usb, when the main keyboard is on USB) is too risky without more comprehensive USB keyboard handling.

[eloquence]

This was addressed in our documentation for now via freedomofpress/securedrop-workstation-docs#112. Since this is ultimately an upstream issue, I would suggest we remove it from our project board for now, but keep it open for tracking purposes.

[ghost]

While understandable, do note this breaks the entire process when installing via sdw-admin on an existent Qubes 4.1 setup with a USB keyboard (such as NUC system). The result is being locked out of access first, and then all screen output once the screensaver kicks in. Not ideal.

[marmarek]

Anyway, for the current released ISO, I don't have better solution than create sys-usb manually after installation. For future ISO (maybe even including 4.1.1), I'll consider relaxing the check to allow creating sys-usb in case of USB keyboard, not hard disable it.

FWIW, 4.1.2 installer will allow creating sys-usb even if USB keyboard is present. In such case, it will list detected "keyboards" (which helps identifying if that's really a keyboard, not another device like in this issue), and then gives an option to enable qubes.InputKeyboard service (same thing as qvm.usb-keyboard salt state does). It looks like this: https://openqa.qubes-os.org/tests/58856#step/firstboot/10

[ghost]

Anyway, for the current released ISO, I don't have better solution than create sys-usb manually after installation. For future ISO (maybe even including 4.1.1), I'll consider relaxing the check to allow creating sys-usb in case of USB keyboard, not hard disable it.

FWIW, 4.1.2 installer will allow creating sys-usb even if USB keyboard is present. In such case, it will list detected "keyboards" (which helps identifying if that's really a keyboard, not another device like in this issue), and then gives an option to enable qubes.InputKeyboard service (same thing as qvm.usb-keyboard salt state does). It looks like this: https://openqa.qubes-os.org/tests/58856#step/firstboot/10

I don't see how this is an upstream issue BTW, as the invoked command responsible for the loss of sys-usb is the Securedrop Workstation's installer own.

Do you have any suggestions as to how we can work around this without making a setup inoperable?

[marmarek]

The original issue was about not creating sys-usb because of some device was detected as a keyboard (in that case, it was some card reader), but no actual USB keyboard was in use. The new installer feature handles this case.

As for a system where you do need to use USB keyboard, that's a different story. There are two things here:

• allow qubes.InputKeyboard service (note it is a security tradeoff - when using an USB keyboard with sys-usb, your sys-usb will see keyboard events, and be able to inject them) - this one can be done from the installer, so, just don't override it
• disarm automatic attachment of USB devices (AFAIR you have some udev rules for that) - it's probably matter of adding ENV{ID_INPUT_KEYBOARD}!="1" to relevant udev rule.
  But generally, it's better to avoid systems where the only keyboard is on USB, and there is only one USB controller.

[ghost]

• But generally, it's better to avoid systems where the only keyboard is on USB, and there is only one USB controller.

Agreed, although this introduces a significant amount of complexity for administrators, as you need to properly review the hardware and how the controllers are connected. In some cases there is usually two controllers (I should verify), but it isn't rare to find systems with only one (for NUCs and similar setups).

I have reproduced the issue using the securedrop-workstation updater. I suspect the culprit is how the sys-usb shutdown is handled... (since they update the base template, and then restart each one of its users). So, I contest again the blame on Qubes itself. This is very much a securedrop-workstation issue.

I don't have the time to test a new install right now, but 4.1 has handled USB keyboard setups fine for me.

Unrelated to this is something that would be a good PR: have an UI that allows configuring one specific USB storage device for SDW, and leave the rest of them alone. The whole architecture of repeating sys-usb functionality with another qube is cumbersome and it also does not quite solve the issue (in the end all we want is to limit/assign specific USB devices to one target qube, which could be something to work on upstream, but since they are using their own sys-usb type qube they can do that on their own and contribute it to upstream if it is sound)...

[sssoleileraaa]

The original issue was about not creating sys-usb because of some device was detected as a keyboard (in that case, it was some card reader), but no actual USB keyboard was in use. The new installer feature handles this case.

Hurray! I'll be able to test the new installer (guessing it's part of Qubes release 4.1.2-rc1) soon. Thanks for the update. :)

[sssoleileraaa]

As for a system where you do need to use USB keyboard, that's a different story.

Agreed, and I would be happy to create a separate issue for that next week in the securedrop-workstation repo, but I first want to check with @gonzalo-bulnes and others on the SecureDrop team if that's something we care about.

[gonzalo-bulnes]

Replying to @creviera's question:

As for a system where you do need to use USB keyboard, that's a different story.

Agreed, and I would be happy to create a separate issue for that next week in the securedrop-workstation repo, but I first want to check with @gonzalo-bulnes and others on the SecureDrop team if that's something we care about.

Short answer: this is a low priority IMHO.

Longer answer:

In the context of the SecureDrop Workstation, I do not care much about making sure the Qubes OS installer creates a sys-usb when a USB keyboard is attached. Sure, creating it afterwards is an extra step, but to me, the fact that it is included in the docs seem enough, for the following two reasons:

• the SD Workstation that I personally use is a development and testing machine, so some manual setup is acceptable to me
• outside of scenarios like mine: I don't think we want to recommend anyone from setting up a SecureDrop Workstation on a machine that's more difficult to physically secure than a laptop, even in the scope of the current SecureDrop Workstation pilot program.
  • In particular: if sourcing hardware is a problem, I believe we should look at ways to support organizations in need of adequate hardware, rather than defaulting to machines like NUCs. (I'm not sure what your circumstances are @vogelfreiheit , and I don't think this is the place to ask about them, but I hear you.)