Bug 1511234 - [RFE] Hook for booting from Passthrough Devices
Summary: [RFE] Hook for booting from Passthrough Devices
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: vdsm
Classification: oVirt
Component: General
Version: ---
Hardware: x86_64
OS: Linux
low
medium
Target Milestone: ---
: ---
Assignee: Germano Veit Michel
QA Contact: Liran Rotenberg
URL:
Whiteboard:
Depends On: 1575824
Blocks: 1608152
TreeView+ depends on / blocked
 
Reported: 2017-11-08 23:51 UTC by Germano Veit Michel
Modified: 2022-03-10 17:24 UTC (History)
10 users (show)

Fixed In Version: v4.20.10
Doc Type: Enhancement
Doc Text:
The new boot_hostdev hook allows virtual machines to boot from passed through host devices such as NIC VF's, PCI-E SAS/RAID Cards, SCSI devices for example without requiring a normal bootable disk from a Red Hat Virtualization storage domain or direct LUN.
Clone Of:
Environment:
Last Closed: 2019-01-30 12:39:08 UTC
oVirt Team: Virt
Embargoed:
sbonazzo: ovirt-4.3-
lrotenbe: testing_plan_complete+
rbarry: devel_ack-


Attachments (Terms of Use)
vdsm log (710.11 KB, application/x-xz)
2018-05-01 11:08 UTC, Liran Rotenberg
no flags Details
pci_logs (1.20 MB, application/x-xz)
2018-07-29 13:36 UTC, Liran Rotenberg
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-45086 0 None None None 2022-03-10 17:24:01 UTC
oVirt gerrit 84408 0 master MERGED hooks: add boot_hostdev 2020-09-07 08:09:21 UTC
oVirt gerrit 91043 0 master MERGED hooks: make boot_hostdev fail if hostdev not found 2020-09-07 08:09:22 UTC
oVirt gerrit 91074 0 master MERGED hooks: fix boot_hostdev usb non-hub device 2020-09-07 08:09:22 UTC
oVirt gerrit 92905 0 ovirt-4.2 MERGED hooks: fix boot_hostdev usb non-hub device 2020-09-07 08:09:21 UTC
oVirt gerrit 92906 0 ovirt-4.2 MERGED hooks: make boot_hostdev fail if hostdev not found 2020-09-07 08:09:21 UTC

Description Germano Veit Michel 2017-11-08 23:51:23 UTC
1. Proposed title of this feature request

Boot from Passthrough (hostdev) devices.

3. What is the nature and description of the request?

Allow Virtual Machines to boot from passed through host devices such as NIC VFs, PCI-E SAS/RAID Cards, SCSI devices etc. Without requiring a normal bootable disk from RHV SD/Direct LUN.

This seems to already be implemented in VDSM, but not in the engine.

In lib/vdsm/virt/vmdevices/hostdevice.py these classes seem to support bootOrder:
  class PciDevice
  class UsbDevice
  class ScsiDevice

Comment 9 Liran Rotenberg 2018-05-01 11:08:01 UTC
Created attachment 1429157 [details]
vdsm log

Tried to verify on:
vdsm-4.20.27-1.el7ev.x86_64
vdsm-hook-boot_hostdev-4.20.27-1.el7ev.noarch
ovirt-engine-4.2.3.3-0.1.el7.noarch

Steps:
1. Enable host passthrough devices
2. Connect a bootable OS device to the host
3. Check the host did recognize the device, and take the name of it(for example usb_3_11)
4. Add a VM
5. In the VM -> Host devices -> Add device -> Add the selected device from step 3.
6. Set custom properties to the VM, boot_hostdev with the device from step 3.
7. Start VM.

Result:
The VM didn't boot from the device given.

Additional info:
After checking, it is possible to boot from the device only by checking the 'Enable menu to select boot device' box in the VM boot properties.
When the VM starts, we need to manually enter to the boot menu and select the device.
This is the same to a VM with a bootable disk and without it. 

The expectation is that if we choose this hook and set it on the VM, we will boot from it without the boot menu.
Or, if we do have a bootable device(let's say CD-ROM) and it didn't boot from it, by the booting order it will try to boot from the host device.

Adding the vdsm log.

Comment 10 Red Hat Bugzilla Rules Engine 2018-05-01 11:08:59 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 11 Germano Veit Michel 2018-05-02 02:07:47 UTC
Indeed it tried to boot from disk instead of the USB device

        <hostdev managed="no" mode="subsystem" type="usb">                                                                                                                                     
            <source>                                                                                                                                                                           
                <address bus="3" device="6" />                                                                                                                                                 
            </source>                                                                                                                                                                          
            <alias name="ua-8a4bdbda-577a-4a38-8831-73fa3a2c7e20" />                                                                                                                           
            <address bus="0" port="1" type="usb" />                                                                                                                                            
        </hostdev>                                                                                                                                                                             
                
        <disk device="disk" snapshot="no" type="file">                                                                                                                                         
            <target bus="ide" dev="hda" />                                                                                                                                                     
            <source file="/rhev/data-center/...." />                                                                                                         
            <driver cache="none" error_policy="stop" io="threads" name="qemu" type="raw" />                                                                                                    
            <alias name="ua-ff9a3c11-563a-42e7-a8ec-ad24348e91c7" />                                                                                                                           
            <address bus="0" controller="0" target="0" type="drive" unit="0" />                                                                                                                
            <boot order="1" />                                                                                                                                                                 
            <serial>ff9a3c11-563a-42e7-a8ec-ad24348e91c7</serial>                                                                                                                              
        </disk>    


Why? here is the custom property:

{'boot_hostdev': 'usb_3_12'}

If the device is usb_3_6, why did you add a property with usb_3_12? Looks like you told the VM to boot from usb_3_12, but there is no usb_3_12, only usb_3_6.

Could you please clarify?

Comment 12 Liran Rotenberg 2018-05-02 07:18:03 UTC
The USB is recognized by the VM.

The problem is that I can boot from it if and only if I start the VM -> Open a console session -> Enter to the boot menu -> Select the USB device.

This doesn't matter if I have a bootable device or not (for example I set CD-ROM with nothing as the boot device with the use of the hook).

If the user uses this hook I would expect one of these:
1. The VM will boot from the device without checking other bootable device (setting the host device as the first boot order).
2. The VM will boot with the normal boot order set in the VM properties(from the engine), and set the host device to be the last one in the boot order. In this case If it didn't find a bootable device, in the end it will try to boot from the host device.
3. Adding the option to set it on the VM properties boot sequence (will solve options 1 and 2 by user decision).

Comment 13 Germano Veit Michel 2018-05-03 00:20:17 UTC
(In reply to Liran Rotenberg from comment #12)
> The USB is recognized by the VM.
Yes, this is not related to the problem.

> The problem is that I can boot from it if and only if I start the VM -> Open
> a console session -> Enter to the boot menu -> Select the USB device.
Yes, because you added a wrong custom option for a device that does not exist.

Let me show it to you again:

1) Custom option:
    <custom>
        <boot_hostdev>usb_3_12</boot_hostdev>
    </custom>

2) Device:
        <hostdev managed="no" mode="subsystem" type="usb">
            <source>
                <address bus="3" device="6" />
            </source>
            <alias name="ua-8a4bdbda-577a-4a38-8831-73fa3a2c7e20" />
            <address bus="0" port="1" type="usb" />
        </hostdev>

Your hostdev is usb_3_6, and in the custom option you configured usb_3_12, which does not exist.

If the device does not exist the hook will do nothing, which is what is happening here. We can modify to fail the start of the VM with an error saying the device specified in the custom option does not exist to make it clear the configuration is wrong. What do you think?

> This doesn't matter if I have a bootable device or not (for example I set
> CD-ROM with nothing as the boot device with the use of the hook).
Correct, not related.


> If the user uses this hook I would expect one of these:
> 1. The VM will boot from the device without checking other bootable device
> (setting the host device as the first boot order).
It does this, as long as you specify a device that exists in the custom option.

> 2. The VM will boot with the normal boot order set in the VM properties(from
> the engine), and set the host device to be the last one in the boot order.
> In this case If it didn't find a bootable device, in the end it will try to
> boot from the host device.
Don't see much use case on this, it is not what the customer requested, but it could be implemented. Currently it will set the host device with boot order=1 and move everything else up by 1.

> 3. Adding the option to set it on the VM properties boot sequence (will
> solve options 1 and 2 by user decision).
A VDSM hook is not related to the web administration portal. I agree this would be the best, but its not in the scope of this RFE. If this is implemented we can remove this hook, which is just a workaround to make VMs boot from hostdevs.

Comment 14 Germano Veit Michel 2018-05-03 00:33:04 UTC
By the way, I just tested it on my labs with usb_usb1 (device=1, bus=1) and it works. But usb_3_6 (or usb_3_12) will not work, the hook does not work with a bus other than 1. So your test will fail even if you use the correct custom option.

So please let me know what you think about failing the VM start if the device does not exist and I'll submit a new patch.

Comment 15 Sandro Bonazzola 2018-05-04 11:00:01 UTC
oVirt 4.2.1 has been released and the bug is back to assigned state.
Resetting target milestone to '---' for allowing to re-evaluate this bug.

Comment 16 Germano Veit Michel 2018-05-08 03:53:41 UTC
I created BZ1575824 to track the unusual naming of USB devices on bus 1. Because I not sure the case of usb_usbX vs usb_X_Y should be handled here, most likely the engine should have consistent naming.

Comment 17 Liran Rotenberg 2018-05-08 06:43:42 UTC
In my point of view, if the device is not there try to keep booting from the boot sequence.

Michal and Martin, can you please provide your opinion?

Comment 18 Michal Skrivanek 2018-05-08 10:21:32 UTC
(In reply to Liran Rotenberg from comment #17)
> In my point of view, if the device is not there try to keep booting from the
> boot sequence.
> 
> Michal and Martin, can you please provide your opinion?

it's a hook. you need to know what you are doing and use the proper device.
The additional fix is fine, it's better to be explicit about wrong settings

Comment 19 Germano Veit Michel 2018-05-09 05:53:59 UTC
Posted 2 patches:

Gerrit 91043: abort vm start if device not found
Gerrit 91074: fix case where usb device is not a hub

Comment 20 Germano Veit Michel 2018-07-08 23:05:19 UTC
Michal,

Need your help here. We have 3 patches attached to this RFE. Just one was backported to vdsm-4.20. Should I backport the other 2 (currently on master) to 4.2.z to close this RFE in 4.2.z or shall we re-target it to 4.3?

Not entirely sure on the correct process to follow here.

Thanks

Comment 21 Michal Skrivanek 2018-07-09 07:42:24 UTC
Hi Germano, I just posted the backports now, they're simple and self-contained, it should make it into 4.2.5 

Liran, we settled for failing the VM start if that device doesn't exist

Comment 22 Germano Veit Michel 2018-07-10 22:43:38 UTC
Thanks Michal!

Comment 23 Liran Rotenberg 2018-07-26 09:05:23 UTC
Hi,
I was able to verify it with USB device.
But, this is not user friendly.

I put the USB device in the host and it got the ID: usb_3_12.

In order to get the ID to set on the custom properties of the VM I first needed to run on the host:
# vdsm-client Host hostdevListByCaps
Then i got:
"usb_3_12": {
        "params": {
            "product": "TransMemory-Mini / Kingston DataTraveler 2.0 Stick", 
            "vendor": "Toshiba Corp.", 
            "product_id": "0x6544", 
            "parent": "usb_usb3", 
            "vendor_id": "0x0930", 
            "driver": "usb", 
            "capability": "usb_device", 
            "is_assignable": "true", 
            "address": {
                "device": "6", 
                "bus": "3"
            }
        }
    }

So the actual USB device need to be set on the VM custom properties is not usb_3_12, it is usb_3_6.

I don't think that a user need to move into using CLI to get the device ID.
Any thoughts?

Comment 24 Michal Skrivanek 2018-07-26 09:25:20 UTC
why do you keep adding usb_3_12? host address has nothing to do with guest address. You're supposed to boot the guest, not the host:)
You see the guest address in the report above, or VM xml, or inside guest(e.g. lsusb).

Comment 26 Michal Skrivanek 2018-07-27 10:15:46 UTC
ah, so you're complaining about the difference in the assigned device in UI and what you need to pass to the hook, right?

Comment 27 Liran Rotenberg 2018-07-29 13:36:03 UTC
Created attachment 1471357 [details]
pci_logs

(In reply to Michal Skrivanek from comment #26)
> ah, so you're complaining about the difference in the assigned device in UI
> and what you need to pass to the hook, right?

Yes. I must go into the CLI in order to know the parameter to set in the hook.

Unfortunately, It does work in case of USB and SCSI.
In each case I physically connected the device, Refreshed the host Caps, Pinned the VM to the host, Attached the device to the VM. 
For USB I did it using:
# vdsm-client Host hostdevListByCaps
Searching for the device id in the UI and getting the right address.
With SCSI device, I had no issues, the UI and the address were the same.

But, for PCI it didn't work.
The steps:
1. Enable SR-IOV on host
2. Create new network
# Network->Networks->New
# Give a name to the network
# Create
3. Set network profile
# Network->vNIC Profiles->Edit the network->Set Passthrough
# optional check the SR-IOV is working->set the VM NIC to the created profile
# check that the vm is starting with a NIC
4. Attach the SR-IOV device to the VM
# VM->Host Devices->Add device
# Add the SR-IOV device(pci_0000_09_10_1 in my case)
5. Set the hook
# VM->Edit->Custom Properties->Set boot_hostdev->Set pci device
6. Run VM

Results:
I tried to add more than one NIC, occured once with libvirt traceback which probably caused by adding the same NIC (physical and VF).
Also, I tried to run with pci_0000_09_10_1, pci_0000_08_00_1, pci_0000_08_00_0
And combinations of the devices passthrough from host.

None of these combination worked, I keep getting "Could not find device pci_0000_09_10_1".
Many times the VF disappeared from the host, probably hitting:
https://bugzilla.redhat.com/show_bug.cgi?id=1446058 

logs added including the vdsm-client Host hostdevListByCaps.

Comment 28 Liran Rotenberg 2018-07-29 13:38:55 UTC
I forgot to add the versions:
vdsm-4.20.35-1.el7ev.x86_64
vdsm-hook-boot_hostdev-4.20.35-1.el7ev.noarch
ovirt-engine-4.2.5.2-0.1.el7ev.noarch

Comment 29 Germano Veit Michel 2018-07-30 02:06:22 UTC
Michal and Liran,

Thanks for the tests.

Michal, I'm starting to agree with Liran that this is getting far from easy to use and there will always be cases where the hook doesnt find the device. As we discussed via email now we have an additional problem that the device name (coming from libvirt?) does not always map to the device address on the host. So parsing it to find the address of the device is useless and results in the hook failing. Also, on comment #27 I see Liran used a SR-IOV NIC VF to boot, the hook doesnt support VF, I would need to add it. I'm sure there will be more cases, parsing the device name has too many cases I think.

I'm thinking of simplifying this. What about the user sets the custom property to 'True' and the hook simply adds boot order to all hostdev devices, following this order:
1. USB 
2. NIC (Network/PXE)
3. SCSI (Disk)

So if a VM has a single boot disk, the boot order goes from this:

1. Hard Disk

to this

1. Pass-through USB
2. Pass-through NIC
3. Pass-through SCSI
4. Hard Disk

This way, we can stop parsing the device names to find the address, and also should fix the complaint by Liran that this is difficult to use.

What do you think?

Comment 30 Michal Skrivanek 2018-07-30 12:28:16 UTC
I'm not sure what has failed here. There was no requirement on comfort of usability, and when you use the right address it is working.

I agree there's quite some room for improvement, but that's not going to happen in 4.2.5, so please open a separate bug for that if you feel it's important enough.

Comment 31 Liran Rotenberg 2018-07-30 12:58:06 UTC
(In reply to Germano Veit Michel from comment #29)
 
> Michal, I'm starting to agree with Liran that this is getting far from easy
> to use and there will always be cases where the hook doesnt find the device.
> As we discussed via email now we have an additional problem that the device
> name (coming from libvirt?) does not always map to the device address on the
> host. So parsing it to find the address of the device is useless and results
> in the hook failing. Also, on comment #27 I see Liran used a SR-IOV NIC VF
> to boot, the hook doesnt support VF, I would need to add it. I'm sure there
> will be more cases, parsing the device name has too many cases I think.


In comment #0 and bug doc text:
Allow Virtual Machines to boot from passed through host devices such as NIC VFs, PCI-E SAS/RAID Cards, SCSI devices etc. Without requiring a normal bootable disk from RHV SD/Direct LUN.

Because of the NIC VFs I tried SR-IOV. If it doesn't support NIC VFs the doc of this bug need to be changed.

(In reply to Michal Skrivanek from comment #30)
> I'm not sure what has failed here. There was no requirement on comfort of
> usability, and when you use the right address it is working.
> 
> I agree there's quite some room for improvement, but that's not going to
> happen in 4.2.5, so please open a separate bug for that if you feel it's
> important enough.

Agreed about the comfort and a separate bug for it.
Along with my comment above about NIC VFs, I did try now again with a normal NIC connected to PXE. 
It's the same pci_0000_08_00_0 / pci_0000_08_00_1, without the VF.
I tried combination of those devices added to the VM(only one of them and both) and I tried each parameter set on the hook: pci_0000_08_00_0 and pci_0000_08_00_1 as it seems right by the vdsm-client Host hostdevListByCaps command.
I also tried just for the tryout their parent pci-e root port: pci_0000_00_1c_4.

The VM failed to start, getting device not found by the hook. For me PCI option doesn't work.

I hope that answers your question.

Comment 32 Michal Skrivanek 2018-07-30 14:00:05 UTC
ok. Up to Germano if he's fine with the current state (without SR-IOV). If PCI VFIO doesn't work either (better to add details and logs though, as it can be any other reason)
If the current state is not ok and you do not want to document it at all we can send this back to backlog.

Comment 33 Germano Veit Michel 2018-07-31 00:35:15 UTC
(In reply to Michal Skrivanek from comment #32)
> ok. Up to Germano if he's fine with the current state (without SR-IOV).
This was developed to a customer looking to pass a normal PCIE device (SCSI controller). I added usb and scsi as a "bonus". 

Adding support to SR-IOV devices needs another [simple] patch, currently the hook does not know how to parse pci_0000_00_1c_4, where the last digit should be the virtual function. It only knows pci_0000_00_1c. So any device with VF wont work as Liron found. The hook is parsing the names to find the source address, so this approach will always be open to missing some cases.

Maybe instead of adding the device name, the user should put the whole source dev string as seen on libvirt XML? The hook only compares and if the same set the boot flag. So we don't have to deal with parsing libvirt names and are not limited to device types.

> If the current state is not ok and you do not want to document it at all we
> can send this back to backlog.
Well, the customer waiting for this is also waiting for UEFI/Q35 support. So we are not in a rush at all.

So... to discuss:
1) Investigate why the device naming does not correspond to the address anymore. If its not supposed to, then the hook is hard to use.
2) Delay this to another 4.2.z or even 4.3. As stated above, the customer that wants to use this also needs other features not yet fully supported for his solution.
3) Think of another way to implement this to make it easier for the user (Liron's complaint).
4) Think about adding the sourcedev address in the custom property field, as seen in the VM XML. With this we remove the device name parsing to guess the address, simplifying the hook. Since the user already needs to find the address as Liron found, it won't make it harder to use and will make the hook much simpler.

Comment 36 Ryan Barry 2019-01-21 14:54:35 UTC
Re-targeting to 4.3.1 since it is missing a patch, an acked blocker flag, or both

Comment 38 Ryan Barry 2019-01-30 12:38:36 UTC
Germano, is this still relevant?

If it is, please clarify a little bit -- which passthrough devices (disk, PXE from NIC, passthrough from USB, etc) are intended to be supported?

Comment 40 Red Hat Bugzilla Rules Engine 2019-01-30 12:39:08 UTC
Development has indicated this request is declined. You may appeal this decision by reopening this request.

Comment 41 Germano Veit Michel 2019-01-30 23:10:40 UTC
(In reply to Ryan Barry from comment #38)
> Germano, is this still relevant?
> 
> If it is, please clarify a little bit -- which passthrough devices (disk,
> PXE from NIC, passthrough from USB, etc) are intended to be supported?

Hi Ryan,

The use case for the attached ticket is covered (pci hostdev, not SR-IOV). usb and scsi should generally work too.
The user needs to find the proper hostdev address from the XML, its not very user friendly but should work if setup correctly.

The current state is good enough for what the customer requested, even has some bonus features. We can always improve this further later if another customer needs something similar.


Note You need to log in before you can comment on or make changes to this bug.