Bug 2141364
| Summary: | libvirt doesn't catch mdevs created thru sysfs [rhel-9.1.0.z] | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | RHEL Program Management Team <pgm-rhel-tools> |
| Component: | libvirt | Assignee: | Jonathon Jongsma <jjongsma> |
| libvirt sub component: | General | QA Contact: | zhentang <zhetang> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | bdobreli, bsawyers, camorris, chhu, dyuan, egallen, fjin, gveitmic, jdenemar, jsuchane, kchamart, lmen, smooney, virt-maint, xuzhang, yafu, ymankad, zhetang |
| Version: | 9.0 | Keywords: | Regression, Triaged, ZStream |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | libvirt-8.5.0-7.1.el9_1 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 2109450 | Environment: | |
| Last Closed: | 2023-01-23 15:18:08 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2109450 | ||
| Bug Blocks: | |||
|
Comment 1
Jonathon Jongsma
2022-11-10 20:39:11 UTC
Tested on on OSP17.0 with libvirt packages:
[heat-admin@compute-0 ~]$ sudo podman exec -it nova_virtqemud rpm -qa | grep libvirt| grep driver
libvirt-daemon-driver-nwfilter-8.5.0-7.1.el9_1.x86_64
libvirt-daemon-driver-qemu-8.5.0-7.1.el9_1.x86_64
libvirt-daemon-driver-storage-core-8.5.0-7.1.el9_1.x86_64
libvirt-daemon-driver-nodedev-8.5.0-7.1.el9_1.x86_64
libvirt-daemon-driver-secret-8.5.0-7.1.el9_1.x86_64
Test steps:
1. Prepare the vGPU environment on OSP17.0 with RHEL9.1 libvirt,qemu-kvm,kernel
(undercloud) [stack@dell-per740-66 ~]$ ssh heat-admin.24.19
[heat-admin@compute-0 ~]$ lspci|grep VGA
03:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller (rev 04)
3d:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
3e:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
[heat-admin@compute-0 ~]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/name
GRID M60-2Q
[heat-admin@compute-0 ~]$ uuid=$(uuidgen)
[heat-admin@compute-0 ~]$ cd /sys/class/mdev_bus/0000:3d:00.0/mdev_supported_types/nvidia-18
[heat-admin@compute-0 nvidia-18]$ sudo chmod 666 create
[heat-admin@compute-0 nvidia-18]$ sudo echo $uuid
10a750fe-e839-481b-96d5-54e9d3eaa786
[heat-admin@compute-0 nvidia-18]$ sudo echo $uuid > create
[heat-admin@compute-0 nvidia-18]$ ls ../../| grep $uuid
10a750fe-e839-481b-96d5-54e9d3eaa786
2. Check in nova_virtqemud, mdev device is present in the list of node devices
(undercloud) [stack@dell-per740-66 ~]$ ssh heat-admin.24.19
[heat-admin@compute-0 ~]$ sudo podman exec -it nova_virtqemud python3
Python 3.9.10 (main, Feb 9 2022, 00:00:00)
[GCC 11.2.1 20220127 (Red Hat 11.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>> conn = libvirt.open('qemu:///system')
>>> conn.listDevices('mdev')
['mdev_10a750fe_e839_481b_96d5_54e9d3eaa786_0000_3d_00_0']
Add more test results: 3. Delete the mdev device by using uuid, check the available_instances and `virsh nodedev-list` outputs are correct [heat-admin@compute-0 nvidia-18]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/available_instances 3 [heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-list --cap mdev mdev_10a750fe_e839_481b_96d5_54e9d3eaa786_0000_3d_00_0 [heat-admin@compute-0 nvidia-18]$ sudo chmod 666 /sys/bus/mdev/devices/10a750fe-e839-481b-96d5-54e9d3eaa786/remove [heat-admin@compute-0 nvidia-18]$ sudo echo 1 > /sys/bus/mdev/devices/10a750fe-e839-481b-96d5-54e9d3eaa786/remove [heat-admin@compute-0 nvidia-18]$ sudo ls /sys/class/mdev_bus/0000:3d:00.0| grep $uuid [heat-admin@compute-0 nvidia-18]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/available_instances 4 [heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-list --cap mdev No output Checking in nova_virtqemud: [heat-admin@compute-0 ~]$ sudo podman exec -it nova_virtqemud python3 Python 3.9.10 (main, Feb 9 2022, 00:00:00) [GCC 11.2.1 20220127 (Red Hat 11.2.1-9)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import libvirt >>> conn = libvirt.open('qemu:///system') >>> conn.listDevices('mdev') [] 4. Create the mdev device by virsh commands, check the `virsh nodedev-create,list,dumpxml` outputs are correct [heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud cat mdev.xml <device> <parent>pci_0000_3d_00_0</parent> <capability type='mdev'> <type id='nvidia-18'/> <uuid>c71395b9-0484-46af-9f01-7b00edfe5038</uuid> </capability> </device> [heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-create mdev.xml Node device mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0 created from mdev.xml [heat-admin@compute-0 nvidia-18]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/available_instances 3 [heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-list --cap mdev mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0 [heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-dumpxml mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0 <device> <name>mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0</name> <path>/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/0000:3c:08.0/0000:3d:00.0/c71395b9-0484-46af-9f01-7b00edfe5038</path> <parent>pci_0000_3d_00_0</parent> <driver> <name>nvidia-vgpu-vfio</name> </driver> <capability type='mdev'> <type id='nvidia-18'/> <uuid>c71395b9-0484-46af-9f01-7b00edfe5038</uuid> <parent_addr>0000:3d:00.0</parent_addr> <iommuGroup number='138'/> </capability> </device> Checking in nova_virtqemud: >>> conn.listDevices('mdev') ['mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0'] 5. Delete the mdev device by virsh commands, check the `virsh nodedev-destroy,list` outputs are correct [heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-destroy mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0 Destroyed node device 'mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0' [heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-list --cap mdev [heat-admin@compute-0 nvidia-18]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/available_instances 4 Checking in nova_virtqemud: >>> conn.listDevices('mdev') [] 6. Create the max number of available_instances mdev devices, check in nova_virtqemud, all the mdev devices are present in the list of node devices [heat-admin@compute-0 nvidia-18]$ sudo ls /sys/class/mdev_bus/0000:3d:00.0 0dd92cbb-07c1-4739-8832-f31c27cd1147 broken_parity_status device iommu_group modalias reset resource5 8ac81344-447e-46b3-bfc5-4ef03e4aa742 class dma_mask_bits irq msi_bus reset_method revision 8ed4d103-deee-464e-977d-b169617ff672 config driver link msi_irqs resource subsystem ...... boot_vga d4da684d-1f63-4916-937a-1d9e12525baf [heat-admin@compute-0 nvidia-18]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/available_instances 0 [heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-list --cap mdev mdev_0dd92cbb_07c1_4739_8832_f31c27cd1147_0000_3d_00_0 mdev_8ac81344_447e_46b3_bfc5_4ef03e4aa742_0000_3d_00_0 mdev_8ed4d103_deee_464e_977d_b169617ff672_0000_3d_00_0 mdev_d4da684d_1f63_4916_937a_1d9e12525baf_0000_3d_00_0 Check in virtqemud: >>> conn.listDevices('mdev') ['mdev_8ed4d103_deee_464e_977d_b169617ff672_0000_3d_00_0', 'mdev_8ac81344_447e_46b3_bfc5_4ef03e4aa742_0000_3d_00_0', 'mdev_0dd92cbb_07c1_4739_8832_f31c27cd1147_0000_3d_00_0', 'mdev_d4da684d_1f63_4916_937a_1d9e12525baf_0000_3d_00_0'] 7. Delete all the mdev devices by `echo 1 > /sys/bus/mdev/devices/***/remove` check the available_instances and `virsh nodedev-list` outputs are correct [heat-admin@compute-0 0000:3d:00.0]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/available_instances 4 [heat-admin@compute-0 0000:3d:00.0]$ sudo podman exec -it nova_virtqemud virsh nodedev-list --cap mdev No output Check in virtqemud: >>> conn.listDevices('mdev') [] Tested on on OSP17.0 with libvirt packages:
[heat-admin@compute-0 ~]$ sudo podman exec -it nova_virtqemud rpm -qa | grep libvirt| grep driver
libvirt-daemon-driver-nwfilter-8.5.0-7.1.el9_1.x86_64
libvirt-daemon-driver-qemu-8.5.0-7.1.el9_1.x86_64
libvirt-daemon-driver-storage-core-8.5.0-7.1.el9_1.x86_64
libvirt-daemon-driver-nodedev-8.5.0-7.1.el9_1.x86_64
libvirt-daemon-driver-secret-8.5.0-7.1.el9_1.x86_64
some udevadm monitor output here just for reference
1.Created mdev using virsh command
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud cat mdev.xml
<device>
<parent>pci_0000_3d_00_0</parent>
<capability type='mdev'>
<type id='nvidia-18'/>
<uuid>c71395b9-0484-46af-9f01-7b00edfe5038</uuid>
</capability>
</device>
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-create mdev.xml
Node device mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0 created from mdev.xml
on another tab
[heat-admin@compute-0 ~]$ udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent
KERNEL[516975.794758] add /devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/0000:3c:08.0/0000:3d:00.0/c71395b9-0484-46af-9f01-7b00edfe5038 (mdev)
UDEV [516975.797323] add /devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/0000:3c:08.0/0000:3d:00.0/c71395b9-0484-46af-9f01-7b00edfe5038 (mdev)
KERNEL[516975.899282] add /devices/virtual/vfio/139 (vfio)
KERNEL[516975.899332] bind /devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/0000:3c:08.0/0000:3d:00.0/c71395b9-0484-46af-9f01-7b00edfe5038 (mdev)
UDEV [516975.900515] bind /devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/0000:3c:08.0/0000:3d:00.0/c71395b9-0484-46af-9f01-7b00edfe5038 (mdev)
UDEV [516975.904917] add /devices/virtual/vfio/139 (vfio)
2. virsh nodedev related commands
scenario:
undefine mdev and create with same xml
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-info mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0
Name: mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0
Parent: pci_0000_3d_00_0
Active: yes
Persistent: no
Autostart: no
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-destroy mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0
Destroyed node device 'mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0'
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-create mdev.xml
Node device mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0 created from mdev.xml
scenario:
define then create mdev use virsh nodedev-cmd
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-define mdev.xml
Node device 'mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0' defined from 'mdev.xml'
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-create mdev.xml
Node device mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0 created from mdev.xml
[heat-admin@compute-0 ~]$ udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent
KERNEL[521083.705462] add /devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/0000:3c:08.0/0000:3d:00.0/c71395b9-0484-46af-9f01-7b00edfe5038 (mdev)
UDEV [521083.708322] add /devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/0000:3c:08.0/0000:3d:00.0/c71395b9-0484-46af-9f01-7b00edfe5038 (mdev)
KERNEL[521083.810082] add /devices/virtual/vfio/138 (vfio)
KERNEL[521083.810117] bind /devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/0000:3c:08.0/0000:3d:00.0/c71395b9-0484-46af-9f01-7b00edfe5038 (mdev)
UDEV [521083.811261] bind /devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/0000:3c:08.0/0000:3d:00.0/c71395b9-0484-46af-9f01-7b00edfe5038 (mdev)
UDEV [521083.815385] add /devices/virtual/vfio/138 (vfio)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (libvirt bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:0311 |