RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2141365 - libvirt doesn't catch mdevs created thru sysfs [rhel-9.0.0.z]
Summary: libvirt doesn't catch mdevs created thru sysfs [rhel-9.0.0.z]
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: libvirt
Version: 9.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Jonathon Jongsma
QA Contact: zhentang
URL:
Whiteboard:
Depends On: 2109450
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-11-09 15:37 UTC by RHEL Program Management Team
Modified: 2022-12-13 16:22 UTC (History)
18 users (show)

Fixed In Version: libvirt-8.0.0-8.2.el9_0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2109450
Environment:
Last Closed: 2022-12-13 16:10:06 UTC
Type: ---
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-138854 0 None None None 2022-11-09 16:11:18 UTC
Red Hat Product Errata RHBA-2022:8982 0 None None None 2022-12-13 16:10:10 UTC

Comment 1 Jonathon Jongsma 2022-11-10 20:35:45 UTC 
posted for review: https://gitlab.com/redhat/rhel/src/libvirt/-/merge_requests/51
Comment 2 chhu 2022-11-18 02:57:40 UTC 
Tested on on OSP17.0 with libvirt packages:
[heat-admin@compute-0 ~]$ sudo podman exec -it nova_virtqemud rpm -qa | grep libvirt| grep driver
libvirt-daemon-driver-nwfilter-8.0.0-8.2.el9_0.x86_64
libvirt-daemon-driver-nodedev-8.0.0-8.2.el9_0.x86_64
libvirt-daemon-driver-qemu-8.0.0-8.2.el9_0.x86_64
libvirt-daemon-driver-secret-8.0.0-8.2.el9_0.x86_64
libvirt-daemon-driver-storage-core-8.0.0-8.2.el9_0.x86_64

Test steps:
1. Prepare the vGPU environment on OSP17.0
(undercloud) [stack@dell-per740-66 ~]$ ssh heat-admin.24.23
[heat-admin@compute-0 ~]$ lspci|grep VGA
03:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller (rev 04)
3d:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
3e:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
[heat-admin@compute-0 ~]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/name
GRID M60-2Q
[heat-admin@compute-0 ~]$ uuid=$(uuidgen)
[heat-admin@compute-0 ~]$ cd /sys/class/mdev_bus/0000:3d:00.0/mdev_supported_types/nvidia-18
[heat-admin@compute-0 nvidia-18]$ sudo chmod 666 create
[heat-admin@compute-0 nvidia-18]$ sudo echo $uuid
2890cda7-21d3-4106-acee-c238004966b8
[heat-admin@compute-0 nvidia-18]$ sudo echo $uuid > create
[heat-admin@compute-0 nvidia-18]$ ls ../../| grep $uuid
2890cda7-21d3-4106-acee-c238004966b8

2. Check in nova_virtqemud, mdev device is present in the list of node devices
(undercloud) [stack@dell-per740-66 ~]$ ssh heat-admin.24.23
[heat-admin@compute-0 ~]$ sudo podman exec -it nova_virtqemud python3
Python 3.9.10 (main, Feb  9 2022, 00:00:00) 
[GCC 11.2.1 20220127 (Red Hat 11.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>> conn = libvirt.open('qemu:///system')
>>> conn.listDevices('mdev')
['mdev_2890cda7_21d3_4106_acee_c238004966b8_0000_3d_00_0']
Comment 3 chhu 2022-11-18 03:12:32 UTC 
Add more test results:
3. Delete the mdev device by using uuid,
   check the available_instances and `virsh nodedev-list` outputs are correct
[heat-admin@compute-0 nvidia-18]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/available_instances
3
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-list --cap mdev
mdev_2890cda7_21d3_4106_acee_c238004966b8_0000_3d_00_0

[heat-admin@compute-0 nvidia-18]$ sudo chmod 666 /sys/bus/mdev/devices/2890cda7-21d3-4106-acee-c238004966b8/remove
[heat-admin@compute-0 nvidia-18]$ sudo echo 1 > /sys/bus/mdev/devices/2890cda7-21d3-4106-acee-c238004966b8/remove
[heat-admin@compute-0 nvidia-18]$ sudo ls /sys/class/mdev_bus/0000:3d:00.0| grep $uuid
[heat-admin@compute-0 nvidia-18]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/available_instances
4
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-list --cap mdev
No output

Checking in nova_virtqemud:
[heat-admin@compute-0 ~]$ sudo podman exec -it nova_virtqemud python3
>>> import libvirt
>>> conn = libvirt.open('qemu:///system')
>>> conn.listDevices('mdev')
[]

4. Create the mdev device by virsh commands,
   check the `virsh nodedev-create,list,dumpxml` outputs are correct
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud cat mdev.xml
<device>
  <parent>pci_0000_3d_00_0</parent>
  <capability type='mdev'>
    <type id='nvidia-18'/>
    <uuid>c71395b9-0484-46af-9f01-7b00edfe5038</uuid>
  </capability>
</device>
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-create mdev.xml
Node device mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0 created from mdev.xml

[heat-admin@compute-0 nvidia-18]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/available_instances 
3
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-list --cap mdev
mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0

[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-dumpxml mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0
<device>
  <name>mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0</name>
  <path>/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/0000:3c:08.0/0000:3d:00.0/c71395b9-0484-46af-9f01-7b00edfe5038</path>
  <parent>pci_0000_3d_00_0</parent>
  <driver>
    <name>nvidia-vgpu-vfio</name>
  </driver>
  <capability type='mdev'>
    <type id='nvidia-18'/>
    <uuid>c71395b9-0484-46af-9f01-7b00edfe5038</uuid>
    <iommuGroup number='138'/>
  </capability>
</device>

Checking in nova_virtqemud:
>>> conn.listDevices('mdev')
['mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0']

5. Delete the mdev device by virsh commands,
   check the `virsh nodedev-destroy,list` outputs are correct
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-destroy mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0
Destroyed node device 'mdev_c71395b9_0484_46af_9f01_7b00edfe5038_0000_3d_00_0'
[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-list --cap mdev
No output
[heat-admin@compute-0 nvidia-18]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/available_instances 
4
Checking in nova_virtqemud:
>>> conn.listDevices('mdev')
[]
Comment 4 chhu 2022-11-18 03:20:22 UTC 
6. Create the max number of available_instances mdev devices,
   check in nova_virtqemud, all the mdev devices are present in the list of node devices
[heat-admin@compute-0 nvidia-18]$ ls ../../
0ba71129-5db5-40a3-8c0d-a0b8ca15fccb  broken_parity_status                  device           iommu_group           modalias     reset         resource5
8b3cd491-f863-41f3-b4f9-9a0969fdf564  c86e9994-5f0d-4941-8efd-e4d41a2b3c0a  dma_mask_bits    irq                   msi_bus      reset_method  revision
8e3454b5-f19e-4447-88c2-73eab11f8797  class ......

[heat-admin@compute-0 nvidia-18]$ cat /sys/class/mdev_bus/0000\:3d\:00.0/mdev_supported_types/nvidia-18/available_instances 
0

[heat-admin@compute-0 ~]$ sudo podman exec -it nova_virtqemud python3
Python 3.9.10 (main, Feb  9 2022, 00:00:00) 
[GCC 11.2.1 20220127 (Red Hat 11.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>> conn = libvirt.open('qemu:///system')
>>> conn.listDevices('mdev')
['mdev_c86e9994_5f0d_4941_8efd_e4d41a2b3c0a_0000_3d_00_0', 'mdev_8b3cd491_f863_41f3_b4f9_9a0969fdf564_0000_3d_00_0', 'mdev_0ba71129_5db5_40a3_8c0d_a0b8ca15fccb_0000_3d_00_0', 'mdev_8e3454b5_f19e_4447_88c2_73eab11f8797_0000_3d_00_0']

[heat-admin@compute-0 nvidia-18]$ sudo podman exec -it nova_virtqemud virsh nodedev-list --cap mdev
mdev_0ba71129_5db5_40a3_8c0d_a0b8ca15fccb_0000_3d_00_0
mdev_8b3cd491_f863_41f3_b4f9_9a0969fdf564_0000_3d_00_0
mdev_8e3454b5_f19e_4447_88c2_73eab11f8797_0000_3d_00_0
mdev_c86e9994_5f0d_4941_8efd_e4d41a2b3c0a_0000_3d_00_0
Comment 12 errata-xmlrpc 2022-12-13 16:10:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (libvirt bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8982
Note You need to log in before you can comment on or make changes to this bug.