Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1965140

Summary:	RFE: Support to define automatically start mdev device after reboot by 'virsh nodedev-define'
Product:	Red Hat Enterprise Linux 8	Reporter:	yafu <yafu>
Component:	libvirt	Assignee:	Jonathon Jongsma <jjongsma>
Status:	CLOSED ERRATA	QA Contact:	zhentang <zhetang>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.4	CC:	jdenemar, jsuchane, lmen, smitterl, virt-maint, xuzhang, yalzhang, zhguo
Target Milestone:	rc	Keywords:	FutureFeature, Triaged
Target Release:	---	Flags:	pm-rhel: mirror+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-7.8.0-1.module+el8.6.0+12978+7d7a0321	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2002761 (view as bug list)		Environment:
Last Closed:	2022-05-10 13:18:42 UTC	Type:	Feature Request
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:	7.8.0
Embargoed:
Bug Depends On:
Bug Blocks:	1980343, 2002761

Description yafu 2021-05-27 00:45:05 UTC

Description of problem:
RFE: Support to define automatically start mdev device after reboot by 'virsh nodedev-define'

Version-Release number of selected component (if applicable):
libvirt-daemon-7.3.0-1.module+el8.5.0+11004+f4810536.x86_64

How reproducible:
100%

Steps to Reproduce:
1.mdevctl can define mdev device automatically start after reboot:
mdevctl define --uuid=8e21b099-b7a0-4c79-ad9e-6744362e2bee --parent=0000:84:00.0 --type=nvidia-12 -a

2.#mdevctl list -d
8e21b099-b7a0-4c79-ad9e-6744362e2bee 0000:84:00.0 nvidia-12 auto

3.

Actual results:


Expected results:
Libvirt should also support define mdev device automatically start after reboot, for now, mdev device defined by 'virsh nodedev-define' is the inactive state after reboot.

Additional info:

Comment 2 Jonathon Jongsma 2021-05-27 13:20:21 UTC

In progress

Comment 3 smitterl 2021-06-01 11:08:25 UTC

I think it's worth mentioning that the documentation around this seems to need some help with, should we open another BZ for this?

I believe there's no documentation available about nodedev-define/undefine.
- nothing on manpage https://gitlab.com/libvirt/libvirt/-/blob/master/docs/manpages/virsh.rst#nodedev-commands
- nothing on official docs https://gitlab.com/libvirt/libvirt/-/blob/master/docs/formatnode.html.in

Actually, the official docs seem to explicitly mention that only nodedev-list and nodedev-dumpxml are supported for mdev types:
https://gitlab.com/libvirt/libvirt/-/blob/master/docs/drvnodedev.html.in#L206

"The support of mediated device's framework in libvirt's node device driver covers the following features:

list available mediated devices on the host (Since 3.4.0)
display device details (Since 3.4.0)"

The documentation then goes on to describe the CCW manual mediated device setup. However, not all steps are necessary anymore, the following snippet will define / start a device after the css device has been bound (directly or via driver override) to vfio_ccw:
# cat mdev.xml
<device>
  <name>mdev_c189ac13_b340_4caa_9c73_e18beb671df0</name>
  <path>/sys/devices/css0/0.0.26aa/c189ac13-b340-4caa-9c73-e18beb671df0</path>
  <parent>css_0_0_26aa</parent>
  <driver>
    <name>vfio_mdev</name>
  </driver>
  <capability type='mdev'>
    <type id='vfio_ccw-io'/>
    <uuid>c189ac13-b340-4caa-9c73-e18beb671df0</uuid>
    <iommuGroup number='0'/>
  </capability>
</device>
# virsh nodedev-list |grep mdev
(empty)
# virsh nodedev-define mdev.xml
Node device 'mdev_c189ac13_b340_4caa_9c73_e18beb671df0' defined from 'mdev.xml'
# virsh nodedev-list --inactive
mdev_c189ac13_b340_4caa_9c73_e18beb671df0
# virsh nodedev-start mdev_c189ac13_b340_4caa_9c73_e18beb671df0
Device mdev_c189ac13_b340_4caa_9c73_e18beb671df0 started
# virsh nodedev-list |grep mdev
mdev_c189ac13_b340_4caa_9c73_e18beb671df0

Comment 4 yafu 2021-06-15 07:07:08 UTC

(In reply to smitterl from comment #3)
> I think it's worth mentioning that the documentation around this seems to
> need some help with, should we open another BZ for this?
> 
> I believe there's no documentation available about nodedev-define/undefine.
> - nothing on manpage
> https://gitlab.com/libvirt/libvirt/-/blob/master/docs/manpages/virsh.
> rst#nodedev-commands
> - nothing on official docs
> https://gitlab.com/libvirt/libvirt/-/blob/master/docs/formatnode.html.in
> 
> Actually, the official docs seem to explicitly mention that only
> nodedev-list and nodedev-dumpxml are supported for mdev types:
> https://gitlab.com/libvirt/libvirt/-/blob/master/docs/drvnodedev.html.in#L206
> 
> "The support of mediated device's framework in libvirt's node device driver
> covers the following features:
> 
> list available mediated devices on the host (Since 3.4.0)
> display device details (Since 3.4.0)"
> 
> The documentation then goes on to describe the CCW manual mediated device
> setup. However, not all steps are necessary anymore, the following snippet
> will define / start a device after the css device has been bound (directly
> or via driver override) to vfio_ccw:
> # cat mdev.xml
> <device>
>   <name>mdev_c189ac13_b340_4caa_9c73_e18beb671df0</name>
>  
> <path>/sys/devices/css0/0.0.26aa/c189ac13-b340-4caa-9c73-e18beb671df0</path>
>   <parent>css_0_0_26aa</parent>
>   <driver>
>     <name>vfio_mdev</name>
>   </driver>
>   <capability type='mdev'>
>     <type id='vfio_ccw-io'/>
>     <uuid>c189ac13-b340-4caa-9c73-e18beb671df0</uuid>
>     <iommuGroup number='0'/>
>   </capability>
> </device>
> # virsh nodedev-list |grep mdev
> (empty)
> # virsh nodedev-define mdev.xml
> Node device 'mdev_c189ac13_b340_4caa_9c73_e18beb671df0' defined from
> 'mdev.xml'
> # virsh nodedev-list --inactive
> mdev_c189ac13_b340_4caa_9c73_e18beb671df0
> # virsh nodedev-start mdev_c189ac13_b340_4caa_9c73_e18beb671df0
> Device mdev_c189ac13_b340_4caa_9c73_e18beb671df0 started
> # virsh nodedev-list |grep mdev
> mdev_c189ac13_b340_4caa_9c73_e18beb671df0

Hi Sebastian, 

There is existing bug for the man page:
Bug 1964223 - No info in 'man virsh' about 'virsh nodedev-define/nodedev-undefine', 'virsh nodedev-start', 'virsh nodedev-list --inactive/--all'

Comment 5 John Ferlan 2021-08-10 15:18:43 UTC

There's been no recent update, moving from 8.5.0 to the backlog to be re-evaluated during 8.6.0 planning

Comment 6 John Ferlan 2021-09-09 16:11:58 UTC

Bulk update: Move RHEL-AV bugs to RHEL8 with existing RHEL9 clone.

NB: Created the RHEL9 clone because I see some vfio_ccw bugs related to this one and since those need only to be resolved for RHEL8, this bug needs to stay in RHEL8.

Comment 7 Jonathon Jongsma 2021-09-15 15:47:41 UTC

Support for autostarting mediated devices has been merged upstream and will be included in the upcoming libvirt 7.8.0 release. These are the relevant commits:

0cb4550305 virsh: add nodedev-info
24ffeef09c nodedev: Implement virNodeDeviceIsPersistent()/IsActive()
6d0b85d792 api: add virNodeDeviceIsPersistent()/IsActive()
16d078eded virsh: add nodedev-autostart
592031a580 nodedev: Add tests for mdevctl autostart command
dc60f91530 nodedev: implement virNodeDevice(Get|Set)Autostart()
c6607a25b9 api: add virNodeDevice(Get|Set)Autostart()

Comment 8 smitterl 2021-10-04 08:42:29 UTC

Jonathon, do you happen to know more or less when this will become available downstream for 9.0 / 8.6? I need this item to plan for depending TestOnly items. Thank you!

Comment 9 Jonathon Jongsma 2021-10-04 16:44:19 UTC

7.8.0 was released upstream on friday October 1. I don't know exactly when it will be packaged downstream. Jiri?

Comment 14 zhentang 2021-11-12 08:32:51 UTC

tested on libvirt-7.9.0-1.module+el8.6.0+13150+28339563

--------- Define mdev device and autostart ------------

# vi mdev.xml
<device>
  <name>mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0</name>
  <path>/sys/devices/pci0000:80/0000:80:02.0/0000:82:00.0/0000:83:08.0/0000:84:00.0/8e21b099-b7a0-4c79-ad9e-6744362e2bee</path>
  <parent>pci_0000_84_00_0</parent>
  <driver>
    <name>vfio_mdev</name>
  </driver>
  <capability type='mdev'>
    <type id='nvidia-12'/>
    <uuid>8e21b099-b7a0-4c79-ad9e-6744362e2bee</uuid>
    <iommuGroup number='0'/>
  </capability>
</device>

# virsh nodedev-define mdev.xml
Node device 'mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0' defined from 'mdev.xml'

# virsh nodedev-info mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0 
Name:           mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0
Parent:         pci_0000_84_00_0
Active:         yes
Persistent:     yes
Autostart:      no

# virsh nodedev-autostart mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0

# virsh nodedev-info mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0 
Name:           mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0
Parent:         pci_0000_84_00_0
Active:         yes
Persistent:     yes
Autostart:      yes

# mdevctl list -d
8e21b099-b7a0-4c79-ad9e-6744362e2bee 0000:84:00.0 nvidia-12 auto (active)

# reboot

# mdevctl list -d
8e21b099-b7a0-4c79-ad9e-6744362e2bee 0000:84:00.0 nvidia-12 auto (active)
# virsh nodedev-info mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0 
Name:           mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0
Parent:         pci_0000_84_00_0
Active:         yes
Persistent:     yes
Autostart:      yes

-------------------------------disable autostart-----------------
# virsh nodedev-autostart --disable mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0 
Device mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0 unmarked as autostarted

# virsh nodedev-info mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0 
Name:           mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0
Parent:         pci_0000_84_00_0
Active:         yes
Persistent:     yes
Autostart:      no

# reboot 

# virsh nodedev-info mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0 
Name:           mdev_8e21b099_b7a0_4c79_ad9e_6744362e2bee_0000_84_00_0
Parent:         pci_0000_84_00_0
Active:         no
Persistent:     yes
Autostart:      no

# mdevctl list -d
8e21b099-b7a0-4c79-ad9e-6744362e2bee 0000:84:00.0 nvidia-12 manual

Comment 16 errata-xmlrpc 2022-05-10 13:18:42 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1759