RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2036669 - DEVICE_DELETED event is not delivered for device frontend if -device is configured via JSON
Summary: DEVICE_DELETED event is not delivered for device frontend if -device is confi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: qemu-kvm
Version: 9.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Kevin Wolf
QA Contact: Yiqian Wei
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-03 14:40 UTC by Peter Krempa
Modified: 2022-05-17 12:31 UTC (History)
23 users (show)

Fixed In Version: qemu-kvm-6.2.0-5.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-17 12:25:11 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Gitlab redhat/centos-stream/src qemu-kvm merge_requests 62 0 None None None 2022-01-19 14:07:10 UTC
Red Hat Issue Tracker RHELPLAN-106761 0 None None None 2022-01-03 14:41:47 UTC
Red Hat Product Errata RHBA-2022:2307 0 None None None 2022-05-17 12:25:55 UTC

Description Peter Krempa 2022-01-03 14:40:35 UTC 
Description of problem:
When -device is used with a JSON object as argument, and the device in question is then removed no DEVICE_DELETED event is emitted for such device.

Version-Release number of selected component (if applicable):
qemu-6.2 upstream release.

How reproducible:
always

Steps to Reproduce:
1. Configure a device via -device JSON. E.g.:

-device '{"driver":"e1000","netdev":"hostnet0","id":"net0","mac":"52:54:00:f1:cd:5f","bus":"pci.0","addr":"0xc"}' \

or

-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:f1:cd:5f","bus":"pci.0","addr":"0x3"}' \


2. Unplug the device via device_del:

{"execute":"device_del","arguments":{"id":"net0"},"id":"libvirt-424"}

3.

Actual results:

In case of e1000, no DEVICE_DELETED event is delivered at all. For virtio-net-pci, only the virtio-backend deletion event is delivered:

2022-01-03 13:49:10.661+0000: 3074909: info : qemuMonitorJSONIOProcessLine:236 : QEMU_MONITOR_RECV_EVENT: mon=0x7fff9c1c88b0 event={"timestamp": {"seconds": 1641217750, "microseconds": 661314}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/net0/virtio-backend"}}


Expected results:

DEVICE_DELETED events for the frontend are delivered:

With: -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:f1:cd:5f,bus=pci.0,addr=0xc

2022-01-03 14:19:19.586+0000: 3116343: info : qemuMonitorJSONIOProcessLine:236 : QEMU_MONITOR_RECV_EVENT: mon=0x7fff9c1c88b0 event={"timestamp": {"seconds": 1641219559, "microseconds": 586854}, "event": "DEVICE_DELETED", "data": {"device": "net0", "path": "/machine/peripheral/net0"}}


With: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:f1:cd:5f,bus=pci.0,addr=0x3 \

2022-01-03 13:53:36.926+0000: 3089522: info : qemuMonitorJSONIOProcessLine:236 : QEMU_MONITOR_RECV_EVENT: mon=0x7fff9c1c88b0 event={"timestamp": {"seconds": 1641218016, "microseconds": 926457}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/net0/virtio-backend"}}
2022-01-03 13:53:36.977+0000: 3089522: info : qemuMonitorJSONIOProcessLine:236 : QEMU_MONITOR_RECV_EVENT: mon=0x7fff9c1c88b0 event={"timestamp": {"seconds": 1641218016, "microseconds": 977140}, "event": "DEVICE_DELETED", "data": {"device": "net0", "path": "/machine/peripheral/net0"}}


Additional info:

The above applies for all devices including disks and others. Network devices were picked just as an example.


Note that after this is fixed, libvirt will require a new capability flag (either on 'device_add' or on 'DEVICE_DELETED') which will be used instead of 'json-cli' for detecting that we can really use the JSON syntax as for now we'll have to revert the support to make sure that libvirt works properly with qemu-6.2.
Comment 1 Peter Krempa 2022-01-03 14:41:24 UTC 
Kevin, please have a look at this one.
Comment 2 Peter Krempa 2022-01-03 15:43:04 UTC 
The original libvirt bug report: https://bugzilla.redhat.com/show_bug.cgi?id=2035237
Comment 4 Yanghang Liu 2022-01-04 03:23:15 UTC 
It's likely the same bug of Bug 2033279 - [wrb][qemu-kvm 6.2] The hot-unplugged device can not be hot-plugged back
Comment 7 Jing Qi 2022-01-10 05:44:42 UTC 
We met the similar issue with guest agent -

It can't really detach guest agent device in a live domain in rhel9

Version-Release number of selected component (if applicable):

libvirt-daemon-7.10.0-1.el9.x86_64

qemu-kvm-6.2.0-1.el9.x86_64

How reproducible:
100%



Steps to Reproduce:



1. Start a domain with guest agent  configuration, below is partial from  virsh dumpxml * - 

channel.xml

 <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-2-avocado-vt-vm1/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>

2. Detach guest agent device in live domain

virsh detach-device avocado-vt-vm1 channel.xml --live

Device detached successfully



3. virsh dumpxml again - 

 The output still have the device above.


4. Checked the log and there is "Device 'channel0' not found" error -
2022-01-10 04:42:23.428+0000: 506525: debug : qemuDomainObjBeginJobInternal:837 : Starting job: job=modify agentJob=none asyncJob=none (vm=0x7f6364090840 name=avocado-vt-vm1, current job=none agentJob=none async=none)
2022-01-10 04:42:23.428+0000: 506525: debug : qemuDomainObjBeginJobInternal:889 : Started job: modify (async=none vm=0x7f6364090840 name=avocado-vt-vm1)
2022-01-10 04:42:23.429+0000: 506525: debug : qemuDomainObjEnterMonitorInternal:5850 : Entering monitor (mon=0x7f6364083460 vm=0x7f6364090840 name=avocado-vt-vm1)
2022-01-10 04:42:23.429+0000: 506525: debug : qemuMonitorDelDevice:2837 : devalias=channel0
2022-01-10 04:42:23.429+0000: 506525: debug : qemuMonitorDelDevice:2839 : mon:0x7f6364083460 vm:0x7f6364090840 fd:26
2022-01-10 04:42:23.429+0000: 506525: info : qemuMonitorSend:952 : QEMU_MONITOR_SEND_MSG: mon=0x7f6364083460 msg={"execute":"device_del","arguments":{"id":"channel0"},"id":"libvirt-405"}^M
 fd=-1
2022-01-10 04:42:23.429+0000: 506707: info : qemuMonitorIOWrite:437 : QEMU_MONITOR_IO_WRITE: mon=0x7f6364083460 buf={"execute":"device_del","arguments":{"id":"channel0"},"id":"libvirt-405"}^M
 len=75 ret=75 errno=0
2022-01-10 04:42:23.430+0000: 506707: debug : qemuMonitorJSONIOProcessLine:222 : Line [{"id": "libvirt-405", "error": {"class": "DeviceNotFound", "desc": "Device 'channel0' not found"}}]
2022-01-10 04:42:23.430+0000: 506707: info : qemuMonitorJSONIOProcessLine:241 : QEMU_MONITOR_RECV_REPLY: mon=0x7f6364083460 reply={"id": "libvirt-405", "error": {"class": "DeviceNotFound", "desc": "Device 'channel0' not found"}}
2022-01-10 04:42:23.430+0000: 506525: debug : qemuDomainObjExitMonitorInternal:5875 : Exited monitor (mon=0x7f6364083460 vm=0x7f6364090840 name=avocado-vt-vm1)
2022-01-10 04:42:23.430+0000: 506525: debug : qemuDomainDeleteDevice:133 : Detaching of device channel0 failed and no event arrived
2022-01-10 04:42:28.430+0000: 506525: debug : qemuDomainObjEnterMonitorInternal:5850 : Entering monitor (mon=0x7f6364083460 vm=0x7f6364090840 name=avocado-vt-vm1)
2022-01-10 04:42:28.430+0000: 506525: debug : qemuMonitorGetDeviceAliases:3867 : aliases=0x7f63afd55790
2022-01-10 04:42:28.430+0000: 506525: debug : qemuMonitorGetDeviceAliases:3869 : mon:0x7f6364083460 vm:0x7f6364090840 fd:26
2022-01-10 04:42:28.430+0000: 506525: info : qemuMonitorSend:952 : QEMU_MONITOR_SEND_MSG: mon=0x7f6364083460 msg={"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-406"}^M
 fd=-1
2022-01-10 04:42:28.430+0000: 506707: info : qemuMonitorIOWrite:437 : QEMU_MONITOR_IO_WRITE: mon=0x7f6364083460 buf={"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-406"}^M
 len=86 ret=86 errno=0
Comment 9 Yanghang Liu 2022-01-26 16:09:13 UTC 
(In reply to Yanghang Liu from comment #4)
> It's likely the same bug of Bug 2033279 - [wrb][qemu-kvm 6.2] The hot-unplugged device can not be hot-plugged back


Hi Kevin,

It seems that I have open a same bug against RHEL8.6 -- Bug 2033279 - [wrb][qemu-kvm 6.2] The hot-unplugged device can not be hot-plugged back

Could you please help have a look at the above bug and fix this issue in RHEL8.6 as well ?
Comment 10 Yanan Fu 2022-01-27 10:30:37 UTC 
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.
Comment 11 Kevin Wolf 2022-01-27 11:48:16 UTC 
(In reply to Yanghang Liu from comment #9)
> Could you please help have a look at the above bug and fix this issue in
> RHEL8.6 as well ?

The description of bug 2033279 looks unrelated (no JSON -device, and DEVICE_DELETED is correctly delivered). However, this bug exists in 8.6, too. If you confirm that the description of the other bug is still accurate (i.e. it is indeed unrelated), you can clone this one instead.
Comment 16 errata-xmlrpc 2022-05-17 12:25:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: qemu-kvm), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2307
Note You need to log in before you can comment on or make changes to this bug.