RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1428893 - domain definition is not updated after a delayed VCPU unplug by the guest
Summary: domain definition is not updated after a delayed VCPU unplug by the guest
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: Luyao Huang
URL:
Whiteboard:
Depends On: 1427801
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-03 15:00 UTC by Peter Krempa
Modified: 2017-08-02 00:03 UTC (History)
7 users (show)

Fixed In Version: libvirt-3.2.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1427801
Environment:
Last Closed: 2017-08-01 17:24:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1846 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2017-08-01 18:02:50 UTC

Description Peter Krempa 2017-03-03 15:00:40 UTC
+++ This bug was initially created as a clone of Bug #1427801 +++

Description of problem:
After a VCPU unplug request times out initially the domain definition is not updated once the guest actually unplugs the vCPUS.

Afterwards further vCPU unplug requests fail with: 
internal error: unable to execute QEMU command 'device_del': Device 'vcpu8' not found

Version-Release number of selected component (if applicable):
Upstream libvirt compiled at
Repo: https://github.com/libvirt/libvirt.git 
Branch: master 
Commit: ca1f38545750d597c75c9773723c716483b03e5c
#libvirtd --version
libvirtd (libvirt) 3.1.0

Also applies to 3.2.0.el7.


How reproducible:
Always

Steps to Reproduce:
1. Define a guest(Fedora25) with current 8 vcpus and 240 max vcpus with following topology
2. virsh setvcpus virt-tests-vm1 16 --live
3. virsh setvcpus virt-tests-vm1 8 --live
error: operation failed: vcpu unplug request timed out

^^ vcpu unplug took long, the command timed out, the failure is expected

4. the guest actually unplugs the vcpus

5. virsh setvcpus virt-tests-vm1 8 --live
error: internal error: unable to execute QEMU command 'device_del': Device 'vcpu8' not found

^^ this should not happen

Actual results:
Cpu unplug  errors out leaving hotplug/unplug operation not functional once a vcpu unplug request times out.


Expected results:
Cpu hotplug/unplug should work fine

Additional info:
It may be necessary to somehow make the guest deny the vcpu unplug. I could not reproduce it.
Note that the bug is not ppc64 specific. The same would happen on intel if the first attempt was rejected.

Guest XML:
<domain type='kvm' id='5'>
  <name>virt-tests-vm1</name>
  <uuid>496d2299-4a36-4403-a583-6e07bec87070</uuid>
  <memory unit='KiB'>33554432</memory>
  <currentMemory unit='KiB'>33554432</currentMemory>
  <vcpu placement='static' current='8'>240</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='ppc64le' machine='pseries-2.9'>hvm</type>
    <boot dev='hd'/>
  </os>
  <cpu>
    <topology sockets='2' cores='15' threads='8'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/sath/runAvocadoFVTTest/avocado-fvt-wrapper/data/avocado-vt/images/f25-ppc64le.qcow2'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='pci-ohci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:61:62:63'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target port='0'/>
      <alias name='serial0'/>
      <address type='spapr-vio' reg='0x30000000'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
      <address type='spapr-vio' reg='0x30000000'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/f25ppc64le.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
    <panic model='pseries'/>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c62,c845</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c62,c845</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

Comment 2 Peter Krempa 2017-03-10 08:12:50 UTC
Upstream fixed by:

commit 8af68ea47830b8d32907dc50c6ca4869d14bb862
Author: Peter Krempa <pkrempa>
Date:   Fri Mar 3 16:04:57 2017 +0100

    qemu: hotplug: Reset device removal waiting code after vCPU unplug
    
    If the delivery of the DEVICE_DELETED event for the vCPU being deleted
    would time out, the code would not call 'qemuDomainResetDeviceRemoval'.
    
    Since the waiting thread did not unregister itself prior to stopping the
    waiting the monitor code would try to wake it up instead of dispatching
    it to the event worker. As a result the unplug process would not be
    completed and the definition would not be updated.

Comment 5 Luyao Huang 2017-05-10 02:37:42 UTC
Verify this bug with libvirt-3.2.0-4.el7.x86_64:

1. prepare a guest which have hotpluggable vcpu:

  <vcpu placement='static'>10</vcpu>
  <vcpus>
    <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='1' enabled='yes' hotpluggable='no' order='2'/>
    <vcpu id='2' enabled='yes' hotpluggable='no' order='3'/>
    <vcpu id='3' enabled='yes' hotpluggable='no' order='4'/>
    <vcpu id='4' enabled='yes' hotpluggable='no' order='5'/>
    <vcpu id='5' enabled='yes' hotpluggable='yes' order='6'/>
    <vcpu id='6' enabled='yes' hotpluggable='yes' order='7'/>
    <vcpu id='7' enabled='yes' hotpluggable='yes' order='8'/>
    <vcpu id='8' enabled='yes' hotpluggable='yes' order='9'/>
    <vcpu id='9' enabled='yes' hotpluggable='yes' order='10'/>


2. login guest and make a big workload for guest cpu:

IN GUEST:

# ./memcpueater

3. hot-unplug vcpu:

# virsh setvcpus r7 5
error: operation failed: vcpu unplug request timed out

4. recheck guest xml:

# virsh dumpxml r7

  <vcpus>
    <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='1' enabled='yes' hotpluggable='no' order='2'/>
    <vcpu id='2' enabled='yes' hotpluggable='no' order='3'/>
    <vcpu id='3' enabled='yes' hotpluggable='no' order='4'/>
    <vcpu id='4' enabled='yes' hotpluggable='no' order='5'/>
    <vcpu id='5' enabled='yes' hotpluggable='yes' order='6'/>
    <vcpu id='6' enabled='yes' hotpluggable='yes' order='7'/>
    <vcpu id='7' enabled='yes' hotpluggable='yes' order='8'/>
    <vcpu id='8' enabled='yes' hotpluggable='yes' order='9'/>
    <vcpu id='9' enabled='yes' hotpluggable='yes' order='10'/>
  </vcpus>

5. stop the cpu eater in guest and do hot-unplug again:

# virsh setvcpus r7 5


6. recheck guest xml:

# virsh dumpxml r7
...
  <vcpu placement='static' current='5'>10</vcpu>
  <vcpus>
    <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='1' enabled='yes' hotpluggable='no' order='2'/>
    <vcpu id='2' enabled='yes' hotpluggable='no' order='3'/>
    <vcpu id='3' enabled='yes' hotpluggable='no' order='4'/>
    <vcpu id='4' enabled='yes' hotpluggable='no' order='5'/>
    <vcpu id='5' enabled='no' hotpluggable='yes'/>
    <vcpu id='6' enabled='no' hotpluggable='yes'/>
    <vcpu id='7' enabled='no' hotpluggable='yes'/>
    <vcpu id='8' enabled='no' hotpluggable='yes'/>
    <vcpu id='9' enabled='no' hotpluggable='yes'/>
  </vcpus>

Comment 6 errata-xmlrpc 2017-08-01 17:24:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 7 errata-xmlrpc 2017-08-02 00:03:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846


Note You need to log in before you can comment on or make changes to this bug.