Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1645022

Summary: Failed to hot unplug memory
Product: Red Hat Enterprise Virtualization Manager Reporter: Pedut <pchocron>
Component: ovirt-engineAssignee: Milan Zamazal <mzamazal>
Status: CLOSED NOTABUG QA Contact: Pedut <pchocron>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: mavital, michal.skrivanek, pchocron, ratamir, rbarry, Rhev-m-bugs
Target Milestone: ovirt-4.3.0Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-06 13:02:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1496395    
Attachments:
Description Flags
logs
none
updated logs
none
updated logs
none
updated logs
none
updated logs none

Description Pedut 2018-11-01 09:02:22 UTC
Created attachment 1499864 [details]
logs

Description of problem:
Failed to hot unplug memory after memory hotplug. 

Version-Release number of selected component (if applicable):
Engine Version: 4.3.0-0.0.el7
OS Version: RHEL - 7.6 - 4.el7
LIBVIRT Version: libvirt-4.5.0-10.el7
VDSM Version: vdsm-4.30.1-29.el7

How reproducible:
100%

Steps to Reproduce:
1. Create VM with OS and run it
2. Hotplug memory to VM

Actual results:
hot unplug completed with error.

Expected results:
hot unplug completed fine.

Additional info:
2018-11-01 10:17:11,218+0200 ERROR (jsonrpc/1) [api] FINISH hotunplugMemory error=Failed to hotunplug memory: {'reason': 'operation failed: unplug of device was rejected by the guest', 'vmId': 'dc5144fe-d4c6-4ef6-846c-0c24e2497e43'} (api:131)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 444, in hotunplugMemory
    return self.vm.hotunplugMemory(params)
  File "<string>", line 2, in hotunplugMemory
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 101, in method
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3389, in hotunplugMemory
    raise exception.HotunplugMemFailed(str(e), vmId=self.id)
HotunplugMemFailed: Failed to hotunplug memory: {'reason': 'operation failed: unplug of device was rejected by the guest', 'vmId': 'dc5144fe-d4c6-4ef6-846c-0c24e2497e43'}

Comment 1 Michal Skrivanek 2018-11-02 05:37:08 UTC
do nit file that as rhv bug

Comment 2 Ryan Barry 2018-11-02 10:27:33 UTC
Can you please attach the qemu log? Maybe platform

Comment 3 Michal Skrivanek 2018-11-02 12:57:14 UTC
as the message indicates the guest refused the unplug. Please do some troubleshooting on the guest side and/or provide logs. if you're not sure get the sosreport

Comment 4 Pedut 2018-11-04 11:25:57 UTC
Created attachment 1501250 [details]
updated logs

Contains vdsm log, qemu log, libvirt log, engine log and sosreport.

Comment 5 Michal Skrivanek 2018-11-06 07:58:57 UTC
there are two memory devices:
    <memory model=\'dimm\'>
      <target>
        <size unit=\'KiB\'>131072</size>
        <node>0</node>
      </target>
      <alias name=\'ua-0addd216-72f7-4982-b6b6-2a81eba925ae\'/>
      <address type=\'dimm\' slot=\'0\' base=\'0x100000000\'/>
    </memory>
    <memory model=\'dimm\'>
      <target>
        <size unit=\'KiB\'>917504</size>
        <node>0</node>
      </target>
      <alias name=\'ua-d1eff842-8a1d-44ec-874c-b8e2a5f6e73b\'/>
      <address type=\'dimm\' slot=\'1\' base=\'0x108000000\'/>
    </memory>

the 128MB block is fixed and cannot be removed, you can only unplug dimm1 896MB.
However in the unplug call you're removing the 128 block:
<memory model=dimm><target><size unit=KiB>131072</size><node>0</node></target><alias name=ua-0addd216-72f7-4982-b6b6-2a81eba925ae /></memory>

hence it fails. Please retest with valid request.

Comment 6 Pedut 2018-11-06 09:58:14 UTC
I reset with a valid request and got the same error.

Comment 7 Michal Skrivanek 2018-11-06 10:08:40 UTC
great, so logs for that?

Comment 8 Pedut 2018-11-06 11:40:58 UTC
Created attachment 1502411 [details]
updated logs

Contains vdsm log, qemu log, libvirt log, engine log and sosreport.

Comment 9 Michal Skrivanek 2018-11-06 11:48:30 UTC
now you do plug 128MB, then immediately 896MB, and then you again try to hot unplug the 128 unremovable block

by valid, I meant a reasonable valid request, e.g., plug in 1GB, and remove the slot with 896MB

Comment 10 Michal Skrivanek 2018-11-06 12:56:49 UTC
ah, both should work actually. Thanks Milan for reminding me. Then we're down to the guest OS not letting go of the memory. That's perfectly ok for guest OS to do that if it wants to

Comment 11 Michal Skrivanek 2018-11-06 12:59:33 UTC
we need logs from the guest. Please attach logs(or sosreport) from the guest as requested in comment #3

Comment 12 Pedut 2018-11-08 11:33:03 UTC
Created attachment 1503303 [details]
updated logs

Comment 13 Michal Skrivanek 2018-11-08 13:25:28 UTC
i see selinux violations on hot plug. Can you please try to set selinux to permissive in the guest *before* you do the initial hot plug, and then do unplug. And again gather the sosreport from guest to compare? Thanks!

Comment 14 Michal Skrivanek 2018-11-08 15:17:33 UTC
also, please make sure you install ovirt-guest-agent

Comment 15 Pedut 2018-11-11 13:45:58 UTC
Hi Michal,

I reproduced it again after I set selinux to permissive in the guest and checked that the ovirt-guest-agent is installed.

[root@dhcp163-130 ~]# cat /etc/selinux/config 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted


[root@dhcp163-130 ~]# rpm -qa | grep guest
qemu-guest-agent-2.12.0-2.el7.x86_64
ovirt-guest-agent-common-1.0.14-1.20180608132658.gitdcb03c3.el7.noarch

Comment 16 Pedut 2018-11-11 14:03:54 UTC
Created attachment 1504386 [details]
updated logs

Comment 17 Michal Skrivanek 2018-11-12 08:31:57 UTC
unfortunately it needs to be the el7 released ovirt-guest-agent, not a random master upstream build.

Comment 18 Pedut 2018-11-12 11:05:40 UTC
I changed the ovirt-guest-agent to the el7 release and the hot unplug worked as expected.

Comment 19 Michal Skrivanek 2018-11-12 11:25:47 UTC
so, works like a charm:)
There's one more test which might be very useful - on RHEL 8 guest this should work out of the box without the guest agent