Created attachment 1499864 [details] logs Description of problem: Failed to hot unplug memory after memory hotplug. Version-Release number of selected component (if applicable): Engine Version: 4.3.0-0.0.el7 OS Version: RHEL - 7.6 - 4.el7 LIBVIRT Version: libvirt-4.5.0-10.el7 VDSM Version: vdsm-4.30.1-29.el7 How reproducible: 100% Steps to Reproduce: 1. Create VM with OS and run it 2. Hotplug memory to VM Actual results: hot unplug completed with error. Expected results: hot unplug completed fine. Additional info: 2018-11-01 10:17:11,218+0200 ERROR (jsonrpc/1) [api] FINISH hotunplugMemory error=Failed to hotunplug memory: {'reason': 'operation failed: unplug of device was rejected by the guest', 'vmId': 'dc5144fe-d4c6-4ef6-846c-0c24e2497e43'} (api:131) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 444, in hotunplugMemory return self.vm.hotunplugMemory(params) File "<string>", line 2, in hotunplugMemory File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 101, in method return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3389, in hotunplugMemory raise exception.HotunplugMemFailed(str(e), vmId=self.id) HotunplugMemFailed: Failed to hotunplug memory: {'reason': 'operation failed: unplug of device was rejected by the guest', 'vmId': 'dc5144fe-d4c6-4ef6-846c-0c24e2497e43'}
do nit file that as rhv bug
Can you please attach the qemu log? Maybe platform
as the message indicates the guest refused the unplug. Please do some troubleshooting on the guest side and/or provide logs. if you're not sure get the sosreport
Created attachment 1501250 [details] updated logs Contains vdsm log, qemu log, libvirt log, engine log and sosreport.
there are two memory devices: <memory model=\'dimm\'> <target> <size unit=\'KiB\'>131072</size> <node>0</node> </target> <alias name=\'ua-0addd216-72f7-4982-b6b6-2a81eba925ae\'/> <address type=\'dimm\' slot=\'0\' base=\'0x100000000\'/> </memory> <memory model=\'dimm\'> <target> <size unit=\'KiB\'>917504</size> <node>0</node> </target> <alias name=\'ua-d1eff842-8a1d-44ec-874c-b8e2a5f6e73b\'/> <address type=\'dimm\' slot=\'1\' base=\'0x108000000\'/> </memory> the 128MB block is fixed and cannot be removed, you can only unplug dimm1 896MB. However in the unplug call you're removing the 128 block: <memory model=dimm><target><size unit=KiB>131072</size><node>0</node></target><alias name=ua-0addd216-72f7-4982-b6b6-2a81eba925ae /></memory> hence it fails. Please retest with valid request.
I reset with a valid request and got the same error.
great, so logs for that?
Created attachment 1502411 [details] updated logs Contains vdsm log, qemu log, libvirt log, engine log and sosreport.
now you do plug 128MB, then immediately 896MB, and then you again try to hot unplug the 128 unremovable block by valid, I meant a reasonable valid request, e.g., plug in 1GB, and remove the slot with 896MB
ah, both should work actually. Thanks Milan for reminding me. Then we're down to the guest OS not letting go of the memory. That's perfectly ok for guest OS to do that if it wants to
we need logs from the guest. Please attach logs(or sosreport) from the guest as requested in comment #3
Created attachment 1503303 [details] updated logs
i see selinux violations on hot plug. Can you please try to set selinux to permissive in the guest *before* you do the initial hot plug, and then do unplug. And again gather the sosreport from guest to compare? Thanks!
also, please make sure you install ovirt-guest-agent
Hi Michal, I reproduced it again after I set selinux to permissive in the guest and checked that the ovirt-guest-agent is installed. [root@dhcp163-130 ~]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=permissive # SELINUXTYPE= can take one of three values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted [root@dhcp163-130 ~]# rpm -qa | grep guest qemu-guest-agent-2.12.0-2.el7.x86_64 ovirt-guest-agent-common-1.0.14-1.20180608132658.gitdcb03c3.el7.noarch
Created attachment 1504386 [details] updated logs
unfortunately it needs to be the el7 released ovirt-guest-agent, not a random master upstream build.
I changed the ovirt-guest-agent to the el7 release and the hot unplug worked as expected.
so, works like a charm:) There's one more test which might be very useful - on RHEL 8 guest this should work out of the box without the guest agent