Created attachment 1717506 [details]
engine log

Description of problem:
When request engine to HotUnplug multiple memory devices in given size, such as 15x256M, it unplugs the same one 256M memory device 15 times, but not all the 15 memory devices. 
Each time engine reports Hotunplug of memory device(with the same id) was successfully requested on VM, memory size updated to desired value, though HotUnplug action actually failed:

2020-09-26 23:08:52,477+03 INFO  [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-281) [vms_update_de5618c5-1dda-4605] Lock Acquired to object 'EngineLock:{exclusiveLocks='[memory_hotplug_test=VM_NAME]', sharedLocks='[65dad12f-6f02-424b-9e89-5e52da24c6b8=VM]'}'
2020-09-26 23:08:52,505+03 INFO  [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-281) [vms_update_de5618c5-1dda-4605] Running command: UpdateVmCommand internal: false. Entities affected :  ID: 65dad12f-6f02-424b-9e89-5e52da24c6b8 Type: VMAction group EDIT_VM_PROPERTIES with role type USER
2020-09-26 23:08:52,526+03 INFO  [org.ovirt.engine.core.vdsbroker.HotUnplugMemoryVDSCommand] (default task-281) [56038c5d] START, HotUnplugMemoryVDSCommand(HostName = host_mixed_3, Params:{hostId='e926be11-3e35-4e8a-9671-6fa7176e8132'}), log id: 1993cc7c
2020-09-26 23:08:52,585+03 INFO  [org.ovirt.engine.core.vdsbroker.HotUnplugMemoryVDSCommand] (default task-281) [56038c5d] FINISH, HotUnplugMemoryVDSCommand, return: , log id: 1993cc7c
2020-09-26 23:08:52,601+03 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-281) [56038c5d] EVENT_ID: MEMORY_HOT_UNPLUG_SUCCESSFULLY_REQUESTED(2,046), Hot unplug of memory device (e85017bb-7e82-45b3-8330-6f4d1b06f174) of size 256MB was successfully requested on VM 'memory_hotplug_test'. Physical memory guaranteed updated from 4864MB to 4608MB}.
2020-09-26 23:09:14,075+03 INFO  [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-281) [vms_update_12296b3c-d744-4401] Lock Acquired to object 'EngineLock:{exclusiveLocks='[memory_hotplug_test=VM_NAME]', sharedLocks='[65dad12f-6f02-424b-9e89-5e52da24c6b8=VM]'}'
2020-09-26 23:09:14,115+03 INFO  [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-281) [vms_update_12296b3c-d744-4401] Running command: UpdateVmCommand internal: false. Entities affected :  ID: 65dad12f-6f02-424b-9e89-5e52da24c6b8 Type: VMAction group EDIT_VM_PROPERTIES with role type USER
2020-09-26 23:09:14,156+03 INFO  [org.ovirt.engine.core.vdsbroker.HotUnplugMemoryVDSCommand] (default task-281) [75856d8b] START, HotUnplugMemoryVDSCommand(HostName = host_mixed_3, Params:{hostId='e926be11-3e35-4e8a-9671-6fa7176e8132'}), log id: 6ddd55ee
2020-09-26 23:09:14,168+03 WARN  [org.ovirt.engine.core.vdsbroker.HotUnplugMemoryVDSCommand] (default task-281) [75856d8b] Unexpected return value: Status [code=85, message=Failed to hotunplug memory: {'vmId': '65dad12f-6f02-424b-9e89-5e52da24c6b8', 'reason': "device not found: model 'dimm' memory device not present in the domain configuration"}]
2020-09-26 23:09:14,168+03 ERROR [org.ovirt.engine.core.vdsbroker.HotUnplugMemoryVDSCommand] (default task-281) [75856d8b] Failed in 'HotUnplugMemoryVDS' method
2020-09-26 23:09:14,169+03 WARN  [org.ovirt.engine.core.vdsbroker.HotUnplugMemoryVDSCommand] (default task-281) [75856d8b] Unexpected return value: Status [code=85, message=Failed to hotunplug memory: {'vmId': '65dad12f-6f02-424b-9e89-5e52da24c6b8', 'reason': "device not found: model 'dimm' memory device not present in the domain configuration"}]
2020-09-26 23:09:14,182+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-281) [75856d8b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_3 command HotUnplugMemoryVDS failed: Failed to hotunplug memory: {'vmId': '65dad12f-6f02-424b-9e89-5e52da24c6b8', 'reason': "device not found: model 'dimm' memory device not present in the domain configuration"}
2020-09-26 23:09:14,182+03 ERROR [org.ovirt.engine.core.vdsbroker.HotUnplugMemoryVDSCommand] (default task-281) [75856d8b] Command 'HotUnplugMemoryVDSCommand(HostName = host_mixed_3, Params:{hostId='e926be11-3e35-4e8a-9671-6fa7176e8132'})' execution failed: VDSGenericException: VDSErrorException: Failed to HotUnplugMemoryVDS, error = Failed to hotunplug memory: {'vmId': '65dad12f-6f02-424b-9e89-5e52da24c6b8', 'reason': "device not found: model 'dimm' memory device not present in the domain configuration"}, code = 85
2020-09-26 23:09:14,182+03 INFO  [org.ovirt.engine.core.vdsbroker.HotUnplugMemoryVDSCommand] (default task-281) [75856d8b] FINISH, HotUnplugMemoryVDSCommand, return: , log id: 6ddd55ee
2020-09-26 23:09:14,196+03 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-281) [75856d8b] EVENT_ID: MEMORY_HOT_UNPLUG_SUCCESSFULLY_REQUESTED(2,046), Hot unplug of memory device (e85017bb-7e82-45b3-8330-6f4d1b06f174) of size 256MB was successfully requested on VM 'memory_hotplug_test'. Physical memory guaranteed updated from 4608MB to 4352MB}.


Version-Release number of selected component (if applicable):
ovirt-engine-4.4.3.3-0.19.el8ev.noarch
vdsm-4.40.31-1.el8ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Run VM (build on base of template OS  Linux release 8.2 (Ootpa) ) with:
   Memory Size= 1024 MB; Maximum memory = 16384 MB; Physical Memory Guaranteed = 512 MB.
   kernel parameter "movable_node".
2. Hotplug 15x256M memory devices(update VM 15 times with +256M memory size each time via API)
3. HotUnplug 15x256M memory devices(update VM 15 times with -256M memory size each time via API)


Actual results:
1. After step 3, VM memory size was updated to 1024M on engine side, but `free -m` on VM showed only one 256M memory device was removed.
2. In engine log:
1) the same memory device of size 256MB was unplugged 15 times.
2) only the first unplug action succeeded, the other 14 times failed because the memory device not present in the domain configuration.
3) engine still shows Hot unplug of memory device of size 256MB was successfully requested on VM, memory size reduced 256M, though unplug action actually failed.


Expected results:
1. All memory devices, not only the first one, should be removed on VM.
2. Memory size shouldn't be reduced on engine side when the unplug action actually failed.


Additional info: