Description of problem: When hot plugging memory in increments then hot unplugging the same memory in increments on a RHEL 8 VM, after repeating the same method several times, the Defined Memory value in RHV-M and the free command on the VM go out of sync and display different values. Virtual DIMMs also remain available on RHV-M for hot unplugging. How reproducible: Fully Steps to Reproduce: 1. Build a RHEL 8.3 VM with the following memory configuration. Memory Size: 4096 MB Maximum Memory: 32768 MB Physical Memory Guaranteed: 4096 MB 2. Configure RHV environment and VM as per pre-requisites. Hot Plugging Memory (https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html/virtual_machine_management_guide/sect-virtual_memory#Hot_Plugging_Virtual_Memory) Hot Unplugging Memory (https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4//html-single/virtual_machine_management_guide/sect-virtual_memory#Hot_Unplugging_Virtual_Memory) 3. Hot plug and hot unplug memory via the REST API in increments. - increase to 8 GB *wait* - increase to 12 GB *wait* - increase to 16 GB *wait* - decrease to 12 GB *wait* - decrease to 8 GB *wait* - decrease to 4 GB *wait* 4. On RHV-M, monitor the 'Defined Memory' value. It will increase during hot plugging and decrease during unplugging. Compute -> Virtual Machines -> select VM > General tab On RHV-M, monitor the memory devices. New virtual DIMMs will be created during hot plugging and removed during unplugging. Compute -> Virtual Machines -> select VM > Vm Devices tab On the VM, monitor the memory values during hot plugging and hot unplugging. # watch free -m 5. Repeat hot plugging and hot unplugging process several times. Actual results: Hot plugging and hot unplugging will increase and decrease memory as expected. However, after several attempts, notice that the memory displayed in free command will be different from the value displayed on the RHV-M UI. > RHV-M displays Defined Memory: 4096 MB > VM displays ~~~ # free -m total used free shared buff/cache available Mem: 28309 886 26929 10 492 26399 ~~~ In addition, the virtual DIMMs remain available on RHV-M and the 'Hot Unplug' button is also visible. Compute -> Virtual Machines -> select VM > Vm Devices tab Clicking on the 'Hot Unplug' button for a specific DIMM unplugs some of the memory and this is reflected by the free command running in the VM. ~~~ # free -m total used free shared buff/cache available Mem: 24213 821 22899 10 492 22525 ~~~ On RHV-M, the 'Defined Memory' field does not change and still displays 4096 MB. Expected results: 1. RHV-M Defined Memory value should match values displayed by the free command in the VM. 2. Virtual DIMMs for already unplugged memory should not remain visible on RHV-M. Compute -> Virtual Machines -> select VM > Vm Devices tab Additional info:
I'm afraid this is just a consequence of unplugging being cooperative and not very reliable. When the guest doesn't want to let go of the DIMM we try to unplug we have no way how to enforce that. Can you please confirm the minimal reproducer and guest OS used?
Hi Michal, Guest OS used is RHEL 8.3. Not aware of other reproducers aside from the one described.
Looking at https://github.com/oVirt/ovirt-engine/blob/7a9899aae5e776fb8d054b51777245e1f9f63a2f/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/HotUnplugMemoryCommand.java#L63, Engine apparently reduces Defined Memory based (among other) on the success of VM.hotunplugMemory Vdsm call. But success return value from that call doesn't necessarily mean a successful DIMM removal. Vdsm calls libvirt to detach the requested DIMM, libvirt initiates the guest OS action and if nothing happens within 5 seconds then it returns success. Vdsm then returns success immediately, without waiting for the corresponding device removal event from libvirt. Then one of the following can happen: a) The DIMM gets removed after some time and everything is fine. b) The user looks at the memory in the meantime and can see the discrepancy between Engine and the guest OS because Engine has already subtracted the value while the DIMM is still plugged in. This is a temporary situation, resulting in a) or c) sooner or later. c) The DIMM removal eventually fails. The discrepancy described in b) becomes permanent. DIMM removal can take a long time, I've seen sometimes >20 seconds in the logs. Even if Vdsm waited for the device removal libvirt event, there is a timeout (30 s by default), the event could come too late and a failure would be reported despite a possible later success. Then we could get the opposite problem, reporting more memory than actually present, if the device removal eventually succeeds. A better option might be to update the memory in Engine based on the domain XML contents. Engine would update Defined Memory only once it gets notified about a DIMM device removal (or has other reason to re-read current memory from the domain XML). What do you think?
Thank you Milan. Engine updating Defined Memory only once it gets notified about a DIMM device removal is in my opinion a better approach. The expected outcomes: 1. If the DIMM is not unplugged then the Defined Memory value should not change. 2. The Defined Memory value on RHV should match the output of free command on the guest. 3. When plugging and unplugging via the API, you specify the new memory value and hope for the best. There is no response/message/confirmation if it worked or not. It would be helpful to have a response or something communicated to the user somehow.
(In reply to Milan Zamazal from comment #7) > A better option might be to update the memory in Engine based on the domain > XML contents. Engine would update Defined Memory only once it gets notified > about a DIMM device removal (or has other reason to re-read current memory > from the domain XML). What do you think? Conceptually, it makes sense but practically, I'm afraid it's problematic to implement this. The 'memory' field is part of vm_static that contains fields that are set by the user and are not expected to change by VDSM - so triggering the unplug operation and skip the update of vm_static until we get the domain xml without the DIMM device would be risky and different than what we typically do. Another option is to think about hot-unplug memory as an asynchronous operation - 1. trigger the hot-unplug 2. update 'memory' in vm_static 3. wait for the DIMM to be removed 4. if the DIMM device removal fails, undo the update But that is also problematic for two reasons: 1. if we update the 'memory' field in vm_static, the discrepancy between what the user sees in webadmin and from within the guest, would still remain 2. in order to monitor the operation as an async operation, we would need VDSM to report a job or something the engine can track - otherwise, the operation may get stuck indefinitely (and the VM would remained locked) So I think that if it's a matter of discrepancy in the time frame between triggering update-VM on the engine side and when it's applied within the guest - we'll need to live with that because fixing it is an RFE that will be quite complicate to implement. We always set a desired state and then try to achieve it - for example when adding a disk, the disk entity is inserted to the database before we allocate the volume on the storage. unfortunately, we can't easily add a status like we have for disk to indicate that the change wasn't applied yet (the 'LOCKED' state of the disk) also for the defined memory. If the hot-unplug fails, we'll need a reliable way to identify that tell it to the engine. Sam, can you please tell if the discrepancy between what the user saw in RHV and from within the guest sorted out after some time (note that as Milan wrote above, it can take some time) or did the hot-unplug fail?
(In reply to Arik from comment #9) > Another option is to think about hot-unplug memory as an asynchronous > operation - > 1. trigger the hot-unplug > 2. update 'memory' in vm_static > 3. wait for the DIMM to be removed > 4. if the DIMM device removal fails, undo the update > > But that is also problematic for two reasons: It's also problematic that if the operation doesn't fail within the 5 second libvirt timeout then it will never fail, it will hang indefinitely. If the operation eventually succeeds then we will get a device removal event (or can detect that in the domain XML if Vdsm wasn't running at the given moment). If the operation fails then we won't get anything. The only way to handle that is to add our own timeout but the question is what a good timeout would be, I can imagine that e.g. hot unplugging a 256 GB DIMM on a busy system can take quite time. > If the hot-unplug fails, we'll need a reliable way to identify that tell it to the engine. I don't think we have a reliable way to do that.
Yeah, handling the negative flow is tricky and to the best of my knowledge we didn't get reports on that (and hot-unplug memory is there for several years now). So I think the question here is whether it is really the issue here - if this ticket is just about the what the user sees in oVirt/RHV and within the guest for a limited time frame, I'd just say that it was designed that way. It may sound like a bug in the design but kubernetes does the same thing by design..
Hi Sam, thank you for clarification, I think I know now what you may be observing. Let's talk about e.g. the following scenario: - VM starts with 1024 MB RAM. - 256 MB + 256 MB + 512 MB RAM is hot plugged. - 256 MB RAM is hot unplugged 4 times in a quick sequence using REST API. The result may be that not all the hot plugged DIMMs are removed due to an error like this reported by Vdsm: libvirt.libvirtError: device not found: model 'dimm' memory device not present in the domain configuration ... vdsm.common.exception.HotunplugMemFailed: Failed to hotunplug memory: {'vmId': '630b9cae-a983-4ab0-a9ac-6b8728f8014d', 'reason': "device not found: model 'dimm' memory device not present in the domain configuration"} Apparently, the problem is that Engine tries to unplug the same DIMM on subsequent hot unplug attempts, because it hasn't been informed about its removal yet it and it wrongly assumes it's free to unplug. Another problem is that Engine decreases Defined Memory even when the hot unplug API call fails immediately. This also explains why the followup attempt to remove a DIMM manually from VM devices succeeds -- the DIMM is free to unplug, because different DIMMs were used for the preceding failed hot unplug attempts. Sam, can you please check that this is indeed the problem observed on your and customer's setups? Namely that the Vdsm error above is present in vdsm.log? If so, I think the double hot unplug of the same DIMM should be fixed in Engine. We can also consider what to do about Defined Memory when a memory hot unplug API call fails with an error. A user workaround is not to attempt another hot unplug attempt until the previous hot unplug is completed (i.e. all the hot unplugged DIMMs disappear from the list of devices in Engine). If the hot unplug fails for a different reason than the one above, I'd like to know whether the hot unplug API call succeeded and if yes then why the hot unplug failed (should be visible in journal or dmesg).
Thanks Milan for your explanation. The scenario is correct with a slight adjustment. During hot plug, the same amount of memory is added each time - 256 MB + 256 MB + 256 MB RAM + 256 MB RAM. Hot unplug is not done in quick sequence but rather with a time delay (e.g. wait 10 secs or 1 minute) before the next hot unplug attempt. Based on your reproducer, it seems the outcome is still the same whether a time delay is applied or not. When hot unplug fails, VDSM on the host displays the same error including traceback. ~~~ 2021-04-29 11:26:18,618-0400 INFO (jsonrpc/6) [virt.vm] (vmId='e9346556-22de-4037-ad95-1250d5ec5d37') Hotunplug memory xml: <?xml version='1.0' encoding='utf-8'?> <memory model="dimm"><target><size unit="KiB">4194304</size><node>0</node></target><alias name="ua-21fd464d-8780-4f22-854a-72b507623a23" /><address base="0x380000000" slot="2" type="dimm" /></memory> (vm:3112) 2021-04-29 11:26:18,620-0400 ERROR (jsonrpc/6) [api] FINISH hotunplugMemory error=Failed to hotunplug memory: {'vmId': 'e9346556-22de-4037-ad95-1250d5ec5d37', 'reason': "device not found: model 'dimm' memory device not present in the domain configuration"} (api:131) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 3121, in hotunplugMemory self._dom.detachDevice(device_xml) File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in f ret = attr(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1343, in detachDevice if ret == -1: raise libvirtError ('virDomainDetachDevice() failed', dom=self) libvirt.libvirtError: device not found: model 'dimm' memory device not present in the domain configuration During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/API.py", line 419, in hotunplugMemory return self.vm.hotunplugMemory(params) File "<decorator-gen-249>", line 2, in hotunplugMemory File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 101, in method return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 3127, in hotunplugMemory raise exception.HotunplugMemFailed(str(e), vmId=self.id) vdsm.common.exception.HotunplugMemFailed: Failed to hotunplug memory: {'vmId': 'e9346556-22de-4037-ad95-1250d5ec5d37', 'reason': "device not found: model 'dimm' memory device not present in the domain configuration"} 2021-04-29 11:26:18,624-0400 INFO (jsonrpc/6) [api.virt] FINISH hotunplugMemory return={'status': {'code': 85, 'message': 'Failed to hotunplug memory: {\'vmId\': \'e9346556-22de-4037-ad95-1250d5ec5d37\', \'reason\': "device not found: model \'dimm\' memory device not present in the domain configuration"}'}} from=::ffff:172.16.42.11,38564, flow_id=5271a546, vmId=e9346556-22de-4037-ad95-1250d5ec5d37 (api:54) 2021-04-29 11:26:18,624-0400 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.hotunplugMemory failed (error 85) in 0.01 seconds (__init__:312) ~~~ When hot unplug fails, the free command on the guest reports more memory than was actually hot plugged. Do you know the cause of this?
(In reply to Sam Wachira from comment #16) > The scenario is correct with a slight adjustment. > During hot plug, the same amount of memory is added each time - 256 MB + 256 > MB + 256 MB RAM + 256 MB RAM. > Hot unplug is not done in quick sequence but rather with a time delay (e.g. > wait 10 secs or 1 minute) before the next hot unplug attempt. It can take some time, depending on the memory used. If I wait for each of the DIMMs to disappear in the web UI then I don't experience any problems. > When hot unplug fails, VDSM on the host displays the same error including > traceback. Good, thanks for checking, so the cause of the problem has been identified. > When hot unplug fails, the free command on the guest reports more memory > than was actually hot plugged. Do you know the cause of this? This is really weird. If the hot unplug fails due to this bug then the hot unplug request shouldn't reach the guest at all. In theory, there can be a bug in `free' or elsewhere (there was in the past), but it works for me. You can always check the true memory using `virsh'.
Verified with: ovirt-engine-4.4.9.1-0.13.el8ev.noarch vdsm-4.40.90.2-1.el8ev.x86_64 Steps: 1. Create and run a RHEL 8.5 VM with: Memory Size: 1024 MB Maximum Memory: 4096 MB Physical Memory Guaranteed: 1024 MB 2. Hot plug 256MB RAM 10 times via REST API. 3. Hot unplug 256MB RAM 10 times in a quick sequence via REST API. Results: 1. All 10 hot plugged 256MB DIMMs get hot unplugged successfully. 2. No DIMM is hot unplugged multiple times. 3. RHV-M Defined Memory value matches values displayed by the free command in the VM.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: RHV Manager (ovirt-engine) security update [ovirt-4.4.9]), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:4626