Description of problem: We got a report about an environment where the user tries to change the CPU type for the hosted-engine virtual machine with no actual results. After investigations we understood that the issues comes from here: MainThread::DEBUG::2019-03-21 10:02:00,865::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) {u'status': u'OK', u'lease': {u'path': u'/rhev/data-center/mnt/storage.internal.viada.de:_rhv4-volume/3ee26ff5-afb1-412a-89ef-a289ac109fed/images/0eba3025-70de-4adc-9857-05b77155bfc9/d301ed77-f3d8-4af2-9f1f-7dcb116d3c01.lease', u'owners': [], u'version': None, u'offset': 0}, u'domain': u'3ee26ff5-afb1-412a-89ef-a289ac109fed', u'capacity': u'134217728', u'voltype': u'LEAF', u'description': u'{"Updated":true,"Size":30720,"Last Updated":"Wed Mar 06 14:35:25 CET 2019","Storage Domains":[{"uuid":"3ee26ff5-afb1-412a-89ef-a289ac109fed"}],"Disk Description":"OVF_STORE"}', u'parent': u'00000000-0000-0000-0000-000000000000', u'format': u'RAW', u'generation': 0, u'image': u'0eba3025-70de-4adc-9857-05b77155bfc9', u'uuid': u'd301ed77-f3d8-4af2-9f1f-7dcb116d3c01', u'disktype': u'OVFS', u'legality': u'LEGAL', u'mtime': u'0', u'apparentsize': u'30720', u'truesize': u'30720', u'type': u'PREALLOCATED', u'children': [], u'pool': u'', u'ctime': u'1550073785'} MainThread::DEBUG::2019-03-21 10:02:01,086::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) {u'status': u'OK', u'lease': {u'path': u'/rhev/data-center/mnt/storage.internal.viada.de:_rhv4-volume/3ee26ff5-afb1-412a-89ef-a289ac109fed/images/556be753-69bd-413c-87e2-e170d3fca9db/5cd0085a-094d-4806-a251-1af759241b4d.lease', u'owners': [], u'version': None, u'offset': 0}, u'domain': u'3ee26ff5-afb1-412a-89ef-a289ac109fed', u'capacity': u'134217728', u'voltype': u'LEAF', u'description': u'{"Updated":true,"Size":30720,"Last Updated":"Wed Mar 06 14:35:25 CET 2019","Storage Domains":[{"uuid":"3ee26ff5-afb1-412a-89ef-a289ac109fed"}],"Disk Description":"OVF_STORE"}', u'parent': u'00000000-0000-0000-0000-000000000000', u'format': u'RAW', u'generation': 0, u'image': u'556be753-69bd-413c-87e2-e170d3fca9db', u'uuid': u'5cd0085a-094d-4806-a251-1af759241b4d', u'disktype': u'OVFS', u'legality': u'LEGAL', u'mtime': u'0', u'apparentsize': u'30720', u'truesize': u'30720', u'type': u'PREALLOCATED', u'children': [], u'pool': u'', u'ctime': u'1550073786'} look at MainThread::DEBUG::2019-03-21 10:02:00,865 ... "Last Updated":"Wed Mar 06 14:35:25 CET 2019" MainThread::DEBUG::2019-03-21 10:02:01,086 ... "Last Updated":"Wed Mar 06 14:35:25 CET 2019" on 2019-03-21 ovirt-ha-agent is still continuously parsing the OVF_STORE but engine updated it last time only on 2019-03-06 (15 days before). In the mean time the user is trying to edit the hosted-engine VM with no apparent results. Version-Release number of selected component (if applicable): 4.2.8 How reproducible: ? Steps to Reproduce: 1. check OVF_STORE updates periodicity on running engines 2. 3. Actual results: on an active storage domain, the engine updated the OVF_STORE 15 days ago Expected results: the engine updates the OVF_STORE files every time cycle that set in OvfUpdateIntervalInMinutes (default 60 minutes). Additional info:
According to https://bugzilla.redhat.com/show_bug.cgi?id=1585986#c59 it seems that changes at cluster level are not increasing VMs generation numbers and so a new OVF_STORE content is not copied to the shared storage.
(In reply to Simone Tiraboschi from comment #2) > According to https://bugzilla.redhat.com/show_bug.cgi?id=1585986#c59 it > seems that changes at cluster level are not increasing VMs generation > numbers and so a new OVF_STORE content is not copied to the shared storage. For HE it's true, because the cluster upgrade code is not touching HE VM. So IMHO works as designed. Does it need to change? I would strongly suggest to explore updating all VMs including HE, but there might be still some gaps I'm not aware of preventing this
(In reply to Michal Skrivanek from comment #3) > For HE it's true, because the cluster upgrade code is not touching HE VM. So > IMHO works as designed. Does it need to change? I would strongly suggest to > explore updating all VMs including HE, but there might be still some gaps > I'm not aware of preventing this I tend to agree that's the best option: I don't see any reason why the hosted-engine VM should be different on this aspect. For sure we don't have something like next run configuration but we can also discuss about that.
I tested this issue by commenting out the isHostedEngine call here: https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/VmCommand.java#L174 and then setting a break point here: https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/storage/ovfstore/OvfDataUpdater.java#L112-L115 and in the ovfUpdate function here: https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/storage/ovfstore/OvfDataUpdater.java#L69 the former of which has the code: updateTimerJob = schedulerService.scheduleWithFixedDelay(this::ovfUpdate, 0, Config.<Long> getValue(ConfigValues.OvfUpdateIntervalInMinutes), TimeUnit.MINUTES); Editing the VM does call into the triggerNow() which calls the schedulerService.scheduleWithFixedDelay function, but does not enter the ovfUpdate function. It has been suggested that this is an infrastructure issue. Maybe you can take a look.
Why comment this out? shouldUpdateHostedEngineOvf should also trigger any time the HE VM is updated, and this should be tested by leaving the code intact, changing the HE CPU level (or other), then checking whether a new OVA is generated.
(In reply to Ryan Barry from comment #6) > Why comment this out? shouldUpdateHostedEngineOvf should also trigger any > time the HE VM is updated, and this should be tested by leaving the code > intact, changing the HE CPU level (or other), then checking whether a new > OVA is generated. Commenting was just for testing more easily to verify that the problem is in the scheduling.
Well, maybe. On the other bug (about DC levels and other changes), the OVA update interval was tested and seems to behave. It's not really a valid test for the given scenario (specifically about HE) if HE isn't tested. In particular, at least shouldUpdateHostedEngineOvf needs to be traced
The verification now is waiting for https://bugzilla.redhat.com/show_bug.cgi?id=1828944 resolution. Though I suspect that even after this fix we still can't change one on of parameters for HE VM - compatibility version, CPU name, CPU architecture, bios type (will bring "locked values" error), which is required for this verification.
verified on rhv-4.4.1-1. HE VM ID - 0e5b6512-8f2b-4864-8d4c-54bf3c4d1099 In Edit cluster window: 1. Replace 'Intel Skylake Server Family' to 'Intel Skylake Client Family' - update messages appear in engine.log. 2020-06-02 15:52:00,341+03 INFO [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-10) [39b5bac9] Running command: UpdateVmCommand internal: true. Entities affected : ID: 0e5b6512-8f2b-4864-8d4c-54bf3c4d1099 Type: VMAction group EDIT_VM_PROPERTIES with role type USER 2020-06-02 15:52:00,588+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [4cd86ff3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[2406b065-b2cd-4253-92fb-fe0682089c29=STORAGE]', sharedLocks='[0825bd31-a44e-46ff-b2f4-abfab700705e=OVF_UPDATE]'}' 2020-06-02 15:52:00,590+03 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (default task-10) [1004c7be] Lock freed to object 'EngineLock:{exclusiveLocks='[6c0e62b2-6259-4bf6-b3c9-6ec1ee24a222=TEMPLATE, 61696f84-67ba-48d4-a248-7b7c85f1e7cb=TEMPLATE]', sharedLocks='[bc1c71fa-2825-426b-8a3c-64a8bcac9fe3=VM, 2f88fd08-274d-41ca-8820-4b824b37fbd1=VM, 30bcd9aa-a473-4e05-be79-bca8a8fa7421=VM, e2dbe932-2e6b-4cfd-843f-f58ae231cf46=VM, 0e5b6512-8f2b-4864-8d4c-54bf3c4d1099=VM, ca71eb49-bc87-4e94-aacd-866a020f779f=VM, 9ca8bc50-d88e-42de-a469-35f794c4edbf=VM]'}' 2020-06-02 15:52:00,618+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [4cd86ff3] ++ description={"Updated":false,"Last Updated":"Tue Jun 02 15:46:57 IDT 2020","Storage Domains":[{"uuid":"2406b065-b2cd-4253-92fb-fe0682089c29"}],"Disk Description":"OVF_STORE"} 2020-06-02 15:52:00,992+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [4cd86ff3] ++ description={"Updated":true,"Size":14336,"Last Updated":"Tue Jun 02 15:52:00 IDT 2020","Storage Domains":[{"uuid":"2406b065-b2cd-4253-92fb-fe0682089c29"}],"Disk Description":"OVF_STORE"} 2020-06-02 15:52:01,035+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [4cd86ff3] ++ description={"Updated":false,"Last Updated":"Tue Jun 02 15:46:57 IDT 2020","Storage Domains":[{"uuid":"2406b065-b2cd-4253-92fb-fe0682089c29"}],"Disk Description":"OVF_STORE"} 2020-06-02 15:52:01,356+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [4cd86ff3] ++ description={"Updated":true,"Size":14336,"Last Updated":"Tue Jun 02 15:52:00 IDT 2020","Storage Domains":[{"uuid":"2406b065-b2cd-4253-92fb-fe0682089c29"}],"Disk Description":"OVF_STORE"} 2020-06-02 15:52:01,411+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [4cd86ff3] Lock freed to object 'EngineLock:{exclusiveLocks='[2406b065-b2cd-4253-92fb-fe0682089c29=STORAGE]', sharedLocks='[0825bd31-a44e-46ff-b2f4-abfab700705e=OVF_UPDATE]'}' 2020-06-02 15:52:01,417+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [76b38c11] Lock Acquired to object 'EngineLock:{exclusiveLocks='[8eedc386-b74b-451a-9945-f539a34b549e=STORAGE]', sharedLocks='[0825bd31-a44e-46ff-b2f4-abfab700705e=OVF_UPDATE]'}' 2020-06-02 15:52:01,433+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [76b38c11] ++ description={"Updated":false,"Last Updated":"Tue Jun 02 15:46:57 IDT 2020","Storage Domains":[{"uuid":"8eedc386-b74b-451a-9945-f539a34b549e"}],"Disk Description":"OVF_STORE"} 2020-06-02 15:52:03,524+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [76b38c11] ++ description={"Updated":true,"Size":25600,"Last Updated":"Tue Jun 02 15:52:01 IDT 2020","Storage Domains":[{"uuid":"8eedc386-b74b-451a-9945-f539a34b549e"}],"Disk Description":"OVF_STORE"} 2020-06-02 15:52:03,897+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [76b38c11] ++ description={"Updated":false,"Last Updated":"Tue Jun 02 15:46:57 IDT 2020","Storage Domains":[{"uuid":"8eedc386-b74b-451a-9945-f539a34b549e"}],"Disk Description":"OVF_STORE"} 2. Replace BIOS Type from 'Q35 Chipset with Legasy BIOS' to '...UEFI BIOS'. 2020-06-02 15:44:02,327+03 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (default task-10) [5c9b2729-03d8-409e-910c-a73a6a1478ea] Lock Acquired to object 'EngineLock:{exclusiveLocks='[6c0e62b2-6259-4bf6-b3c9-6ec1ee24a222=TEMPLATE, 61696f84-67ba-48d4-a248-7b7c85f1e7cb=TEMPLATE]', sharedLocks='[bc1c71fa-2825-426b-8a3c-64a8bcac9fe3=VM, 2f88fd08-274d-41ca-8820-4b824b37fbd1=VM, 30bcd9aa-a473-4e05-be79-bca8a8fa7421=VM, e2dbe932-2e6b-4cfd-843f-f58ae231cf46=VM, 0e5b6512-8f2b-4864-8d4c-54bf3c4d1099=VM, ca71eb49-bc87-4e94-aacd-866a020f779f=VM, 9ca8bc50-d88e-42de-a469-35f794c4edbf=VM]'}' 2020-06-02 15:44:02,601+03 INFO [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-10) [60f33f8c] Running command: UpdateVmCommand internal: true. Entities affected : ID: 0e5b6512-8f2b-4864-8d4c-54bf3c4d1099 Type: VMAction group EDIT_VM_PROPERTIES with role type USER 2020-06-02 15:44:02,628+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-10) [1c1d42a2] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[0825bd31-a44e-46ff-b2f4-abfab700705e=OVF_UPDATE]', sharedLocks=''}' 2020-06-02 15:44:02,628+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-10) [1c1d42a2] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[0825bd31-a44e-46ff-b2f4-abfab700705e=OVF_UPDATE]', sharedLocks=''}' 2020-06-02 15:44:02,640+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-10) [1c1d42a2] Attempting to update VM OVFs in Data Center 'golden_env_mixed'
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: RHV Manager (ovirt-engine) 4.4 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3247