Description of problem: In an environment containing only two storage domain (including hosted engine storage domain), if the current master storage domain is put into maintenance mode, then the HE VM will shutdown abruptly. This is because the sanlock for the hosted engine storage is also removed when we put the regular master storage domain in maintenance mode. Deactivating the master storage domain is followed by SpmStopVDSCommand and DisconnectStoragePoolVDSCommand. ==== 2016-11-26 06:46:53,122 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (DefaultQuartzScheduler3) [42944d24] START, DeactivateStorageDomainVDSCommand( DeactivateStorageDomainVDSCommandParameters:{runAsync='true', storagePoolId='00000001-0001-0001-0001-000000000149', ignoreFailoverLimit='false', storageDomainId='918f7d4e-114f-4dcf-872e-7c872b698862', masterDomainId='00000000-0000-0000-0000-000000000000', masterVersion='8'}), log id: 7ed960c0 2016-11-26 06:46:54,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (DefaultQuartzScheduler3) [42944d24] FINISH, DeactivateStorageDomainVDSCommand, log id: 7ed960c0 2016-11-26 06:46:54,154 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (DefaultQuartzScheduler3) [42944d24] START, SpmStopOnIrsVDSCommand( SpmStopOnIrsVDSCommandParameters:{runAsync='true', storagePoolId='00000001-0001-0001-0001-000000000149', ignoreFailoverLimit='false'}), log id: 40501a64 2016-11-26 06:46:55,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler3) [42944d24] SpmStopVDSCommand::Stopping SPM on vds 'hosted_engine_1', pool id '00000001-0001-0001-0001-000000000149' 2016-11-26 06:46:56,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [42944d24] START, DisconnectStoragePoolVDSCommand(HostName = hosted_engine_1, DisconnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e26f3591-d015-44dc-8def-845324201d0a', storagePoolId='00000001-0001-0001-0001-000000000149', vds_spm_id='1'}), log id: 40736e8 ==== As a part of DisconnectStoragePoolVDSCommand, the sanlock of hosted engine storage domain is also released in addition to the master storage domain. Thread-28::INFO::2016-11-26 10:14:12,841::clusterlock::259::Storage.SANLock::(releaseHostId) Releasing host id for domain dda867f2-c61d-4621-964d-0acf209b28bf (id: 1) Thread-26::INFO::2016-11-26 10:14:12,842::clusterlock::259::Storage.SANLock::(releaseHostId) Releasing host id for domain 918f7d4e-114f-4dcf-872e-7c872b698862 (id: 1) Thread-26::DEBUG::2016-11-26 10:14:14,843::clusterlock::270::Storage.SANLock::(releaseHostId) Host id for domain 918f7d4e-114f-4dcf-872e-7c872b698862 released successfully (id: 1) Thread-28::DEBUG::2016-11-26 10:14:14,844::clusterlock::270::Storage.SANLock::(releaseHostId) Host id for domain dda867f2-c61d-4621-964d-0acf209b28bf released successfully (id: 1) Since sanlock is removed, the HE qemu-kvm process will be killed. 2016-11-26 10:14:13+0530 30589 [734]: s12 kill 24371 sig 9 count 1 2016-11-26 10:14:13+0530 30589 [734]: dead 24371 ci 2 count 1 libvirtEventLoop::INFO::2016-11-26 10:14:13,827::vm::1308::virt.vm::(setDownStatus) vmId=`3375cd24-44d7-43cd-b149-96fcc554d42b`::Changed state to Down: Lost connection with qemu process (code=2) This is working good in RHEV 3.6 (vdsm-4.17.26-0.el7ev.noarch) where the HE storage domain will still remain active even though it doesn't get master role which I believe is the desired behavior. Here releasing sanlock on the HE storage domain will fail as below and the VM won't be effected. Cannot release host id: ('6f6248a6-5f75-449d-8387-424368289b5f', SanlockException(16, 'Sanlock lockspace remove failure', 'Device or resource busy')) In the newer vdsm version, we are are passing unused=False to rem_lockspace and hence the lock is released even when we are having acquired resource in the lockspace which is causing this issue. def releaseHostId(self, hostId, async=False): self._domainLock.releaseHostId(hostId, async, False) def releaseHostId(self, hostId, async, unused): with self._lock: self.log.info("Releasing host id for domain %s (id: %s)", self._sdUUID, hostId) try: sanlock.rem_lockspace(self._sdUUID, hostId, self._idsPath, async=async, unused=unused) I think the behavior was changes as per https://gerrit.ovirt.org/#/c/43549/ Version-Release number of selected component (if applicable): vdsm-4.18.13-1.el7ev.x86_64 How reproducible: 100% Steps to Reproduce: 1. In an HE environment with two storage domain, put the master storage domain in maintenance mode. 2. The HE VM will go down abruptly within few seconds. Actual results: HE VM going down when master storage domain is put into maintenance mode. Expected results: HE VM should remain up. Additional info:
The issue can also cause outage of the running VMs with disks in the HE storage domain.
I've found even worser scenario, in which if you have two hosted-engine-hosts, lets say A and B, and assume that A is running HE-VM, and B is in local maintenance, then you're setting your single master storage domain in to the maintenance, then there is no other hosts in hosted engine host cluster with positive HA-score and HE-VM never goes up again. I've found a work around to manually remove local maintenance from host B from CLI and then ha-agent started HE-VM, once it's score became positive.
(In reply to Nikolai Sednev from comment #4) > I've found even worser scenario, in which if you have two > hosted-engine-hosts, lets say A and B, and assume that A is running HE-VM, > and B is in local maintenance, then you're setting your single master > storage domain in to the maintenance, then there is no other hosts in hosted > engine host cluster with positive HA-score and HE-VM never goes up again. > > I've found a work around to manually remove local maintenance from host B > from CLI and then ha-agent started HE-VM, once it's score became positive. Yes, tracked here: https://bugzilla.redhat.com/1394570
Adam, does this make any sense to you?
Looks like the sdm indirection code has dropped an optional parameter which means the correct value of unused=True is being omitted and the function ends up using False as the default. Patch forthcoming. Thank you nijin for such a thorough initial analysis of the issue!
Proposed fix for the issue uploaded: https://gerrit.ovirt.org/67715 (master) Can you please test with this patch to see if the issue is resolved?
Simone, can you explain why the hosted engine vm is having a disk on the master storage domain? When deactivating a domain, we expect any process holding a lease on this domain to be killed. Otherwise we will not be able to disconnect from the storage server of this storage domain or reconnect again because of open files. The proposed fix looks correct, this was unintended behavior change, but I think this is only the trigger that revealed the root cause, which is hosted engine using a lease on the wrong storage domain. Can we have the output of: sanlock client status -D When this setup is up and the hosted engine domain is running, and then again after we deactivate the master storage domain, and hosted engine vm get killed?
(In reply to nijin ashok from comment #0) > Deactivating the master storage domain is followed by SpmStopVDSCommand and > DisconnectStoragePoolVDSCommand. Disconnecting storage pool deactivate *all* the storage domain, not only the master domain. > As a part of DisconnectStoragePoolVDSCommand, the sanlock of hosted engine > storage domain is also released in addition to the master storage domain. Expected > Thread-28::INFO::2016-11-26 > 10:14:12,841::clusterlock::259::Storage.SANLock::(releaseHostId) Releasing > host id for domain dda867f2-c61d-4621-964d-0acf209b28bf (id: 1) > Thread-26::INFO::2016-11-26 > 10:14:12,842::clusterlock::259::Storage.SANLock::(releaseHostId) Releasing > host id for domain 918f7d4e-114f-4dcf-872e-7c872b698862 (id: 1) > Thread-26::DEBUG::2016-11-26 > 10:14:14,843::clusterlock::270::Storage.SANLock::(releaseHostId) Host id for > domain 918f7d4e-114f-4dcf-872e-7c872b698862 released successfully (id: 1) > Thread-28::DEBUG::2016-11-26 > 10:14:14,844::clusterlock::270::Storage.SANLock::(releaseHostId) Host id for > domain dda867f2-c61d-4621-964d-0acf209b28bf released successfully (id: 1) Expected > Since sanlock is removed, the HE qemu-kvm process will be killed. > > 2016-11-26 10:14:13+0530 30589 [734]: s12 kill 24371 sig 9 count 1 > 2016-11-26 10:14:13+0530 30589 [734]: dead 24371 ci 2 count 1 > > libvirtEventLoop::INFO::2016-11-26 > 10:14:13,827::vm::1308::virt.vm::(setDownStatus) > vmId=`3375cd24-44d7-43cd-b149-96fcc554d42b`::Changed state to Down: Lost > connection with qemu process (code=2) Expected > This is working good in RHEV 3.6 (vdsm-4.17.26-0.el7ev.noarch) where the HE > storage domain will still remain active even though it doesn't get master > role which I believe is the desired behavior. Here releasing sanlock on the > HE storage domain will fail as below and the VM won't be effected. This may work in 3.6 since the hosted engine storage domain is not part of the pool, so engine does not disconnect it. In 4.0, the hosted engine storage domain is part of the pool, but engine must detect it and never deactivate it. This seems to be a bug in engine, related to importing the hosted storage domain into engine. > Cannot release host id: ('6f6248a6-5f75-449d-8387-424368289b5f', > SanlockException(16, 'Sanlock lockspace remove failure', 'Device or resource > busy')) This error means that the storage domain is inactive, but there is a process using it, so we will either fail to disconnect from this storage server, or fail to reconnect again because of open files. When you put a storage domain to maintenance, we want sanlock to kill all processes using this storage. Otherwise, how can you perform maintenace on this stoage? RHV must have full control on its storage. > I think the behavior was changes as per https://gerrit.ovirt.org/#/c/43549/ Good catch, but I think this little bug just exposed the real bug, which is engine deactivating the hosted engine storage domain. > Expected results: > > HE VM should remain up. The expected result is that hosted engine storage domain cannot be deactivate in engine, with a clear error message why this is disabled.
(In reply to Nir Soffer from comment #9) > Simone, can you explain why the hosted engine vm is having a disk on the > master > storage domain? For sure we don't handle or manage it on ovirt-ha-agent side. On a cold boot for instance, ovirt-ha-agent will simply connect the storage server for the hosted-engine storage domain, it will start monitoring the hosted-engine storage domain, it will prepare all the images on the hosted-engine storage domain and than it could consume the hosted-engine lockspace and metadata area and eventually start the engine VM. No resources on other storage domains, neither the master one, are required. If it will really get any lease on the master storage domain (I'll check on a reproducer) on my opinion this will happen only when the engine, once it's up, is going to connect the other storage domains including the master storage domain and hosted-engine one for the second time. So, for what I understood, the root cause could be due to a different behavior in the engine connecting other storage domain on an hosted-engine host or explicitly disconnecting also the hosted-engine storage domain entering pool maintenance.
Hello Adam, Thanks for the quick fix. I tested this in my environment and it works good. The HE VM stays online when I put master storage domain in maintenance mode.
Hello Nir, I am just adding some more information as per my test if it helps. This happens only with Data Center with two storage domain - A HE storage domain and normal regular storage domain which is having master role. If I put the master storage domain in maintenance mode, the engine initiate a SpmStopVDSCommand and DisconnectStoragePoolVDSCommand. This is because we never allow the HE storage to become master and the Data Center don't have any other regular storage domain. As a part of the DisconnectStoragePoolVDSCommand, we are removing the sanlock in the HE storage domain too. Also HE VM is not having any lease on any other storage domain. In 3.6, if I put the regular master storage domain in maintenance mode, HE storage domain will be still active although it doesn't get master role. Same happened now in RHV 4.0 when I applied fix provided by Adam.
The vdsm patch is merged, but this bug is not fixed. The vdsm fix return vdsm to the previous behavior, which hides the fact that engine wrongly deactivate the hosted engine storage domain. Allon, this is an ovirt-engine bug, see comment 10.
There's too many things going on here. Raz - let's take VDSM with the merged patch (937e58f18c0615b8d201fb40d3e7d98bb980cda5) and test engine 4.0 and 4.1 against it, and open an engine bug against the remaining issues if we have them.
We have two issues here: 1. regression in vdsm, changing the behavior when deactivating a storage domain 2. regression in engine, deactivating the hosted engine storage domain. This bug is tracking the vdsm side, the engine side is tracked in bug 1402789.
I've set Master data storage domain in to maintenance several times, while second data storage domain remained active and Master shifted to second data storage domain without restarting the engine. My data storage domains were placed on NFS storage, just like HE was deployed on NFS. Works for me on these components on hosts: ovirt-vmconsole-host-1.0.4-1.el7ev.noarch mom-0.5.8-1.el7ev.noarch ovirt-hosted-engine-setup-2.1.0-0.0.master.20161221071755.git46cacd3.el7.centos.noarch ovirt-setup-lib-1.1.0-1.el7.centos.noarch libvirt-client-2.0.0-10.el7_3.2.x86_64 ovirt-release41-pre-4.1.0-0.6.beta2.20161221025826.gitc487776.el7.centos.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch qemu-kvm-rhev-2.6.0-28.el7_3.2.x86_64 ovirt-hosted-engine-ha-2.1.0-0.0.master.20161221070856.20161221070854.git387fa53.el7.centos.noarch rhevm-appliance-20161116.0-1.el7ev.noarch sanlock-3.4.0-1.el7.x86_64 ovirt-host-deploy-1.6.0-0.0.master.20161215101008.gitb76ad50.el7.centos.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch ovirt-imageio-common-0.5.0-0.201611201242.gitb02532b.el7.centos.noarch vdsm-4.18.999-1218.gitd36143e.el7.centos.x86_64 ovirt-imageio-daemon-0.5.0-0.201611201242.gitb02532b.el7.centos.noarch Linux version 3.10.0-514.2.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Wed Nov 16 13:15:13 EST 2016 Linux 3.10.0-514.2.2.el7.x86_64 #1 SMP Wed Nov 16 13:15:13 EST 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.3 (Maipo) On engine: ovirt-engine-setup-plugin-ovirt-engine-common-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-imageio-proxy-0.5.0-0.201611201242.gitb02532b.el7.centos.noarch ovirt-iso-uploader-4.1.0-0.0.master.20160909154152.git14502bd.el7.centos.noarch ovirt-engine-userportal-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-dbscripts-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-extensions-api-impl-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-imageio-common-0.5.0-0.201611201242.gitb02532b.el7.centos.noarch ovirt-host-deploy-1.6.0-0.0.master.20161215101008.gitb76ad50.el7.centos.noarch python-ovirt-engine-sdk4-4.1.0-0.1.a0.20161215git77fce51.el7.centos.x86_64 ovirt-host-deploy-java-1.6.0-0.0.master.20161215101008.gitb76ad50.el7.centos.noarch ovirt-release41-pre-4.1.0-0.6.beta2.20161221025826.gitc487776.el7.centos.noarch ovirt-setup-lib-1.1.0-1.el7.centos.noarch ovirt-engine-extension-aaa-jdbc-1.1.2-1.el7.noarch ovirt-engine-dwh-setup-4.1.0-0.0.master.20161129154019.el7.centos.noarch ovirt-imageio-proxy-setup-0.5.0-0.201611201242.gitb02532b.el7.centos.noarch ovirt-engine-tools-backup-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-websocket-proxy-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-setup-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-backend-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-tools-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-webadmin-portal-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-restapi-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-vmconsole-proxy-helper-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-setup-plugin-ovirt-engine-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-wildfly-overlay-10.0.0-1.el7.noarch ovirt-engine-cli-3.6.9.2-1.el7.centos.noarch ovirt-web-ui-0.1.1-2.el7.centos.x86_64 ovirt-engine-setup-base-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-vmconsole-1.0.4-1.el7.centos.noarch ovirt-engine-dwh-4.1.0-0.0.master.20161129154019.el7.centos.noarch ovirt-engine-setup-plugin-websocket-proxy-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-hosts-ansible-inventory-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-engine-dashboard-1.1.0-0.4.20161128git5ed6f96.el7.centos.noarch ovirt-engine-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-guest-agent-common-1.0.13-1.20161220085008.git165fff1.el7.centos.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7.centos.noarch ovirt-engine-wildfly-10.1.0-1.el7.x86_64 ovirt-engine-lib-4.1.0-0.3.beta2.20161221085908.el7.centos.noarch ovirt-vmconsole-proxy-1.0.4-1.el7.centos.noarch Linux version 3.10.0-514.2.2.el7.x86_64 (builder.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Dec 6 23:06:41 UTC 2016 Linux 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux CentOS Linux release 7.3.1611 (Core)