Bug 1292178

Summary: StorageDomainIsMemberOfPool prevents hosted-engine upgrade
Product: [oVirt] ovirt-hosted-engine-ha Reporter: Simone Tiraboschi <stirabos>
Component: GeneralAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED CURRENTRELEASE QA Contact: Artyom <alukiano>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 1.3.3.3CC: alukiano, bugs, didi, fdeutsch, gklein, lveyde, msivak, nsoffer, rgolan, rmartins, sbonazzo, stirabos, ylavi
Target Milestone: ovirt-3.6.1Keywords: Regression, Triaged, ZStream
Target Release: 1.3.3.4Flags: rule-engine: ovirt-3.6.z+
rule-engine: blocker+
ylavi: planning_ack+
sbonazzo: devel_ack+
gklein: testing_ack+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: integration
Fixed In Version: Doc Type: Bug Fix
Doc Text:
After patch https://gerrit.ovirt.org/#/c/49598/ VDSM raises a new StorageDomainIsMemberOfPool exception when stopMonitoringDomain verb got called on a storage domain which is connected to a storagePool. Ignoring it because we know that we are still in safe condition since in that case VDSM already owns that storage domain.
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-13 14:38:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1284954    

Description Simone Tiraboschi 2015-12-16 16:36:13 UTC
Description of problem:
after patch https://gerrit.ovirt.org/#/c/49598/ VDSM raises a new StorageDomainIsMemberOfPool exception when stopMonitoringDomain verb got called on a storage domain which is connected to a storagePool.

Here we have to distinct issue:
1. hosted-engine storage domains from 3.5 where connected to a bootstrap storage pool that nobody uses. To perform the upgrade the HA agent should perform different actions including stopping the monitoring of that domain to start the SPM there to be able to create a new volume here for the hosted-engine configuration. The issue is that now stopMonitoringDomain fails.

2. starting from oVirt 3.6.1 RC4, the engine auto imports the hosted-engine storage domain, that means that it attaches the hosted-engine storage domain to the storage pool of the datacenter that contains the hosted-engine hosts.
So after that, the hosted-engine storage domain will be connected to a storage poll and so the stopMonitoringDomain verb will always fails.

Version-Release number of selected component (if applicable):
vdsm-4.17.13-1

How reproducible:
100%

Steps to Reproduce:
1. Deploy hosted-engine with vdsm-4.17.13-1
2.
3.

Actual results:
MainThread::INFO::2015-12-16 15:55:03,331::upgrade::879::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade) Upgrading to current version
MainThread::INFO::2015-12-16 15:55:05,406::upgrade::659::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_stopMonitoringDomain) Stop monitoring domain
MainThread::ERROR::2015-12-16 15:55:05,420::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Failed stopping monitoring domain: Storage domain is member of pool: 'domain=1c6698e1-bc9c-4169-8432-1a36c2d12bce'' - trying to restart agent


Expected results:
The system successfully upgrades from hosted-engine 3.5 to 3.6.

Additional info:

Comment 1 Nir Soffer 2015-12-16 17:18:20 UTC
(In reply to Simone Tiraboschi from comment #0)
> Description of problem:
> after patch https://gerrit.ovirt.org/#/c/49598/ VDSM raises a new
> StorageDomainIsMemberOfPool exception when stopMonitoringDomain verb got
> called on a storage domain which is connected to a storagePool.
> 
> Here we have to distinct issue:
> 1. hosted-engine storage domains from 3.5 where connected to a bootstrap
> storage pool that nobody uses. To perform the upgrade the HA agent should
> perform different actions including stopping the monitoring of that domain
> to start the SPM there to be able to create a new volume here for the
> hosted-engine configuration. The issue is that now stopMonitoringDomain
> fails.

stopMonitoringDomain must fail when you try to stop a domain which is part
of the pool. The fact that it did not fail before was a bug in vdsm.

We will not change this behavior.

You should not stop such domain monitor, or detect the failure and 
continue normally. This domain belong now to vdsm, and you cannot 
stop monitoring it, because vdsm owns it.

> 2. starting from oVirt 3.6.1 RC4, the engine auto imports the hosted-engine
> storage domain, that means that it attaches the hosted-engine storage domain
> to the storage pool of the datacenter that contains the hosted-engine hosts.
> So after that, the hosted-engine storage domain will be connected to a
> storage poll and so the stopMonitoringDomain verb will always fails.

Indeed, this is expected as explained above.

There is not regression here, hosted engine code should be fixed.
Changing component.

Comment 2 Red Hat Bugzilla Rules Engine 2015-12-17 07:56:00 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 3 Artyom 2015-12-27 13:39:43 UTC
Verified
Upgrade from 3.5 - Red Hat Enterprise Virtualization Hypervisor release 7.1 (20151015.0.el7ev)
==============================================================================================
ovirt-hosted-engine-ha-1.2.7.2-1.el7ev.noarch
ovirt-hosted-engine-setup-1.2.6.1-1.el7ev.noarch
vdsm-4.16.27-1.el7ev.x86_64

to 3.6 - Red Hat Enterprise Virtualization Hypervisor (Beta) release 7.2 (20151221.1.el7ev)
==============================================================================================
ovirt-hosted-engine-setup-1.3.1.3-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.3.6-1.el7ev.noarch
vdsm-4.17.13-1.el7ev.noarch

1) Install RHEV-H 3.5
2) Deploy hosted-engine on one host with NFS storage
3) Enable global maintenance(on host hosted-engine --set-maintenance --mode=global)
4) Upgrade engine to 3.6
5) Power off engine vm(hosted-engine --vm-poweroff)
6) Upgrade host to RHEV-H 3.6 via usb-key

Upgrade succeed and after host upgrade ovirt-he-agent and ovirt-ha-broker up
Also checked upgrade on RHEL.

Comment 4 Sandro Bonazzola 2016-01-13 14:38:20 UTC
oVirt 3.6.1 has been released, closing current release