Bug 909967 - engine: we fail to move host from non-operational to maintenance when storage is disconnected on host
Summary: engine: we fail to move host from non-operational to maintenance when storage...
Keywords:
Status: CLOSED DUPLICATE of bug 910005
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.1.3
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 3.3.0
Assignee: Oved Ourfali
QA Contact:
URL:
Whiteboard: infra
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-02-11 14:51 UTC by Dafna Ron
Modified: 2016-02-10 19:13 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-06-09 08:41:57 UTC
oVirt Team: Infra
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
logs (405.42 KB, application/x-gzip)
2013-02-11 14:51 UTC, Dafna Ron
no flags Details

Description Dafna Ron 2013-02-11 14:51:33 UTC
Created attachment 696112 [details]
logs

Description of problem:

I created 1 iscsi storage domain on 2 hosts cluster -> blocked the domain in both hosts using iptables 
1 host stayed in up state and I was able to move it maintenance
the other host became non-operation and when we try to put it in maintenance we fail and go back to non-operational. 

2013-02-11 15:58:07,217 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) Host encounter a problem moving to maintenance mode. The Host status will change to Non operational status.

although this is a different scenrio, this may be connected to bug: https://bugzilla.redhat.com/show_bug.cgi?id=874019 
which was fixed from 3.2

Version-Release number of selected component (if applicable):

si27

How reproducible:

100%

Steps to Reproduce:
1. install two hsots with vdsm-4.10.2-1.3.el6.x86_64 and create a pool with 1 iscsi storage domain
2. block connectivity to the storage domain from both hosts
3. try to put both hosts in maintenance 
  
Actual results:

we fail to move the non-operational host to maintenance 

Expected results:

we should be able to move the non-operational host to mainetanance 

Additional info: logs 

013-02-11 15:58:07,345 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-52) [7eca4e51] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 5d82a420-743a-11e2-a041-001a4a16971d Type: VDS
2013-02-11 15:58:07,347 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [7eca4e51] START, SetVdsStatusVDSCommand(HostName = cougar01, HostId = 5d82a420-743a-11e2-a041-001a4a16971d, status=NonOperational, nonOperationalReason=NONE), log id: 563f0f43
2013-02-11 15:58:07,349 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [7eca4e51] FINISH, SetVdsStatusVDSCommand, log id: 563f0f43
2013-02-11 15:58:07,430 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [7eca4e51] ResourceManager::RerunFailedCommand: Error: VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Failed to initialize storage, vds = 5d82a420-743a-11e2-a041-001a4a16971d : cougar01
2013-02-11 15:58:07,430 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [7eca4e51] VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Failed to initialize storage: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Failed to initialize storage
        at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:212) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.MaintananceVdsCommand.ProcessStorageOnVdsInactive(MaintananceVdsCommand.java:178) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.VdsEventListener.vdsMovedToMaintanance(VdsEventListener.java:69) [engine-bll.jar:]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
:

Comment 6 Yaniv Bronhaim 2013-06-09 08:41:57 UTC
This bug is targeted to 3.3, 
The issue that represent here is related to vdsm startup, if you look in cougar01 log you can see the repeated print:
MainThread::WARNING::2013-02-11 13:58:55,482::supervdsm::183::SuperVdsmProxy::(_connect) Connect to svdsm failed [Errno 2] No such file or directory
MainThread::DEBUG::2013-02-11 13:58:56,484::supervdsm::179::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm

This was fixed for 3.2 as part https://bugzilla.redhat.com/show_bug.cgi?id=910005 and changed all over again as part of 3.3 features - http://www.ovirt.org/Features/Supervdsm_service

*** This bug has been marked as a duplicate of bug 910005 ***


Note You need to log in before you can comment on or make changes to this bug.