Created attachment 696112[details]
logs
Description of problem:
I created 1 iscsi storage domain on 2 hosts cluster -> blocked the domain in both hosts using iptables
1 host stayed in up state and I was able to move it maintenance
the other host became non-operation and when we try to put it in maintenance we fail and go back to non-operational.
2013-02-11 15:58:07,217 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) Host encounter a problem moving to maintenance mode. The Host status will change to Non operational status.
although this is a different scenrio, this may be connected to bug: https://bugzilla.redhat.com/show_bug.cgi?id=874019
which was fixed from 3.2
Version-Release number of selected component (if applicable):
si27
How reproducible:
100%
Steps to Reproduce:
1. install two hsots with vdsm-4.10.2-1.3.el6.x86_64 and create a pool with 1 iscsi storage domain
2. block connectivity to the storage domain from both hosts
3. try to put both hosts in maintenance
Actual results:
we fail to move the non-operational host to maintenance
Expected results:
we should be able to move the non-operational host to mainetanance
Additional info: logs
013-02-11 15:58:07,345 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-52) [7eca4e51] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5d82a420-743a-11e2-a041-001a4a16971d Type: VDS
2013-02-11 15:58:07,347 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [7eca4e51] START, SetVdsStatusVDSCommand(HostName = cougar01, HostId = 5d82a420-743a-11e2-a041-001a4a16971d, status=NonOperational, nonOperationalReason=NONE), log id: 563f0f43
2013-02-11 15:58:07,349 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [7eca4e51] FINISH, SetVdsStatusVDSCommand, log id: 563f0f43
2013-02-11 15:58:07,430 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [7eca4e51] ResourceManager::RerunFailedCommand: Error: VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Failed to initialize storage, vds = 5d82a420-743a-11e2-a041-001a4a16971d : cougar01
2013-02-11 15:58:07,430 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [7eca4e51] VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Failed to initialize storage: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Failed to initialize storage
at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:212) [engine-bll.jar:]
at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [engine-bll.jar:]
at org.ovirt.engine.core.bll.MaintananceVdsCommand.ProcessStorageOnVdsInactive(MaintananceVdsCommand.java:178) [engine-bll.jar:]
at org.ovirt.engine.core.bll.VdsEventListener.vdsMovedToMaintanance(VdsEventListener.java:69) [engine-bll.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
:
This bug is targeted to 3.3,
The issue that represent here is related to vdsm startup, if you look in cougar01 log you can see the repeated print:
MainThread::WARNING::2013-02-11 13:58:55,482::supervdsm::183::SuperVdsmProxy::(_connect) Connect to svdsm failed [Errno 2] No such file or directory
MainThread::DEBUG::2013-02-11 13:58:56,484::supervdsm::179::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm
This was fixed for 3.2 as part https://bugzilla.redhat.com/show_bug.cgi?id=910005 and changed all over again as part of 3.3 features - http://www.ovirt.org/Features/Supervdsm_service
*** This bug has been marked as a duplicate of bug 910005 ***