Bug 882825 - Failed Hot-Plug and Hot-UnPlug Direct LUN in FC DC environment
Failed Hot-Plug and Hot-UnPlug Direct LUN in FC DC environment
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.1.0
x86_64 Linux
unspecified Severity urgent
: ---
: 3.2.0
Assigned To: Maor
vvyazmin@redhat.com
storage
: Regression, ZStream
Depends On:
Blocks: 888039 915537
  Show dependency treegraph
 
Reported: 2012-12-03 02:52 EST by vvyazmin@redhat.com
Modified: 2016-02-10 15:26 EST (History)
13 users (show)

See Also:
Fixed In Version: sf3
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 888039 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
## Logs rhevm, vdsm, screen-shots (1.44 MB, application/x-gzip)
2012-12-03 02:52 EST, vvyazmin@redhat.com
no flags Details

  None (edit)
Description vvyazmin@redhat.com 2012-12-03 02:52:38 EST
Created attachment 656423 [details]
## Logs rhevm, vdsm, screen-shots

Description of problem:
Failed Hot-Plug and Hot-UnPlug Direct LUN in FC DC environment

Version-Release number of selected component (if applicable):

RHEVM 3.0 environment:
RHEVM: rhevm-3.0.7_0001-2.el6_3.x86_64 
VDSM: vdsm-4.9-113.4.el6_3.x86_64 
LIBVIRT: libvirt-0.9.10-21.el6_3.6.x86_64 
QUMU-KVM: qemu-kvm-rhev-0.12.1.2-2.295.el6_3.5.x86_64

RHEVM 3.1 environment:
RHEVM: rhevm-3.1.0-32.el6ev.noarch
VDSM: vdsm-4.9.6-44.0.el6_3.x86_64 
LIBVIRT: libvirt-0.9.10-21.el6_3.6.x86_64 
QUMU-KVM: qemu-kvm-rhev-0.12.1.2-2.295.el6_3.5.x86_64 
SANLOCK: sanlock-2.3-4.el6_3.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Upgrade RHEVM from 3.0 to 3.1 environment
2. Create VM with disks on multiple storage domain in FC DC environment
3. Hot-Plug and Hot-UnPlug Direct LUN 
  
Actual results:
Failed Hot-Plug and Hot-UnPlug Direct LUN in FC DC environment

Expected results:
Succeed Hot-Plug and Hot-UnPlug Direct LUN in FC DC environment

Additional info:

No problem found on iSCSI DC environment

/var/log/vdsm/vdsm.log
Thread-18962::DEBUG::2012-12-02 18:11:19,502::BindingXMLRPC::171::vds::(wrapper) [10.35.97.166]
Thread-18962::DEBUG::2012-12-02 18:11:19,503::task::588::TaskManager.Task::(_updateState) Task=`d6e2fdb6-9a81-491b-936e-18aa2c4cd893`::moving from state init -> state preparing
Thread-18962::INFO::2012-12-02 18:11:19,503::logUtils::37::dispatcher::(wrapper) Run and protect: getDeviceList(storageType=3, options={})
Thread-18962::DEBUG::2012-12-02 18:11:19,503::misc::1044::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-18962::DEBUG::2012-12-02 18:11:19,504::misc::1046::SamplingMethod::(__call__) Got in to sampling method
Thread-18962::DEBUG::2012-12-02 18:11:19,504::misc::1044::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-18962::DEBUG::2012-12-02 18:11:19,504::misc::1046::SamplingMethod::(__call__) Got in to sampling method
Thread-18962::DEBUG::2012-12-02 18:11:19,504::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-18962::DEBUG::2012-12-02 18:11:19,527::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
Thread-18962::DEBUG::2012-12-02 18:11:19,528::misc::1054::SamplingMethod::(__call__) Returning last result

/var/log/ovirt-engine/engine.log
2012-12-02 18:11:31,184 ERROR [org.ovirt.engine.core.bll.AttachDiskToVmCommand] (http-/0.0.0.0:8443-12) [684fca0b] Command org.ovirt.engine.core.bll.AttachDiskToVmCommand throw 
exception: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at java.util.ArrayList.rangeCheck(ArrayList.java:604) [rt.jar:1.7.0_09-icedtea]
        at java.util.ArrayList.get(ArrayList.java:382) [rt.jar:1.7.0_09-icedtea]
        at org.ovirt.engine.core.bll.AbstractDiskVmCommand.performPlugCommnad(AbstractDiskVmCommand.java:58) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.AttachDiskToVmCommand.ExecuteVmCommand(AttachDiskToVmCommand.java:122) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.VmCommand.executeCommand(VmCommand.java:78) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:876) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:973) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1372) [engine-bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:142) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:109) [engine-utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:991) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:291) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:350) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.Backend.RunAction(Backend.java:299) [engine-bll.jar:]
        at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source) [:1.7.0_09-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.ovirt.engine.core.utils.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:11) [engine-utils.jar:]
        at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) [:1.7.0_09-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-3]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-3]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.3.Final-redhat-3]
        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:181) [jboss-as-ee.jar:7.1.3.Final-redhat-3]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-3]
        at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view8.RunAction(Unknown Source) [engine-common.jar:]
        at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.RunAction(GenericApiGWTServiceImpl.java:120)
        at sun.reflect.GeneratedMethodAccessor209.invoke(Unknown Source) [:1.7.0_09-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
        at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:196)
        at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:161)
        at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:222)
        at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec.jar:1.0.1.Final-redhat-2]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.1.Final-redhat-2]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
       at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489)
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:372)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:679)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931)
        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]

2012-12-02 18:11:31,203 ERROR [org.ovirt.engine.core.bll.AttachDiskToVmCommand] (http-/0.0.0.0:8443-12) [684fca0b] Transaction rolled-back for command: org.ovirt.engine.core.bll.AttachDiskToVmCommand.
Comment 5 Maor 2012-12-04 11:12:21 EST
We also get the same exception in attach and detach flows for FC disk
Comment 6 Maor 2012-12-04 12:39:01 EST
Posted patch upstream at http://gerrit.ovirt.org/#/c/9724/
Comment 8 Maor 2012-12-17 09:09:29 EST
Merged fix with change-id I6414445b9a74be299205ff7fc9a21d1388a29687
Comment 10 vvyazmin@redhat.com 2013-01-16 10:50:52 EST
RHEVM 3.2 - SF03 environment:

RHEVM: rhevm-3.2.0-4.el6ev.noarch
VDSM: vdsm-4.10.2-3.0.el6ev.x86_64
LIBVIRT: libvirt-0.10.2-13.el6.x86_64
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64
SANLOCK: sanlock-2.6-2.el6.x86_64
Comment 11 Itamar Heim 2013-06-11 04:50:27 EDT
3.2 has been released
Comment 12 Itamar Heim 2013-06-11 04:50:27 EDT
3.2 has been released
Comment 13 Itamar Heim 2013-06-11 04:50:27 EDT
3.2 has been released
Comment 14 Itamar Heim 2013-06-11 04:54:56 EDT
3.2 has been released
Comment 15 Itamar Heim 2013-06-11 05:25:17 EDT
3.2 has been released

Note You need to log in before you can comment on or make changes to this bug.