Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 908365

Summary:

engine: we fail to end LiveMigrateDisk command with exception: java.lang.ArrayIndexOutOfBoundsException: -1

Product:

Red Hat Enterprise Virtualization Manager

Reporter:

Dafna Ron <dron>

Component:

ovirt-engine

Assignee:

Maor <mlipchuk>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Leonid Natapov <lnatapov>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

3.2.0

CC:

abaron, acathrow, amureini, dyasny, hateya, iheim, lpeer, Rhev-m-bugs, scohen, yeylon, ykaul

Target Milestone:

---

Flags:

scohen: Triaged+

Target Release:

3.2.0

Hardware:

x86_64

OS:

Linux

Whiteboard:

storage

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

Storage

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
logs	none

Description Dafna Ron 2013-02-06 14:24:55 UTC

Created attachment 693957 [details]
logs

Description of problem:

I live migrated several disks and the command failed for some of the vms. 
engine fails to end the command with exception: java.lang.ArrayIndexOutOfBoundsException: -1

since we keep trying to end the command, not only the log is filled with this error but the event log keeps reporting the command as failed. 

Version-Release number of selected component (if applicable):

sf5
vdsm-4.10.2-5.0.el6ev.x86_64
libvirt-0.10.2-18.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64

How reproducible:

100%

Steps to Reproduce:
1. create 3 iscsi storage domains and 20 vm's 
2. live migrate all the vms to 2 domains
3.
  
Actual results:

some of the vm's fail to migrate and we are getting 'java.lang.ArrayIndexOutOfBoundsException: -1' in the engine logs
also, the event log keeps reporting the same actions as failed. 

Expected results:

we should be able to end the actions

Additional info: logs

2013-02-06 14:46:02,904 ERROR [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-3-thread-30) EntityAsyncTask::EndCommandAction [within thread]: EndAction for action type LiveMigrateDisk threw an exception: java.lang.ArrayIndexOutOfBoundsException: -1
        at java.util.Arrays$ArrayList.get(Arrays.java:2866) [rt.jar:1.7.0_09-icedtea]
        at org.ovirt.engine.core.bll.CommandBase.getCurrentTaskHandler(CommandBase.java:1700) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.internalEndWithFailure(CommandBase.java:534) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:472) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1470) [engine-bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:108) [engine-utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:416) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.Backend.endAction(Backend.java:377) [engine-bll.jar:]
        at sun.reflect.GeneratedMethodAccessor198.invoke(Unknown Source) [:1.7.0_09-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.ovirt.engine.core.utils.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:11) [engine-utils.jar:]
        at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) [:1.7.0_09-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]

Comment 1 Maor 2013-03-17 17:19:45 UTC

Which sf did it reproduced, it seams it does not reproduce any more.

Comment 2 Haim 2013-03-17 21:22:45 UTC

(In reply to comment #1)
> Which sf did it reproduced, it seams it does not reproduce any more.

bug was opened on SF5, are you running the exact repro steps? are you aware of a patch in this area that might have fixed the bug?

Comment 3 Maor 2013-03-21 13:11:25 UTC

(In reply to comment #2)
> (In reply to comment #1)
> > Which sf did it reproduced, it seams it does not reproduce any more.
> 
> bug was opened on SF5, are you running the exact repro steps? are you aware
> of a patch in this area that might have fixed the bug?

Yes, I created 3 ISCSI domains, and ran 12 VMs and moved their disks to two domains as described.
I found some patches witch might solved it, but I can't be sure.
I will attach the log of the reproduced scenario.

Comment 5 Haim 2013-03-21 13:27:09 UTC

since Maor can't reproduce and we suspect a fix in this area, moving to ON_QA which will be verified by my guys, we will re-open if necessary.

Comment 6 Leonid Natapov 2013-03-28 11:52:01 UTC

sf11

Comment 8 Itamar Heim 2013-06-11 08:56:13 UTC

3.2 has been released

Comment 9 Itamar Heim 2013-06-11 08:56:13 UTC

3.2 has been released

Comment 10 Itamar Heim 2013-06-11 08:56:24 UTC

3.2 has been released

Comment 11 Itamar Heim 2013-06-11 08:58:55 UTC

3.2 has been released

Comment 12 Itamar Heim 2013-06-11 09:28:22 UTC

3.2 has been released