1646161 – IndexOutOfBoundsException: Index: 0, Size: 0 during import of VM

Bug 1646161 - IndexOutOfBoundsException: Index: 0, Size: 0 during import of VM

Summary: IndexOutOfBoundsException: Index: 0, Size: 0 during import of VM

Keywords:
Status:	CLOSED CANTFIX
Alias:	None
Product:	ovirt-engine
Classification:	oVirt
Component:	BLL.Storage
Sub Component:
Version:	4.2.7.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	urgent
Target Milestone:	ovirt-4.3.4
Target Release:	---
Assignee:	shani
QA Contact:	Elad
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-11-05 09:47 UTC by Petr Kubica
Modified:	2019-04-07 09:43 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Clone Of:
Environment:
Last Closed:	2019-04-07 09:43:18 UTC
oVirt Team:	Storage
Embargoed:
Dependent Products:
Flags:	rule-engine: ovirt-4.3+

Attachments	(Terms of Use)
lvscan and engine.log - scan disk (283.63 KB, text/plain) 2018-11-12 13:03 UTC, Petr Kubica	no flags	Details
View All

Description Petr Kubica 2018-11-05 09:47:17 UTC

Description of problem:
We had a huge iSCSI domain with more than hundred VMs (<4TB domain), we wanted to move that domain into another datacenter (domain wasn't master at migration time) via documentation [1].
After domain was attached into second datacenter, lots of VMs are missing in VM Import tab at imported domain and there is lots of VMs which were deleted few days before migration.
We want to import only VMs which was on storage domain before detaching domain from the first datacenter. Half of these VMs was imported successfully but the second half isn't. It fails with message General command validation failure and in engine log I can see [2] exception.
Many of VMs, which was in first datacenter before detaching the domain aren't listed in Import VMs tab.

Short info what happened: There are 4 groups of VMs after migration
- VMs which was sucessfully imported after migration of domain
- VMs listed (and should be there) but couldn't be imported - message General command validation failure
- VMs which are listed but they're was deleted before detaching domain 
- VMs which was on storage domain before detaching, but after attaching in secondary datacenter these VMs are not listed anywhere.

[1] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/html/administration_guide/sect-importing_existing_storage_domains#Migrating_SD_between_DC_Same_Env

[2] 2018-11-05 10:17:59,560+01 INFO  [org.ovirt.engine.core.bll.exportimport.ImportVmFromConfigurationCommand] (default task-185) [97009a46-235b-498a-9ecd-f570d0b6b626] Lock Acquired to object 'EngineLock:{exclusiveLocks='[b85142f5-4aac-412a-ae4d-aff541c33ae8=VM, brq-w2k12r2=VM_NAME]', sharedLocks='[b85142f5-4aac-412a-ae4d-aff541c33ae8=REMOTE_VM]'}'
2018-11-05 10:17:59,662+01 ERROR [org.ovirt.engine.core.bll.exportimport.ImportVmFromConfigurationCommand] (default task-185) [97009a46-235b-498a-9ecd-f570d0b6b626] Error during ValidateFailure.: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at java.util.ArrayList.rangeCheck(ArrayList.java:657) [rt.jar:1.8.0_181]
	at java.util.ArrayList.get(ArrayList.java:433) [rt.jar:1.8.0_181]
	at org.ovirt.engine.core.bll.validator.ImportValidator.validateStorageExistsForMemoryDisks(ImportValidator.java:144) [bll.jar:]
	at org.ovirt.engine.core.bll.exportimport.ImportVmFromConfigurationCommand.isValidDisks(ImportVmFromConfigurationCommand.java:151) [bll.jar:]
	at org.ovirt.engine.core.bll.exportimport.ImportVmFromConfigurationCommand.validate(ImportVmFromConfigurationCommand.java:103) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:792) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.validateOnly(CommandBase.java:381) [bll.jar:]
	at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.canRunActions(PrevalidatingMultipleActionsRunner.java:113) [bll.jar:]
	at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.invokeCommands(PrevalidatingMultipleActionsRunner.java:99) [bll.jar:]
	at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.execute(PrevalidatingMultipleActionsRunner.java:76) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runMultipleActionsImpl(Backend.java:596) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runMultipleActions(Backend.java:566) [bll.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_181]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_181]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_181]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:78)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor266.invoke(Unknown Source) [:1.8.0_181]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_181]
	at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:264) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:379) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl.jar:2.4.7.Final-redhat-1]
	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
	at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runMultipleActions(Unknown Source) [common.jar:]
	at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runMultipleActions(GenericApiGWTServiceImpl.java:161)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_181]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_181]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_181]
	at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:587)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:333)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:303)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373)
	at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [jboss-servlet-api_3.1_spec.jar:1.0.0.Final-redhat-1]
	at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.service(GenericApiGWTServiceImpl.java:78)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec.jar:1.0.0.Final-redhat-1]
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
	at org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) [utils.jar:]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.utils.servlet.CachingFilter.doFilter(CachingFilter.java:133) [utils.jar:]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:73) [branding.jar:]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:65) [utils.jar:]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:65)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1501)
	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1501)
	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1501)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:330)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]

2018-11-05 10:17:59,676+01 INFO  [org.ovirt.engine.core.bll.exportimport.ImportVmFromConfigurationCommand] (default task-185) [97009a46-235b-498a-9ecd-f570d0b6b626] Lock freed to object 'EngineLock:{exclusiveLocks='[b85142f5-4aac-412a-ae4d-aff541c33ae8=VM, brq-w2k12r2=VM_NAME]', sharedLocks='[b85142f5-4aac-412a-ae4d-aff541c33ae8=REMOTE_VM]'}'


Version-Release number of selected component (if applicable):
ovirt-engine-4.2.7.3-0.1.el7ev.noarch

Steps to Reproduce:
1. detach iSCSI domain from first datacenter with hundreds of VMs
2. attach domain in second datacenter

Additional info:
severity set to urgent due to lost of data

Comment 3 Red Hat Bugzilla Rules Engine 2018-11-06 12:57:27 UTC

This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 10 Petr Kubica 2018-11-12 09:28:48 UTC

> Short info what happened: There are 4 groups of VMs after migration
> - VMs which was sucessfully imported after migration of domain
no action item here
> - VMs listed (and should be there) but couldn't be imported - message
> General command validation failure
already WA, I think this bug is related to this issue

These 2 are still blockers in our production, should we create separate bugs?
> - VMs which are listed but they're deleted before detaching domain 
These VMs metadata were incorrect on SD, and had to be fixed from backup of DB.
Afterwards VMs were successfully restored.

> - VMs which was on storage domain before detaching, but after attaching in
> secondary datacenter these VMs are not listed anywhere.
>
These VMs can be seen in backup-DB, however, in current DB after attaching SD they are not listed. On the SD we can see them, however we suspect wrong metadata as oVirt wont list them.

Any WA here? 

Furthermore,

scanning discs on the storage domain, is not finding all the discs that we can see on SD (lvdisplay), can this be cause again by wrong metadata, or can we suspect some missing crucial data from storage domain, thus potentially lost of data?

Comment 11 Eyal Shenitzky 2018-11-12 10:06:54 UTC

> These 2 are still blockers in our production, should we create separate bugs?
> > - VMs which are listed but they're deleted before detaching domain 
> These VMs metadata were incorrect on SD, and had to be fixed from backup of
> DB.
> Afterwards VMs were successfully restored.

No, this is expected since the OVF update failed
> 
> > - VMs which was on storage domain before detaching, but after attaching in
> > secondary datacenter these VMs are not listed anywhere.
> >
> These VMs can be seen in backup-DB, however, in current DB after attaching
> SD they are not listed. On the SD we can see them, however we suspect wrong
> metadata as oVirt wont list them.
> 
> Any WA here? 

Just import their disks and recreate the VMs
> 
> Furthermore,
> 
> scanning discs on the storage domain, is not finding all the discs that we
> can see on SD (lvdisplay), can this be cause again by wrong metadata, or can
> we suspect some missing crucial data from storage domain, thus potentially
> lost of data?
Do you mean scanning disk using oVirt?

Comment 12 Petr Kubica 2018-11-12 13:02:23 UTC

(In reply to Eyal Shenitzky from comment #11)
> > These 2 are still blockers in our production, should we create separate bugs?
> > > - VMs which are listed but they're deleted before detaching domain 
> > These VMs metadata were incorrect on SD, and had to be fixed from backup of
> > DB.
> > Afterwards VMs were successfully restored.
> 
> No, this is expected since the OVF update failed
> > 
> > > - VMs which was on storage domain before detaching, but after attaching in
> > > secondary datacenter these VMs are not listed anywhere.
> > >
> > These VMs can be seen in backup-DB, however, in current DB after attaching
> > SD they are not listed. On the SD we can see them, however we suspect wrong
> > metadata as oVirt wont list them.
> > 
> > Any WA here? 
> 
> Just import their disks and recreate the VMs
Their disks are not listed too (in disk import).

> > 
> > Furthermore,
> > 
> > scanning discs on the storage domain, is not finding all the discs that we
> > can see on SD (lvdisplay), can this be cause again by wrong metadata, or can
> > we suspect some missing crucial data from storage domain, thus potentially
> > lost of data?
> Do you mean scanning disk using oVirt?
Yes, I do. I will attach logs from scanning disk.

Comment 13 Petr Kubica 2018-11-12 13:03:15 UTC

Created attachment 1504710 [details]
lvscan and engine.log - scan disk

Comment 14 Tal Nisan 2019-01-07 12:16:32 UTC

We've tried to reproduce it in the scenario described and in other scenarios concerning missing memory disks (on the same domain, different domain, missing domain, domain on another pool) and they all worked correctly.
The only way to reach the exception is when the memory disk entity exists yet a corresponding image entity is missing from the database which is a situation that should never happen and we could not reproduce.
For now we cannot reproduce and thus cannot find a solution so we're postponing to 4.3, if you can provide any other data please do

Comment 15 Petr Kubica 2019-01-08 07:22:34 UTC

(In reply to Tal Nisan from comment #14)
> We've tried to reproduce it in the scenario described and in other scenarios
> concerning missing memory disks (on the same domain, different domain,
> missing domain, domain on another pool) and they all worked correctly.
> The only way to reach the exception is when the memory disk entity exists
> yet a corresponding image entity is missing from the database which is a
> situation that should never happen and we could not reproduce.
> For now we cannot reproduce and thus cannot find a solution so we're
> postponing to 4.3, if you can provide any other data please do

We still have environment where did it happen if it could help. This environment exists from RHEV 3.3 
I could provide an access to this environment. Ping me on IRC or send me email about details if you're interested.

Comment 16 RHEL Program Management 2019-03-11 15:16:38 UTC

This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 17 Tal Nisan 2019-03-12 11:01:16 UTC

Not a blocker, this is a corner case

Comment 18 Tal Nisan 2019-04-07 09:43:18 UTC

This is a corner case in which an image chain is not updated correctly in the database, most likely due to a failed live merge in older Engine/VDSM versions, once the situation exists there's not much to do aside for following the manual steps mentioned in the comments, closing the bug due to that

Note You need to log in before you can comment on or make changes to this bug.