Bug 999024 - [engine-backend] EJBComponentUnavailableException when engine crashes after task of CreateSnapshotVDSCommand was sent
[engine-backend] EJBComponentUnavailableException when engine crashes after t...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.3.0
x86_64 Unspecified
unspecified Severity medium
: ---
: 3.4.0
Assigned To: Martin Perina
Tareq Alayan
infra
: Triaged
Depends On:
Blocks: rhev3.4beta 1142926
  Show dependency treegraph
 
Reported: 2013-08-20 10:35 EDT by Elad
Modified: 2016-02-10 14:14 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-12 07:54:35 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs (928.30 KB, application/x-gzip)
2013-08-20 10:35 EDT, Elad
no flags Details
engine.log (1.26 MB, application/x-gzip)
2013-08-25 10:25 EDT, Elad
no flags Details
server.log (45 bytes, application/x-gzip)
2013-09-10 02:46 EDT, Elad
no flags Details

  None (edit)
Description Elad 2013-08-20 10:35:58 EDT
Created attachment 788521 [details]
logs

Description of problem:
When engine looses it's connection with host after CreateSnapshotVDSCommand was send, it throws this exception in log:

2013-08-20 16:01:40,877 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) EJBComponentUnavailableException: JBAS014559: Invocation cannot proceed as component is shutting down: org.jboss.as.ejb3.component.EJBComponentUnavailableException: JBAS014559: Invocation cannot proceed as component is shutting down
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:59) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]

Version-Release number of selected component (if applicable):
rhevm-3.3.0-0.15.master.el6ev.noarch

How reproducible:
100%

Steps to Reproduce:
On a block pool:
- create a snapshot to vm and restart vdsm right after the task starts


Actual results:
When vdsm crashes after engine send the createVolume command, engine throws the mentioned exception. 
There are no other issues, just the exception in log

Expected results:
Engine should revert the CreateSnapshotVDSCommand without throwing these exception 


Additional info:
logs
Comment 1 Elad 2013-08-20 11:02:48 EDT
CORRECTION:
Engine throws this exception after it crashes after CreateSnapshotVDSCommand was send.
So the reproduction step is to create a snapshot to vm and restart ovirt-engine right after the task starts on spm.
Comment 2 Barak 2013-08-25 09:59:27 EDT
Does the operation succeeds ?
Comment 3 Elad 2013-08-25 10:23:58 EDT
No, engine performs roll-back for CreateAllSnapshotsFromVmCommand. The roll-back ends successfully. The only problem here is the log issue

Attaching the relevant engine.log
Comment 4 Elad 2013-08-25 10:25:28 EDT
Created attachment 790093 [details]
engine.log
Comment 5 Roy Golan 2013-09-10 02:21:43 EDT
(In reply to Elad from comment #1)
> CORRECTION:
> Engine throws this exception after it crashes after CreateSnapshotVDSCommand
> was send.
> So the reproduction step is to create a snapshot to vm and restart
> ovirt-engine right after the task starts on spm.

1st please supply the server.log as well

2nd The engine doesn't crash. it shuts down so the EJB components (LockManager, DB connection manager etc) throws exceptions with a clear message (component is shutting down)

The real issue here is WHY the engine shuts down? 

bug  962180 with possibly similar case so I want to know if thats a real issue.
Comment 6 Elad 2013-09-10 02:44:27 EDT
(In reply to Roy Golan from comment #5)
> (In reply to Elad from comment #1)
> > CORRECTION:
> > Engine throws this exception after it crashes after CreateSnapshotVDSCommand
> > was send.
> > So the reproduction step is to create a snapshot to vm and restart
> > ovirt-engine right after the task starts on spm.
> 
> 1st please supply the server.log as well
> 
> 2nd The engine doesn't crash. it shuts down so the EJB components
> (LockManager, DB connection manager etc) throws exceptions with a clear
> message (component is shutting down)
> 
> The real issue here is WHY the engine shuts down? 
That was part of the my test - the restart was done manually. The only issue is the ugly exception


server.log will be attached
Comment 7 Elad 2013-09-10 02:46:30 EDT
Created attachment 795856 [details]
server.log
Comment 8 Roy Golan 2013-09-10 02:54:32 EDT
(In reply to Elad from comment #6)
> (In reply to Roy Golan from comment #5)
> > (In reply to Elad from comment #1)
> > > CORRECTION:
> > > Engine throws this exception after it crashes after CreateSnapshotVDSCommand
> > > was send.
> > > So the reproduction step is to create a snapshot to vm and restart
> > > ovirt-engine right after the task starts on spm.
> > 
> > 1st please supply the server.log as well
> > 
> > 2nd The engine doesn't crash. it shuts down so the EJB components
> > (LockManager, DB connection manager etc) throws exceptions with a clear
> > message (component is shutting down)
> > 
> > The real issue here is WHY the engine shuts down? 
> That was part of the my test - the restart was done manually. The only issue
> is the ugly exception
> 

not an issue.
> 
> server.log will be attached
Comment 9 Martin Perina 2013-12-12 07:54:35 EST
(In reply to Elad from comment #0)
> Created attachment 788521 [details]
> logs
> 
> Description of problem:
> When engine looses it's connection with host after CreateSnapshotVDSCommand
> was send, it throws this exception in log:
> 
> 2013-08-20 16:01:40,877 ERROR
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-74) EJBComponentUnavailableException:
> JBAS014559: Invocation cannot proceed as component is shutting down:
> org.jboss.as.ejb3.component.EJBComponentUnavailableException: JBAS014559:
> Invocation cannot proceed as component is shutting down
>         at
> org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.
> processInvocation(ShutDownInterceptorFactory.java:59)
> [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
>         at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
> [jboss-invocation.jar:1.1.1.Final-redhat-2]

This is standard JBoss error message when you try to invoke some method on EJB during JBoss shutdown. Since the rollback was successful, there's no error.

Note You need to log in before you can comment on or make changes to this bug.