Bug 1064499

Created attachment 862464 [details]
engine, vdsm, libvirt and qemu logs

Description of problem:
I have a scenario in which I had 2 VMs that were failed to resume from a 'paused' state due to an AttributeError (reported here https://bugzilla.redhat.com/show_bug.cgi?id=1064471). I stopped the VMs from  UI, and they were reported as 'down' by engine and vdsm after I powered it off although libvirt reported them as 'paused' (reported here  https://bugzilla.redhat.com/show_bug.cgi?id=1063336)
When I tried to start the VM again, engine failed with SQL exception.


Version-Release number of selected component (if applicable):
ovirt-engine-3.4.0-0.7.beta2.el6.noarch
postgresql-8.4.18-1.el6_4.x86_64


Steps to Reproduce:
On shared storage DC with iSCSI domain
have a running VM with disk 
1. block connectivity from host to the storage server and wait for the domain to become 'inactive' and the VM to enter to 'paused' state 
2. resume connectivity to the storage server and wait for the domain to become 'active'. Try to resume the VM, it will fail
3. power-off the VM and try to start it again


Actual results:
Resuming the VM from 'paused' state fails with an AtrributeError as reported here https://bugzilla.redhat.com/show_bug.cgi?id=1064471.
After stopping the VMs, engine and vdsm report them as down:

## select vm_name,status from vm

 vm_name  | status
----------+--------
 shared-2 |      0
 shared-1 |      0

## vdsClient -s 0 list table

6ec3beea-c5a4-4e1b-93cb-37879754f0b7  29262  1                    Up
455425c8-d79f-4f18-85aa-48b516e1751a   5567  shared-3             Up

But libvirt reports the VMs as 'paused':

## virsh -r list

 Id    Name                           State
----------------------------------------------------
 6     shared-1                       paused
 7     shared-2                       paused
 8     1                              running
 10    shared-3                       running


QEMU processes are still running on the host:

## pgrep qemu

5567
20552
21055
29262

When I started the VMs again, I got this exception in engine.log:

2014-02-12 18:37:46,766 ERROR [org.ovirt.engine.core.bll.job.JobRepositoryImpl] (org.ovirt.thread.pool-6-thread-46) Failed to save step 533c8d4c-19df-46af-8a70-f031b99e964c, VALIDATING.: org.springframework.dao.Da
taIntegrityViolationException: CallableStatementCallback; SQL [{call insertstep(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}]; ERROR: insert or update on table "step" violates foreign key constraint "fk_step_job"
  Detail: Key (job_id)=(ab401dcd-1103-4dbb-ae46-a14438616647) is not present in table "job".
  Where: SQL statement "INSERT INTO step( step_id, parent_step_id, job_id, step_type, description, step_number, status, start_time, end_time, correlation_id, external_id, external_system_type, is_external) VALUES
(  $1 ,  $2 ,  $3 ,  $4 ,  $5 ,  $6 ,  $7 ,  $8 ,  $9 ,  $10 ,  $11 ,  $12 ,  $13 )"
PL/pgSQL function "insertstep" line 2 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: insert or update on table "step" violates foreign key constraint "fk_step_job"
  Detail: Key (job_id)=(ab401dcd-1103-4dbb-ae46-a14438616647) is not present in table "job".
  Where: SQL statement "INSERT INTO step( step_id, parent_step_id, job_id, step_type, description, step_number, status, start_time, end_time, correlation_id, external_id, external_system_type, is_external) VALUES
(  $1 ,  $2 ,  $3 ,  $4 ,  $5 ,  $6 ,  $7 ,  $8 ,  $9 ,  $10 ,  $11 ,  $12 ,  $13 )"
PL/pgSQL function "insertstep" line 2 at SQL statement
        at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:245) [spring-jdbc.jar:3.1.1.RELEASE]

 


Additional info: engine, vdsm, libvirt and qemu logs