Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 975946

Summary: [rhevm] Failed run Live Storage Migration for large amount of disks - Error checking for a transaction
Product: Red Hat Enterprise Virtualization Manager Reporter: vvyazmin <vvyazmin>
Component: ovirt-engineAssignee: Ayal Baron <abaron>
Status: CLOSED WONTFIX QA Contact: Aharon Canan <acanan>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.2.0CC: abaron, acanan, acathrow, amureini, hateya, iheim, jkt, lpeer, Rhev-m-bugs, scohen, yeylon, yzaslavs
Target Milestone: ---Flags: abaron: Triaged+
Target Release: 3.2.6   
Hardware: x86_64   
OS: Linux   
Whiteboard: storage
Fixed In Version: is1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-09 18:33:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 985043    
Bug Blocks:    
Attachments:
Description Flags
## Logs rhevm
none
## Logs rhevm, vdsm, libvirt, thread dump, superVdsm
none
## Logs rhevm, vdsm, libvirt, thread dump, vds_table none

Description vvyazmin@redhat.com 2013-06-19 16:16:09 UTC 
Created attachment 763036 [details]
## Logs rhevm

Summary: Failed run  Live Storage Migration for more than 50 disks

Description of problem: Failed run  Live Storage Migration for more than 50 disks


Version-Release number of selected component (if applicable):
RHEVM 3.2 - SF18 environment:

RHEVM: rhevm-3.2.1-0.31.el6ev.noarch
VDSM: vdsm-4.10.2-23.0.el6ev.x86_64
LIBVIRT: libvirt-0.10.2-18.el6_4.5.x86_64
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.355.el6_4.5.x86_64
SANLOCK: sanlock-2.6-2.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Create FCP Data-center
2. Create 100 VM with 1 disk with 'VirtIO' interface
3. Power on all those VM
4. Run Live Storage Migration (LSM) from DC-01 to DC-02 to all 100 disks
  
Relevant information:
I succeed run LSM for 50 disk's

Actual results:
Failed run LSM for all disks

Expected results:
Successful run LSM

Impact on user:

Workaround: 
Run LSM for with bulk maximum 30 disk.

Additional info:

/var/log/ovirt-engine/engine.log

2013-06-17 13:04:58,524 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (pool-4-thread-221) [148d0935] START, CreateSnapshotVDSCommand( storagePoolId = f5b6630b-4b7a-4e8c-952a-c2aa3b7fe1d5, ignoreFailoverLimit 
= false, compatabilityVersion = 3.2, storageDomainId = eab483e7-745f-4eb2-bde4-cc9dd77eb67f, imageGroupId = c03c1791-02b5-4f5a-99b8-0bd27086460f, imageSizeInBytes = 21474836480, volumeFormat = COW, newImageId = 571c40fc-e645-46c8-b453-f6
b3892efbb1, newImageDescription = , imageId = b311316b-d149-4b9c-8cf6-40d9a7810c03, sourceImageGroupId = c03c1791-02b5-4f5a-99b8-0bd27086460f), log id: 73be1c51
2013-06-17 13:04:59,974 ERROR [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] (Transaction Reaper Worker 0) Transaction rolled-back for command: org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.

2013-06-17 13:09:52,636 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (pool-4-thread-104) [54ffa8eb] Command org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw exception: javax.ejb.EJBException: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:191) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:237) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:218) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at 

/var/log/vdsm/vdsm.log
Comment 1 Allon Mureinik 2013-07-09 12:33:28 UTC 
In current upstream I could not find any VDSM call from within the transaction.
The last one was removed by 2575a223515a4f984157e8017e272cdd5ac98db0.

Moving to ON_QA to test if this indeed fixed the issue.
If so - please flag VERIFIED.
If not - please reopen with new engine+vdsm logs.
Comment 2 vvyazmin@redhat.com 2013-07-17 09:26:20 UTC 
Failed, verified on RHEVM 3.3 - IS5 environment:

RHEVM: rhevm-3.3.0-0.7.master.el6ev.noarch
VDSM: vdsm-4.11.0-121.git082925a.el6.x86_64
LIBVIRT: libvirt-0.10.2-18.el6_4.9.x86_64
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.355.el6_4.5.x86_64
SANLOCK: sanlock-2.6-2.el6.x86_64

This bug depend on BZ985043 - can't create LSM snapshot
Comment 3 vvyazmin@redhat.com 2013-07-23 08:37:01 UTC 
Failed, tested on RHEVM 3.3 - IS6 environment:

RHEVM: rhevm-3.3.0-0.9.master.el6ev.noarch
VDSM: vdsm-4.12.0-rc1.12.git8ee6885.el6.x86_64
LIBVIRT: libvirt-0.10.2-18.el6_4.9.x86_64
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.355.el6_4.5.x86_64
SANLOCK: sanlock-2.6-2.el6.x86_64

Logs attached
Comment 4 vvyazmin@redhat.com 2013-07-23 08:37:47 UTC 
Created attachment 777229 [details]
## Logs rhevm, vdsm, libvirt, thread dump, superVdsm
Comment 5 vvyazmin@redhat.com 2013-07-23 08:53:49 UTC 
Created attachment 777233 [details]
## Logs rhevm, vdsm, libvirt, thread dump, vds_table
Comment 6 Yair Zaslavsky 2013-08-26 15:02:33 UTC 
I would recommend to change CreateAllSnapshotsFromVmCommand to be non transactive...
Comment 9 Ayal Baron 2014-01-09 18:33:23 UTC 
Closing old bugs