Bug 975946 - [rhevm] Failed run Live Storage Migration for large amount of disks - Error checking for a transaction
[rhevm] Failed run Live Storage Migration for large amount of disks - Error c...
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.2.0
x86_64 Linux
unspecified Severity high
: ---
: 3.2.6
Assigned To: Ayal Baron
Aharon Canan
storage
:
Depends On: 985043
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-19 12:16 EDT by vvyazmin@redhat.com
Modified: 2016-02-10 12:53 EST (History)
12 users (show)

See Also:
Fixed In Version: is1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-09 13:33:23 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
abaron: Triaged+


Attachments (Terms of Use)
## Logs rhevm (332.79 KB, application/x-xz)
2013-06-19 12:16 EDT, vvyazmin@redhat.com
no flags Details
## Logs rhevm, vdsm, libvirt, thread dump, superVdsm (4.37 MB, application/x-gzip)
2013-07-23 04:37 EDT, vvyazmin@redhat.com
no flags Details
## Logs rhevm, vdsm, libvirt, thread dump, vds_table (5.08 MB, application/x-gzip)
2013-07-23 04:53 EDT, vvyazmin@redhat.com
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 13994 None None None Never

  None (edit)
Description vvyazmin@redhat.com 2013-06-19 12:16:09 EDT
Created attachment 763036 [details]
## Logs rhevm

Summary: Failed run  Live Storage Migration for more than 50 disks

Description of problem: Failed run  Live Storage Migration for more than 50 disks


Version-Release number of selected component (if applicable):
RHEVM 3.2 - SF18 environment:

RHEVM: rhevm-3.2.1-0.31.el6ev.noarch
VDSM: vdsm-4.10.2-23.0.el6ev.x86_64
LIBVIRT: libvirt-0.10.2-18.el6_4.5.x86_64
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.355.el6_4.5.x86_64
SANLOCK: sanlock-2.6-2.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Create FCP Data-center
2. Create 100 VM with 1 disk with 'VirtIO' interface
3. Power on all those VM
4. Run Live Storage Migration (LSM) from DC-01 to DC-02 to all 100 disks
  
Relevant information:
I succeed run LSM for 50 disk's

Actual results:
Failed run LSM for all disks

Expected results:
Successful run LSM

Impact on user:

Workaround: 
Run LSM for with bulk maximum 30 disk.

Additional info:

/var/log/ovirt-engine/engine.log

2013-06-17 13:04:58,524 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (pool-4-thread-221) [148d0935] START, CreateSnapshotVDSCommand( storagePoolId = f5b6630b-4b7a-4e8c-952a-c2aa3b7fe1d5, ignoreFailoverLimit 
= false, compatabilityVersion = 3.2, storageDomainId = eab483e7-745f-4eb2-bde4-cc9dd77eb67f, imageGroupId = c03c1791-02b5-4f5a-99b8-0bd27086460f, imageSizeInBytes = 21474836480, volumeFormat = COW, newImageId = 571c40fc-e645-46c8-b453-f6
b3892efbb1, newImageDescription = , imageId = b311316b-d149-4b9c-8cf6-40d9a7810c03, sourceImageGroupId = c03c1791-02b5-4f5a-99b8-0bd27086460f), log id: 73be1c51
2013-06-17 13:04:59,974 ERROR [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] (Transaction Reaper Worker 0) Transaction rolled-back for command: org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.

2013-06-17 13:09:52,636 ERROR [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (pool-4-thread-104) [54ffa8eb] Command org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw exception: javax.ejb.EJBException: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:191) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:237) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:218) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at 

/var/log/vdsm/vdsm.log
Comment 1 Allon Mureinik 2013-07-09 08:33:28 EDT
In current upstream I could not find any VDSM call from within the transaction.
The last one was removed by 2575a223515a4f984157e8017e272cdd5ac98db0.

Moving to ON_QA to test if this indeed fixed the issue.
If so - please flag VERIFIED.
If not - please reopen with new engine+vdsm logs.
Comment 2 vvyazmin@redhat.com 2013-07-17 05:26:20 EDT
Failed, verified on RHEVM 3.3 - IS5 environment:

RHEVM: rhevm-3.3.0-0.7.master.el6ev.noarch
VDSM: vdsm-4.11.0-121.git082925a.el6.x86_64
LIBVIRT: libvirt-0.10.2-18.el6_4.9.x86_64
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.355.el6_4.5.x86_64
SANLOCK: sanlock-2.6-2.el6.x86_64

This bug depend on BZ985043 - can't create LSM snapshot
Comment 3 vvyazmin@redhat.com 2013-07-23 04:37:01 EDT
Failed, tested on RHEVM 3.3 - IS6 environment:

RHEVM: rhevm-3.3.0-0.9.master.el6ev.noarch
VDSM: vdsm-4.12.0-rc1.12.git8ee6885.el6.x86_64
LIBVIRT: libvirt-0.10.2-18.el6_4.9.x86_64
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.355.el6_4.5.x86_64
SANLOCK: sanlock-2.6-2.el6.x86_64

Logs attached
Comment 4 vvyazmin@redhat.com 2013-07-23 04:37:47 EDT
Created attachment 777229 [details]
## Logs rhevm, vdsm, libvirt, thread dump, superVdsm
Comment 5 vvyazmin@redhat.com 2013-07-23 04:53:49 EDT
Created attachment 777233 [details]
## Logs rhevm, vdsm, libvirt, thread dump, vds_table
Comment 6 Yair Zaslavsky 2013-08-26 11:02:33 EDT
I would recommend to change CreateAllSnapshotsFromVmCommand to be non transactive...
Comment 9 Ayal Baron 2014-01-09 13:33:23 EST
Closing old bugs

Note You need to log in before you can comment on or make changes to this bug.