Bug 958898 - Cassandra Data Migration - migration of large sql table has data losses
Summary: Cassandra Data Migration - migration of large sql table has data losses
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Operations Network
Classification: JBoss
Component: Installer
Version: JON 3.2
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: JON 3.2.0
Assignee: Stefan Negrea
QA Contact: Mike Foley
URL:
Whiteboard:
Depends On:
Blocks: 951619
TreeView+ depends on / blocked
 
Reported: 2013-05-02 15:44 UTC by Armine Hovsepyan
Modified: 2015-09-03 00:01 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)
top-beforeMigration (167.68 KB, image/png)
2013-05-15 14:54 UTC, Armine Hovsepyan
no flags Details
top-duringMigration (500.49 KB, image/png)
2013-05-15 14:54 UTC, Armine Hovsepyan
no flags Details

Description Armine Hovsepyan 2013-05-02 15:44:19 UTC
Description of problem:
migration of large sql table  has data losses

Version-Release number of selected component (if applicable):
jenkins build 188

How reproducible:
always

Steps to Reproduce:
1. run migrator jar against sql table having > 900 000 rows (my data -- 983990)
2.
3.
  
Actual results:
migrated only 714416

Expected results:
all data should be migrated 

Additional info:
when there are few data (my data -- 369635) migration worked correctly - all the data was migrated from postgres to cassandra

Comment 1 Stefan Negrea 2013-05-03 07:23:48 UTC
Updated all the queries to avoid any potential overlap between them. Please retest.

Comment 2 Armine Hovsepyan 2013-05-03 20:42:41 UTC
reopening 
jenkins build with logs here->  http://jenkins.jonqe.lab.eng.bos.redhat.com:9080/job/Migrator_Run/30/console

Comment 3 Stefan Negrea 2013-05-07 21:16:27 UTC
Got the following error from the SQL host you are using for tests:

198232 [Thread-5] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper  - ERROR: could not write block 1 of temporary file: No space left on device
  Hint: Perhaps out of disk space?


Please make sure there is enough space on the disk....

Comment 4 Armine Hovsepyan 2013-05-09 13:13:47 UTC
verified.

created an env with 8gb ram and 20 gb disk space - migration passing with no exception.

Comment 5 Armine Hovsepyan 2013-05-13 17:09:51 UTC
reopned

retested with 4gb ram 30db disk space (25% of overall disk space is used) - migration fails for 10 mln rows from postgresql --> http://jenkins.jonqe.lab.eng.bos.redhat.com:9080/job/Migrator_Run/47/console

Comment 6 Armine Hovsepyan 2013-05-15 14:54:14 UTC
Created attachment 748359 [details]
top-beforeMigration

Comment 7 Armine Hovsepyan 2013-05-15 14:54:48 UTC
Created attachment 748360 [details]
top-duringMigration

Comment 8 Armine Hovsepyan 2013-05-15 14:56:35 UTC
When migration for large data (>68 mln rows in postgres) starts - postgres takes 98% CPU usage (16 gb ram 50 gb disk size) and migration fails with could not extract ResultSet exception.

please have a look into logs in jenkins -> http://jenkins.jonqe.lab.eng.bos.redhat.com:9080/job/Migrator_Run/55/console

Comment 9 Stefan Negrea 2013-05-17 20:50:04 UTC
The log show that the estimation process was causing the timeout. Added a few more updates to the increase the timeout for count queries.

Comment 10 Armine Hovsepyan 2013-05-20 11:29:12 UTC
When migration for large data (>68 mln rows in postgres) starts - postgres takes 98% CPU usage (16 gb ram 50 gb disk size) and migration fails with could not extract ResultSet exception.

please have a look into logs in jenkins -> http://jenkins.jonqe.lab.eng.bos.redhat.com:9080/job/Migrator_Run/  61 & 62 builds. 

migrator jar from rhq-cassandra jenkins build 228.

Comment 11 Stefan Negrea 2013-06-26 18:14:05 UTC
This is an environmental issue due to insufficient disk space for the data. Please increase the disk space and try again the test case. The amount of new data created by migration will be comparable to the existing SQL data. Please expand the disk accordingly.

Comment 12 Armine Hovsepyan 2013-06-27 10:05:12 UTC
verified.
For details please have a look into Migration Run jenkins build 82.


Note You need to log in before you can comment on or make changes to this bug.