Bug 958898

Summary: Cassandra Data Migration - migration of large sql table has data losses
Product: [JBoss] JBoss Operations Network Reporter: Armine Hovsepyan <ahovsepy>
Component: InstallerAssignee: Stefan Negrea <snegrea>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: JON 3.2CC: hrupp, jsanda, mfoley
Target Milestone: ---   
Target Release: JON 3.2.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 951619    
Attachments:
Description Flags
top-beforeMigration
none
top-duringMigration none

Description Armine Hovsepyan 2013-05-02 15:44:19 UTC 
Description of problem:
migration of large sql table  has data losses

Version-Release number of selected component (if applicable):
jenkins build 188

How reproducible:
always

Steps to Reproduce:
1. run migrator jar against sql table having > 900 000 rows (my data -- 983990)
2.
3.
  
Actual results:
migrated only 714416

Expected results:
all data should be migrated 

Additional info:
when there are few data (my data -- 369635) migration worked correctly - all the data was migrated from postgres to cassandra
Comment 1 Stefan Negrea 2013-05-03 07:23:48 UTC 
Updated all the queries to avoid any potential overlap between them. Please retest.
Comment 2 Armine Hovsepyan 2013-05-03 20:42:41 UTC 
reopening 
jenkins build with logs here->  http://jenkins.jonqe.lab.eng.bos.redhat.com:9080/job/Migrator_Run/30/console
Comment 3 Stefan Negrea 2013-05-07 21:16:27 UTC 
Got the following error from the SQL host you are using for tests:

198232 [Thread-5] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper  - ERROR: could not write block 1 of temporary file: No space left on device
  Hint: Perhaps out of disk space?


Please make sure there is enough space on the disk....
Comment 4 Armine Hovsepyan 2013-05-09 13:13:47 UTC 
verified.

created an env with 8gb ram and 20 gb disk space - migration passing with no exception.
Comment 5 Armine Hovsepyan 2013-05-13 17:09:51 UTC 
reopned

retested with 4gb ram 30db disk space (25% of overall disk space is used) - migration fails for 10 mln rows from postgresql --> http://jenkins.jonqe.lab.eng.bos.redhat.com:9080/job/Migrator_Run/47/console
Comment 6 Armine Hovsepyan 2013-05-15 14:54:14 UTC 
Created attachment 748359 [details]
top-beforeMigration
Comment 7 Armine Hovsepyan 2013-05-15 14:54:48 UTC 
Created attachment 748360 [details]
top-duringMigration
Comment 8 Armine Hovsepyan 2013-05-15 14:56:35 UTC 
When migration for large data (>68 mln rows in postgres) starts - postgres takes 98% CPU usage (16 gb ram 50 gb disk size) and migration fails with could not extract ResultSet exception.

please have a look into logs in jenkins -> http://jenkins.jonqe.lab.eng.bos.redhat.com:9080/job/Migrator_Run/55/console
Comment 9 Stefan Negrea 2013-05-17 20:50:04 UTC 
The log show that the estimation process was causing the timeout. Added a few more updates to the increase the timeout for count queries.
Comment 10 Armine Hovsepyan 2013-05-20 11:29:12 UTC 
When migration for large data (>68 mln rows in postgres) starts - postgres takes 98% CPU usage (16 gb ram 50 gb disk size) and migration fails with could not extract ResultSet exception.

please have a look into logs in jenkins -> http://jenkins.jonqe.lab.eng.bos.redhat.com:9080/job/Migrator_Run/  61 & 62 builds. 

migrator jar from rhq-cassandra jenkins build 228.
Comment 11 Stefan Negrea 2013-06-26 18:14:05 UTC 
This is an environmental issue due to insufficient disk space for the data. Please increase the disk space and try again the test case. The amount of new data created by migration will be comparable to the existing SQL data. Please expand the disk accordingly.
Comment 12 Armine Hovsepyan 2013-06-27 10:05:12 UTC 
verified.
For details please have a look into Migration Run jenkins build 82.