Bug 1354156

Summary: RGW process crash during multisite sync operation
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: shilpa <smanjara>
Component: RGWAssignee: Casey Bodley <cbodley>
Status: CLOSED ERRATA QA Contact: shilpa <smanjara>
Severity: high Docs Contact:
Priority: unspecified    
Version: 2.0CC: cbodley, ceph-eng-bugs, hnallurv, kbader, kdreyer, mbenjamin, owasserm, sweil, yehuda
Target Milestone: rc   
Target Release: 2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.2-23.el7cp Ubuntu: ceph_10.2.2-18redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-23 19:43:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1353972    

Description shilpa 2016-07-10 07:08:54 UTC 
Description of problem:
While syncing data from master to non-master zone, the non-master rgw process crashed with a Segfault.

Version-Release number of selected component (if applicable):
ceph-radosgw-10.2.2-15.el7cp.x86_64

Steps to Reproduce:
1. Configure rgw1 as master zone and upload around 300G of data to master zone.
2. Configure rgw2 as the second zone and start syncing data from rgw1. 
3. Add a delay of about 150ms between the two nodes.


Actual results:
Around 300G of data was uploaded on rgw1 before starting the sync and rgw2 had no data in it. Once the sync started, another 300G were added to rgw1. Process crashed on rgw2 after syncing 300G of data. 

2016-07-09 22:06:38.136383 7fdd437fe700 -1 *** Caught signal (Segmentation fault) **
 in thread 7fdd437fe700 thread_name:radosgw

 ceph version 10.2.2-15.el7cp (60cd52496ca02bdde9c2f4191e617f75166d87b6)
 1: (()+0x54e22a) [0x7fde5db4822a]
 2: (()+0xf100) [0x7fde5cf79100]
 3: (RGWCoroutinesStack::wakeup()+0xa) [0x7fde5d8c44aa]
 4: (RGWBucketShardFullSyncCR::operate()+0x8a1) [0x7fde5d971531]
 5: (RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x7e) [0x7fde5d8c3a2e]
 6: (RGWCoroutinesManager::run(std::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x3f1) [0x7fde5d8c59d1]
 7: (RGWCoroutinesManager::run(RGWCoroutine*)+0x70) [0x7fde5d8c6590]
 8: (RGWRemoteDataLog::run_sync(int, rgw_data_sync_status&)+0x352) [0x7fde5d96d912]
 9: (RGWDataSyncProcessorThread::process()+0x49) [0x7fde5da30289]
 10: (RGWRadosThread::Worker::entry()+0x133) [0x7fde5d9d5083]
 11: (()+0x7dc5) [0x7fde5cf71dc5]
 12: (clone()+0x6d) [0x7fde5c57bced]
Comment 3 Casey Bodley 2016-07-18 17:17:39 UTC 
The fix has been cherry-picked to ceph-2-rhel-patches.
Comment 7 shilpa 2016-07-28 16:37:33 UTC 
Not seen the crash since ceph-10.2.2-23. Moving to verified.
Comment 9 errata-xmlrpc 2016-08-23 19:43:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1755.html