Bug 1354156 - RGW process crash during multisite sync operation
Summary: RGW process crash during multisite sync operation
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW
Version: 2.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: 2.0
Assignee: Casey Bodley
QA Contact: shilpa
Depends On:
Blocks: 1353972
TreeView+ depends on / blocked
Reported: 2016-07-10 07:08 UTC by shilpa
Modified: 2017-07-31 20:59 UTC (History)
9 users (show)

Fixed In Version: RHEL: ceph-10.2.2-23.el7cp Ubuntu: ceph_10.2.2-18redhat1xenial
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2016-08-23 19:43:52 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 16665 0 None None None 2016-07-12 19:51:56 UTC
Red Hat Product Errata RHBA-2016:1755 0 normal SHIPPED_LIVE Red Hat Ceph Storage 2.0 bug fix and enhancement update 2016-08-23 23:23:52 UTC

Description shilpa 2016-07-10 07:08:54 UTC
Description of problem:
While syncing data from master to non-master zone, the non-master rgw process crashed with a Segfault.

Version-Release number of selected component (if applicable):

Steps to Reproduce:
1. Configure rgw1 as master zone and upload around 300G of data to master zone.
2. Configure rgw2 as the second zone and start syncing data from rgw1. 
3. Add a delay of about 150ms between the two nodes.

Actual results:
Around 300G of data was uploaded on rgw1 before starting the sync and rgw2 had no data in it. Once the sync started, another 300G were added to rgw1. Process crashed on rgw2 after syncing 300G of data. 

2016-07-09 22:06:38.136383 7fdd437fe700 -1 *** Caught signal (Segmentation fault) **
 in thread 7fdd437fe700 thread_name:radosgw

 ceph version 10.2.2-15.el7cp (60cd52496ca02bdde9c2f4191e617f75166d87b6)
 1: (()+0x54e22a) [0x7fde5db4822a]
 2: (()+0xf100) [0x7fde5cf79100]
 3: (RGWCoroutinesStack::wakeup()+0xa) [0x7fde5d8c44aa]
 4: (RGWBucketShardFullSyncCR::operate()+0x8a1) [0x7fde5d971531]
 5: (RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x7e) [0x7fde5d8c3a2e]
 6: (RGWCoroutinesManager::run(std::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x3f1) [0x7fde5d8c59d1]
 7: (RGWCoroutinesManager::run(RGWCoroutine*)+0x70) [0x7fde5d8c6590]
 8: (RGWRemoteDataLog::run_sync(int, rgw_data_sync_status&)+0x352) [0x7fde5d96d912]
 9: (RGWDataSyncProcessorThread::process()+0x49) [0x7fde5da30289]
 10: (RGWRadosThread::Worker::entry()+0x133) [0x7fde5d9d5083]
 11: (()+0x7dc5) [0x7fde5cf71dc5]
 12: (clone()+0x6d) [0x7fde5c57bced]

Comment 3 Casey Bodley 2016-07-18 17:17:39 UTC
The fix has been cherry-picked to ceph-2-rhel-patches.

Comment 7 shilpa 2016-07-28 16:37:33 UTC
Not seen the crash since ceph-10.2.2-23. Moving to verified.

Comment 9 errata-xmlrpc 2016-08-23 19:43:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.