Bug 1359712 – A master zone switch requires radosgw to be restarted

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1359712 - A master zone switch requires radosgw to be restarted

Summary:	A master zone switch requires radosgw to be restarted

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat Ceph Storage
Classification:	Red Hat
Component:	RGW (Show other bugs)
Sub Component:
Version:	2.0
Hardware:	Unspecified Unspecified

Priority	high Severity unspecified
Target Milestone:	rc
Target Release:	2.1
Assigned To:	Casey Bodley
QA Contact:	Rachana Patel
Docs Contact:	Bara Ancincova

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:	1322504 1383917
	Show dependency tree / graph

Reported:	2016-07-25 06:39 EDT by shilpa
Modified:	2017-07-30 11:58 EDT (History)
CC List:	11 users (show)

See Also:
Fixed In Version:	RHEL: ceph-10.2.3-2.el7cp Ubuntu: ceph_10.2.3-3redhat1xenial
Doc Type:	Bug Fix
Doc Text:	.A restart of the radosgw process is no longer required after switching the zone from master to non-master When a non-master zone was promoted to the master zone, all I/0 requests became unresponsive until the `radosgw` process was restarted on both zones. Consequently, the I/0 requests timed out. The underlying source code has been modified, and restarting `radosgw` is no longer required in the described situation.
Story Points:	---
Clone Of:
Environment:
Last Closed:	2016-11-22 14:28:53 EST
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2016:2815	normal	SHIPPED_LIVE	Moderate: Red Hat Ceph Storage security, bug fix, and enhancement update	2017-03-21 22:06:33 EDT

Groups: None (edit)

Description shilpa 2016-07-25 06:39:27 EDT

Description of problem:
When the zone is switched from master to non-master, all I/O requests hang until a rgw process restart on both the zones

Version-Release number of selected component (if applicable):
ceph-radosgw-10.2.2-27.el7cp.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Modify non-master zone with '--master' flag.
radosgw-admin zone modify --rgw-zonegroup=us --rgw-zone=us-2
--access_key=secret --secret=secret --endpoints=http://magna059:80 --default
--master
2. Update and commit the period


Actual results:
A radosgw restart should not be expected with a period configuration change. However all the I/O's hang until a process restart.

Additional info:

Traceback (most recent call last):
  File "s3del.py", line 21, in <module>
    conn.delete_bucket(buck.name)
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 641, in delete_bucket
    response = self.make_request('DELETE', bucket, headers=headers)
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
    retry_handler=retry_handler
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1028, in _mexe
    raise BotoServerError(response.status, response.reason, body)
boto.exception.BotoServerError: BotoServerError: 504 Gateway Time-out
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

Comment 7 Casey Bodley 2016-09-08 14:01:57 EDT

Hi Shilpa,

We have upstream testing that makes multiple changes to the master zone without restarting gateways, and we haven't seen it hit this issue. Can you try to reproduce the issue with the latest build?

Comment 8 shilpa 2016-09-09 04:10:27 EDT

(In reply to Casey Bodley from comment #7)
> Hi Shilpa,
> 
> We have upstream testing that makes multiple changes to the master zone
> without restarting gateways, and we haven't seen it hit this issue. Can you
> try to reproduce the issue with the latest build?

Hi Casey, 

Sure, I will try it on 2.0 Async build?

Comment 11 shilpa 2016-11-04 06:29:57 EDT

Verified on 10.2.3-12. No gateway restart is required to switch master.

Comment 14 shilpa 2016-11-22 08:44:51 EST

Hi Bara,

It looks good to me.

Comment 16 errata-xmlrpc 2016-11-22 14:28:53 EST

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2815.html

Note You need to log in before you can comment on or make changes to this bug.