1349746 – [upgrade] on upgrade from 1.3.2 to 2.0 on ubuntu, zone setup seems to have changed

Bug 1349746 - [upgrade] on upgrade from 1.3.2 to 2.0 on ubuntu, zone setup seems to have changed

Summary: [upgrade] on upgrade from 1.3.2 to 2.0 on ubuntu, zone setup seems to have ch...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat
Component:	RGW
Sub Component:
Version:	2.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	2.0
Assignee:	Orit Wasserman
QA Contact:	ceph-qe-bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-06-24 07:13 UTC by Tejas
Modified:	2017-07-30 15:59 UTC (History)
CC List:	11 users (show)
Fixed In Version:	RHEL: ceph-10.2.2-9.el7cp Ubuntu: ceph_10.2.2-7redhat1xenial
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-08-23 19:42:35 UTC
Target Upstream Version:
Dependent Products:

Attachments	(Terms of Use)
rgw-log (5.86 MB, text/plain) 2016-07-05 10:28 UTC, Tejas	no flags	Details
user info command (58.08 KB, text/plain) 2016-07-05 10:38 UTC, Tejas	no flags	Details
Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2016:1755	normal	SHIPPED_LIVE	Red Hat Ceph Storage 2.0 bug fix and enhancement update	2016-08-23 23:23:52 UTC

Description Tejas 2016-06-24 07:13:59 UTC

Description of problem:
After upgrade from ceph 1.3.2 to ceph 2.0 on ubuntu,
the default zone setup has chaged in 2.0
I am not able to create new users, or do any IO.

Version-Release number of selected component (if applicable):
ceph 10.2.2-5redhat1trusty

How reproducible:
Always

Steps to Reproduce:
1. Upgrade the cluster from ceph 1.3.2 to ceph 2.0
2. od a rgw upgrade
3. RGW user create, IO not happening after upgrade

Found a upstream tracker similar to this:
http://tracker.ceph.com/issues/15597


Expected results:
RGW should handle the upgrade

Additional info:

root@magna077:~# radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full
2016-06-24 07:00:17.852066 7ff7199d2900  0 RGWZoneParams::create(): error creating default zone params: (17) File exists
2016-06-24 07:00:18.245546 7ff7199d2900  0 ERROR: rgw_put_system_obj() oid=.meta:user:testuser:_rJnCMln9ob7m03foN2M7A0S:2) returned ret=-22
could not create subuser: unable to create subuser, unable to store user info2016-06-24 07:00:18.245566 7ff7199d2900  0 ERROR: put_entry: store_in_heap() key=testuser returned ret=-22

oot@magna077:~# radosgw-admin zone get
unable to initialize zone: (2) No such file or directory


Log:

2016-06-24 06:29:08.911787 7f2a0bfc7700  0 ERROR: no socket server point defined, cannot start fcgi frontend
2016-06-24 06:29:26.223617 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:26.233857 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:26.234526 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:26.234537 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:26.234549 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:26.234756 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:26.234795 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:26 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
2016-06-24 06:29:26.752485 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:26.756476 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:26.756554 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:26.756563 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:26.756578 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:26.756734 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:26.756779 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:26 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
2016-06-24 06:29:26.853022 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:26.855720 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:26.855799 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:26.855808 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:26.855823 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:26.855995 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:26.856037 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:26 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
2016-06-24 06:29:27.214789 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:27.217477 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:27.217560 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:27.217569 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:27.217582 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:27.217717 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:27.217759 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:27 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
2016-06-24 06:29:31.841204 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:31.843974 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:31.844053 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:31.844062 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:31.844076 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:31.844226 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:31.844273 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:31 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
2016-06-24 06:29:42.181923 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:42.184887 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:42.184975 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:42.184984 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:42.185001 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:42.185169 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:42.185220 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:42 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
2016-06-24 06:29:52.633822 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:52.636667 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:52.636753 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:52.636763 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:52.636778 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:52.636944 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:52.636992 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:52 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic

Comment 2 Orit Wasserman 2016-06-27 11:22:41 UTC

It was fixed upstream http://tracker.ceph.com/issues/15937, it is being backported to jewel https://github.com/ceph/ceph/pull/9294.

Comment 3 Christina Meno 2016-06-27 17:03:47 UTC

Casey,

Would you please get this into the downstream patches today?

Comment 12 Tejas 2016-07-05 10:28:21 UTC

Created attachment 1176362 [details]
rgw-log

Comment 13 Tejas 2016-07-05 10:38:31 UTC

Created attachment 1176364 [details]
user info command

Comment 15 Tejas 2016-07-05 12:15:30 UTC

I will open a new bug for this.

Opened Bug 1352888

Comment 16 Tejas 2016-07-05 12:24:12 UTC

This issue is resolved as per Orit. 
So moving this bug to Verified.
Verified in build.
ceph version 10.2.2-15.el7cp

Comment 18 errata-xmlrpc 2016-08-23 19:42:35 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1755.html

Note You need to log in before you can comment on or make changes to this bug.