Bug 1349746

Summary: [upgrade] on upgrade from 1.3.2 to 2.0 on ubuntu, zone setup seems to have changed
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Tejas <tchandra>
Component: RGWAssignee: Orit Wasserman <owasserm>
Status: CLOSED ERRATA QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.0CC: cbodley, ceph-eng-bugs, gmeno, hnallurv, kbader, kdreyer, mbenjamin, owasserm, sweil, tchandra, tserlin
Target Milestone: rc   
Target Release: 2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.2-9.el7cp Ubuntu: ceph_10.2.2-7redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-23 19:42:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
rgw-log
none
user info command none

Description Tejas 2016-06-24 07:13:59 UTC 
Description of problem:
After upgrade from ceph 1.3.2 to ceph 2.0 on ubuntu,
the default zone setup has chaged in 2.0
I am not able to create new users, or do any IO.

Version-Release number of selected component (if applicable):
ceph 10.2.2-5redhat1trusty

How reproducible:
Always

Steps to Reproduce:
1. Upgrade the cluster from ceph 1.3.2 to ceph 2.0
2. od a rgw upgrade
3. RGW user create, IO not happening after upgrade

Found a upstream tracker similar to this:
http://tracker.ceph.com/issues/15597


Expected results:
RGW should handle the upgrade

Additional info:

root@magna077:~# radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full
2016-06-24 07:00:17.852066 7ff7199d2900  0 RGWZoneParams::create(): error creating default zone params: (17) File exists
2016-06-24 07:00:18.245546 7ff7199d2900  0 ERROR: rgw_put_system_obj() oid=.meta:user:testuser:_rJnCMln9ob7m03foN2M7A0S:2) returned ret=-22
could not create subuser: unable to create subuser, unable to store user info2016-06-24 07:00:18.245566 7ff7199d2900  0 ERROR: put_entry: store_in_heap() key=testuser returned ret=-22

oot@magna077:~# radosgw-admin zone get
unable to initialize zone: (2) No such file or directory


Log:

2016-06-24 06:29:08.911787 7f2a0bfc7700  0 ERROR: no socket server point defined, cannot start fcgi frontend
2016-06-24 06:29:26.223617 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:26.233857 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:26.234526 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:26.234537 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:26.234549 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:26.234756 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:26.234795 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:26 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
2016-06-24 06:29:26.752485 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:26.756476 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:26.756554 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:26.756563 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:26.756578 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:26.756734 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:26.756779 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:26 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
2016-06-24 06:29:26.853022 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:26.855720 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:26.855799 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:26.855808 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:26.855823 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:26.855995 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:26.856037 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:26 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
2016-06-24 06:29:27.214789 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:27.217477 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:27.217560 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:27.217569 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:27.217582 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:27.217717 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:27.217759 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:27 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
2016-06-24 06:29:31.841204 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:31.843974 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:31.844053 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:31.844062 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:31.844076 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:31.844226 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:31.844273 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:31 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
2016-06-24 06:29:42.181923 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:42.184887 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:42.184975 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:42.184984 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:42.185001 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:42.185169 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:42.185220 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:42 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
2016-06-24 06:29:52.633822 7f2b69ffb700  1 ====== starting new request req=0x7f2b69ff57d0 =====
2016-06-24 06:29:52.636667 7f2b69ffb700  0 WARNING: couldn't find acl header for object, generating default
2016-06-24 06:29:52.636753 7f2b69ffb700  0 sending create_bucket request to master zonegroup
2016-06-24 06:29:52.636763 7f2b69ffb700  0 ERROR: endpoints not configured for upstream zone
2016-06-24 06:29:52.636778 7f2b69ffb700  0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-06-24 06:29:52.636944 7f2b69ffb700  1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ======
2016-06-24 06:29:52.636992 7f2b69ffb700  1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:52 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
Comment 2 Orit Wasserman 2016-06-27 11:22:41 UTC 
It was fixed upstream http://tracker.ceph.com/issues/15937, it is being backported to jewel https://github.com/ceph/ceph/pull/9294.
Comment 3 Christina Meno 2016-06-27 17:03:47 UTC 
Casey,

Would you please get this into the downstream patches today?
Comment 12 Tejas 2016-07-05 10:28:21 UTC 
Created attachment 1176362 [details]
rgw-log
Comment 13 Tejas 2016-07-05 10:38:31 UTC 
Created attachment 1176364 [details]
user info command
Comment 15 Tejas 2016-07-05 12:15:30 UTC 
I will open a new bug for this.

Opened Bug 1352888
Comment 16 Tejas 2016-07-05 12:24:12 UTC 
This issue is resolved as per Orit. 
So moving this bug to Verified.
Verified in build.
ceph version 10.2.2-15.el7cp
Comment 18 errata-xmlrpc 2016-08-23 19:42:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1755.html