Description of problem: After upgrade from ceph 1.3.2 to ceph 2.0 on ubuntu, the default zone setup has chaged in 2.0 I am not able to create new users, or do any IO. Version-Release number of selected component (if applicable): ceph 10.2.2-5redhat1trusty How reproducible: Always Steps to Reproduce: 1. Upgrade the cluster from ceph 1.3.2 to ceph 2.0 2. od a rgw upgrade 3. RGW user create, IO not happening after upgrade Found a upstream tracker similar to this: http://tracker.ceph.com/issues/15597 Expected results: RGW should handle the upgrade Additional info: root@magna077:~# radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full 2016-06-24 07:00:17.852066 7ff7199d2900 0 RGWZoneParams::create(): error creating default zone params: (17) File exists 2016-06-24 07:00:18.245546 7ff7199d2900 0 ERROR: rgw_put_system_obj() oid=.meta:user:testuser:_rJnCMln9ob7m03foN2M7A0S:2) returned ret=-22 could not create subuser: unable to create subuser, unable to store user info2016-06-24 07:00:18.245566 7ff7199d2900 0 ERROR: put_entry: store_in_heap() key=testuser returned ret=-22 oot@magna077:~# radosgw-admin zone get unable to initialize zone: (2) No such file or directory Log: 2016-06-24 06:29:08.911787 7f2a0bfc7700 0 ERROR: no socket server point defined, cannot start fcgi frontend 2016-06-24 06:29:26.223617 7f2b69ffb700 1 ====== starting new request req=0x7f2b69ff57d0 ===== 2016-06-24 06:29:26.233857 7f2b69ffb700 0 WARNING: couldn't find acl header for object, generating default 2016-06-24 06:29:26.234526 7f2b69ffb700 0 sending create_bucket request to master zonegroup 2016-06-24 06:29:26.234537 7f2b69ffb700 0 ERROR: endpoints not configured for upstream zone 2016-06-24 06:29:26.234549 7f2b69ffb700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2016-06-24 06:29:26.234756 7f2b69ffb700 1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ====== 2016-06-24 06:29:26.234795 7f2b69ffb700 1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:26 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic 2016-06-24 06:29:26.752485 7f2b69ffb700 1 ====== starting new request req=0x7f2b69ff57d0 ===== 2016-06-24 06:29:26.756476 7f2b69ffb700 0 WARNING: couldn't find acl header for object, generating default 2016-06-24 06:29:26.756554 7f2b69ffb700 0 sending create_bucket request to master zonegroup 2016-06-24 06:29:26.756563 7f2b69ffb700 0 ERROR: endpoints not configured for upstream zone 2016-06-24 06:29:26.756578 7f2b69ffb700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2016-06-24 06:29:26.756734 7f2b69ffb700 1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ====== 2016-06-24 06:29:26.756779 7f2b69ffb700 1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:26 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic 2016-06-24 06:29:26.853022 7f2b69ffb700 1 ====== starting new request req=0x7f2b69ff57d0 ===== 2016-06-24 06:29:26.855720 7f2b69ffb700 0 WARNING: couldn't find acl header for object, generating default 2016-06-24 06:29:26.855799 7f2b69ffb700 0 sending create_bucket request to master zonegroup 2016-06-24 06:29:26.855808 7f2b69ffb700 0 ERROR: endpoints not configured for upstream zone 2016-06-24 06:29:26.855823 7f2b69ffb700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2016-06-24 06:29:26.855995 7f2b69ffb700 1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ====== 2016-06-24 06:29:26.856037 7f2b69ffb700 1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:26 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic 2016-06-24 06:29:27.214789 7f2b69ffb700 1 ====== starting new request req=0x7f2b69ff57d0 ===== 2016-06-24 06:29:27.217477 7f2b69ffb700 0 WARNING: couldn't find acl header for object, generating default 2016-06-24 06:29:27.217560 7f2b69ffb700 0 sending create_bucket request to master zonegroup 2016-06-24 06:29:27.217569 7f2b69ffb700 0 ERROR: endpoints not configured for upstream zone 2016-06-24 06:29:27.217582 7f2b69ffb700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2016-06-24 06:29:27.217717 7f2b69ffb700 1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ====== 2016-06-24 06:29:27.217759 7f2b69ffb700 1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:27 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic 2016-06-24 06:29:31.841204 7f2b69ffb700 1 ====== starting new request req=0x7f2b69ff57d0 ===== 2016-06-24 06:29:31.843974 7f2b69ffb700 0 WARNING: couldn't find acl header for object, generating default 2016-06-24 06:29:31.844053 7f2b69ffb700 0 sending create_bucket request to master zonegroup 2016-06-24 06:29:31.844062 7f2b69ffb700 0 ERROR: endpoints not configured for upstream zone 2016-06-24 06:29:31.844076 7f2b69ffb700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2016-06-24 06:29:31.844226 7f2b69ffb700 1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ====== 2016-06-24 06:29:31.844273 7f2b69ffb700 1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:31 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic 2016-06-24 06:29:42.181923 7f2b69ffb700 1 ====== starting new request req=0x7f2b69ff57d0 ===== 2016-06-24 06:29:42.184887 7f2b69ffb700 0 WARNING: couldn't find acl header for object, generating default 2016-06-24 06:29:42.184975 7f2b69ffb700 0 sending create_bucket request to master zonegroup 2016-06-24 06:29:42.184984 7f2b69ffb700 0 ERROR: endpoints not configured for upstream zone 2016-06-24 06:29:42.185001 7f2b69ffb700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2016-06-24 06:29:42.185169 7f2b69ffb700 1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ====== 2016-06-24 06:29:42.185220 7f2b69ffb700 1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:42 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic 2016-06-24 06:29:52.633822 7f2b69ffb700 1 ====== starting new request req=0x7f2b69ff57d0 ===== 2016-06-24 06:29:52.636667 7f2b69ffb700 0 WARNING: couldn't find acl header for object, generating default 2016-06-24 06:29:52.636753 7f2b69ffb700 0 sending create_bucket request to master zonegroup 2016-06-24 06:29:52.636763 7f2b69ffb700 0 ERROR: endpoints not configured for upstream zone 2016-06-24 06:29:52.636778 7f2b69ffb700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2016-06-24 06:29:52.636944 7f2b69ffb700 1 ====== req done req=0x7f2b69ff57d0 op status=-5 http_status=500 ====== 2016-06-24 06:29:52.636992 7f2b69ffb700 1 civetweb: 0x7f2b8c001420: 10.8.128.77 - - [24/Jun/2016:06:29:52 +0000] "PUT /bigbucket/ HTTP/1.1" 500 0 - Boto/2.40.0 Python/2.7.6 Linux/3.19.0-61-generic
It was fixed upstream http://tracker.ceph.com/issues/15937, it is being backported to jewel https://github.com/ceph/ceph/pull/9294.
Casey, Would you please get this into the downstream patches today?
Created attachment 1176362 [details] rgw-log
Created attachment 1176364 [details] user info command
I will open a new bug for this. Opened Bug 1352888
This issue is resolved as per Orit. So moving this bug to Verified. Verified in build. ceph version 10.2.2-15.el7cp
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1755.html