Bug 1305387 - Unable to start radosgw on RHEL 6.6 [NEEDINFO]
Unable to start radosgw on RHEL 6.6
Status: CLOSED WONTFIX
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW (Show other bugs)
1.2.3
x86_64 Linux
unspecified Severity high
: rc
: 1.2.4
Assigned To: Yehuda Sadeh
ceph-qe-bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-07 17:29 EST by Rachana Patel
Modified: 2017-07-30 11:45 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-13 12:42:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
yehuda: needinfo? (racpatel)


Attachments (Terms of Use)

  None (edit)
Description Rachana Patel 2016-02-07 17:29:58 EST
Description of problem:
========================
On rhel 6.6 installed ceph 1.2.3. radosgw is failing with below error
[ubuntu@magna117 ~]$  sudo /usr/bin/radosgw start
2016-02-07 17:25:20.531570 7fc84afee820 -1 WARNING: libcurl doesn't support curl_multi_wait()
2016-02-07 17:25:20.531572 7fc84afee820 -1 WARNING: cross zone / region transfer performance may be affected



Version-Release number of selected component (if applicable):
============================================================
[ubuntu@magna117 ~]$ rpm -qa | grep ceph
ceph-common-0.80.8-17.el6cp.x86_64
ceph-radosgw-0.80.8-17.el6cp.x86_64



How reproducible:
=================
always


Steps to Reproduce:
==================

1.Creted ceph cluster on RHEL 6.6 (ceph 1.2.3)
2. followed below doc for radosgw
https://access.redhat.com/documentation/en/red-hat-ceph-storage/version-1.2.3/red-hat-ceph-storage-123-installation-guide-for-rhel-x86-64/installation-guide-for-rhel-x86-64

3.radosgw service fails with below error
https://access.redhat.com/documentation/en/red-hat-ceph-storage/version-1.2.3/red-hat-ceph-storage-123-installation-guide-for-rhel-x86-64/installation-guide-for-rhel-x86-64

Actual results:
================
rgw service not starting


Expected results:


Additional info:
=================

[ubuntu@magna117 ~]$  sudo /usr/bin/radosgw -d --debug-rgw 20 --debug-ms 1 start
2016-02-07 17:21:40.986556 7f7b0a3ca820  0 ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7), process radosgw, pid 9834
2016-02-07 17:21:40.986563 7f7b0a3ca820 -1 WARNING: libcurl doesn't support curl_multi_wait()
2016-02-07 17:21:40.986565 7f7b0a3ca820 -1 WARNING: cross zone / region transfer performance may be affected
2016-02-07 17:21:41.009773 7f7b0a3ca820  1 -- :/0 messenger.start
2016-02-07 17:21:41.010666 7f7b0a3ca820 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2016-02-07 17:21:41.010671 7f7b0a3ca820  0 librados: client.admin initialization error (2) No such file or directory
2016-02-07 17:21:41.010693 7f7b0a3ca820  1 -- :/1009834 mark_down_all
2016-02-07 17:21:41.010772 7f7b0a3ca820  1 -- :/1009834 shutdown complete.
2016-02-07 17:21:41.010821 7f7b0a3ca820 -1 Couldn't init storage provider (RADOS)
Comment 2 Ken Dreyer (Red Hat) 2016-02-08 18:00:24 EST
I'm pretty sure we won't fix the curl stuff, since RHEL 6 is unlikely to backport  curl_multi_wait().

The problem seems to be with the missing keyring... Yehuda?
Comment 3 Brad Hubbard 2016-02-08 19:55:44 EST
These are just warnings, they are not fatal.

static void check_curl()
{
  derr << "WARNING: libcurl doesn't support curl_multi_wait()" << dendl;
  derr << "WARNING: cross zone / region transfer performance may be affected" << dendl;
}

The real problem is here I believe...

340 int MonClient::init()
341 {
...
365   if (auth_supported->is_supported_auth(CEPH_AUTH_CEPHX)) {
366     r = keyring->from_ceph_context(cct);
367     if (r == -ENOENT) {
368       auth_supported->remove_supported_auth(CEPH_AUTH_CEPHX);
369       if (auth_supported->get_supported_set().size() > 0) {
370 »·······r = 0;
371 »·······no_keyring_disabled_cephx = true;
372       } else {
373 »·······lderr(cct) << "ERROR: missing keyring, cannot use cephx for authentication" << dendl;

Leading to...

204 int librados::RadosClient::connect()
205 {
...
255   err = monclient.init();
256   if (err) {
257     ldout(cct, 0) << conf->name << " initialization error " << cpp_strerror(-err) << dendl;
258     shutdown();
259     goto out;
260   }

Could you check
https://access.redhat.com/documentation/en/red-hat-ceph-storage/version-1.2.3/red-hat-ceph-storage-123-ceph-object-gateway-for-rhel-x86-64/#configure_ceph_object_gateway
especially section 2.5, and make sure your keys are set up correctly?
Comment 4 Brad Hubbard 2016-02-09 22:27:58 EST
If we look at /etc/init.d/ceph-radosgw we see the following.

# bash -x /etc/init.d/ceph-radosgw start 2>&1|grep client.radosgw
+ PREFIX=client.radosgw.
++ ceph-conf --list-sections client.radosgw.
++ ceph-conf -n client.radosgw.magna117 'auto start'
++ ceph-conf -n client.radosgw.magna117 host
++ ceph-conf -n client.radosgw.magna117 user
++ /usr/bin/radosgw -n client.radosgw.magna117 --show-config-value log_file
+ log_file=/var/log/radosgw/client.radosgw.gateway.log
+ '[' -n /var/log/radosgw/client.radosgw.gateway.log ']'
+ '[' '!' -e /var/log/radosgw/client.radosgw.gateway.log ']'
+ daemon --user=apache '/usr/bin/radosgw -n client.radosgw.magna117'
+ '[' '/usr/bin/radosgw -n client.radosgw.magna117' '!=' '/usr/bin/radosgw -n client.radosgw.magna117' ']'
+ base='radosgw -n client.radosgw.magna117'
+ __pids_var_run 'radosgw -n client.radosgw.magna117' ''
+ local 'base=radosgw -n client.radosgw.magna117'
+ local 'pid_file=/var/run/radosgw -n client.radosgw.magna117.pid'
++ /usr/bin/dirname /var/run/radosgw -n client.radosgw.magna117.pid
+ '[' -f '/var/run/radosgw -n client.radosgw.magna117.pid' ']'
+ runuser -s /bin/bash apache -c 'ulimit -S -c 0 >/dev/null 2>&1 ; /usr/bin/radosgw -n client.radosgw.magna117'
+ success 'radosgw -n client.radosgw.magna117 startup'
+ echo 'Starting client.radosgw.magna117...'
Starting client.radosgw.magna117...

So it calculates some of the name from "hostname -s" so when starting we need to specify the name and we also need to add a key for the *actual* name, client.radosgw.magna117, not client.radosgw.gateway.

So I added a key per the docs and can now start the gateway fine.

# radosgw -d -n client.radosgw.magna117 startup
2016-02-09 22:25:19.911201 7f8af7e07820  0 ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7), process radosgw, pid 26602
2016-02-09 22:25:19.911207 7f8af7e07820 -1 WARNING: libcurl doesn't support curl_multi_wait()
2016-02-09 22:25:19.911209 7f8af7e07820 -1 WARNING: cross zone / region transfer performance may be affected
2016-02-09 22:25:20.109311 7f8ac28fa700  0 ERROR: can't get key: ret=-2
2016-02-09 22:25:20.109330 7f8ac28fa700  0 ERROR: sync_all_users() returned ret=-2
2016-02-09 22:25:20.109582 7f8af7e07820  0 framework: fastcgi
2016-02-09 22:25:20.109605 7f8af7e07820  0 framework conf key: socket_port, val: 9000
2016-02-09 22:25:20.109611 7f8af7e07820  0 framework conf key: socket_host, val: 0.0.0.0
2016-02-09 22:25:20.109616 7f8af7e07820  0 starting handler: fastcgi

Rachana, Have another go at setting this up, only this time when following the docs instead of "client.radosgw.gateway" use "client.radosgw.magna117" and use "-n client.radosgw.magna117" when starting the gateway. If that works we can see whether a doc bug is needed or where we go from there.
Comment 5 Rachana Patel 2016-02-10 18:14:26 EST
tried solution suggested in Comment#3 but it didn't solve problem
Comment 6 Brad Hubbard 2016-02-10 18:28:04 EST
(In reply to Rachana Patel from comment #5)
> tried solution suggested in Comment#3 but it didn't solve problem

Do you mean comment#4 ?

The rados gateway was starting up and running fine on magna117 when I logged out?

Are you still having trouble with this? If so, I may need to set up a rhel6 cluster and document the steps, if any, that differ from the docs to get this going.
Comment 7 Rachana Patel 2016-02-12 11:52:42 EST
(In reply to Brad Hubbard from comment #6)
> (In reply to Rachana Patel from comment #5)
> > tried solution suggested in Comment#3 but it didn't solve problem
> 
> Do you mean comment#4 ?
> 
> The rados gateway was starting up and running fine on magna117 when I logged
> out?
> 
> Are you still having trouble with this? If so, I may need to set up a rhel6
> cluster and document the steps, if any, that differ from the docs to get
> this going.

Sorry, for typo, I meant Comment#4.

Yes, it was working on that node(magna117), But when I set up new cluster  and followed Doc it didn't work in that. So looks like something is different than Doc and better to document steps again.
Comment 8 Brad Hubbard 2016-02-12 16:25:46 EST
(In reply to Rachana Patel from comment #7)
> Sorry, for typo, I meant Comment#4.

No problem.

> 
> Yes, it was working on that node(magna117), But when I set up new cluster 
> and followed Doc it didn't work in that. So looks like something is
> different than Doc and better to document steps again.

I see. I'll get onto this on Monday, thanks for your patience.
Comment 11 Ken Dreyer (Red Hat) 2016-05-13 12:42:56 EDT
1.2 is going EOL very soon; closing

Note You need to log in before you can comment on or make changes to this bug.