Bug 1387872

Summary:	Ceph Radosgw fails to start in IPv6 environment
Product:	Red Hat OpenStack	Reporter:	Marius Cornea <mcornea>
Component:	openstack-tripleo-heat-templates	Assignee:	Giulio Fidente <gfidente>
Status:	CLOSED ERRATA	QA Contact:	Yogev Rabl <yrabl>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	10.0 (Newton)	CC:	dbecker, gfidente, jschluet, jslagle, kschinck, mburns, morazi, rhel-osp-director-maint, scohen, seb, yrabl
Target Milestone:	rc	Keywords:	Triaged
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-5.1.0-3.el7ost puppet-tripleo-5.4.0-2.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-12-14 16:24:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Marius Cornea 2016-10-22 21:43:59 UTC

Description of problem:
Ceph Radosgw fails to start in IPv6 environment. /var/log/ceph/radosgw.log shows:
2016-10-22 21:14:06.584029 7f9b2746f9c0  0 civetweb: 0x7f9b276cddc0: set_ports_option: fd00:fd00:fd00:3000::12:8080: invalid port spec. Expecting list of: [IP_ADDRESS:]PORT[s|r]

cat /etc/ceph/ceph.conf 

[global]
osd_pool_default_pgp_num = 32
osd_pool_default_min_size = 1
auth_service_required = cephx
mon_initial_members = overcloud-serviceapi-0,overcloud-serviceapi-1,overcloud-serviceapi-2
fsid = 9ec38a7e-9896-11e6-a11d-5254004bd9e6
cluster_network = fd00:fd00:fd00:4000::1b/64
auth_supported = cephx
auth_cluster_required = cephx
mon_host = [fd00:fd00:fd00:3000::12],[fd00:fd00:fd00:3000::1a],[fd00:fd00:fd00:3000::22]
auth_client_required = cephx
osd_pool_default_size = 3
osd_pool_default_pg_num = 32
ms_bind_ipv6 = True
public_network = fd00:fd00:fd00:3000::12/64

[mon.overcloud-serviceapi-0]
public_addr = fd00:fd00:fd00:3000::12

[client.radosgw.gateway]
user = apache
rgw_frontends = civetweb port=fd00:fd00:fd00:3000::12:8080
log_file = /var/log/ceph/radosgw.log
host = overcloud-serviceapi-0
keyring = /etc/ceph/ceph.client.radosgw.gateway.keyring
rgw_keystone_token_cache_size = 500
rgw_keystone_url = http://192.168.0.12:35357
rgw_s3_auth_use_keystone = True
rgw_keystone_admin_token = T7dQrP9vM7ReXX43GQQutxnZP
rgw_keystone_accepted_roles = admin,_member_,Member


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-5.0.0-0.6.0rc3.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy IPv6 overcloud with Radosgw enabled
2. Upload object with swift client

Actual results:
Upload fails:
('Connection aborted.', BadStatusLine("''",))

Expected results:
Successful object upload.

Comment 2 seb 2016-10-25 10:37:13 UTC

AFAIK this is purely a Ceph issue, this is not related to the Ceph DFG.

Comment 3 Giulio Fidente 2016-10-25 12:19:37 UTC

I think we want the civetweb binding address in brackets. I will attempt a fix later in the day unless keith or jonh pick this up earlier.

Comment 4 Giulio Fidente 2016-10-27 09:39:50 UTC

Given the Ceph bug, we need both the tripleo-heat-templates fix and Ceph >= 10.2.3 for this to work.

Comment 5 seb 2016-10-31 11:09:00 UTC

This is weird since http://tracker.ceph.com/issues/16945 is part of rhcs-2.0

Comment 6 Giulio Fidente 2016-10-31 11:43:46 UTC

(In reply to seb from comment #5)
> This is weird since http://tracker.ceph.com/issues/16945 is part of rhcs-2.0

ack, we need the THT fix too

Comment 7 seb 2016-10-31 11:55:14 UTC

What's THT?

Comment 8 Keith Schincke 2016-11-01 12:25:20 UTC

I will be testing this as a ceph/jewel cluster today.
1) Ceph 10.2.3 should fix the RGW civetweb issue.
2) tripleo-heat-templates (THT) may need patch: https://review.openstack.org/#/c/390569/

Comment 9 Jon Schlueter 2016-11-12 06:17:38 UTC

https://review.openstack.org/#/c/395785/ has merged to master of puppet-tripleo

Comment 10 Giulio Fidente 2016-11-15 13:17:37 UTC

puppet-tripleo and tripleo-heat-templates patches merged into master branch, waiting on newton backports to merge

Comment 11 Giulio Fidente 2016-11-15 19:59:55 UTC

patches merged into newton branch

Comment 15 Yogev Rabl 2016-12-01 20:56:38 UTC

Verified on openstack-tripleo-heat-templates-5.1.0-6.el7ost.noarch

Comment 17 errata-xmlrpc 2016-12-14 16:24:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html