Bug 2111680

Summary:	cephadm --config initial-ceph.conf no longer supports comma delimited networks for routed traffic
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	John Fulton <johfulto>
Component:	Cephadm	Assignee:	Adam King <adking>
Status:	CLOSED ERRATA	QA Contact:	Aditya Ramteke <aramteke>
Severity:	medium	Docs Contact:	Akash Raj <akraj>
Priority:	unspecified
Version:	5.2	CC:	adking, akraj, cephqe-warriors, fpantano, hjensas, jparoly, msaini, vereddy
Target Milestone:	---
Target Release:	6.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-17.2.6-5.el9cp	Doc Type:	Bug Fix
Doc Text:	.Bootstrap no longer fails if a comma-separated list of quoted IPs are passed in as the public network in the initial Ceph configuration Previously,`cephadm` bootstrap would improperly parse comma-delimited lists of IP addresses, if the list was quoted. Due to this, the bootstrap would fail if a comma-separated list of quoted IP addresses, for example, ’172.120.3.0/24,172.117.3.0/24,172.118.3.0/24,172.119.3.0/24’, was provided as the `public_network` in the initial Ceph configuration passed to bootstrap with the `--config` parameter. With this fix, you can enter the comma-separated lists of quoted IPs into the initial Ceph configuration passed to bootstrap for the `public_network` or `cluster_network`, and it works as expected.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-06-15 09:15:36 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2180567
Bug Blocks:	2192813

Description John Fulton 2022-07-27 19:57:17 UTC

In 5.1 I could do the following:

(undercloud) [stack@undercloud-0 ~]$ cat /home/stack/initial-ceph.conf
[global]
public_network = '172.120.3.0/24,172.117.3.0/24,172.118.3.0/24,172.119.3.0/24'
cluster_network = '172.120.4.0/24,172.117.4.0/24,172.118.4.0/24,172.119.4.0/24'
ms_bind_ipv4 = true
ms_bind_ipv6 = false
(undercloud) [stack@undercloud-0 ~]$ 

./cephadm bootstrap --config initial-ceph.conf ...

and my Ceph cluster was able to bootstrap without a problem. 

However with cephadm-16.2.8-79.el9cp.noarch and 5-259 from the Ceph 5.2 beta, I intead see the following error and the bootstrap fails:

ERROR: Invalid public_network '172.120.3.0/24,172.117.3.0/24,172.118.3.0/24,172.119.3.0/24' parameter: '172.120.3.0/24 invalid: \"'172.120.3.0/24\" does not appear to be an IPv4 or IPv6 network, 172.119.3.0/24' is not in CIDR format (address/netmask)

Comment 1 John Fulton 2022-07-27 20:03:12 UTC

WORKAROUND:

bootstrap with "--config initial-ceph.conf" and then use command like the following:

  ceph config set global public_network '172.117.3.0/24,172.118.3.0/24,172.119.3.0/24,172.120.3.0/24'
  ceph config set global cluster_network '172.117.4.0/24,172.118.4.0/24,172.119.4.0/24,172.120.4.0/24'

The OSDs (which are on Ceph Storage Nodes on a routed network) are then able to join the ceph cluster.

Though customers can use a workaround, OpenStack documentation has supported the comma delimited method in the initial ceph.conf for a "one command deploy". 

Thus, they might perceive this as a regression.

Comment 2 John Fulton 2022-07-27 20:05:11 UTC

(In reply to John Fulton from comment #1)
> WORKAROUND:
> 
> bootstrap with "--config initial-ceph.conf" and then use..

I meant bootstrap WITHOUT the initial ceph conf.

Comment 23 errata-xmlrpc 2023-06-15 09:15:36 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 6.1 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:3623