When using IPv6 for my public and cluster network my mon is able to bootstrap (if I use the fix from BZ 2002639). However I end up with 0 OSDs when I follow the standard procedure to add an OSD host by running a `ceph orch` command. Using journalctl of the OSD's systemd_unit shows it looking for an IPv4 address in an IPv6 network: "unable to find any IPv4 address in networks 'fd00:fd00:fd00:3000::/64' interfaces" I can work around this by running the following before adding any OSD hosts: ceph config set osd ms_bind_ipv4 false Though we have a workaround, the need to do the above breaks automated deployments. Either cephadm should take care of the above for the user or pick_address.cc should detect that it has an IPv6 address and do the right thing without requiring the user to set ms_bind_ipv4 false.
This bug has broken OSP17 IPv6 CI jobs with Ceph. We have a workaround on the tripleo/director side [1] but we'd rather not merge a workaround into TripleO and request that Ceph fix this issue. [1] https://review.opendev.org/c/openstack/tripleo-ansible/+/814064
> unable to find any IPv4 address in networks 'fd00:fd00:fd00:3000::/64' interfaces '' comes from pick_address.cc. Neha, do you think we can fix this there?
Created attachment 1842147 [details] output of "journalctl -u ceph-3ac0e06a-470c-11ec-babe-52540007205b"
Setting ms_bind options automatically (in pick_address.cc, cephadm, or tripleo) based on detection of the environment is non-trivial and if done incorrectly could break dual stack support [1]. TripleO [2] with ceph-ansible [3] didn't have dual stack support for Ceph (pick either v4 or v6). cephadm shouldn't break dual stack support in order to go back to how it was. Instead, tripleo, like cephadm, shouldn't block dual stack and instead it should allow the user to pick the v4 and v6 binding options too. [1] https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv4-ipv6-dual-stack-mode [2] https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/ceph-ansible/ceph-base.yaml#L598-L602 [3] https://github.com/ceph/ceph-ansible/blob/83a8dd5a6a1f9ffe43b2a75d7b49775e34c58f24/roles/ceph-config/templates/ceph.conf.j2#L10-L13
If you want to make pick_address.cc set the ms_bind options automatically based on detection of the environment, then feel free to re-open this bug. Otherwise OpenStack users can deploy OSDs in v6 (only) by using the following: parameter_defaults: CephConfigOverrides: global: ms_bind_ipv4: false ms_bind_ipv6: true