Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2107114

Summary: [RHOS17] OSDs are failing to start after node reboot with IPv6 network
Product: Red Hat OpenStack Reporter: Mikolaj Ciecierski <mciecier>
Component: tripleo-ansibleAssignee: John Fulton <johfulto>
Status: CLOSED DUPLICATE QA Contact: Yogev Rabl <yrabl>
Severity: high Docs Contact:
Priority: high    
Version: 17.0 (Wallaby)CC: aschoen, ceph-eng-bugs, gmeno, johfulto, kthakre, nthomas, ykaul
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-07-21 13:41:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mikolaj Ciecierski 2022-07-14 11:34:57 UTC 
Description of problem:
Performing update of OSP17 composable with network-protocol ipv6 and deployed ceph. After reboot of first OSD node, performed as the last step of the update procedure, all OSDs are failing to start. 
According to documentation [1] when network_data.yaml contains 'ipv6: true', ms_bind_ipv4 should be set false and ms_bind_ipv6 should be set to true. However in config dump[2] there is only ms_bind_ipv6 set, and ms_bind_ipv4 is not(the default value is true). This is probably the reason why OSD is trying to bind to IPv4 address.

In journactl I can see:

journalctl -xeu ceph-3e818ffc-fc2e-5bbc-8acb-804c4370d17c.service
Jul 14 11:19:26 cephstorage-1 ceph-3e818ffc-fc2e-5bbc-8acb-804c4370d17c-osd-13[105063]: debug 2022-07-14T11:19:26.627+0000 7fb1ec307200  1 bdev(0x55a19a02f400 /var/lib/ceph/osd/ceph-13/block) open size 34355544064 (0x7ffc00000, 32 GiB) b>
Jul 14 11:19:26 cephstorage-1 ceph-3e818ffc-fc2e-5bbc-8acb-804c4370d17c-osd-13[105063]: debug 2022-07-14T11:19:26.627+0000 7fb1ec307200  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-13/block size 32 GiB
Jul 14 11:19:26 cephstorage-1 ceph-3e818ffc-fc2e-5bbc-8acb-804c4370d17c-osd-13[105063]: debug 2022-07-14T11:19:26.628+0000 7fb1ec307200  1 bdev(0x55a19a02f400 /var/lib/ceph/osd/ceph-13/block) close
Jul 14 11:19:26 cephstorage-1 ceph-3e818ffc-fc2e-5bbc-8acb-804c4370d17c-osd-13[105063]: debug 2022-07-14T11:19:26.646+0000 7fb1ec307200  1 bdev(0x55a19a02f800 /var/lib/ceph/osd/ceph-13/block) close
Jul 14 11:19:26 cephstorage-1 ceph-3e818ffc-fc2e-5bbc-8acb-804c4370d17c-osd-13[105063]: debug 2022-07-14T11:19:26.901+0000 7fb1ec307200  0 starting osd.13 osd_data /var/lib/ceph/osd/ceph-13 /var/lib/ceph/osd/ceph-13/journal
Jul 14 11:19:26 cephstorage-1 ceph-3e818ffc-fc2e-5bbc-8acb-804c4370d17c-osd-13[105063]: debug 2022-07-14T11:19:26.904+0000 7fb1ec307200 -1 unable to find any IPv4 address in networks 'fd00:fd00:fd00:3000::/64' interfaces ''
Jul 14 11:19:26 cephstorage-1 ceph-3e818ffc-fc2e-5bbc-8acb-804c4370d17c-osd-13[105063]: debug 2022-07-14T11:19:26.904+0000 7fb1ec307200 -1 Failed to pick public address.
Jul 14 11:19:26 cephstorage-1 podman[105087]: 2022-07-14 11:19:26.97891986 +0000 UTC m=+0.045991096 container died ffc1c96cb74f07d104cdebec73924de0f1647dd28ef66cf75b674155fc447301 (image=undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rh>
Jul 14 11:19:27 cephstorage-1 podman[105087]: 2022-07-14 11:19:27.005086538 +0000 UTC m=+0.072157756 container remove ffc1c96cb74f07d104cdebec73924de0f1647dd28ef66cf75b674155fc447301 (image=undercloud-0.ctlplane.redhat.local:8787/rh-osbs>
Jul 14 11:19:27 cephstorage-1 systemd[1]: ceph-3e818ffc-fc2e-5bbc-8acb-804c4370d17c.service: Main process exited, code=exited, status=1/FAILURE

 An ExecStart= process belonging to unit ceph-3e818ffc-fc2e-5bbc-8acb-804c4370d17c.service has exited.
The process' exit code is 'exited' and its exit status is 1.





Version-Release number of selected component (if applicable):


How reproducible:
always, when ipv6 network is used

Steps to Reproduce:
1.Deploy OSP17 with ceph
2.Reboot one ceph OSD node
3.Run `sudo cephadm shell --  ceph osd tree` to check status of osds

Actual results:
All osds are down after reboot

[heat-admin@controller-0 ~]$ sudo cephadm shell --  ceph osd tree                                                                                                                                                                            
Inferring fsid 3e818ffc-fc2e-5bbc-8acb-804c4370d17c
Using recent ceph image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph@sha256:0eb00dcba8ab47ff957166e1a2018be0905c11ed30b7c71247893423353986ee                                                                                       
WARN[0000]  binary not found, container dns will not be enabled
ID  CLASS  WEIGHT   TYPE NAME               STATUS  REWEIGHT  PRI-AFF
-1         0.46783  root default
-3         0.15594      host cephstorage-0
 0    hdd  0.03119          osd.0               up   1.00000  1.00000
 1    hdd  0.03119          osd.1               up   1.00000  1.00000
 2    hdd  0.03119          osd.2               up   1.00000  1.00000
 3    hdd  0.03119          osd.3               up   1.00000  1.00000
 4    hdd  0.03119          osd.4               up   1.00000  1.00000
-5         0.15594      host cephstorage-1
 5    hdd  0.03119          osd.5             down         0  1.00000
 7    hdd  0.03119          osd.7             down         0  1.00000
 9    hdd  0.03119          osd.9             down         0  1.00000
11    hdd  0.03119          osd.11            down         0  1.00000
13    hdd  0.03119          osd.13            down   1.00000  1.00000
-7         0.15594      host cephstorage-2
 6    hdd  0.03119          osd.6               up   1.00000  1.00000
 8    hdd  0.03119          osd.8               up   1.00000  1.00000
10    hdd  0.03119          osd.10              up   1.00000  1.00000
12    hdd  0.03119          osd.12              up   1.00000  1.00000
14    hdd  0.03119          osd.14              up   1.00000  1.00000


Expected results:
All osds should start after reboot of the machine 

Additional info:
[1]https://docs.openstack.org/project-deploy-guide/tripleo-docs/wallaby/features/deployed_ceph.html#example-ipv6
[2]
[heat-admin@controller-0 ~]$ sudo cephadm shell --  ceph config dump
Inferring fsid 3e818ffc-fc2e-5bbc-8acb-804c4370d17c
Using recent ceph image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph@sha256:0eb00dcba8ab47ff957166e1a2018be0905c11ed30b7c71247893423353986ee
WARN[0000]  binary not found, container dns will not be enabled 
WHO                                     MASK  LEVEL     OPTION                                 VALUE                                                                                                                           RO
global                                        advanced  cluster_network                        fd00:fd00:fd00:4000::/64                                                                                                        * 
global                                        basic     container_image                        undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph@sha256:0eb00dcba8ab47ff957166e1a2018be0905c11ed30b7c71247893423353986ee  * 
global                                        advanced  ms_bind_ipv6                           true                                                                                                                              
global                                        advanced  osd_pool_default_pg_num                32                                                                                                                                
global                                        advanced  osd_pool_default_pgp_num               32                                                                                                                                
global                                        advanced  osd_pool_default_size                  3               
...
Comment 1 RHEL Program Management 2022-07-14 11:35:03 UTC 
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Comment 3 John Fulton 2022-07-20 12:32:36 UTC 
OSP17 does not use ceph-ansible so this shouldn't be filed under ceph/ceph-ansible
Comment 4 John Fulton 2022-07-20 12:35:50 UTC 
This sounds a like a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2104647. 
Please try to reproduce by doing a new deploy with tripleo-ansible-3.3.1-0.20220708201820.fa5422f.el8ost installed on your undercloud and then do the reboot.
If the bug reproduces, then please provide needinfo me access to a live system so I can investigate.
If the bug does not reproduce, then please close this as a duplicate of 2104647.
Comment 5 John Fulton 2022-07-20 12:50:38 UTC 
For an example of how to correctly deploy ceph w/ OSP17 w/ IPv6 please see this:

  https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/deployed_ceph.html#example-ipv6

we need to make sure IR is passing your network_data.yaml to 'openstack overcloud ceph deploy' and that before the reboot that your ceph configuration (ssh controller-0; sudo cephadm shell -- ceph config dump) has ms_bind_ipv4 = False and ms_bind_ipv6 = True. The ms_bind settings will be configured correctly by director provided the network_data.yaml file is passed by IR as per the patch below from December:

  https://review.gerrithub.io/c/redhat-openstack/infrared/+/528642
Comment 7 Khomesh Thakre 2022-07-21 13:41:10 UTC 

*** This bug has been marked as a duplicate of bug 2104647 ***