Bug 2282254

Summary: Mon stay in CrashLoopBackOff when deploying ODF single stack ipv6
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Shay Rozen <srozen>
Component: rookAssignee: Blaine Gardner <brgardne>
Status: CLOSED ERRATA QA Contact: Shay Rozen <srozen>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.16CC: brgardne, ebenahar, odf-bz-bot, tnielsen
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: ODF 4.16.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.16.0-110 Doc Type: No Doc Update
Doc Text:
If this bug requires documentation, please select an appropriate Doc Type value.
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-07-17 13:23:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Shay Rozen 2024-05-21 20:27:12 UTC
Description of problem (please be detailed as possible and provide log
snippests):
ODF deployment with single stack ipv6 fail to deploy because mons don't come up.

Version of all relevant components (if applicable):
OCP 4.16.0-rc.1
ODF 4.16.0-102

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Can't deploy.

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
No

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Install OCP with single stack ipv6
2. Create storagesystem and storagecluster with:
spec:
  network:
    ipFamily: IPv6
    dualStack: false

Actual results:
Mons are in CrashLoopBackOff

rook-ceph-mon-a-7c969b89d-2t4st                           1/2     CrashLoopBackOff   17 (2m57s ago)   67m
rook-ceph-mon-b-78bb759896-bccgf                          1/2     CrashLoopBackOff   25 (3m36s ago)   111m
rook-ceph-mon-c-58f64786b-c99ts                           1/2     CrashLoopBackOff   21 (4m15s ago)   90m

Expected results:
Mons should come up and ODF deployed


Additional info:

Seems like the port got into the bind ip address, instead of [d13:6d22:0:5::e:3300]:3300 it should be [d13:6d22:0:5::e]:3000
Public address seems fine [d13:6d21:29dc:3::e6fc]:3300

from mon log
debug 2024-05-21T17:57:57.173+0000 7f7879db1b00  0 starting mon.a rank 0 at public addrs v2:[d13:6d21:29dc:3::e6fc]:3300/0 at bind addrs [v2:[d13:6d22:0:5::e:3300]:3300/0,v1:[d13:6d22:0:5::e:3300]:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 30da6c5f-0147-4267-9883-6f7bbc6e7dbc
debug 2024-05-21T17:57:57.174+0000 7f7879db1b00  1 mon.a@-1(???) e0 preinit fsid 30da6c5f-0147-4267-9883-6f7bbc6e7dbc
debug 2024-05-21T17:57:57.174+0000 7f7879db1b00  1 mon.a@-1(???) e0  initial_members a, filtering seed monmap
debug 2024-05-21T17:57:57.176+0000 7f7879db1b00 -1  Processor -- bind unable to bind to v2:[d13:6d22:0:5::e:3300]:3300/0: (99) Cannot assign requested address
debug 2024-05-21T17:57:57.176+0000 7f7879db1b00 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
debug 2024-05-21T17:58:02.177+0000 7f7879db1b00 -1  Processor -- bind unable to bind to v2:[d13:6d22:0:5::e:3300]:3300/0: (99) Cannot assign requested address
debug 2024-05-21T17:58:02.177+0000 7f7879db1b00 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
debug 2024-05-21T17:58:07.177+0000 7f7879db1b00 -1  Processor -- bind unable to bind to v2:[d13:6d22:0:5::e:3300]:3300/0: (99) Cannot assign requested address
debug 2024-05-21T17:58:07.177+0000 7f7879db1b00 -1  Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address
debug 2024-05-21T17:58:07.177+0000 7f7879db1b00 -1 unable to bind monitor to [v2:[d13:6d22:0:5::e:3300]:3300/0,v1:[d13:6d22:0:5::e:3300]:6789/0]

Comment 9 Shay Rozen 2024-05-30 07:44:11 UTC
Verified on 4.16.0-110

rook-ceph-mon-a-6c9cb964d-hfrxc                                   2/2     Running     0             23m
rook-ceph-mon-b-75dc88b466-kgtpq                                  2/2     Running     0             23m
rook-ceph-mon-c-77f9bff476-4wh69                                  2/2     Running     0             22m

Comment 11 errata-xmlrpc 2024-07-17 13:23:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:4591