Bug 2282254 - Mon stay in CrashLoopBackOff when deploying ODF single stack ipv6
Summary: Mon stay in CrashLoopBackOff when deploying ODF single stack ipv6
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: rook
Version: 4.16
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ODF 4.16.0
Assignee: Blaine Gardner
QA Contact: Shay Rozen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-05-21 20:27 UTC by Shay Rozen
Modified: 2024-07-17 13:23 UTC (History)
4 users (show)

Fixed In Version: 4.16.0-110
Doc Type: No Doc Update
Doc Text:
If this bug requires documentation, please select an appropriate Doc Type value.
Clone Of:
Environment:
Last Closed: 2024-07-17 13:23:32 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github red-hat-storage rook pull 656 0 None open BUG 2282254: mon: fix bind addr when IPv6 and msgr2 are enabled 2024-05-23 17:21:43 UTC
Red Hat Product Errata RHSA-2024:4591 0 None None None 2024-07-17 13:23:35 UTC

Description Shay Rozen 2024-05-21 20:27:12 UTC
Description of problem (please be detailed as possible and provide log
snippests):
ODF deployment with single stack ipv6 fail to deploy because mons don't come up.

Version of all relevant components (if applicable):
OCP 4.16.0-rc.1
ODF 4.16.0-102

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Can't deploy.

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
No

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Install OCP with single stack ipv6
2. Create storagesystem and storagecluster with:
spec:
  network:
    ipFamily: IPv6
    dualStack: false

Actual results:
Mons are in CrashLoopBackOff

rook-ceph-mon-a-7c969b89d-2t4st                           1/2     CrashLoopBackOff   17 (2m57s ago)   67m
rook-ceph-mon-b-78bb759896-bccgf                          1/2     CrashLoopBackOff   25 (3m36s ago)   111m
rook-ceph-mon-c-58f64786b-c99ts                           1/2     CrashLoopBackOff   21 (4m15s ago)   90m

Expected results:
Mons should come up and ODF deployed


Additional info:

Seems like the port got into the bind ip address, instead of [d13:6d22:0:5::e:3300]:3300 it should be [d13:6d22:0:5::e]:3000
Public address seems fine [d13:6d21:29dc:3::e6fc]:3300

from mon log
debug 2024-05-21T17:57:57.173+0000 7f7879db1b00  0 starting mon.a rank 0 at public addrs v2:[d13:6d21:29dc:3::e6fc]:3300/0 at bind addrs [v2:[d13:6d22:0:5::e:3300]:3300/0,v1:[d13:6d22:0:5::e:3300]:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 30da6c5f-0147-4267-9883-6f7bbc6e7dbc
debug 2024-05-21T17:57:57.174+0000 7f7879db1b00  1 mon.a@-1(???) e0 preinit fsid 30da6c5f-0147-4267-9883-6f7bbc6e7dbc
debug 2024-05-21T17:57:57.174+0000 7f7879db1b00  1 mon.a@-1(???) e0  initial_members a, filtering seed monmap
debug 2024-05-21T17:57:57.176+0000 7f7879db1b00 -1  Processor -- bind unable to bind to v2:[d13:6d22:0:5::e:3300]:3300/0: (99) Cannot assign requested address
debug 2024-05-21T17:57:57.176+0000 7f7879db1b00 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
debug 2024-05-21T17:58:02.177+0000 7f7879db1b00 -1  Processor -- bind unable to bind to v2:[d13:6d22:0:5::e:3300]:3300/0: (99) Cannot assign requested address
debug 2024-05-21T17:58:02.177+0000 7f7879db1b00 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
debug 2024-05-21T17:58:07.177+0000 7f7879db1b00 -1  Processor -- bind unable to bind to v2:[d13:6d22:0:5::e:3300]:3300/0: (99) Cannot assign requested address
debug 2024-05-21T17:58:07.177+0000 7f7879db1b00 -1  Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address
debug 2024-05-21T17:58:07.177+0000 7f7879db1b00 -1 unable to bind monitor to [v2:[d13:6d22:0:5::e:3300]:3300/0,v1:[d13:6d22:0:5::e:3300]:6789/0]

Comment 9 Shay Rozen 2024-05-30 07:44:11 UTC
Verified on 4.16.0-110

rook-ceph-mon-a-6c9cb964d-hfrxc                                   2/2     Running     0             23m
rook-ceph-mon-b-75dc88b466-kgtpq                                  2/2     Running     0             23m
rook-ceph-mon-c-77f9bff476-4wh69                                  2/2     Running     0             22m

Comment 11 errata-xmlrpc 2024-07-17 13:23:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:4591


Note You need to log in before you can comment on or make changes to this bug.