Bug 2173161 - ODF-console can not start when you disable IPv6 on Node with kernel parameter.
Summary: ODF-console can not start when you disable IPv6 on Node with kernel parameter.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: odf-operator
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ODF 4.13.0
Assignee: Sanjal Katiyar
QA Contact: Shay Rozen
URL:
Whiteboard:
: 2139785 (view as bug list)
Depends On:
Blocks: 2154341
TreeView+ depends on / blocked
 
Reported: 2023-02-24 11:00 UTC by Raimund Sacherer
Modified: 2024-05-23 07:31 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, when IPv6 was disabled at node's kernel level, `IPv6 listen` directive of `nginx` configuration for the `odf-console` pod gave an error. As a result, OpenShift Data Foundation was stuck with `odf-console` not available and `odf-console` is in `CrashLoopBackOff` errors. With this fix, all the `nginx` configurations are added through the ConfigMap created by the `odf-operator`. You need to perform the following steps to overcome the issue: 1. Go to ConfigMap: "odf-console-nginx-conf". 2. Comment out the followig line as shown: ---- # listen [::]:9001 ssl; ---- 3. Restart the `odf-console` pod by deleting it. 4. Refresh the user interface.
Clone Of:
Environment:
Last Closed: 2023-06-21 15:24:01 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github red-hat-storage odf-operator pull 305 0 None Merged add nginx configurations for the odf-console 2023-04-06 07:59:18 UTC
Github red-hat-storage odf-operator pull 306 0 None Merged Bug 2139785:[release-4.13] add nginx configurations for the odf-console 2023-04-06 07:59:20 UTC
Red Hat Product Errata RHBA-2023:3742 0 None None None 2023-06-21 15:24:27 UTC

Description Raimund Sacherer 2023-02-24 11:00:26 UTC
Description of problem (please be detailed as possible and provide log
snippests):
ODF-console can not start when you disable IPv6 on Node with kernel parameter. (more details in the reproducable steps section).

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
You have to activate IPv6 again. IPv6 can not be disabled.

Is there any workaround available to the best of your knowledge?
Activate IPv6 again.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Not tested


Steps to Reproduce:

ODF Console is running:
```
[jose@rhte-router ~]$ oc get pods -A -o wide| grep odf-console
openshift-storage                                  odf-console-6b6db469d9-h2fjl                                      1/1     Running     0               44h    10.130.2.6     mutua-5pdst-infra-wkzwv    <none>           <none>
```


We create a machine config to disable IPV6 with a kernel argument (as described in KCS: https://access.redhat.com/solutions/5513091):
```
[jose@rhte-router ~]$ cat disableIPV6.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-openshift-machineconfig-worker-kargs2
spec:
  kernelArguments:
   -  ipv6.disable=1
```


After applying this configuration snippet and waiting for all worker nodes to apply the config and reboot, ODF-console does not start anymore:
```
openshift-storage                                  odf-console-6b6db469d9-nvhlt                                      0/1     CrashLoopBackOff   8 (92s ago)     17m     10.130.2.11    mutua-5pdst-infra-wkzwv    <none>           <none>
```

Logs indicate that NGINX tries to listen on socket `[::]` which, as IPv6 is disabled, fails with an error:
```
[jose@rhte-router ~]$ oc logs --prefix=true --all-containers=true -n openshift-storage odf-console-6b6db469d9-nvhlt
[pod/odf-console-6b6db469d9-nvhlt/odf-console] 2023/02/24 10:06:12 [emerg] 1#0: socket() [::]:9001 failed (97: Address family not supported by protocol)
[pod/odf-console-6b6db469d9-nvhlt/odf-console] nginx: [emerg] socket() [::]:9001 failed (97: Address family not supported by protocol)
```


Actual results:
ODF-console not starting/Crashing

Expected results:
ODF-console starting without issues


Additional info:

Comment 3 Scott Ostapovicz 2023-03-14 12:55:27 UTC
As far as I know, "::" is the IPv6 unspecified address, and should not be used to reference any specific host.  But in any case, it is a an IPv6 address, so if you trying to disable IPv6 what is the expectation here when you then use an IPv6 address?  I guess I do not understand this issue.  The behavior seems correct to me.

Comment 4 Raimund Sacherer 2023-03-14 13:20:20 UTC
(In reply to Scott Ostapovicz from comment #3)
> As far as I know, "::" is the IPv6 unspecified address, and should not be
> used to reference any specific host.  But in any case, it is a an IPv6
> address, so if you trying to disable IPv6 what is the expectation here when
> you then use an IPv6 address?  I guess I do not understand this issue.  The
> behavior seems correct to me.

Hi Scott, 

The issue is that CU for internal requirements or compliance want to disable IPV6 on all hosts on their system. Doing this breaks the odf-console, as soon as you deactivate IPV6 on the host via kernel parameter, odf-console does not start anymore. 

The problem seems to be the default configuration of nginx in odf-console. It seems to expect that IPV6 is present, if not, nginx does not start.

Best regards
Raimund

Comment 5 Scott Ostapovicz 2023-03-14 14:36:04 UTC
So an address somewhere in the stack is still an IPv6 address even though the script nominally disables IPv6.  Perhaps you can suss this out @tnielsen ?

Comment 10 Sanjal Katiyar 2023-03-30 12:22:16 UTC
*** Bug 2139785 has been marked as a duplicate of this bug. ***

Comment 18 Shay Rozen 2023-05-29 12:11:38 UTC
Check on 4.13.0-205. Changed the configmap as above. Restarted odf console. Still in CrashLoopBackOff. Moving to assigned.

Comment 20 Sanjal Katiyar 2023-05-29 13:56:47 UTC
Hi Shay,
I have added a detailed recording https://bugzilla.redhat.com/show_bug.cgi?id=2173161#c19 on how to test this (basically video version of https://bugzilla.redhat.com/show_bug.cgi?id=2173161#c11)
Feel free to connect with me offline, if there is still any issue you might be facing.

Thanks.

Comment 21 Shay Rozen 2023-05-29 15:52:48 UTC
Hi Sanjal. I've commented the wrong line. Looks like it is working now.
I saw that no doc update for this BZ. Who is responsible to create KCS for this hack.

Comment 22 Sanjal Katiyar 2023-05-30 05:43:48 UTC
Added the doc text, also adding need_info on reporter for getting help with KCS doc.

Comment 23 Mudit Agarwal 2023-06-14 10:52:18 UTC
Sanjal/Kusum

Doc text should be a bug fix and not a known issue as we have fixed something here.
If there are steps which some one needs to follow while disabling IPv6 then these steps should be added to the deployment guide or the guide where we are providing enabling/disabling steps for IPv6

Comment 24 Sanjal Katiyar 2023-06-14 11:05:05 UTC
Yeah, my intension for adding it as "Known Issue" is exactly as you mentioned:
Earlier there was no workaround to fix this issue (other than enabling IPv6 again on the nodes). As per this BZ, we added the functionality so that users can now use a workaround manually.

I will let doc team decide the correct place for this issue and the correct "Doc Type" as well.

Thanks.

Comment 26 Sanjal Katiyar 2023-06-14 11:31:08 UTC
Thanks Mudit/Anjana/Kusuma... updated...

Comment 28 errata-xmlrpc 2023-06-21 15:24:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.13.0 enhancement and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3742


Note You need to log in before you can comment on or make changes to this bug.