Bug 1995468 - Nodes can't resolved IPv4 address in dual stack configuration
Summary: Nodes can't resolved IPv4 address in dual stack configuration
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.8
Hardware: All
OS: All
unspecified
medium
Target Milestone: ---
: 4.9.0
Assignee: Yossi Boaron
QA Contact: Rei
URL:
Whiteboard:
Depends On:
Blocks: 1996573
TreeView+ depends on / blocked
 
Reported: 2021-08-19 08:33 UTC by Rei
Modified: 2021-10-18 17:47 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-18 17:47:26 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift baremetal-runtimecfg pull 148 0 None None None 2021-08-19 13:25:23 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:47:38 UTC

Description Rei 2021-08-19 08:33:57 UTC
Description of problem:
In case you deployed dual stack BM setup nodes can't resolved IPv4 address

OpenShift release version:
[rhalle@ocp-edge36 ~]$ oc version
Client Version: 4.8.0-0.nightly-2021-08-17-004424
Server Version: 4.8.0-0.nightly-2021-08-17-004424
Kubernetes Version: v1.21.1+9807387


Cluster Platform:
BM

How reproducible:
Deploy ocp on BM
Run:
oc debug node/master-0-0 --image registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/rhel8/support-tools:latest -- bash -c 'chroot /host dig master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com A | grep -A 1 'ANSWER SECTION'                        | grep -v 'ANSWER SECTION' | awk '{print $NF}'' --insecure-skip-tls-verify




Actual results:
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 5a22c279593559ef (echoed)
;; QUESTION SECTION:
;master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com. IN A

Expected results:
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
--
;; ANSWER SECTION:
master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com. 30 IN A 192.168.123.58


Impact of the problem:
Dual stack nodes can't communicate with IPv4 protocol

Additional info:



** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report.  You may also mark the bug private if you wish.

Comment 1 Rei 2021-08-19 08:59:00 UTC
This is regression from OCP-4.7 in 4.7 resolution is for IPv4
oc version
Client Version: 4.7.0-0.nightly-2021-08-18-141355
Server Version: 4.7.0-0.nightly-2021-08-18-141355
Kubernetes Version: v1.20.0+4593a24
[root@seal23 ~]# oc get node
NAME                                              STATUS   ROLES    AGE   VERSION
master-0-0.ocp-edge-cluster-0.qe.lab.redhat.com   Ready    master   15m   v1.20.0+4593a24
master-0-1.ocp-edge-cluster-0.qe.lab.redhat.com   Ready    master   15m   v1.20.0+4593a24
master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com   Ready    master   15m   v1.20.0+4593a24
[root@seal23 ~]# watch oc get bmh -n openshift-machine-api
[root@seal23 ~]# oc get node
NAME                                              STATUS   ROLES    AGE     VERSION
master-0-0.ocp-edge-cluster-0.qe.lab.redhat.com   Ready    master   26m     v1.20.0+4593a24
master-0-1.ocp-edge-cluster-0.qe.lab.redhat.com   Ready    master   26m     v1.20.0+4593a24
master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com   Ready    master   26m     v1.20.0+4593a24
worker-0-0.ocp-edge-cluster-0.qe.lab.redhat.com   Ready    worker   2m47s   v1.20.0+4593a24
worker-0-1.ocp-edge-cluster-0.qe.lab.redhat.com   Ready    worker   98s     v1.20.0+4593a24
[root@seal23 ~]# oc debug node/master-0-1.ocp-edge-cluster-0.qe.lab.redhat.com --image registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/rhel8/support-tools:latest -- bash -c 'chroot /host dig master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com A' --insecure-skip-tls-verify
Starting pod/master-0-1ocp-edge-cluster-0qelabredhatcom-debug ...
To use host binaries, run `chroot /host`

; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48136
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 78fcb8b250274b72 (echoed)
;; QUESTION SECTION:
;master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com. IN A

;; ANSWER SECTION:
master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com. 30 IN A 192.168.123.145

;; Query time: 0 msec
;; SERVER: 192.168.123.144#53(192.168.123.144)
;; WHEN: Thu Aug 19 08:54:54 UTC 2021
;; MSG SIZE  rcvd: 151


Removing debug pod ...
[root@seal23 ~]# ^A^AAAA
oc debug node/master-0-1.ocp-edge-cluster-0.qe.lab.redhat.com --image registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/rhel8/support-tools:latest -- bash -c 'chroot /host dig master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com AAAA' --insecure-skip-tls-verify
Starting pod/master-0-1ocp-edge-cluster-0qelabredhatcom-debug ...
To use host binaries, run `chroot /host`

; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35570
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 94a4e5668d1b0adb (echoed)
;; QUESTION SECTION:
;master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com. IN AAAA

;; ANSWER SECTION:
master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com. 30 IN A 192.168.123.145

;; Query time: 0 msec
;; SERVER: 192.168.123.144#53(192.168.123.144)
;; WHEN: Thu Aug 19 08:55:12 UTC 2021
;; MSG SIZE  rcvd: 151

Comment 3 Rei 2021-08-22 12:41:13 UTC
Verify:

[root@seal23 ~]# oc debug node/master-0-1.ocp-edge-cluster-0.qe.lab.redhat.com --image registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/rhel8/support-tools:latest -- bash -c 'chroot /host ping master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com -6' --insecure-skip-tls-verify
Starting pod/master-0-1ocp-edge-cluster-0qelabredhatcom-debug ...
To use host binaries, run `chroot /host`
PING master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com(master-0-2 (fd2e:6f44:5dd8::65)) 56 data bytes
64 bytes from master-0-2 (fd2e:6f44:5dd8::65): icmp_seq=1 ttl=64 time=2.43 ms
64 bytes from master-0-2 (fd2e:6f44:5dd8::65): icmp_seq=2 ttl=64 time=0.406 ms
64 bytes from master-0-2 (fd2e:6f44:5dd8::65): icmp_seq=3 ttl=64 time=0.367 ms
64 bytes from master-0-2 (fd2e:6f44:5dd8::65): icmp_seq=4 ttl=64 time=0.256 ms
^C
Removing debug pod ...
[root@seal23 ~]# oc debug node/master-0-1.ocp-edge-cluster-0.qe.lab.redhat.com --image registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/rhel8/support-tools:latest -- bash -c 'chroot /host ping master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com -4' --insecure-skip-tls-verify
Starting pod/master-0-1ocp-edge-cluster-0qelabredhatcom-debug ...
To use host binaries, run `chroot /host`
PING master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com (192.168.123.121) 56(84) bytes of data.
64 bytes from master-0-2 (192.168.123.121): icmp_seq=1 ttl=64 time=0.946 ms
64 bytes from master-0-2 (192.168.123.121): icmp_seq=2 ttl=64 time=0.259 ms
64 bytes from master-0-2 (192.168.123.121): icmp_seq=3 ttl=64 time=0.216 ms
64 bytes from master-0-2 (192.168.123.121): icmp_seq=4 ttl=64 time=0.402 ms
^C
Removing debug pod ...
[root@seal23 ~]# oc debug node/master-0-1.ocp-edge-cluster-0.qe.lab.redhat.com --image registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/rhel8/support-tools:latest -- bash -c 'chroot /host dig master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com ' --insecure-skip-tls-verify
Starting pod/master-0-1ocp-edge-cluster-0qelabredhatcom-debug ...
To use host binaries, run `chroot /host`

; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16163
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 672e0ff3f32b86bc (echoed)
;; QUESTION SECTION:
;master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com. IN A

;; ANSWER SECTION:
master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com. 13 IN A 192.168.123.121

;; Query time: 0 msec
;; SERVER: 192.168.123.55#53(192.168.123.55)
;; WHEN: Sun Aug 22 12:38:42 UTC 2021
;; MSG SIZE  rcvd: 151


Removing debug pod ...
[root@seal23 ~]# oc debug node/master-0-1.ocp-edge-cluster-0.qe.lab.redhat.com --image registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/rhel8/support-tools:latest -- bash -c 'chroot /host dig master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com AAAA' --insecure-skip-tls-verify
Starting pod/master-0-1ocp-edge-cluster-0qelabredhatcom-debug ...
To use host binaries, run `chroot /host`

; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44879
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: bf5dc7b3500906b2 (echoed)
;; QUESTION SECTION:
;master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com. IN AAAA

;; ANSWER SECTION:
master-0-2.ocp-edge-cluster-0.qe.lab.redhat.com. 30 IN AAAA fd2e:6f44:5dd8::65

;; Query time: 0 msec
;; SERVER: 192.168.123.55#53(192.168.123.55)
;; WHEN: Sun Aug 22 12:38:54 UTC 2021
;; MSG SIZE  rcvd: 163


Removing debug pod ...
[root@seal23 ~]# oc version
Client Version: 4.9.0-0.nightly-2021-08-22-005117
Server Version: 4.9.0-0.nightly-2021-08-22-070405
Kubernetes Version: v1.22.0-rc.0+5c2f7cd

Comment 6 errata-xmlrpc 2021-10-18 17:47:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.