Bug 1931974
| Summary: | Operators cannot access kubeapi endpoint on OVNKubernetes on ipv6 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Osher De Paz <odepaz> | ||||
| Component: | Networking | Assignee: | Antoni Segura Puimedon <asegurap> | ||||
| Networking sub component: | runtime-cfg | QA Contact: | nshidlin <nshidlin> | ||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||
| Severity: | high | ||||||
| Priority: | high | CC: | aconstan, asegurap, shardy | ||||
| Version: | 4.8 | ||||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.8.0 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: |
Cause: When bootstrapping an IPv6 using cluster in environments where both IPv4 and IPv6 addressing is available, Kubelet could choose any stack.
Consequence: The times it picked IPv4, the address registered as an endpoint to the Kubernetes IP would not match the protocol of the service (IPv6 on an IPv6 installation) and OVN would fail to load balance to it.
Fix: Detect when the cluster being deployed is an IPv6 one and use that information to indicate to Kubelet that it should preferably bind to an IPv6 address.
Result: Cluster installation consistently works for IPv6 clusters in environments where IPv4 and IPv6 addressing is available.
|
Story Points: | --- | ||||
| Clone Of: | |||||||
| : | 1980866 (view as bug list) | Environment: | |||||
| Last Closed: | 2021-07-27 22:47:39 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1980866 | ||||||
| Attachments: |
|
||||||
|
Description
Osher De Paz
2021-02-23 16:57:35 UTC
Hi Could you provide me with a kubeconfig / must-gather to the cluster in exhibiting the problems stated? Thanks in advance, Alexander Also, please retry with the latest 4.8 version. We've had some IPv6 fixes coming in with the latest downstream merge: https://github.com/openshift/ovn-kubernetes/pull/440 Hi
So, I've had a look at this with the helper of a reproducer that Osher provided me with, and I saw the following:
[root@f13-h23-b04-5039ms assisted-test-infra]# oc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 2003:db8::1 <none> 443/TCP 126m
[root@f13-h23-b04-5039ms assisted-test-infra]# oc get ep
NAME ENDPOINTS AGE
kubernetes 10.88.0.1:6443 126m
I.e: the kube-apiserver running on the bootstrap node during the cluster creation phase has an IPv4 address on this IPv6 single-stack cluster, this is the reason ovnkube-master does not add the service to its load balancer items.
I've had a look at that kube-apiserver container and I saw the following:
$ crictl inspect $KUBE_APISERVER_CONTAINER
"info": {
"sandboxID": "b4f5090d01424f7629e97198176a6d473a1f7fdc1b6e561b57e6b91ebb8fed11",
"pid": 38933,
"runtimeSpec": {
"ociVersion": "1.0.2-dev",
"process": {
"user": {
"uid": 0,
"gid": 0
},
"args": [
"/bin/bash",
"-ec",
"hyperkube kube-apiserver --openshift-config=/etc/kubernetes/config/kube-apiserver-config.yaml --logtostderr=false --alsologtostderr --v=2 --log-file=/var/log/bootstrap-control-plane/kube-apiserver.log --advertise-address=${HOST_IP}\n"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"HOSTNAME=random-hostname-a49e0432-ee46-4e67-8f36-b6548cfe3c84",
"HOST_IP=10.88.0.1",
So, presumably there's a mis-configuration of the ENV: `HOST_IP` which results in it advertising an IPv4 address.
I am thus re-assigning to the KNI team.
/Alex
kubelet is incorrectly picking the podman0 IPv4 bridge address. We'll need to add the nodeip-configuration service to it. Verified using assisted-installer to install 4.8.0-0.nightly-2021-04-18-203506 oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-04-18-203506 True False 105s Cluster version is 4.8.0-0.nightly-2021-04-18-203506 oc get ep NAME ENDPOINTS AGE kubernetes [1001:db8::4f]:6443,[1001:db8::55]:6443 41m oc get ep NAME ENDPOINTS AGE kubernetes [1001:db8::4f]:6443,[1001:db8::55]:6443 42m https://github.com/openshift/installer/pull/4756 refers to https://github.com/openshift/cluster-kube-apiserver-operator/pull/1042 (which is a release-4.7 backport), so do we need to backport the installer PR to 4.7? This is also under discussion via https://github.com/openshift/installer/pull/5013 where we need the TemplateData, I'm wondering if it makes sense to just backport 4756 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |