Bug 1909199 - Tigera collaboration: switching from ipip bgp to native bgp causes cluster degradation for 20 minutes.
Summary: Tigera collaboration: switching from ipip bgp to native bgp causes cluster de...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Ben Bennett
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-18 15:21 UTC by Dirk Porter
Modified: 2024-10-01 17:14 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-29 15:40:55 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Dirk Porter 2020-12-18 15:21:27 UTC 
Description of problem:

Customer is using calico sdn on openshift 4.5. The nodes connect to a leaf and connect to each other using bgp over an ipip tunnel. 

When the customer kills that ipip tunnel to fail over to standard bgp or goes from standard bgp to bgp over ipip, the following operators go into a degraded state: 

openshift-apiserver-operator
controller-manager
authentication 
console
kube-server 
operator-lifecycle-manager-package-server 

Certain commands are unable to be run for instance oc get routes, which I believe is due to a communication issue between the openshift-apiserver and the kube-apiserver. 

After 20 minutes the cluster recovers itself, but I have been unable to discern what is actually 'recovering' 

Errors from kube-apiserver during the downtime: 

2020-11-02T21:29:12.156052555Z E1102 21:29:12.156024       1 controller.go:114] loading OpenAPI spec for "v1.project.openshift.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
2020-11-02T21:29:13.40151479Z E1102 21:29:13.401479       1 controller.go:114] loading OpenAPI spec for "v1.packages.operators.coreos.com" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable

During the downtime the apiservices show that the probes are failing. 

Looking in the kube-proxy logs: 

master-0 pod: 

2020-11-02T19:06:30.62992526Z E1102 19:06:30.623489       1 proxier.go:1546] Failed to execute iptables-restore: exit status 4 (iptables-restore v1.8.4 (nf_tables):
2020-11-02T19:06:30.62992526Z line 413: CHAIN_USER_DEL failed (Device or resource busy): chain KUBE-SEP-QLSSCJ6QXIVS67KP

master-1 pod: 

2020-11-02T19:02:28.168044575Z E1102 19:02:28.168016       1 reflector.go:178] runtime/asm_amd64.s:1357: Failed to list *v1.Service: Get https://api-int.ocp4.contoso.com:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=26242: dial tcp 192.168.1.10:6443: connect: connection refused
2020-11-02T19:10:34.372411891Z E1102 19:10:34.371888       1 proxier.go:1546] Failed to execute iptables-restore: exit status 4 (iptables-restore v1.8.4 (nf_tables):
2020-11-02T19:10:34.372411891Z line 451: CHAIN_USER_DEL failed (Device or resource busy): chain KUBE-SEP-ZFBU47UZLOLD3YIN

master-2 pod: 

2020-11-02T19:02:28.181830497Z E1102 19:02:28.176355       1 reflector.go:178] runtime/asm_amd64.s:1357: Failed to list *v1.Service: Get https://api-int.ocp4.contoso.com:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=26242: dial tcp 192.168.1.10:6443: connect: connection refused
2020-11-02T19:05:10.754841936Z E1102 19:05:10.730204       1 proxier.go:1546] Failed to execute iptables-restore: exit status 4 (iptables-restore v1.8.4 (nf_tables):
2020-11-02T19:05:10.754841936Z line 385: CHAIN_USER_DEL failed (Device or resource busy): chain KUBE-SEP-O7WLIJMTHFXGXEZH
2020-11-02T19:05:10.754841936Z line 386: CHAIN_USER_DEL failed (Device or resource busy): chain KUBE-SEP-WSVRAU2MCJJEW33R

Initially this looked like this bughttps://bugzilla.redhat.com/show_bug.cgi?id=1880680 , however after trying implement the workaround there was no change in the behavior. 

I have several must-gathers that I can put into a google drive and share. 


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. create cluster with calico sdn
2. switch calico from ipip bgp to native bgp


Actual results:

operators degrade commands that use the openshift-apiserver(such as oc get routes) fail for 20 minutes, then come back up 

Expected results:
minimal disruption 
Additional info: I think what the real question here is what is recovering here. After my analysis I have not seen anything that restarts.
Comment 3 Dan Winship 2021-01-11 15:35:58 UTC 
> 2020-11-02T19:05:10.754841936Z line 385: CHAIN_USER_DEL failed (Device or resource busy): chain KUBE-SEP-O7WLIJMTHFXGXEZH
> 2020-11-02T19:05:10.754841936Z line 386: CHAIN_USER_DEL failed (Device or resource busy): chain KUBE-SEP-WSVRAU2MCJJEW33R

> Initially this looked like this bug https://bugzilla.redhat.com/show_bug.cgi?id=1880680 , however after trying implement the workaround there was no change in the behavior. 

What workaround? bz 1880680 should be fixed in current version of OCP. What version of 4.5 are you on?



But anyway, this seems like a Calico problem, which should be investigated with Tigera, not something RH supports directly...
Comment 4 Dan Williams 2021-01-11 20:10:17 UTC 
CHAIN_USER_DEL fixes to iptables landed in 4.5.21 and 4.6.4 FWIW
Comment 5 Pooriya Aghaalitari 2021-01-12 20:18:07 UTC 
Hi Dan,

We have looked at this and the network seems to be up and running without issue. Our investigation shows that the services that extend the kubernetes API are the service that are suffering.
Comment 6 Dan Winship 2021-01-13 12:37:02 UTC 
Are the kube-proxy rules correct then? Does the kubernetes.default service IP work?
Comment 7 Pooriya Aghaalitari 2021-01-14 00:17:11 UTC 
Please see the output below. Following are the cluster operators and apiservices that are impacted.


NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.5.25    True        False         True       29m
cloud-credential                           4.5.25    True        False         False      95m
cluster-autoscaler                         4.5.25    True        False         False      33m
config-operator                            4.5.25    True        False         False      33m
console                                    4.5.25    True        False         True       31m
csi-snapshot-controller                    4.5.25    True        False         False      33m
dns                                        4.5.25    True        False         False      48m
etcd                                       4.5.25    True        False         False      62m
image-registry                             4.5.25    True        False         False      51m
ingress                                    4.5.25    True        False         False      43m
insights                                   4.5.25    True        False         False      51m
kube-apiserver                             4.5.25    True        False         False      59m
kube-controller-manager                    4.5.25    True        False         False      59m
kube-scheduler                             4.5.25    True        False         False      59m
kube-storage-version-migrator              4.5.25    True        False         False      44m
machine-api                                4.5.25    True        False         False      49m
machine-approver                           4.5.25    True        False         False      52m
machine-config                             4.5.25    True        False         False      47m
marketplace                                4.5.25    True        False         False      48m
monitoring                                 4.5.25    False       False         True       5m43s
network                                    4.5.25    True        False         False      63m
node-tuning                                4.5.25    True        False         False      60m
openshift-apiserver                        4.5.25    False       False         False      8m2s
openshift-controller-manager               4.5.25    True        False         False      49m
openshift-samples                          4.5.25    True        False         False      32m
operator-lifecycle-manager                 4.5.25    True        False         False      54m
operator-lifecycle-manager-catalog         4.5.25    True        False         False      54m
operator-lifecycle-manager-packageserver   4.5.25    False       True          False      8m29s
service-ca                                 4.5.25    True        False         False      64m
storage                                    4.5.25    True        False         False      52m




NAME                                      SERVICE                                                      AVAILABLE                      AGE
v1.                                       Local                                                        True                           90m
v1.admissionregistration.k8s.io           Local                                                        True                           90m
v1.apiextensions.k8s.io                   Local                                                        True                           90m
v1.apm.k8s.elastic.co                     Local                                                        True                           28m
v1.apps                                   Local                                                        True                           90m
v1.apps.openshift.io                      openshift-apiserver/api                                      False (FailedDiscoveryCheck)   51m
v1.authentication.k8s.io                  Local                                                        True                           90m
v1.authorization.k8s.io                   Local                                                        True                           90m
v1.authorization.openshift.io             openshift-apiserver/api                                      False (FailedDiscoveryCheck)   51m
v1.autoscaling                            Local                                                        True                           90m
v1.autoscaling.openshift.io               Local                                                        True                           25m
v1.batch                                  Local                                                        True                           90m
v1.build.openshift.io                     openshift-apiserver/api                                      False (FailedDiscoveryCheck)   51m
v1.cloudcredential.openshift.io           Local                                                        True                           21m
v1.config.openshift.io                    Local                                                        True                           21m
v1.console.openshift.io                   Local                                                        True                           25m
v1.coordination.k8s.io                    Local                                                        True                           90m
v1.crd.projectcalico.org                  Local                                                        True                           21m
v1.elasticsearch.k8s.elastic.co           Local                                                        True                           28m
v1.image.openshift.io                     openshift-apiserver/api                                      False (FailedDiscoveryCheck)   51m
v1.imageregistry.operator.openshift.io    Local                                                        True                           21m
v1.ingress.operator.openshift.io          Local                                                        True                           28m
v1.k8s.cni.cncf.io                        Local                                                        True                           28m
v1.kibana.k8s.elastic.co                  Local                                                        True                           28m
v1.machineconfiguration.openshift.io      Local                                                        True                           21m
v1.monitoring.coreos.com                  Local                                                        True                           21m
v1.network.operator.openshift.io          Local                                                        True                           21m
v1.networking.k8s.io                      Local                                                        True                           90m
v1.oauth.openshift.io                     openshift-apiserver/api                                      False (FailedDiscoveryCheck)   51m
v1.operator.openshift.io                  Local                                                        True                           37m
v1.operator.tigera.io                     Local                                                        True                           28m
v1.operators.coreos.com                   Local                                                        True                           21m
v1.packages.operators.coreos.com          openshift-operator-lifecycle-manager/packageserver-service   False (FailedDiscoveryCheck)   48m
v1.project.openshift.io                   openshift-apiserver/api                                      False (FailedDiscoveryCheck)   51m
v1.quota.openshift.io                     openshift-apiserver/api                                      False (FailedDiscoveryCheck)   51m
v1.rbac.authorization.k8s.io              Local                                                        True                           90m
v1.route.openshift.io                     openshift-apiserver/api                                      False (FailedDiscoveryCheck)   51m
v1.samples.operator.openshift.io          Local                                                        True                           21m
v1.scheduling.k8s.io                      Local                                                        True                           90m
v1.security.openshift.io                  openshift-apiserver/api                                      False (FailedDiscoveryCheck)   51m
v1.storage.k8s.io                         Local                                                        True                           90m
v1.template.openshift.io                  openshift-apiserver/api                                      False (FailedDiscoveryCheck)   51m
v1.tuned.openshift.io                     Local                                                        True                           28m
v1.user.openshift.io                      openshift-apiserver/api                                      False (FailedDiscoveryCheck)   51m
v1alpha1.elasticsearch.k8s.elastic.co     Local                                                        True                           28m
v1alpha1.flowcontrol.apiserver.k8s.io     Local                                                        True                           90m
v1alpha1.metal3.io                        Local                                                        True                           89m
v1alpha1.migration.k8s.io                 Local                                                        True                           21m
v1alpha1.operator.openshift.io            Local                                                        True                           21m
v1alpha1.operators.coreos.com             Local                                                        True                           36m
v1alpha1.whereabouts.cni.cncf.io          Local                                                        True                           25m
v1alpha2.operators.coreos.com             Local                                                        True                           28m
v1beta1.admissionregistration.k8s.io      Local                                                        True                           90m
v1beta1.apiextensions.k8s.io              Local                                                        True                           90m
v1beta1.apm.k8s.elastic.co                Local                                                        True                           36m
v1beta1.authentication.k8s.io             Local                                                        True                           90m
v1beta1.authorization.k8s.io              Local                                                        True                           90m
v1beta1.autoscaling.openshift.io          Local                                                        True                           25m
v1beta1.batch                             Local                                                        True                           90m
v1beta1.beat.k8s.elastic.co               Local                                                        True                           21m
v1beta1.certificates.k8s.io               Local                                                        True                           90m
v1beta1.coordination.k8s.io               Local                                                        True                           90m
v1beta1.discovery.k8s.io                  Local                                                        True                           90m
v1beta1.elasticsearch.k8s.elastic.co      Local                                                        True                           32m
v1beta1.enterprisesearch.k8s.elastic.co   Local                                                        True                           21m
v1beta1.events.k8s.io                     Local                                                        True                           90m
v1beta1.extensions                        Local                                                        True                           90m
v1beta1.kibana.k8s.elastic.co             Local                                                        True                           28m
v1beta1.machine.openshift.io              Local                                                        True                           28m
v1beta1.metrics.k8s.io                    openshift-monitoring/prometheus-adapter                      False (FailedDiscoveryCheck)   37m
v1beta1.networking.k8s.io                 Local                                                        True                           90m
v1beta1.node.k8s.io                       Local                                                        True                           90m
v1beta1.policy                            Local                                                        True                           90m
v1beta1.rbac.authorization.k8s.io         Local                                                        True                           90m
v1beta1.scheduling.k8s.io                 Local                                                        True                           90m
v1beta1.snapshot.storage.k8s.io           Local                                                        True                           21m
v1beta1.storage.k8s.io                    Local                                                        True                           90m
v2beta1.autoscaling                       Local                                                        True                           90m
v2beta2.autoscaling                       Local                                                        True                           90m
v3.projectcalico.org                      tigera-system/tigera-api                                     False (FailedDiscoveryCheck)   52m




Kube-proxy rules before I switch from IPIP to Native BGP.


Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.0.1           /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  anywhere             172.30.0.1           /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.101.66        /* openshift-operator-lifecycle-manager/catalog-operator-metrics:https-metrics cluster IP */ tcp dpt:tproxy
KUBE-SVC-A2G2ICINC4ZVGP64  tcp  --  anywhere             172.30.101.66        /* openshift-operator-lifecycle-manager/catalog-operator-metrics:https-metrics cluster IP */ tcp dpt:tproxy
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.0.10          /* openshift-dns/dns-default:dns-tcp cluster IP */ tcp dpt:domain
KUBE-SVC-6BRQXW4I6ZZ3LHZH  tcp  --  anywhere             172.30.0.10          /* openshift-dns/dns-default:dns-tcp cluster IP */ tcp dpt:domain
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.135.246       /* openshift-marketplace/redhat-marketplace:grpc cluster IP */ tcp dpt:50051
KUBE-SVC-UO3GDY73GKWXARGX  tcp  --  anywhere             172.30.135.246       /* openshift-marketplace/redhat-marketplace:grpc cluster IP */ tcp dpt:50051
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.229.29        /* tigera-system/tigera-api:queryserver cluster IP */ tcp dpt:webcache
KUBE-SVC-BXX6NV5PBDEKW23Y  tcp  --  anywhere             172.30.229.29        /* tigera-system/tigera-api:queryserver cluster IP */ tcp dpt:webcache
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.99.200        /* openshift-monitoring/prometheus-k8s:tenancy cluster IP */ tcp dpt:XmlIpcRegSvc
KUBE-SVC-W3K2PRZPP3KE4WYD  tcp  --  anywhere             172.30.99.200        /* openshift-monitoring/prometheus-k8s:tenancy cluster IP */ tcp dpt:XmlIpcRegSvc
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.22.195        /* openshift-machine-api/cluster-autoscaler-operator:metrics cluster IP */ tcp dpt:9192
KUBE-SVC-GGV3SPGNRULALRSD  tcp  --  anywhere             172.30.22.195        /* openshift-machine-api/cluster-autoscaler-operator:metrics cluster IP */ tcp dpt:9192
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.123.13        /* openshift-etcd/etcd:etcd-metrics cluster IP */ tcp dpt:9979
KUBE-SVC-Z7PD6XV52AKYPMA5  tcp  --  anywhere             172.30.123.13        /* openshift-etcd/etcd:etcd-metrics cluster IP */ tcp dpt:9979
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.161.21        /* openshift-kube-controller-manager/kube-controller-manager:https cluster IP */ tcp dpt:https
KUBE-SVC-VQFT5ZCKL2KRMQ3Q  tcp  --  anywhere             172.30.161.21        /* openshift-kube-controller-manager/kube-controller-manager:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.125.24        /* openshift-kube-apiserver/apiserver:https cluster IP */ tcp dpt:https
KUBE-SVC-X7YGTN7QRQI2VNWZ  tcp  --  anywhere             172.30.125.24        /* openshift-kube-apiserver/apiserver:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.207.12        /* openshift-kube-apiserver-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-KHZTXOIJSDOQRG4A  tcp  --  anywhere             172.30.207.12        /* openshift-kube-apiserver-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.22.68         /* openshift-machine-config-operator/machine-config-daemon:metrics cluster IP */ tcp dpt:etlservicemgr
KUBE-SVC-PFY2VR2AT5VQM74G  tcp  --  anywhere             172.30.22.68         /* openshift-machine-config-operator/machine-config-daemon:metrics cluster IP */ tcp dpt:etlservicemgr
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.244.83        /* openshift-ingress-operator/metrics:metrics cluster IP */ tcp dpt:9393
KUBE-SVC-DZZGCZT3USY56SM6  tcp  --  anywhere             172.30.244.83        /* openshift-ingress-operator/metrics:metrics cluster IP */ tcp dpt:9393
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.141.172       /* openshift-console/downloads:http cluster IP */ tcp dpt:http
KUBE-SVC-FPN24U5GX5G2TPXH  tcp  --  anywhere             172.30.141.172       /* openshift-console/downloads:http cluster IP */ tcp dpt:http
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.242.69        /* calico-system/calico-typha:calico-typha cluster IP */ tcp dpt:absolab-tags
KUBE-SVC-RK657RLKDNVNU64O  tcp  --  anywhere             172.30.242.69        /* calico-system/calico-typha:calico-typha cluster IP */ tcp dpt:absolab-tags
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.235.238       /* openshift-ingress/router-internal-default:http cluster IP */ tcp dpt:http
KUBE-SVC-U3LVBEEPLKGG5GBK  tcp  --  anywhere             172.30.235.238       /* openshift-ingress/router-internal-default:http cluster IP */ tcp dpt:http
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.224.190       /* openshift-dns-operator/metrics:metrics cluster IP */ tcp dpt:9393
KUBE-SVC-2TW25BGER7T666BH  tcp  --  anywhere             172.30.224.190       /* openshift-dns-operator/metrics:metrics cluster IP */ tcp dpt:9393
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.67.137        /* openshift-monitoring/prometheus-adapter:https cluster IP */ tcp dpt:https
KUBE-SVC-GDUOWZ6AYLOEFLKA  tcp  --  anywhere             172.30.67.137        /* openshift-monitoring/prometheus-adapter:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.3.84          /* openshift-etcd-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-AV3LJ2I3TMKQAKOJ  tcp  --  anywhere             172.30.3.84          /* openshift-etcd-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.173.140       /* openshift-console/console:https cluster IP */ tcp dpt:https
KUBE-SVC-2O3SXCDVWWS7KYC5  tcp  --  anywhere             172.30.173.140       /* openshift-console/console:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.66.55         /* openshift-monitoring/alertmanager-main:web cluster IP */ tcp dpt:9094
KUBE-SVC-WHIODLEQRXTXJ6C7  tcp  --  anywhere             172.30.66.55         /* openshift-monitoring/alertmanager-main:web cluster IP */ tcp dpt:9094
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.177.155       /* openshift-authentication/oauth-openshift:https cluster IP */ tcp dpt:https
KUBE-SVC-DK4IP773FHBZHRYV  tcp  --  anywhere             172.30.177.155       /* openshift-authentication/oauth-openshift:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.22.195        /* openshift-machine-api/cluster-autoscaler-operator:https cluster IP */ tcp dpt:https
KUBE-SVC-H7AEPRVAHANZXX45  tcp  --  anywhere             172.30.22.195        /* openshift-machine-api/cluster-autoscaler-operator:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.39.154        /* openshift-authentication-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-FWPMMI34GVXXB7IX  tcp  --  anywhere             172.30.39.154        /* openshift-authentication-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.147.109       /* calico-system/calico-node-metrics:calico-metrics-port cluster IP */ tcp dpt:9081
KUBE-SVC-BPJNZGPODTH4UZQI  tcp  --  anywhere             172.30.147.109       /* calico-system/calico-node-metrics:calico-metrics-port cluster IP */ tcp dpt:9081
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.67.222        /* openshift-controller-manager-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-DYEHYI43W4Y6JVSZ  tcp  --  anywhere             172.30.67.222        /* openshift-controller-manager-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.55.28         /* openshift-kube-controller-manager-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-BCVO45GDJF63HKMI  tcp  --  anywhere             172.30.55.28         /* openshift-kube-controller-manager-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.78.95         /* openshift-cluster-storage-operator/csi-snapshot-controller-operator-metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-VBEBQDAER3JW7JUB  tcp  --  anywhere             172.30.78.95         /* openshift-cluster-storage-operator/csi-snapshot-controller-operator-metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.181.186       /* openshift-operator-lifecycle-manager/packageserver-service:5443 cluster IP */ tcp dpt:spss
KUBE-SVC-BOLNPNOKMMIDOV7N  tcp  --  anywhere             172.30.181.186       /* openshift-operator-lifecycle-manager/packageserver-service:5443 cluster IP */ tcp dpt:spss
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.123.13        /* openshift-etcd/etcd:etcd cluster IP */ tcp dpt:etcd-client
KUBE-SVC-7CKPKLVT4G7W7WIT  tcp  --  anywhere             172.30.123.13        /* openshift-etcd/etcd:etcd cluster IP */ tcp dpt:etcd-client
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.40.118        /* openshift-monitoring/grafana:https cluster IP */ tcp dpt:hbci
KUBE-SVC-RD6ZTFGQGXUEWIFM  tcp  --  anywhere             172.30.40.118        /* openshift-monitoring/grafana:https cluster IP */ tcp dpt:hbci
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.188.25        /* openshift-monitoring/thanos-querier:tenancy cluster IP */ tcp dpt:XmlIpcRegSvc
KUBE-SVC-C4CT6K4SQFWI5WLJ  tcp  --  anywhere             172.30.188.25        /* openshift-monitoring/thanos-querier:tenancy cluster IP */ tcp dpt:XmlIpcRegSvc
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.58.237        /* openshift-console-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-6RVLNWC5AKEV5WJD  tcp  --  anywhere             172.30.58.237        /* openshift-console-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.16.101        /* openshift-insights/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-FO4YVUCBKKQXTXB6  tcp  --  anywhere             172.30.16.101        /* openshift-insights/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.64.229        /* openshift-multus/multus-admission-controller:webhook cluster IP */ tcp dpt:https
KUBE-SVC-A3VVZ52UMEGJJFHI  tcp  --  anywhere             172.30.64.229        /* openshift-multus/multus-admission-controller:webhook cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.48.85         /* openshift-config-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-XAHL2OVG46O6QFL7  tcp  --  anywhere             172.30.48.85         /* openshift-config-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.147.109       /* calico-system/calico-node-metrics:calico-bgp-metrics-port cluster IP */ tcp dpt:iua
KUBE-SVC-ZMPNACNGKBKCFXCW  tcp  --  anywhere             172.30.147.109       /* calico-system/calico-node-metrics:calico-bgp-metrics-port cluster IP */ tcp dpt:iua
KUBE-MARK-MASQ  udp  -- !10.128.0.0/14        172.30.0.10          /* openshift-dns/dns-default:dns cluster IP */ udp dpt:domain
KUBE-SVC-BGNS3J6UB7MMLVDO  udp  --  anywhere             172.30.0.10          /* openshift-dns/dns-default:dns cluster IP */ udp dpt:domain
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.216.13        /* openshift-kube-storage-version-migrator-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-D5VYWAE3NWJS4H36  tcp  --  anywhere             172.30.216.13        /* openshift-kube-storage-version-migrator-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.2.157         /* openshift-cloud-credential-operator/controller-manager-service: cluster IP */ tcp dpt:https
KUBE-SVC-LS7JF6SL4ODP2YA4  tcp  --  anywhere             172.30.2.157         /* openshift-cloud-credential-operator/controller-manager-service: cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.147.130       /* openshift-controller-manager/controller-manager:https cluster IP */ tcp dpt:https
KUBE-SVC-ZU5C2KTEVGGF4RWY  tcp  --  anywhere             172.30.147.130       /* openshift-controller-manager/controller-manager:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.132.219       /* openshift-apiserver/api:https cluster IP */ tcp dpt:https
KUBE-SVC-NM6OF7LZYCSWPYSN  tcp  --  anywhere             172.30.132.219       /* openshift-apiserver/api:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.235.238       /* openshift-ingress/router-internal-default:https cluster IP */ tcp dpt:https
KUBE-SVC-PIUKAOOLWSYDMVAC  tcp  --  anywhere             172.30.235.238       /* openshift-ingress/router-internal-default:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.99.200        /* openshift-monitoring/prometheus-k8s:web cluster IP */ tcp dpt:xmltec-xmlmail
KUBE-SVC-DCLNKYLNAMROIJRV  tcp  --  anywhere             172.30.99.200        /* openshift-monitoring/prometheus-k8s:web cluster IP */ tcp dpt:xmltec-xmlmail
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.64.229        /* openshift-multus/multus-admission-controller:metrics cluster IP */ tcp dpt:pcsync-https
KUBE-SVC-HWYXEEIGDEK65VFZ  tcp  --  anywhere             172.30.64.229        /* openshift-multus/multus-admission-controller:metrics cluster IP */ tcp dpt:pcsync-https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.217.176       /* openshift-machine-api/machine-api-operator:https cluster IP */ tcp dpt:pcsync-https
KUBE-SVC-UIDONVFEB6LPHORF  tcp  --  anywhere             172.30.217.176       /* openshift-machine-api/machine-api-operator:https cluster IP */ tcp dpt:pcsync-https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.229.29        /* tigera-system/tigera-api:apiserver cluster IP */ tcp dpt:https
KUBE-SVC-5YT3S4Q5ZQB7MXPI  tcp  --  anywhere             172.30.229.29        /* tigera-system/tigera-api:apiserver cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.235.238       /* openshift-ingress/router-internal-default:metrics cluster IP */ tcp dpt:jetcmeserver
KUBE-SVC-LMGCLHC2KUY6NS4N  tcp  --  anywhere             172.30.235.238       /* openshift-ingress/router-internal-default:metrics cluster IP */ tcp dpt:jetcmeserver
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.34.0          /* openshift-cloud-credential-operator/cco-metrics:cco-metrics cluster IP */ tcp dpt:idonix-metanet
KUBE-SVC-SSFS4UJOKJYBUN2S  tcp  --  anywhere             172.30.34.0          /* openshift-cloud-credential-operator/cco-metrics:cco-metrics cluster IP */ tcp dpt:idonix-metanet
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.0.10          /* openshift-dns/dns-default:metrics cluster IP */ tcp dpt:9154
KUBE-SVC-P2RWE722QPZ5K3VW  tcp  --  anywhere             172.30.0.10          /* openshift-dns/dns-default:metrics cluster IP */ tcp dpt:9154
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.241.232       /* openshift-service-ca-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-Z26MZGDJSJICLYJU  tcp  --  anywhere             172.30.241.232       /* openshift-service-ca-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.175.93        /* openshift-kube-scheduler-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-HH47JV2DWEPNMQEX  tcp  --  anywhere             172.30.175.93        /* openshift-kube-scheduler-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.74.247        /* openshift-operator-lifecycle-manager/olm-operator-metrics:https-metrics cluster IP */ tcp dpt:tproxy
KUBE-SVC-5IJVCVIN67YXVDZB  tcp  --  anywhere             172.30.74.247        /* openshift-operator-lifecycle-manager/olm-operator-metrics:https-metrics cluster IP */ tcp dpt:tproxy
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.130.49        /* openshift-marketplace/redhat-operators:grpc cluster IP */ tcp dpt:50051
KUBE-SVC-SGDZNVXMHJCPEAE2  tcp  --  anywhere             172.30.130.49        /* openshift-marketplace/redhat-operators:grpc cluster IP */ tcp dpt:50051
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.222.148       /* openshift-cluster-version/cluster-version-operator:metrics cluster IP */ tcp dpt:9099
KUBE-SVC-LR44LCGLBA5H46DK  tcp  --  anywhere             172.30.222.148       /* openshift-cluster-version/cluster-version-operator:metrics cluster IP */ tcp dpt:9099
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.195.94        /* openshift-marketplace/marketplace-operator-metrics:https-metrics cluster IP */ tcp dpt:tproxy
KUBE-SVC-TSFFZBTPSVTKQCXM  tcp  --  anywhere             172.30.195.94        /* openshift-marketplace/marketplace-operator-metrics:https-metrics cluster IP */ tcp dpt:tproxy
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.228.190       /* openshift-kube-scheduler/scheduler:https cluster IP */ tcp dpt:https
KUBE-SVC-OGQPOTBHHZMRDA43  tcp  --  anywhere             172.30.228.190       /* openshift-kube-scheduler/scheduler:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.66.55         /* openshift-monitoring/alertmanager-main:tenancy cluster IP */ tcp dpt:XmlIpcRegSvc
KUBE-SVC-YXEMPCT6EJQEIJNP  tcp  --  anywhere             172.30.66.55         /* openshift-monitoring/alertmanager-main:tenancy cluster IP */ tcp dpt:XmlIpcRegSvc
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.145.83        /* openshift-apiserver-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-CIUYVLZDADCHPTYT  tcp  --  anywhere             172.30.145.83        /* openshift-apiserver-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.195.94        /* openshift-marketplace/marketplace-operator-metrics:metrics cluster IP */ tcp dpt:m2mservices
KUBE-SVC-LG3WZOYAKHCJ6X6O  tcp  --  anywhere             172.30.195.94        /* openshift-marketplace/marketplace-operator-metrics:metrics cluster IP */ tcp dpt:m2mservices
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.188.25        /* openshift-monitoring/thanos-querier:web cluster IP */ tcp dpt:xmltec-xmlmail
KUBE-SVC-G5A7ID5ATXHWKRS5  tcp  --  anywhere             172.30.188.25        /* openshift-monitoring/thanos-querier:web cluster IP */ tcp dpt:xmltec-xmlmail
KUBE-NODEPORTS  all  --  anywhere             anywhere             /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL




Kube-proxy rules after I switch from IPIP to Native BGP.

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.123.13        /* openshift-etcd/etcd:etcd cluster IP */ tcp dpt:etcd-client
KUBE-SVC-7CKPKLVT4G7W7WIT  tcp  --  anywhere             172.30.123.13        /* openshift-etcd/etcd:etcd cluster IP */ tcp dpt:etcd-client
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.40.118        /* openshift-monitoring/grafana:https cluster IP */ tcp dpt:hbci
KUBE-SVC-RD6ZTFGQGXUEWIFM  tcp  --  anywhere             172.30.40.118        /* openshift-monitoring/grafana:https cluster IP */ tcp dpt:hbci
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.188.25        /* openshift-monitoring/thanos-querier:tenancy cluster IP */ tcp dpt:XmlIpcRegSvc
KUBE-SVC-C4CT6K4SQFWI5WLJ  tcp  --  anywhere             172.30.188.25        /* openshift-monitoring/thanos-querier:tenancy cluster IP */ tcp dpt:XmlIpcRegSvc
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.58.237        /* openshift-console-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-6RVLNWC5AKEV5WJD  tcp  --  anywhere             172.30.58.237        /* openshift-console-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.16.101        /* openshift-insights/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-FO4YVUCBKKQXTXB6  tcp  --  anywhere             172.30.16.101        /* openshift-insights/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.64.229        /* openshift-multus/multus-admission-controller:webhook cluster IP */ tcp dpt:https
KUBE-SVC-A3VVZ52UMEGJJFHI  tcp  --  anywhere             172.30.64.229        /* openshift-multus/multus-admission-controller:webhook cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.48.85         /* openshift-config-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-XAHL2OVG46O6QFL7  tcp  --  anywhere             172.30.48.85         /* openshift-config-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.147.109       /* calico-system/calico-node-metrics:calico-bgp-metrics-port cluster IP */ tcp dpt:iua
KUBE-SVC-ZMPNACNGKBKCFXCW  tcp  --  anywhere             172.30.147.109       /* calico-system/calico-node-metrics:calico-bgp-metrics-port cluster IP */ tcp dpt:iua
KUBE-MARK-MASQ  udp  -- !10.128.0.0/14        172.30.0.10          /* openshift-dns/dns-default:dns cluster IP */ udp dpt:domain
KUBE-SVC-BGNS3J6UB7MMLVDO  udp  --  anywhere             172.30.0.10          /* openshift-dns/dns-default:dns cluster IP */ udp dpt:domain
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.216.13        /* openshift-kube-storage-version-migrator-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-D5VYWAE3NWJS4H36  tcp  --  anywhere             172.30.216.13        /* openshift-kube-storage-version-migrator-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.2.157         /* openshift-cloud-credential-operator/controller-manager-service: cluster IP */ tcp dpt:https
KUBE-SVC-LS7JF6SL4ODP2YA4  tcp  --  anywhere             172.30.2.157         /* openshift-cloud-credential-operator/controller-manager-service: cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.147.130       /* openshift-controller-manager/controller-manager:https cluster IP */ tcp dpt:https
KUBE-SVC-ZU5C2KTEVGGF4RWY  tcp  --  anywhere             172.30.147.130       /* openshift-controller-manager/controller-manager:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.132.219       /* openshift-apiserver/api:https cluster IP */ tcp dpt:https
KUBE-SVC-NM6OF7LZYCSWPYSN  tcp  --  anywhere             172.30.132.219       /* openshift-apiserver/api:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.235.238       /* openshift-ingress/router-internal-default:https cluster IP */ tcp dpt:https
KUBE-SVC-PIUKAOOLWSYDMVAC  tcp  --  anywhere             172.30.235.238       /* openshift-ingress/router-internal-default:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.99.200        /* openshift-monitoring/prometheus-k8s:web cluster IP */ tcp dpt:xmltec-xmlmail
KUBE-SVC-DCLNKYLNAMROIJRV  tcp  --  anywhere             172.30.99.200        /* openshift-monitoring/prometheus-k8s:web cluster IP */ tcp dpt:xmltec-xmlmail
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.64.229        /* openshift-multus/multus-admission-controller:metrics cluster IP */ tcp dpt:pcsync-https
KUBE-SVC-HWYXEEIGDEK65VFZ  tcp  --  anywhere             172.30.64.229        /* openshift-multus/multus-admission-controller:metrics cluster IP */ tcp dpt:pcsync-https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.217.176       /* openshift-machine-api/machine-api-operator:https cluster IP */ tcp dpt:pcsync-https
KUBE-SVC-UIDONVFEB6LPHORF  tcp  --  anywhere             172.30.217.176       /* openshift-machine-api/machine-api-operator:https cluster IP */ tcp dpt:pcsync-https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.229.29        /* tigera-system/tigera-api:apiserver cluster IP */ tcp dpt:https
KUBE-SVC-5YT3S4Q5ZQB7MXPI  tcp  --  anywhere             172.30.229.29        /* tigera-system/tigera-api:apiserver cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.235.238       /* openshift-ingress/router-internal-default:metrics cluster IP */ tcp dpt:jetcmeserver
KUBE-SVC-LMGCLHC2KUY6NS4N  tcp  --  anywhere             172.30.235.238       /* openshift-ingress/router-internal-default:metrics cluster IP */ tcp dpt:jetcmeserver
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.34.0          /* openshift-cloud-credential-operator/cco-metrics:cco-metrics cluster IP */ tcp dpt:idonix-metanet
KUBE-SVC-SSFS4UJOKJYBUN2S  tcp  --  anywhere             172.30.34.0          /* openshift-cloud-credential-operator/cco-metrics:cco-metrics cluster IP */ tcp dpt:idonix-metanet
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.0.10          /* openshift-dns/dns-default:metrics cluster IP */ tcp dpt:9154
KUBE-SVC-P2RWE722QPZ5K3VW  tcp  --  anywhere             172.30.0.10          /* openshift-dns/dns-default:metrics cluster IP */ tcp dpt:9154
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.241.232       /* openshift-service-ca-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-Z26MZGDJSJICLYJU  tcp  --  anywhere             172.30.241.232       /* openshift-service-ca-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.175.93        /* openshift-kube-scheduler-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-HH47JV2DWEPNMQEX  tcp  --  anywhere             172.30.175.93        /* openshift-kube-scheduler-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.74.247        /* openshift-operator-lifecycle-manager/olm-operator-metrics:https-metrics cluster IP */ tcp dpt:tproxy
KUBE-SVC-5IJVCVIN67YXVDZB  tcp  --  anywhere             172.30.74.247        /* openshift-operator-lifecycle-manager/olm-operator-metrics:https-metrics cluster IP */ tcp dpt:tproxy
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.130.49        /* openshift-marketplace/redhat-operators:grpc cluster IP */ tcp dpt:50051
KUBE-SVC-SGDZNVXMHJCPEAE2  tcp  --  anywhere             172.30.130.49        /* openshift-marketplace/redhat-operators:grpc cluster IP */ tcp dpt:50051
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.222.148       /* openshift-cluster-version/cluster-version-operator:metrics cluster IP */ tcp dpt:9099
KUBE-SVC-LR44LCGLBA5H46DK  tcp  --  anywhere             172.30.222.148       /* openshift-cluster-version/cluster-version-operator:metrics cluster IP */ tcp dpt:9099
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.195.94        /* openshift-marketplace/marketplace-operator-metrics:https-metrics cluster IP */ tcp dpt:tproxy
KUBE-SVC-TSFFZBTPSVTKQCXM  tcp  --  anywhere             172.30.195.94        /* openshift-marketplace/marketplace-operator-metrics:https-metrics cluster IP */ tcp dpt:tproxy
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.228.190       /* openshift-kube-scheduler/scheduler:https cluster IP */ tcp dpt:https
KUBE-SVC-OGQPOTBHHZMRDA43  tcp  --  anywhere             172.30.228.190       /* openshift-kube-scheduler/scheduler:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.66.55         /* openshift-monitoring/alertmanager-main:tenancy cluster IP */ tcp dpt:XmlIpcRegSvc
KUBE-SVC-YXEMPCT6EJQEIJNP  tcp  --  anywhere             172.30.66.55         /* openshift-monitoring/alertmanager-main:tenancy cluster IP */ tcp dpt:XmlIpcRegSvc
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.145.83        /* openshift-apiserver-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-CIUYVLZDADCHPTYT  tcp  --  anywhere             172.30.145.83        /* openshift-apiserver-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.195.94        /* openshift-marketplace/marketplace-operator-metrics:metrics cluster IP */ tcp dpt:m2mservices
KUBE-SVC-LG3WZOYAKHCJ6X6O  tcp  --  anywhere             172.30.195.94        /* openshift-marketplace/marketplace-operator-metrics:metrics cluster IP */ tcp dpt:m2mservices
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.188.25        /* openshift-monitoring/thanos-querier:web cluster IP */ tcp dpt:xmltec-xmlmail
KUBE-SVC-G5A7ID5ATXHWKRS5  tcp  --  anywhere             172.30.188.25        /* openshift-monitoring/thanos-querier:web cluster IP */ tcp dpt:xmltec-xmlmail
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.0.1           /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  anywhere             172.30.0.1           /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.101.66        /* openshift-operator-lifecycle-manager/catalog-operator-metrics:https-metrics cluster IP */ tcp dpt:tproxy
KUBE-SVC-A2G2ICINC4ZVGP64  tcp  --  anywhere             172.30.101.66        /* openshift-operator-lifecycle-manager/catalog-operator-metrics:https-metrics cluster IP */ tcp dpt:tproxy
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.0.10          /* openshift-dns/dns-default:dns-tcp cluster IP */ tcp dpt:domain
KUBE-SVC-6BRQXW4I6ZZ3LHZH  tcp  --  anywhere             172.30.0.10          /* openshift-dns/dns-default:dns-tcp cluster IP */ tcp dpt:domain
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.135.246       /* openshift-marketplace/redhat-marketplace:grpc cluster IP */ tcp dpt:50051
KUBE-SVC-UO3GDY73GKWXARGX  tcp  --  anywhere             172.30.135.246       /* openshift-marketplace/redhat-marketplace:grpc cluster IP */ tcp dpt:50051
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.229.29        /* tigera-system/tigera-api:queryserver cluster IP */ tcp dpt:webcache
KUBE-SVC-BXX6NV5PBDEKW23Y  tcp  --  anywhere             172.30.229.29        /* tigera-system/tigera-api:queryserver cluster IP */ tcp dpt:webcache
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.99.200        /* openshift-monitoring/prometheus-k8s:tenancy cluster IP */ tcp dpt:XmlIpcRegSvc
KUBE-SVC-W3K2PRZPP3KE4WYD  tcp  --  anywhere             172.30.99.200        /* openshift-monitoring/prometheus-k8s:tenancy cluster IP */ tcp dpt:XmlIpcRegSvc
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.22.195        /* openshift-machine-api/cluster-autoscaler-operator:metrics cluster IP */ tcp dpt:9192
KUBE-SVC-GGV3SPGNRULALRSD  tcp  --  anywhere             172.30.22.195        /* openshift-machine-api/cluster-autoscaler-operator:metrics cluster IP */ tcp dpt:9192
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.123.13        /* openshift-etcd/etcd:etcd-metrics cluster IP */ tcp dpt:9979
KUBE-SVC-Z7PD6XV52AKYPMA5  tcp  --  anywhere             172.30.123.13        /* openshift-etcd/etcd:etcd-metrics cluster IP */ tcp dpt:9979
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.161.21        /* openshift-kube-controller-manager/kube-controller-manager:https cluster IP */ tcp dpt:https
KUBE-SVC-VQFT5ZCKL2KRMQ3Q  tcp  --  anywhere             172.30.161.21        /* openshift-kube-controller-manager/kube-controller-manager:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.125.24        /* openshift-kube-apiserver/apiserver:https cluster IP */ tcp dpt:https
KUBE-SVC-X7YGTN7QRQI2VNWZ  tcp  --  anywhere             172.30.125.24        /* openshift-kube-apiserver/apiserver:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.207.12        /* openshift-kube-apiserver-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-KHZTXOIJSDOQRG4A  tcp  --  anywhere             172.30.207.12        /* openshift-kube-apiserver-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.22.68         /* openshift-machine-config-operator/machine-config-daemon:metrics cluster IP */ tcp dpt:etlservicemgr
KUBE-SVC-PFY2VR2AT5VQM74G  tcp  --  anywhere             172.30.22.68         /* openshift-machine-config-operator/machine-config-daemon:metrics cluster IP */ tcp dpt:etlservicemgr
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.244.83        /* openshift-ingress-operator/metrics:metrics cluster IP */ tcp dpt:9393
KUBE-SVC-DZZGCZT3USY56SM6  tcp  --  anywhere             172.30.244.83        /* openshift-ingress-operator/metrics:metrics cluster IP */ tcp dpt:9393
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.141.172       /* openshift-console/downloads:http cluster IP */ tcp dpt:http
KUBE-SVC-FPN24U5GX5G2TPXH  tcp  --  anywhere             172.30.141.172       /* openshift-console/downloads:http cluster IP */ tcp dpt:http
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.242.69        /* calico-system/calico-typha:calico-typha cluster IP */ tcp dpt:absolab-tags
KUBE-SVC-RK657RLKDNVNU64O  tcp  --  anywhere             172.30.242.69        /* calico-system/calico-typha:calico-typha cluster IP */ tcp dpt:absolab-tags
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.235.238       /* openshift-ingress/router-internal-default:http cluster IP */ tcp dpt:http
KUBE-SVC-U3LVBEEPLKGG5GBK  tcp  --  anywhere             172.30.235.238       /* openshift-ingress/router-internal-default:http cluster IP */ tcp dpt:http
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.224.190       /* openshift-dns-operator/metrics:metrics cluster IP */ tcp dpt:9393
KUBE-SVC-2TW25BGER7T666BH  tcp  --  anywhere             172.30.224.190       /* openshift-dns-operator/metrics:metrics cluster IP */ tcp dpt:9393
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.67.137        /* openshift-monitoring/prometheus-adapter:https cluster IP */ tcp dpt:https
KUBE-SVC-GDUOWZ6AYLOEFLKA  tcp  --  anywhere             172.30.67.137        /* openshift-monitoring/prometheus-adapter:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.3.84          /* openshift-etcd-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-AV3LJ2I3TMKQAKOJ  tcp  --  anywhere             172.30.3.84          /* openshift-etcd-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.173.140       /* openshift-console/console:https cluster IP */ tcp dpt:https
KUBE-SVC-2O3SXCDVWWS7KYC5  tcp  --  anywhere             172.30.173.140       /* openshift-console/console:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.66.55         /* openshift-monitoring/alertmanager-main:web cluster IP */ tcp dpt:9094
KUBE-SVC-WHIODLEQRXTXJ6C7  tcp  --  anywhere             172.30.66.55         /* openshift-monitoring/alertmanager-main:web cluster IP */ tcp dpt:9094
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.177.155       /* openshift-authentication/oauth-openshift:https cluster IP */ tcp dpt:https
KUBE-SVC-DK4IP773FHBZHRYV  tcp  --  anywhere             172.30.177.155       /* openshift-authentication/oauth-openshift:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.22.195        /* openshift-machine-api/cluster-autoscaler-operator:https cluster IP */ tcp dpt:https
KUBE-SVC-H7AEPRVAHANZXX45  tcp  --  anywhere             172.30.22.195        /* openshift-machine-api/cluster-autoscaler-operator:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.39.154        /* openshift-authentication-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-FWPMMI34GVXXB7IX  tcp  --  anywhere             172.30.39.154        /* openshift-authentication-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.147.109       /* calico-system/calico-node-metrics:calico-metrics-port cluster IP */ tcp dpt:9081
KUBE-SVC-BPJNZGPODTH4UZQI  tcp  --  anywhere             172.30.147.109       /* calico-system/calico-node-metrics:calico-metrics-port cluster IP */ tcp dpt:9081
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.67.222        /* openshift-controller-manager-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-DYEHYI43W4Y6JVSZ  tcp  --  anywhere             172.30.67.222        /* openshift-controller-manager-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.55.28         /* openshift-kube-controller-manager-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-BCVO45GDJF63HKMI  tcp  --  anywhere             172.30.55.28         /* openshift-kube-controller-manager-operator/metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.78.95         /* openshift-cluster-storage-operator/csi-snapshot-controller-operator-metrics:https cluster IP */ tcp dpt:https
KUBE-SVC-VBEBQDAER3JW7JUB  tcp  --  anywhere             172.30.78.95         /* openshift-cluster-storage-operator/csi-snapshot-controller-operator-metrics:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  tcp  -- !10.128.0.0/14        172.30.155.173       /* openshift-operator-lifecycle-manager/packageserver-service:5443 cluster IP */ tcp dpt:spss
KUBE-SVC-BOLNPNOKMMIDOV7N  tcp  --  anywhere             172.30.155.173       /* openshift-operator-lifecycle-manager/packageserver-service:5443 cluster IP */ tcp dpt:spss
KUBE-NODEPORTS  all  --  anywhere             anywhere             /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Comment 8 Pooriya Aghaalitari 2021-01-14 00:19:36 UTC 
Please let me know if you need any other information. I am also available to look at this issue if there is a need.
Comment 10 Dan Winship 2021-01-18 16:47:37 UTC 
Those bits of iptables output don't really answer the question. You'd want to look at "iptables-save" to get the full picture and see if the iptables rules are as you would expect them to be after the config change. (ie, does 172.30.0.1 actually end up pointing to the correct destination IPs?) And probably other things. I'm not sure exactly what Calico configures in "ipip bgp" and "native bgp" modes, but make sure the IPs, routes, iptables rules, etc, are all as you would expect. The debugging above shows that some pods seem to not have the expected network connectivity. The question is why, and that's a Calico question, not an OCP question.

> 2020-11-02T19:05:10.754841936Z line 386: CHAIN_USER_DEL failed (Device or resource busy): chain KUBE-SEP-WSVRAU2MCJJEW33R
> 
> Initially this looked like this bughttps://bugzilla.redhat.com/show_bug.cgi?id=1880680 , however after trying implement the workaround there was no change in the behavior. 

At the time of those log messages (2020-11-02) they would have to have been running a version of OCP that didn't have the fix for 1880680. However, you mentioned 4.5.25 later, which _does_ have the fix. You should confirm that you are not still seeing the CHAIN_USER_DEL errors in 4.5.25.
Comment 11 Dan Winship 2021-01-18 17:04:10 UTC 
(In reply to Dan Winship from comment #10)
> > 2020-11-02T19:05:10.754841936Z line 386: CHAIN_USER_DEL failed (Device or resource busy): chain KUBE-SEP-WSVRAU2MCJJEW33R
> > 
> > Initially this looked like this bughttps://bugzilla.redhat.com/show_bug.cgi?id=1880680 , however after trying implement the workaround there was no change in the behavior. 
> 
> At the time of those log messages (2020-11-02) they would have to have been
> running a version of OCP that didn't have the fix for 1880680. However, you
> mentioned 4.5.25 later, which _does_ have the fix. You should confirm that
> you are not still seeing the CHAIN_USER_DEL errors in 4.5.25.

Ah, correction to that: the fix is in the iptables client, not in the kernel. So if Calico is shipping its own iptables binary in some container image, that binary needs to be updated to RHEL iptables-1.8.4-10.el8_2.4 or later, or upstream 1.8.6 or later to have the fix for this. ("nft: Fix for concurrent noflush restore calls")
Comment 12 Matthew Whitehead 2021-01-19 14:30:00 UTC 
Pooriya, action item for you: please run 'iptables --version' in the relevant Calico containers and post the version in this bugzilla.
Comment 13 Pooriya Aghaalitari 2021-01-22 20:15:45 UTC 
I will get this done over the weekend and post the output.
Comment 14 David Haupt 2021-01-25 22:17:07 UTC 
Hi

Sorry, I am not 100% up to speed here, but here is the output from Calico Enterprise v3.4.1:

$ oc exec -it -n calico-system calico-node-dbt7j -- iptables --version
iptables v1.8.2 (legacy)
$ kubectl exec -it -n calico-system calico-node-dbt7j -- iptables-nft --version
iptables v1.8.2 (nf_tables)
$ oc version
Client Version: openshift-clients-4.2.2-201910250432-4-g4ac90784
Server Version: 4.5.17
Kubernetes Version: v1.18.3+45b9524
Comment 15 Dan Winship 2021-01-26 16:22:22 UTC 
> $ kubectl exec -it -n calico-system calico-node-dbt7j -- iptables-nft --version
> iptables v1.8.2 (nf_tables)

So yeah, that version of iptables still has the "CHAIN_USER_DEL" bug, which means some of Calico's attempts to push its iptables rules out may fail, so presumably the iptables that are actually present after the config change are incorrect/incomplete (which you could confirm by looking through the iptables rules to see if they're as expected, or else seeing if Calico is logging errors about failing to update the iptables rules).

Updating to iptables 1.8.6 (or 1.8.7 which just came out) should fix this.
Comment 16 David Haupt 2021-01-26 23:06:10 UTC 
Thank you Dan!

We'll circle back internally and provide an update here.
Comment 19 David Haupt 2021-02-05 18:43:15 UTC 
We have raised the priority on our side, the work needed by engineering is targeted for the next sprint.
I'll get back to you with any further updates here.
Comment 20 Pooriya Aghaalitari 2021-02-10 19:32:29 UTC 
Our engineering has agreed to upgrade the iptables. However, we are not super confident that this will fix the issue. I will keep this case updated once iptables is updated and the result is validated.
Comment 21 Pooriya Aghaalitari 2021-02-26 20:16:17 UTC 
Hi Dan,

Could you please let us know where we can get the RPM packages for 1.8.6/1.8.7? Thanks.
Comment 22 Dan Winship 2021-03-01 13:39:30 UTC 
As stated in comment 11, if you are using RHEL/UBI, then you want iptables-1.8.4-10.el8_2.4 or later
Comment 23 Vladislav Walek 2021-03-01 18:00:34 UTC 
(In reply to Dan Winship from comment #15)
> > $ kubectl exec -it -n calico-system calico-node-dbt7j -- iptables-nft --version
> > iptables v1.8.2 (nf_tables)
> 
> So yeah, that version of iptables still has the "CHAIN_USER_DEL" bug, which
> means some of Calico's attempts to push its iptables rules out may fail, so
> presumably the iptables that are actually present after the config change
> are incorrect/incomplete (which you could confirm by looking through the
> iptables rules to see if they're as expected, or else seeing if Calico is
> logging errors about failing to update the iptables rules).
> 
> Updating to iptables 1.8.6 (or 1.8.7 which just came out) should fix this.

Hey Dan,
just quick one, from where  the 1.8.6 or 1.8.7 version can be downloaded?
Thx
Comment 24 Matthew Whitehead 2021-03-01 19:25:32 UTC 
If you are using RHEL8 to build your containers (you might be using RHEL7 or something else), here is how to download iptables-1.8.4-10.el8_2.4.

Login to https://access.redhat.com
Choose "Downloads" in upper left
Choose "RPM Package Search"
In the "keywords" search box put in "iptables"
Click on the architecture you want, likely "X86_64"

Right now the most up-to-date RHEL8 version is 1.8.4-15.el8_3.3 .

The most up-to-date RHEL7 version is 1.4.21-35.el7 but I do not know if that needs/has the fixes you require.
Comment 25 Pooriya Aghaalitari 2021-03-02 18:21:55 UTC 
Thanks for the info. We updated to 1.8.4-15 and the issue still persist. The reason that I asked for 1.8.6/1.8.7 was that it was asked here. https://access.redhat.com/support/cases/#/case/02756914. 
So this issue is not addressed by the update to 1.8.4-15. Please let me know if you need any info. What is the next please?
Comment 26 Dan Winship 2021-03-02 18:40:17 UTC 
See comment 10. If operators are going Degraded, it's presumably because their attempts to connect to the apiserver are failing. Why are they failing? Do the nodes have the iptables rules you expect them to have, at the time you expect them to have them?
Comment 28 Matthew Whitehead 2021-03-03 20:53:21 UTC 
Raw source for iptables I believe is hosted here: https://git.netfilter.org/iptables/
Comment 32 Pooriya Aghaalitari 2021-03-17 18:41:49 UTC 
Hi,

I think we can close this bug as the situation is not happening due to any bug. We are trying to configure bgp right from the beginning and avoid switching between encap and native bgp. Following is a short summary of our troubleshooting.

Setting up the VXLAN or IPIP encapsulation is generally done as part of the network setup of the cluster and it makes most sense to get that configuration correct before creating any non-host networked pods. What appears to be happening here is that we have both host networked and non-host networked pods configured and communicating with each other and then we are changing the networking encapsulation. Any connections that were initiated from host networked pods to non-host networked pods on different nodes will stall: The source address for the originator of the connection is assigned by the kernel at the start of the connection and is based on the egress interface (so either the physical interface or the tunnel device depending on whether you have encapsulation or not). Once the encapsulation mode is altered the source address is either no longer valid, or it adversely impacts the return path (depending on whether you are going from encapsulation->no encapsulation, or no encapsulation->encapsulation). At this point the TCP flows are broken.

This may impact other flows, such as Pods accessing node ports - so where SNAT is involved.

We believe the reason it takes so long to recover is that we are relying on the various TCP timeouts to close down the sockets and force new connections to be created. Depending on the TCP configuration and the applications this could take hours.


Pooriya Aghaalitari,
Solution Architect
Comment 33 Dan Winship 2021-03-29 15:40:55 UTC 
> I think we can close this bug as the situation is not happening due to any bug.

ok
Note You need to log in before you can comment on or make changes to this bug.