Description of problem: Upgrading one IPI installed(GCP) 4.5.39 cluster to 4.6.0-0.nightly-2021-05-15-131411 was failed. Version-Release number of selected component (if applicable): 4.5.39-x86_64--> 4.6.0-0.nightly-2021-05-15-131411 How reproducible: Always Steps to Reproduce: 1.IPI installed one OCP 4.5.39 on GCP with OVNKubernetes network 2.Upgrade to 4.6.0-0.nightly-2021-05-15-131411 Actual results: 1.Upgrade was failed. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.39 True True 17h Unable to apply 4.6.0-0.nightly-2021-05-15-131411: the cluster operator network has not yet successfully rolled out $ oc get no NAME STATUS ROLES AGE VERSION kewang172003-szjp7-master-0.c.openshift-qe.internal Ready master 19h v1.18.3+64fc02b kewang172003-szjp7-master-1.c.openshift-qe.internal Ready master 19h v1.18.3+64fc02b kewang172003-szjp7-master-2.c.openshift-qe.internal Ready master 19h v1.18.3+64fc02b kewang172003-szjp7-worker-a-ktpld.c.openshift-qe.internal Ready worker 18h v1.18.3+64fc02b kewang172003-szjp7-worker-b-pvvn6.c.openshift-qe.internal Ready worker 18h v1.18.3+64fc02b kewang172003-szjp7-worker-c-549vr.c.openshift-qe.internal Ready worker 18h v1.18.3+64fc02b $ oc get co --no-headers | grep -v '.True.*False.*False' dns 4.5.39 True True False 19h image-registry 4.6.0-0.nightly-2021-05-15-131411 True True False 18h monitoring 4.6.0-0.nightly-2021-05-15-131411 False True True 5m17s network 4.5.39 True True True 19h $ oc describe co/network Name: network Namespace: Labels: <none> Annotations: network.operator.openshift.io/last-seen-state: {"DaemonsetStates":[{"Namespace":"openshift-ovn-kubernetes","Name":"ovnkube-node","LastSeenStatus":{"currentNumberScheduled":6,"numberMiss... API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2021-05-17T12:09:04Z Generation: 1 Managed Fields: API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: f:status: .: f:extension: Manager: cluster-version-operator Operation: Update Time: 2021-05-17T12:09:04Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:network.operator.openshift.io/last-seen-state: f:status: f:conditions: f:relatedObjects: f:versions: Manager: cluster-network-operator Operation: Update Time: 2021-05-18T07:20:23Z Resource Version: 507735 Self Link: /apis/config.openshift.io/v1/clusteroperators/network UID: 7b2647cf-3bf0-4fc6-85e2-dcdbc7c555df Spec: Status: Conditions: Last Transition Time: 2021-05-17T14:40:07Z Message: DaemonSet "openshift-ovn-kubernetes/ovnkube-node" rollout is not making progress - last change 2021-05-17T14:28:10Z Reason: RolloutHung Status: True Type: Degraded Last Transition Time: 2021-05-17T12:11:11Z Status: True Type: Upgradeable Last Transition Time: 2021-05-17T14:28:07Z Message: DaemonSet "openshift-multus/network-metrics-daemon" is not available (awaiting 1 nodes) DaemonSet "openshift-ovn-kubernetes/ovnkube-node" update is rolling out (1 out of 6 updated) Reason: Deploying Status: True Type: Progressing Last Transition Time: 2021-05-17T12:12:40Z Status: True Type: Available Extension: <nil> Related Objects: Group: Name: applied-cluster Namespace: openshift-network-operator Resource: configmaps Group: apiextensions.k8s.io Name: network-attachment-definitions.k8s.cni.cncf.io Resource: customresourcedefinitions Group: apiextensions.k8s.io Name: ippools.whereabouts.cni.cncf.io Resource: customresourcedefinitions Group: apiextensions.k8s.io Name: overlappingrangeipreservations.whereabouts.cni.cncf.io Resource: customresourcedefinitions Group: Name: openshift-multus Resource: namespaces Group: rbac.authorization.k8s.io Name: multus Resource: clusterroles Group: Name: multus Namespace: openshift-multus Resource: serviceaccounts Group: rbac.authorization.k8s.io Name: multus Resource: clusterrolebindings Group: rbac.authorization.k8s.io Name: multus-whereabouts Resource: clusterrolebindings Group: rbac.authorization.k8s.io Name: whereabouts-cni Resource: clusterroles Group: Name: cni-binary-copy-script Namespace: openshift-multus Resource: configmaps Group: apps Name: multus Namespace: openshift-multus Resource: daemonsets Group: Name: metrics-daemon-sa Namespace: openshift-multus Resource: serviceaccounts Group: rbac.authorization.k8s.io Name: metrics-daemon-role Resource: clusterroles Group: rbac.authorization.k8s.io Name: metrics-daemon-sa-rolebinding Resource: clusterrolebindings Group: apps Name: network-metrics-daemon Namespace: openshift-multus Resource: daemonsets Group: monitoring.coreos.com Name: monitor-network Namespace: openshift-multus Resource: servicemonitors Group: Name: network-metrics-service Namespace: openshift-multus Resource: services Group: rbac.authorization.k8s.io Name: prometheus-k8s Namespace: openshift-multus Resource: roles Group: rbac.authorization.k8s.io Name: prometheus-k8s Namespace: openshift-multus Resource: rolebindings Group: Name: multus-admission-controller Namespace: openshift-multus Resource: services Group: rbac.authorization.k8s.io Name: multus-admission-controller-webhook Resource: clusterroles Group: rbac.authorization.k8s.io Name: multus-admission-controller-webhook Resource: clusterrolebindings Group: admissionregistration.k8s.io Name: multus.openshift.io Resource: validatingwebhookconfigurations Group: Name: openshift-service-ca Namespace: openshift-network-operator Resource: configmaps Group: apps Name: multus-admission-controller Namespace: openshift-multus Resource: daemonsets Group: monitoring.coreos.com Name: monitor-multus-admission-controller Namespace: openshift-multus Resource: servicemonitors Group: rbac.authorization.k8s.io Name: prometheus-k8s Namespace: openshift-multus Resource: roles Group: rbac.authorization.k8s.io Name: prometheus-k8s Namespace: openshift-multus Resource: rolebindings Group: monitoring.coreos.com Name: prometheus-k8s-rules Namespace: openshift-multus Resource: prometheusrules Group: Name: openshift-ovn-kubernetes Resource: namespaces Group: apiextensions.k8s.io Name: egressfirewalls.k8s.ovn.org Resource: customresourcedefinitions Group: apiextensions.k8s.io Name: egressips.k8s.ovn.org Resource: customresourcedefinitions Group: Name: ovn-kubernetes-node Namespace: openshift-ovn-kubernetes Resource: serviceaccounts Group: rbac.authorization.k8s.io Name: openshift-ovn-kubernetes-node Resource: clusterroles Group: rbac.authorization.k8s.io Name: openshift-ovn-kubernetes-node Resource: clusterrolebindings Group: Name: ovn-kubernetes-controller Namespace: openshift-ovn-kubernetes Resource: serviceaccounts Group: rbac.authorization.k8s.io Name: openshift-ovn-kubernetes-controller Resource: clusterroles Group: rbac.authorization.k8s.io Name: openshift-ovn-kubernetes-controller Resource: clusterrolebindings Group: rbac.authorization.k8s.io Name: openshift-ovn-kubernetes-sbdb Namespace: openshift-ovn-kubernetes Resource: roles Group: rbac.authorization.k8s.io Name: openshift-ovn-kubernetes-sbdb Namespace: openshift-ovn-kubernetes Resource: rolebindings Group: Name: ovnkube-config Namespace: openshift-ovn-kubernetes Resource: configmaps Group: Name: ovnkube-db Namespace: openshift-ovn-kubernetes Resource: services Group: apps Name: ovs-node Namespace: openshift-ovn-kubernetes Resource: daemonsets Group: network.operator.openshift.io Name: ovn Namespace: openshift-ovn-kubernetes Resource: operatorpkis Group: monitoring.coreos.com Name: master-rules Namespace: openshift-ovn-kubernetes Resource: prometheusrules Group: monitoring.coreos.com Name: networking-rules Namespace: openshift-ovn-kubernetes Resource: prometheusrules Group: monitoring.coreos.com Name: monitor-ovn-master-metrics Namespace: openshift-ovn-kubernetes Resource: servicemonitors Group: Name: ovn-kubernetes-master Namespace: openshift-ovn-kubernetes Resource: services Group: monitoring.coreos.com Name: monitor-ovn-node Namespace: openshift-ovn-kubernetes Resource: servicemonitors Group: Name: ovn-kubernetes-node Namespace: openshift-ovn-kubernetes Resource: services Group: rbac.authorization.k8s.io Name: prometheus-k8s Namespace: openshift-ovn-kubernetes Resource: roles Group: rbac.authorization.k8s.io Name: prometheus-k8s Namespace: openshift-ovn-kubernetes Resource: rolebindings Group: policy Name: ovn-raft-quorum-guard Namespace: openshift-ovn-kubernetes Resource: poddisruptionbudgets Group: apps Name: ovnkube-master Namespace: openshift-ovn-kubernetes Resource: daemonsets Group: apps Name: ovnkube-node Namespace: openshift-ovn-kubernetes Resource: daemonsets Group: Name: openshift-network-operator Resource: namespaces Versions: Name: operator Version: 4.5.39 Events: <none> $ oc get pods -A |grep -Ev 'Running|Completed' NAMESPACE NAME READY STATUS RESTARTS AGE default example-6nsh66swsq 0/1 Init:0/1 0 16h openshift-console downloads-68dd868b7d-9q5h9 0/1 CrashLoopBackOff 341 17h openshift-dns dns-default-4bnz5 2/3 CrashLoopBackOff 239 18h openshift-image-registry image-registry-7889d8dc64-gln7z 0/1 CrashLoopBackOff 323 17h openshift-kube-descheduler-operator descheduler-operator-7c45685d87-pw49q 0/1 ImagePullBackOff 0 18h openshift-marketplace certified-operators-h7m5k 0/1 ContainerCreating 0 17h openshift-marketplace community-operators-8zthn 0/1 ContainerCreating 0 17h openshift-marketplace redhat-marketplace-9vl2v 0/1 ContainerCreating 0 17h openshift-marketplace redhat-operators-cpclc 0/1 ContainerCreating 0 17h openshift-monitoring prometheus-adapter-79c9cd9cc4-fmzsj 0/1 ContainerCreating 0 9m50s openshift-multus network-metrics-daemon-5vnhg 0/2 ContainerCreating 0 17h openshift-operators-redhat elasticsearch-operator-794576474b-5np4f 0/1 ImagePullBackOff 0 17h openshift-ovn-kubernetes ovnkube-node-kpnnd 2/3 CrashLoopBackOff 173 17h $ oc describe pod -n openshift-ovn-kubernetes ovnkube-node-kpnnd Name: ovnkube-node-kpnnd Namespace: openshift-ovn-kubernetes Priority: 2000001000 Priority Class Name: system-node-critical Node: kewang172003-szjp7-worker-c-549vr.c.openshift-qe.internal/10.0.32.4 Start Time: Mon, 17 May 2021 22:28:10 +0800 Labels: app=ovnkube-node component=network controller-revision-hash=64469bfcdb kubernetes.io/os=linux openshift.io/component=network pod-template-generation=2 type=infra Annotations: <none> Status: Running IP: 10.0.32.4 IPs: IP: 10.0.32.4 Controlled By: DaemonSet/ovnkube-node Containers: ovn-controller: Container ID: cri-o://5cc2731354c3b8999559df591b41c9bfb82ebfd47bea7efaa16d7abf8c17a4c4 Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca0f5984ef77958226c20a3d91df7381b0b5b30f94af45680610160b04a2198 Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca0f5984ef77958226c20a3d91df7381b0b5b30f94af45680610160b04a2198 Port: <none> Host Port: <none> Command: /bin/bash -c set -e if [[ -f "/env/${K8S_NODE}" ]]; then set -o allexport source "/env/${K8S_NODE}" set +o allexport fi echo "$(date -Iseconds) - starting ovn-controller" exec ovn-controller unix:/var/run/openvswitch/db.sock -vfile:off \ --no-chdir --pidfile=/var/run/ovn/ovn-controller.pid \ -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt \ -vconsole:"${OVN_LOG_LEVEL}" State: Running Started: Mon, 17 May 2021 22:28:30 +0800 Ready: True Restart Count: 0 Requests: cpu: 10m memory: 300Mi Environment: OVN_LOG_LEVEL: info K8S_NODE: (v1:spec.nodeName) Mounts: /env from env-overrides (rw) /etc/openvswitch from etc-openvswitch (rw) /etc/ovn/ from etc-openvswitch (rw) /ovn-ca from ovn-ca (rw) /ovn-cert from ovn-cert (rw) /run/openvswitch from run-openvswitch (rw) /run/ovn/ from run-ovn (rw) /var/lib/openvswitch from var-lib-openvswitch (rw) /var/run/secrets/kubernetes.io/serviceaccount from ovn-kubernetes-node-token-g5hzm (ro) kube-rbac-proxy: Container ID: cri-o://c6f6ea4f132ab48d29996cacc7b982d5d7fe1d2b19a71dc8195d9d372106f028 Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405773b0f8b446c089b47fb16e1ab5fc9c1345c3742d8dc7f3173f521b691862 Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405773b0f8b446c089b47fb16e1ab5fc9c1345c3742d8dc7f3173f521b691862 Port: 9103/TCP Host Port: 9103/TCP Command: /bin/bash -c #!/bin/bash set -euo pipefail TLS_PK=/etc/pki/tls/metrics-cert/tls.key TLS_CERT=/etc/pki/tls/metrics-cert/tls.crt # As the secret mount is optional we must wait for the files to be present. # The service is created in monitor.yaml and this is created in sdn.yaml. # If it isn't created there is probably an issue so we want to crashloop. retries=0 while [[ "${retries}" -lt 100 ]]; do TS=$( curl \ -s \ --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \ -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ "https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/api/v1/namespaces/ovn-kubernetes/services/ovnkube-node" | python -c 'import json,sys; print(json.load(sys.stdin)["metadata"]["creationTimestamp"])' 2>/dev/null || true ) || : if [ -n "${TS}" ]; then break fi (( retries += 1 )) echo $(date -Iseconds) INFO: Failed to get ovnkube-node service from API. Retry "${retries}"/100 1>&2 sleep 20 done if [ "${retries}" -ge 20 ]; then echo $(date -Iseconds) FATAL: Unable to get ovnkube-node service from API. exit 1 fi TS=$(date -d "${TS}" +%s) WARN_TS=$(( ${TS} + $(( 20 * 60)) )) HAS_LOGGED_INFO=0 log_missing_certs(){ CUR_TS=$(date +%s) if [[ "${CUR_TS}" -gt "WARN_TS" ]]; then echo $(date -Iseconds) WARN: ovn-node-metrics-cert not mounted after 20 minutes. elif [[ "${HAS_LOGGED_INFO}" -eq 0 ]] ; then echo $(date -Iseconds) INFO: ovn-node-metrics-cert not mounted. Waiting one hour. HAS_LOGGED_INFO=1 fi } while [[ ! -f "${TLS_PK}" || ! -f "${TLS_CERT}" ]] ; do log_missing_certs sleep 5 done exec /usr/bin/kube-rbac-proxy \ --logtostderr \ --secure-listen-address=:9103 \ --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 \ --upstream=http://127.0.0.1:29103/ \ --tls-private-key-file=${TLS_PK} \ --tls-cert-file=${TLS_CERT} State: Running Started: Tue, 18 May 2021 15:10:09 +0800 Last State: Terminated Reason: Error Message: 06:16:45+00:00 INFO: Failed to get ovnkube-node service from API. Retry 79/100 2021-05-18T06:19:17+00:00 INFO: Failed to get ovnkube-node service from API. Retry 80/100 2021-05-18T06:21:49+00:00 INFO: Failed to get ovnkube-node service from API. Retry 81/100 2021-05-18T06:24:20+00:00 INFO: Failed to get ovnkube-node service from API. Retry 82/100 2021-05-18T06:26:52+00:00 INFO: Failed to get ovnkube-node service from API. Retry 83/100 2021-05-18T06:29:23+00:00 INFO: Failed to get ovnkube-node service from API. Retry 84/100 2021-05-18T06:31:55+00:00 INFO: Failed to get ovnkube-node service from API. Retry 85/100 2021-05-18T06:34:26+00:00 INFO: Failed to get ovnkube-node service from API. Retry 86/100 2021-05-18T06:36:58+00:00 INFO: Failed to get ovnkube-node service from API. Retry 87/100 2021-05-18T06:39:29+00:00 INFO: Failed to get ovnkube-node service from API. Retry 88/100 2021-05-18T06:42:01+00:00 INFO: Failed to get ovnkube-node service from API. Retry 89/100 2021-05-18T06:44:33+00:00 INFO: Failed to get ovnkube-node service from API. Retry 90/100 2021-05-18T06:47:04+00:00 INFO: Failed to get ovnkube-node service from API. Retry 91/100 2021-05-18T06:49:36+00:00 INFO: Failed to get ovnkube-node service from API. Retry 92/100 2021-05-18T06:52:07+00:00 INFO: Failed to get ovnkube-node service from API. Retry 93/100 2021-05-18T06:54:39+00:00 INFO: Failed to get ovnkube-node service from API. Retry 94/100 2021-05-18T06:57:10+00:00 INFO: Failed to get ovnkube-node service from API. Retry 95/100 2021-05-18T06:59:42+00:00 INFO: Failed to get ovnkube-node service from API. Retry 96/100 2021-05-18T07:02:13+00:00 INFO: Failed to get ovnkube-node service from API. Retry 97/100 2021-05-18T07:04:45+00:00 INFO: Failed to get ovnkube-node service from API. Retry 98/100 2021-05-18T07:07:17+00:00 INFO: Failed to get ovnkube-node service from API. Retry 99/100 2021-05-18T07:09:48+00:00 INFO: Failed to get ovnkube-node service from API. Retry 100/100 2021-05-18T07:10:08+00:00 FATAL: Unable to get ovnkube-node service from API. Exit Code: 1 Started: Tue, 18 May 2021 10:57:33 +0800 Finished: Tue, 18 May 2021 15:10:08 +0800 Ready: True Restart Count: 4 Requests: cpu: 10m memory: 20Mi Environment: <none> Mounts: /etc/pki/tls/metrics-cert from ovn-node-metrics-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from ovn-kubernetes-node-token-g5hzm (ro) ovnkube-node: Container ID: cri-o://90c2dc072ef82c94e4bfb7a79e3e9f3c1ea797a0bf6a2702b4d852678c59355d Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca0f5984ef77958226c20a3d91df7381b0b5b30f94af45680610160b04a2198 Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca0f5984ef77958226c20a3d91df7381b0b5b30f94af45680610160b04a2198 Port: 29103/TCP Host Port: 29103/TCP Command: /bin/bash -c set -xe if [[ -f "/env/${K8S_NODE}" ]]; then set -o allexport source "/env/${K8S_NODE}" set +o allexport fi echo "I$(date "+%m%d %H:%M:%S.%N") - waiting for db_ip addresses" cp -f /usr/libexec/cni/ovn-k8s-cni-overlay /cni-bin-dir/ ovn_config_namespace=openshift-ovn-kubernetes echo "I$(date "+%m%d %H:%M:%S.%N") - disable conntrack on geneve port" iptables -t raw -A PREROUTING -p udp --dport 6081 -j NOTRACK iptables -t raw -A OUTPUT -p udp --dport 6081 -j NOTRACK retries=0 while true; do # TODO: change to use '--request-timeout=30s', if https://github.com/kubernetes/kubernetes/issues/49343 is fixed. db_ip=$(timeout 30 kubectl get ep -n ${ovn_config_namespace} ovnkube-db -o jsonpath='{.subsets[0].addresses[0].ip}') if [[ -n "${db_ip}" ]]; then break fi (( retries += 1 )) if [[ "${retries}" -gt 40 ]]; then echo "E$(date "+%m%d %H:%M:%S.%N") - db endpoint never came up" exit 1 fi echo "I$(date "+%m%d %H:%M:%S.%N") - waiting for db endpoint" sleep 5 done echo "I$(date "+%m%d %H:%M:%S.%N") - starting ovnkube-node db_ip ${db_ip}" gateway_mode_flags= # Check to see if ovs is provided by the node. This is only for upgrade from 4.5->4.6 or # openshift-sdn to ovn-kube conversion if grep -q OVNKubernetes /etc/systemd/system/ovs-configuration.service && [ -f /host/var/run/ovs-config-executed ]; then gateway_mode_flags="--gateway-mode local --gateway-interface br-ex" else gateway_mode_flags="--gateway-mode local --gateway-interface none" fi exec /usr/bin/ovnkube --init-node "${K8S_NODE}" \ --nb-address "ssl:10.0.0.3:9641,ssl:10.0.0.4:9641,ssl:10.0.0.5:9641" \ --sb-address "ssl:10.0.0.3:9642,ssl:10.0.0.4:9642,ssl:10.0.0.5:9642" \ --nb-client-privkey /ovn-cert/tls.key \ --nb-client-cert /ovn-cert/tls.crt \ --nb-client-cacert /ovn-ca/ca-bundle.crt \ --nb-cert-common-name "ovn" \ --sb-client-privkey /ovn-cert/tls.key \ --sb-client-cert /ovn-cert/tls.crt \ --sb-client-cacert /ovn-ca/ca-bundle.crt \ --sb-cert-common-name "ovn" \ --config-file=/run/ovnkube-config/ovnkube.conf \ --loglevel "${OVN_KUBE_LOG_LEVEL}" \ --inactivity-probe="${OVN_CONTROLLER_INACTIVITY_PROBE}" \ ${gateway_mode_flags} \ --metrics-bind-address "127.0.0.1:29103" State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Message: c(119): stderr: "" I0518 07:29:52.668615 3694670 node.go:116] Node kewang172003-szjp7-worker-c-549vr.c.openshift-qe.internal connection status = not connected I0518 07:29:53.160635 3694670 ovs.go:164] exec(120): /usr/bin/ovs-appctl --timeout=15 -t /var/run/ovn/ovn-controller.314045.ctl connection-status I0518 07:29:53.167191 3694670 ovs.go:167] exec(120): stdout: "not connected\n" I0518 07:29:53.167229 3694670 ovs.go:168] exec(120): stderr: "" I0518 07:29:53.167243 3694670 node.go:116] Node kewang172003-szjp7-worker-c-549vr.c.openshift-qe.internal connection status = not connected I0518 07:29:53.660640 3694670 ovs.go:164] exec(121): /usr/bin/ovs-appctl --timeout=15 -t /var/run/ovn/ovn-controller.314045.ctl connection-status I0518 07:29:53.666945 3694670 ovs.go:167] exec(121): stdout: "not connected\n" I0518 07:29:53.666979 3694670 ovs.go:168] exec(121): stderr: "" I0518 07:29:53.666989 3694670 node.go:116] Node kewang172003-szjp7-worker-c-549vr.c.openshift-qe.internal connection status = not connected I0518 07:29:54.160676 3694670 ovs.go:164] exec(122): /usr/bin/ovs-appctl --timeout=15 -t /var/run/ovn/ovn-controller.314045.ctl connection-status I0518 07:29:54.167052 3694670 ovs.go:167] exec(122): stdout: "not connected\n" I0518 07:29:54.167135 3694670 ovs.go:168] exec(122): stderr: "" I0518 07:29:54.167145 3694670 node.go:116] Node kewang172003-szjp7-worker-c-549vr.c.openshift-qe.internal connection status = not connected I0518 07:29:54.167163 3694670 ovs.go:164] exec(123): /usr/bin/ovs-appctl --timeout=15 -t /var/run/ovn/ovn-controller.314045.ctl connection-status I0518 07:29:54.172900 3694670 ovs.go:167] exec(123): stdout: "not connected\n" I0518 07:29:54.172932 3694670 ovs.go:168] exec(123): stderr: "" I0518 07:29:54.172942 3694670 node.go:116] Node kewang172003-szjp7-worker-c-549vr.c.openshift-qe.internal connection status = not connected F0518 07:29:54.172965 3694670 ovnkube.go:130] timed out waiting sbdb for node kewang172003-szjp7-worker-c-549vr.c.openshift-qe.internal: timed out waiting for the condition Exit Code: 1 Started: Tue, 18 May 2021 15:28:52 +0800 Finished: Tue, 18 May 2021 15:29:54 +0800 Ready: False Restart Count: 169 Requests: cpu: 10m memory: 300Mi Readiness: exec [test -f /etc/cni/net.d/10-ovn-kubernetes.conf] delay=5s timeout=1s period=5s #success=1 #failure=3 Environment: KUBERNETES_SERVICE_PORT: 6443 KUBERNETES_SERVICE_HOST: api-int.kewang172003.qe.gcp.devcluster.openshift.com OVN_CONTROLLER_INACTIVITY_PROBE: 30000 OVN_KUBE_LOG_LEVEL: 4 K8S_NODE: (v1:spec.nodeName) Mounts: /cni-bin-dir from host-cni-bin (rw) /env from env-overrides (rw) /etc/cni/net.d from host-cni-netd (rw) /etc/openvswitch from etc-openvswitch (rw) /etc/ovn/ from etc-openvswitch (rw) /etc/systemd/system from systemd-units (ro) /host from host-slash (ro) /ovn-ca from ovn-ca (rw) /ovn-cert from ovn-cert (rw) /run/netns from host-run-netns (ro) /run/openvswitch from run-openvswitch (rw) /run/ovn-kubernetes/ from host-run-ovn-kubernetes (rw) /run/ovn/ from run-ovn (rw) /run/ovnkube-config/ from ovnkube-config (rw) /var/lib/cni/networks/ovn-k8s-cni-overlay from host-var-lib-cni-networks-ovn-kubernetes (rw) /var/lib/openvswitch from var-lib-openvswitch (rw) /var/run/secrets/kubernetes.io/serviceaccount from ovn-kubernetes-node-token-g5hzm (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: ... QoS Class: Burstable Node-Selectors: beta.kubernetes.io/os=linux Tolerations: op=Exists Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 60m (x161 over 17h) kubelet Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca0f5984ef77958226c20a3d91df7381b0b5b30f94af45680610160b04a2198" already present on machine Warning Unhealthy 10m (x2025 over 17h) kubelet Readiness probe failed: Warning BackOff 28s (x3872 over 17h) kubelet Back-off restarting failed container ---- Checked the log of above pod ovnkube-node-kpnnd, found many same error, SSL_connect: error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small Expected results: Upgrade should be successful. Additional info: - From must-gahter output: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information. ClusterID: 5c43bc33-c812-4aa9-a038-54e913d40dc1 ClusterVersion: Updating to "4.6.0-0.nightly-2021-05-15-131411" from "4.5.39" for 18 hours: Unable to apply 4.6.0-0.nightly-2021-05-15-131411: the cluster operator monitoring is degraded ClusterOperators: clusteroperator/dns is progressing: At least 1 DNS DaemonSet is progressing. clusteroperator/image-registry is progressing: Progressing: The deployment has not completed clusteroperator/monitoring is not available () because Failed to rollout the stack. Error: running task Updating prometheus-adapter failed: reconciling PrometheusAdapter Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-adapter: expected 3 replicas, got 2 updated replicas clusteroperator/network is degraded because DaemonSet "openshift-ovn-kubernetes/ovnkube-node" rollout is not making progress - last change 2021-05-17T14:28:10Z
This bug blocked this upgrade path test, so added UpgradeBlocker.
The first thing I noticed was the error output from the kube-rbac-proxy container: "2021-05-18T07:10:08+00:00 FATAL: Unable to get ovnkube-node service from API." That reminded me of a 4.6 bug I recently fixed in this PR: https://github.com/openshift/cluster-network-operator/pull/1096 I wanted to verify that the same problem was being seen in our CI with other 4.5->4.6 upgrade jobs, which we do have: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-informing#periodic-ci-openshift-release-master-ci-4.6-upgrade-from-stable-4.5-e2e-aws-ovn-upgrade But, those are permafailing and I don't think they are even getting past the initial cluster install of 4.5 (maybe we need a new BZ for that?). You can see in this clusterversion.yaml that 4.5 is still status Progressing: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.6-upgrade-from-stable-4.5-e2e-aws-ovn-upgrade/1395469310314418176/artifacts/e2e-aws-ovn-upgrade/gather-extra/artifacts/clusterversion.json Anyway, the 4.6 fix from the above PR has made it in to a 4.6 nightly. I think this is the first one that has it: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4.6.0-0.nightly/release/4.6.0-0.nightly-2021-05-21-060114 Would you be able to try upgrading to that image and see if at least that kube-rbac-proxy error is fixed? If the upgrade still fails, we can continue debugging from that reproduction.
Thank you for providing the cluster to debug. Is there any way to keep it around longer? I do not see the message "Unable to get ovnkube-node service from API." anymore which means the 4.6 image used is working around that problem. However, the ovnkube-node pod is still in crashloop because of the "/usr/bin/ovs-appctl --timeout=15 -t /var/run/ovn/ovn-controller.314045.ctl connection-status" command is still failing. I do not know the root cause of the problem yet, but I can see that the networking on the node with the affected pod does seem to have less ports and network namespaces than a similar worker node. For reference, because I'm not sure if this is expected output or not, here is the "ovn-nbctl show" output of the affected node: [root@kewang251211-ht2zn-master-2 ~]# ovn-nbctl show kewang251211-ht2zn-worker-b-6wplf.c.openshift-qe.internal switch c2bfd547-ea18-43b8-9ef8-24f6d0227f00 (kewang251211-ht2zn-worker-b-6wplf.c.openshift-qe.internal) port k8s-kewang251211-ht2zn-worker-b-6wplf.c.openshift-qe.internal addresses: ["5a:2b:f0:66:0d:21 10.128.2.2"] port openshift-monitoring_alertmanager-main-1 addresses: ["dynamic"] port openshift-monitoring_prometheus-adapter-5d6b5646f5-z5krz addresses: ["dynamic"] port openshift-monitoring_grafana-5bf8c576b7-6tstp addresses: ["dynamic"] port openshift-console_downloads-699dd89bc8-chr8p addresses: ["dynamic"] port openshift-dns_dns-default-nhqxx addresses: ["dynamic"] port openshift-monitoring_prometheus-k8s-0 addresses: ["dynamic"] port stor-kewang251211-ht2zn-worker-b-6wplf.c.openshift-qe.internal type: router addresses: ["0a:58:0a:80:02:01"] router-port: rtos-kewang251211-ht2zn-worker-b-6wplf.c.openshift-qe.internal port openshift-multus_network-metrics-daemon-knxzn addresses: ["dynamic"] vs the output of a seemingly working node: [root@kewang251211-ht2zn-master-2 ~]# ovn-nbctl show kewang251211-ht2zn-worker-a-82ntp.c.openshift-qe.internal switch 6254f1f9-3833-4297-af37-fbf05856059f (kewang251211-ht2zn-worker-a-82ntp.c.openshift-qe.internal) port openshift-kube-descheduler-operator_descheduler-operator-56969b9f95-698gl addresses: ["dynamic"] port openshift-marketplace_certified-operators-ql4qx addresses: ["dynamic"] port openshift-monitoring_openshift-state-metrics-676b6b9966-96rcw addresses: ["dynamic"] port openshift-monitoring_thanos-querier-84bb5c55c8-bnghp addresses: ["dynamic"] port openshift-marketplace_redhat-operators-cvxqx addresses: ["dynamic"] port openshift-monitoring_alertmanager-main-2 addresses: ["dynamic"] port openshift-marketplace_redhat-marketplace-tfrpx addresses: ["dynamic"] port openshift-service-catalog-removed_openshift-service-catalog-controller-manager-remover-455q9 addresses: ["dynamic"] port openshift-ingress_router-default-7cb59d77f5-9h7z7 addresses: ["dynamic"] port openshift-console_downloads-699dd89bc8-vjpsj addresses: ["dynamic"] port openshift-monitoring_telemeter-client-5fb489b979-tn9fb addresses: ["dynamic"] port openshift-monitoring_prometheus-adapter-5d6b5646f5-mcmh2 addresses: ["dynamic"] port k8s-kewang251211-ht2zn-worker-a-82ntp.c.openshift-qe.internal addresses: ["1a:ae:a5:a8:be:4a 10.131.0.2"] port openshift-monitoring_kube-state-metrics-7c856ddccd-g568g addresses: ["dynamic"] port openshift-kube-storage-version-migrator_migrator-79cbb9c9bc-6hvfl addresses: ["dynamic"] port openshift-dns_dns-default-bvckj addresses: ["dynamic"] port openshift-multus_network-metrics-daemon-c2rnj addresses: ["dynamic"] port openshift-image-registry_image-registry-69db996b7d-qts9n addresses: ["dynamic"] port stor-kewang251211-ht2zn-worker-a-82ntp.c.openshift-qe.internal type: router addresses: ["0a:58:0a:83:00:01"] router-port: rtos-kewang251211-ht2zn-worker-a-82ntp.c.openshift-qe.internal port openshift-service-catalog-removed_openshift-service-catalog-apiserver-remover-nqkk4 addresses: ["dynamic"] port openshift-marketplace_community-operators-hp7m8 addresses: ["dynamic"] The high level problem is that the ovnkube-node that is crashlooping (as you can see in the original logs of this bug) is that this command returns "not connected": [root@kewang251211-ht2zn-worker-b-6wplf ~]# /usr/bin/ovs-appctl --timeout=15 -t /var/run/ovn/ovn-controller.165523.ctl connection-status not connected [root@kewang251211-ht2zn-worker-b-6wplf ~]# Will need to continue to debug.
Per the UpgradeBlocker from comment 3, ee're asking the following questions to evaluate whether or not this bug warrants blocking an upgrade edge from either the previous X.Y or X.Y.Z. The ultimate goal is to avoid delivering an update which introduces new risk or reduces cluster functionality in any way. Sample answers are provided to give more context and the ImpactStatementRequested label has been added to this bug. When responding, please remove ImpactStatementRequested and set the ImpactStatementProposed label. The expectation is that the assignee answers these questions. Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking? * example: Customers upgrading from 4.y.Z to 4.y+1.z running on GCP with thousands of namespaces, approximately 5% of the subscribed fleet * example: All customers upgrading from 4.y.z to 4.y+1.z fail approximately 10% of the time What is the impact? Is it serious enough to warrant blocking edges? * example: Up to 2 minute disruption in edge routing * example: Up to 90 seconds of API downtime * example: etcd loses quorum and you have to restore from backup How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)? * example: Issue resolves itself after five minutes * example: Admin uses oc to fix things * example: Admin must SSH to hosts, restore from backups, or other non standard admin activities Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)? * example: No, it has always been like this we just never noticed * example: Yes, from 4.y.z to 4.y+1.z Or 4.y.z to 4.y.z+1
asood was able to give me a cluster in this failed state to debug with for a while. Eventually, the cluster stop responding so I couldn't debug further. I noticed that one of the nodes had 14 crashlooping pods. That makes more sense now, considering Arti noticed apiserver crashlooping. The ovnkube-node pod was also crashlooping. here's the list: openshift-apiserver apiserver-5c786757c8-bqnxs 1/2 CrashLoopBackOff 234 19h 10.128.0.23 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-authentication-operator authentication-operator-86c8d8cd87-v46lw 0/1 CrashLoopBackOff 212 19h 10.128.0.61 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-authentication oauth-openshift-7f975577d4-v5n24 0/1 CrashLoopBackOff 233 19h 10.128.0.8 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-cluster-storage-operator cluster-storage-operator-6cff649b8b-dqmpb 0/1 CrashLoopBackOff 211 19h 10.128.0.64 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-cluster-storage-operator csi-snapshot-controller-operator-6b67df7874-l58tn 0/1 CrashLoopBackOff 211 19h 10.128.0.63 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-controller-manager-operator openshift-controller-manager-operator-64967fdf46-7vh2p 0/1 CrashLoopBackOff 211 19h 10.128.0.60 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-etcd-operator etcd-operator-5b8d9b4dc6-jxfbq 0/1 CrashLoopBackOff 212 20h 10.128.0.5 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-kube-apiserver-operator kube-apiserver-operator-865b77f7f4-lzg5w 0/1 CrashLoopBackOff 211 20h 10.128.0.53 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-marketplace marketplace-operator-8699d97b54-m6xgm 0/1 CrashLoopBackOff 273 19h 10.128.0.31 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-oauth-apiserver apiserver-66d86b7df5-h9mjq 0/1 CrashLoopBackOff 233 19h 10.128.0.7 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-operator-lifecycle-manager catalog-operator-748d794d75-lk652 0/1 CrashLoopBackOff 245 19h 10.128.0.65 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-operator-lifecycle-manager olm-operator-5d5f7cdf96-5hnr5 0/1 CrashLoopBackOff 255 19h 10.128.0.62 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-operator-lifecycle-manager packageserver-864fcf94dd-2ps2v 0/1 CrashLoopBackOff 255 19h 10.128.0.9 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> openshift-ovn-kubernetes ovnkube-node-726hf 2/3 CrashLoopBackOff 188 19h 10.0.0.3 asood-5264-gcp-o-n5zln-master-2.c.openshift-qe.internal <none> <none> None of the other nodes (3 master, 4 workers total) had any crashlooping pods. with a debug pod on the affected node, I could see that no flows were programmed on br-int and that I could not reach the next hop out of the ovn-k8s-mp0 port. I also noticed some weirdness around netns that I didn't see on a non-affected node: sh-4.4# ip netns Error: Peer netns reference is invalid. Error: Peer netns reference is invalid. Error: Peer netns reference is invalid. 0c99070e-0ac4-4d72-b102-2a1d3debd7c3 Error: Peer netns reference is invalid. bfc4a4cb-071f-4704-b3dc-8d254d932d34 a745bf6a-af79-4149-ad8e-c158d819fe2e (id: 6) 5d2181df-0721-4b4d-94ee-da0cbadd3493 (id: 3) 4883dc97-4fee-48a6-aa17-ad9fd0669f61 (id: 5) f4feaee9-62aa-4d90-9bf5-9e0bbf66c6d6 (id: 0) 50857577-f1ad-4926-b435-d3d30cfecf9b (id: 25) 8d72fd9e-1d54-4569-8a2d-ac69f9ede225 (id: 26) ab76bc8c-5a39-43ec-b80b-4102ea3daaea (id: 28) 97942227-0858-423e-94d2-0e769fe89df8 (id: 24) 736fabb7-c53a-4215-9953-3ea5e7586347 (id: 17) 077e19ac-4932-455a-9507-c2c9def8b37f (id: 23) c521449b-f0b7-43e0-bb4d-3e46910b0867 (id: 22) 6aae78b5-95a7-45e0-b92d-c0c699e7fafe (id: 21) 10e9873c-a8a4-440e-a73e-92425c4a87c1 (id: 14) 2cd0d214-5a94-4e28-ae51-4a9f3b73a3f7 (id: 4) 6d7f8531-37a1-43f1-a04b-0ec8d4a56c41 (id: 7) 8134511f-a1eb-47aa-b419-e0f071d5f1be (id: 1) 166dd83f-3185-4779-81ea-23f7bbbd6d26 (id: 20) 09c5791d-b65a-492f-abe8-6ff5bf2161dc (id: 2) 70a87f85-d216-457d-b792-8aebca04fd11 (id: 13) 22b8f155-9649-4473-aa43-dcbb9a5500ee (id: 10) also in the ovn-k8s-cni-overlay.log file I saw a lot of errors about namespaces not existing, whereas in a good node that log file was empty. I'm not sure if the namespace issues are even related, just documenting what I discovered today. Hoping to get another cluster to debug after the long weekend.
More notes: On the affected node, there are no flows for br-int: sh-4.4# ovs-ofctl dump-flows br-int sh-4.4# compared to a node that has no crashlooping pods that has 6k flows: sh-4.4# ovs-ofctl dump-flows br-int | wc -l 6244 The ovn-controller logs from the crashlooping ovnkube-node pod has repeating messages like: stream_ssl|WARN|SSL_connect: error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small ./oc logs -nopenshift-ovn-kubernetes ovnkube-node-krh6m ovn-controller 2021-06-01T13:44:35+00:00 - starting ovn-controller 2021-06-01T13:44:35Z|00001|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2021-06-01T13:44:35Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2021-06-01T13:44:35Z|00003|main|INFO|OVN internal version is : [20.12.0-20.16.1-56.0] 2021-06-01T13:44:35Z|00004|main|INFO|OVS IDL reconnected, force recompute. 2021-06-01T13:44:35Z|00005|reconnect|INFO|ssl:10.0.0.4:9642: connecting... 2021-06-01T13:44:35Z|00006|main|INFO|OVNSB IDL reconnected, force recompute. 2021-06-01T13:44:35Z|00007|stream_ssl|WARN|SSL_connect: error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small 2021-06-01T13:44:35Z|00008|reconnect|INFO|ssl:10.0.0.4:9642: connection attempt failed (Protocol error) 2021-06-01T13:44:35Z|00009|reconnect|INFO|ssl:10.0.0.6:9642: connecting... 2021-06-01T13:44:35Z|00010|stream_ssl|WARN|SSL_connect: error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small 2021-06-01T13:44:35Z|00011|reconnect|INFO|ssl:10.0.0.6:9642: connection attempt failed (Protocol error) 2021-06-01T13:44:35Z|00012|reconnect|INFO|ssl:10.0.0.5:9642: connecting... 2021-06-01T13:44:35Z|00013|stream_ssl|WARN|SSL_connect: error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small 2021-06-01T13:44:35Z|00014|reconnect|INFO|ssl:10.0.0.5:9642: connection attempt failed (Protocol error) 2021-06-01T13:44:36Z|00015|reconnect|INFO|ssl:10.0.0.4:9642: connecting... 2021-06-01T13:44:36Z|00016|stream_ssl|WARN|SSL_connect: error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small 2021-06-01T13:44:36Z|00017|reconnect|INFO|ssl:10.0.0.4:9642: connection attempt failed (Protocol error) 2021-06-01T13:44:36Z|00018|reconnect|INFO|ssl:10.0.0.4:9642: waiting 2 seconds before reconnect 2021-06-01T13:44:38Z|00019|reconnect|INFO|ssl:10.0.0.6:9642: connecting... 2021-06-01T13:44:38Z|00020|stream_ssl|WARN|SSL_connect: error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small 2021-06-01T13:44:38Z|00021|reconnect|INFO|ssl:10.0.0.6:9642: connection attempt failed (Protocol error) 2021-06-01T13:44:38Z|00022|reconnect|INFO|ssl:10.0.0.6:9642: waiting 4 seconds before reconnect 2021-06-01T13:44:42Z|00023|reconnect|INFO|ssl:10.0.0.5:9642: connecting... 2021-06-01T13:44:42Z|00024|stream_ssl|WARN|SSL_connect: error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small 2021-06-01T13:44:42Z|00025|reconnect|INFO|ssl:10.0.0.5:9642: connection attempt failed (Protocol error) 2021-06-01T13:44:42Z|00026|reconnect|INFO|ssl:10.0.0.5:9642: continuing to reconnect in the background but suppressing further logging 2021-06-01T13:44:50Z|00027|stream_ssl|WARN|SSL_connect: error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small <snip repeating messages> 2021-06-02T20:47:30Z|13992|stream_ssl|WARN|SSL_connect: error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small the ovn-controller logs from a working node: ❯ ./oc logs -nopenshift-ovn-kubernetes ovnkube-node-4x6gn ovn-controller 2021-06-01T11:59:49Z|00001|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2021-06-01T11:59:49Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory) 2021-06-01T11:59:50Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2021-06-01T11:59:50Z|00004|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2021-06-01T11:59:50Z|00005|main|INFO|OVS IDL reconnected, force recompute. 2021-06-01T11:59:50Z|00006|main|INFO|OVNSB IDL reconnected, force recompute. 2021-06-01T11:59:50Z|00007|reconnect|INFO|ssl:10.0.0.5:9642: connecting... 2021-06-01T11:59:50Z|00008|reconnect|INFO|ssl:10.0.0.5:9642: connected 2021-06-01T11:59:50Z|00009|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch 2021-06-01T11:59:50Z|00010|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2021-06-01T11:59:50Z|00011|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection failed (No such file or directory) 2021-06-01T11:59:50Z|00012|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: waiting 1 seconds before reconnect 2021-06-01T11:59:51Z|00013|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2021-06-01T11:59:51Z|00014|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2021-06-01T11:59:51Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch 2021-06-01T11:59:51Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2021-06-01T11:59:51Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2021-06-01T11:59:52Z|00015|binding|INFO|Claiming lport k8s-asood-611-45-gcpo-cxqgs-worker-c-ndf2w.c.openshift-qe.internal for this chassis. 2021-06-01T11:59:52Z|00016|binding|INFO|k8s-asood-611-45-gcpo-cxqgs-worker-c-ndf2w.c.openshift-qe.internal: Claiming 62:1e:7b:81:61:5a 10.131.0.2 2021-06-01T11:59:53Z|00017|binding|INFO|Claiming lport rtoe-GR_asood-611-45-gcpo-cxqgs-worker-c-ndf2w.c.openshift-qe.internal for this chassis. 2021-06-01T11:59:53Z|00018|binding|INFO|rtoe-GR_asood-611-45-gcpo-cxqgs-worker-c-ndf2w.c.openshift-qe.internal: Claiming 00:00:a9:fe:21:02 169.254.33.2/24 2021-06-01T11:59:53Z|00019|binding|INFO|Claiming lport jtor-GR_asood-611-45-gcpo-cxqgs-worker-c-ndf2w.c.openshift-qe.internal for this chassis. 2021-06-01T11:59:53Z|00020|binding|INFO|jtor-GR_asood-611-45-gcpo-cxqgs-worker-c-ndf2w.c.openshift-qe.internal: Claiming router 2021-06-01T11:59:53Z|00021|binding|INFO|Claiming lport rtoj-GR_asood-611-45-gcpo-cxqgs-worker-c-ndf2w.c.openshift-qe.internal for this chassis. 2021-06-01T11:59:53Z|00022|binding|INFO|rtoj-GR_asood-611-45-gcpo-cxqgs-worker-c-ndf2w.c.openshift-qe.internal: Claiming 0a:58:64:40:05:01 100.64.5.1/29 2021-06-01T11:59:53Z|00023|binding|INFO|Claiming lport etor-GR_asood-611-45-gcpo-cxqgs-worker-c-ndf2w.c.openshift-qe.internal for this chassis. 2021-06-01T11:59:53Z|00024|binding|INFO|etor-GR_asood-611-45-gcpo-cxqgs-worker-c-ndf2w.c.openshift-qe.internal: Claiming 00:00:a9:fe:21:02 2021-06-01T12:00:15Z|00025|binding|INFO|Claiming lport openshift-dns_dns-default-db8m9 for this chassis. 2021-06-01T12:00:15Z|00026|binding|INFO|openshift-dns_dns-default-db8m9: Claiming dynamic 2021-06-01T12:00:18Z|00027|binding|INFO|Claiming lport openshift-image-registry_image-registry-5f7f8695c-mx2zf for this chassis. 2021-06-01T12:00:18Z|00028|binding|INFO|openshift-image-registry_image-registry-5f7f8695c-mx2zf: Claiming dynamic 2021-06-01T12:00:18Z|00029|binding|INFO|Claiming lport openshift-ingress_router-default-5684b58df8-mm9pz for this chassis. 2021-06-01T12:00:18Z|00030|binding|INFO|openshift-ingress_router-default-5684b58df8-mm9pz: Claiming dynamic 2021-06-01T12:00:19Z|00031|binding|INFO|Claiming lport openshift-monitoring_kube-state-metrics-6bdff4f7ff-g9wkw for this chassis. 2021-06-01T12:00:19Z|00032|binding|INFO|openshift-monitoring_kube-state-metrics-6bdff4f7ff-g9wkw: Claiming dynamic 2021-06-01T12:00:20Z|00033|binding|INFO|Claiming lport openshift-monitoring_openshift-state-metrics-6fb5bf5bf-pjmcr for this chassis. 2021-06-01T12:00:20Z|00034|binding|INFO|openshift-monitoring_openshift-state-metrics-6fb5bf5bf-pjmcr: Claiming dynamic 2021-06-01T12:00:20Z|00035|binding|INFO|Claiming lport openshift-monitoring_telemeter-client-695bcf9d57-hgwcw for this chassis. 2021-06-01T12:00:20Z|00036|binding|INFO|openshift-monitoring_telemeter-client-695bcf9d57-hgwcw: Claiming dynamic 2021-06-01T12:00:27Z|00037|binding|INFO|Claiming lport openshift-monitoring_alertmanager-main-0 for this chassis. 2021-06-01T12:00:27Z|00038|binding|INFO|openshift-monitoring_alertmanager-main-0: Claiming dynamic 2021-06-01T12:00:44Z|00039|binding|INFO|Releasing lport openshift-image-registry_image-registry-5f7f8695c-mx2zf from this chassis. 2021-06-01T12:01:36Z|00040|binding|INFO|Claiming lport openshift-monitoring_prometheus-adapter-d85576b7d-v2sjt for this chassis. 2021-06-01T12:01:36Z|00041|binding|INFO|openshift-monitoring_prometheus-adapter-d85576b7d-v2sjt: Claiming dynamic 2021-06-01T12:54:49Z|00042|binding|INFO|Claiming lport openshift-marketplace_qe-app-registry-c867cc446-lr2th for this chassis. 2021-06-01T12:54:49Z|00043|binding|INFO|openshift-marketplace_qe-app-registry-c867cc446-lr2th: Claiming dynamic 2021-06-01T13:04:30Z|00044|binding|INFO|Claiming lport arti-test1_test-rc-jrn5t for this chassis. 2021-06-01T13:04:30Z|00045|binding|INFO|arti-test1_test-rc-jrn5t: Claiming dynamic 2021-06-01T13:19:14Z|00046|binding|INFO|Releasing lport openshift-monitoring_prometheus-adapter-d85576b7d-v2sjt from this chassis. 2021-06-01T13:34:44Z|00047|binding|INFO|Claiming lport openshift-kube-storage-version-migrator_migrator-9b45646f-4mm24 for this chassis. 2021-06-01T13:34:44Z|00048|binding|INFO|openshift-kube-storage-version-migrator_migrator-9b45646f-4mm24: Claiming dynamic 2021-06-01T13:35:25Z|00049|binding|INFO|Releasing lport openshift-marketplace_qe-app-registry-c867cc446-lr2th from this chassis. 2021-06-01T13:35:26Z|00050|binding|INFO|Claiming lport openshift-monitoring_kube-state-metrics-7d9d4bf44-dj924 for this chassis. 2021-06-01T13:35:26Z|00051|binding|INFO|openshift-monitoring_kube-state-metrics-7d9d4bf44-dj924: Claiming dynamic 2021-06-01T13:35:30Z|00052|binding|INFO|Claiming lport openshift-monitoring_telemeter-client-7f9749d88b-g64pc for this chassis. 2021-06-01T13:35:30Z|00053|binding|INFO|openshift-monitoring_telemeter-client-7f9749d88b-g64pc: Claiming dynamic 2021-06-01T13:35:36Z|00054|binding|INFO|Releasing lport openshift-monitoring_kube-state-metrics-6bdff4f7ff-g9wkw from this chassis. 2021-06-01T13:35:41Z|00055|binding|INFO|Releasing lport openshift-monitoring_telemeter-client-695bcf9d57-hgwcw from this chassis. 2021-06-01T13:35:48Z|00056|binding|INFO|Releasing lport openshift-monitoring_openshift-state-metrics-6fb5bf5bf-pjmcr from this chassis. 2021-06-01T13:35:49Z|00057|binding|INFO|Claiming lport openshift-monitoring_prometheus-adapter-5464769995-62q74 for this chassis. 2021-06-01T13:35:49Z|00058|binding|INFO|openshift-monitoring_prometheus-adapter-5464769995-62q74: Claiming dynamic 2021-06-01T13:35:53Z|00059|binding|INFO|Claiming lport openshift-ingress_router-default-d64f74849-8cqpr for this chassis. 2021-06-01T13:35:53Z|00060|binding|INFO|openshift-ingress_router-default-d64f74849-8cqpr: Claiming dynamic 2021-06-01T13:36:19Z|00061|binding|INFO|Claiming lport openshift-monitoring_thanos-querier-67ddc87d56-9sv6f for this chassis. 2021-06-01T13:36:19Z|00062|binding|INFO|openshift-monitoring_thanos-querier-67ddc87d56-9sv6f: Claiming dynamic 2021-06-01T13:37:02Z|00063|binding|INFO|Releasing lport openshift-ingress_router-default-5684b58df8-mm9pz from this chassis. 2021-06-01T13:37:04Z|00064|binding|INFO|Releasing lport openshift-monitoring_alertmanager-main-0 from this chassis. 2021-06-01T13:37:13Z|00065|binding|INFO|Claiming lport openshift-monitoring_alertmanager-main-0 for this chassis. 2021-06-01T13:37:13Z|00066|binding|INFO|openshift-monitoring_alertmanager-main-0: Claiming dynamic 2021-06-01T13:44:10Z|00067|binding|INFO|Claiming lport openshift-multus_network-metrics-daemon-lckrf for this chassis. 2021-06-01T13:44:10Z|00068|binding|INFO|openshift-multus_network-metrics-daemon-lckrf: Claiming dynamic 2021-06-01T13:48:17Z|00004|rconn(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: connection closed by peer 2021-06-01T13:48:17Z|00069|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connection closed by peer 2021-06-01T13:48:17Z|00070|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connection closed by peer 2021-06-01T13:48:18Z|00005|rconn(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2021-06-01T13:48:18Z|00071|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2021-06-01T13:48:18Z|00006|rconn(ovn_pinctrl0)|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection failed (Connection refused) 2021-06-01T13:48:18Z|00072|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection failed (Connection refused) 2021-06-01T13:48:18Z|00007|rconn(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: waiting 2 seconds before reconnect 2021-06-01T13:48:18Z|00073|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: waiting 2 seconds before reconnect 2021-06-01T13:48:18Z|00074|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2021-06-01T13:48:18Z|00075|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory) 2021-06-01T13:48:18Z|00076|reconnect|INFO|unix:/var/run/openvswitch/db.sock: waiting 2 seconds before reconnect 2021-06-01T13:48:20Z|00008|rconn(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2021-06-01T13:48:20Z|00077|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2021-06-01T13:48:20Z|00009|rconn(ovn_pinctrl0)|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection failed (Connection refused) 2021-06-01T13:48:20Z|00010|rconn(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: waiting 4 seconds before reconnect 2021-06-01T13:48:20Z|00078|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection failed (Connection refused) 2021-06-01T13:48:20Z|00079|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: waiting 4 seconds before reconnect 2021-06-01T13:48:20Z|00080|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2021-06-01T13:48:20Z|00081|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory) 2021-06-01T13:48:20Z|00082|reconnect|INFO|unix:/var/run/openvswitch/db.sock: waiting 4 seconds before reconnect 2021-06-01T13:48:24Z|00011|rconn(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2021-06-01T13:48:24Z|00012|rconn(ovn_pinctrl0)|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection failed (Connection refused) 2021-06-01T13:48:24Z|00013|rconn(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: continuing to retry connections in the background but suppressing further logging 2021-06-01T13:48:24Z|00083|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2021-06-01T13:48:24Z|00084|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection failed (Connection refused) 2021-06-01T13:48:24Z|00085|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: continuing to retry connections in the background but suppressing further logging 2021-06-01T13:48:24Z|00086|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2021-06-01T13:48:24Z|00087|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory) 2021-06-01T13:48:24Z|00088|reconnect|INFO|unix:/var/run/openvswitch/db.sock: continuing to reconnect in the background but suppressing further logging 2021-06-01T13:48:32Z|00014|rconn(ovn_pinctrl0)|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection failed (Connection refused) 2021-06-01T13:48:32Z|00089|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection failed (Connection refused) 2021-06-01T13:48:40Z|00015|rconn(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2021-06-01T13:48:40Z|00090|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2021-06-01T13:48:40Z|00091|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2021-06-02T06:52:35Z|00092|binding|INFO|Claiming lport openshift-monitoring_prometheus-adapter-dc589ffcf-28khk for this chassis. 2021-06-02T06:52:35Z|00093|binding|INFO|openshift-monitoring_prometheus-adapter-dc589ffcf-28khk: Claiming dynamic 2021-06-02T06:52:38Z|00094|binding|INFO|Releasing lport openshift-monitoring_prometheus-adapter-5464769995-62q74 from this chassis. 2021-06-02T06:54:03Z|00095|binding|INFO|Claiming lport openshift-monitoring_prometheus-adapter-57675549b6-6bm4d for this chassis. 2021-06-02T06:54:03Z|00096|binding|INFO|openshift-monitoring_prometheus-adapter-57675549b6-6bm4d: Claiming dynamic 2021-06-02T06:54:06Z|00097|binding|INFO|Releasing lport openshift-monitoring_prometheus-adapter-dc589ffcf-28khk from this chassis. 2021-06-02T18:30:41Z|00098|binding|INFO|Claiming lport openshift-monitoring_prometheus-adapter-65c595cb47-kxgvt for this chassis. 2021-06-02T18:30:41Z|00099|binding|INFO|openshift-monitoring_prometheus-adapter-65c595cb47-kxgvt: Claiming dynamic 2021-06-02T18:30:44Z|00100|binding|INFO|Releasing lport openshift-monitoring_prometheus-adapter-57675549b6-6bm4d from this chassis.
also, whatever the problem is, it keeps appending the same two iptables rules to the node. just another symptom of the problem. these two: -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK full iptables: sh-4.4# iptables-save # Generated by iptables-save v1.8.4 on Thu Jun 3 03:36:11 2021 *nat :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :gcp-vips - [0:0] :gcp-vips-local - [0:0] :KUBE-MARK-MASQ - [0:0] :KUBE-MARK-DROP - [0:0] :KUBE-POSTROUTING - [0:0] :KUBE-KUBELET-CANARY - [0:0] :OVN-KUBE-NODEPORT - [0:0] :OVN-KUBE-SNAT-MGMTPORT - [0:0] -A PREROUTING -j OVN-KUBE-NODEPORT -A PREROUTING -m comment --comment "gcp LB vip DNAT" -j gcp-vips -A POSTROUTING -o ovn-k8s-mp0 -j OVN-KUBE-SNAT-MGMTPORT -A POSTROUTING -s 169.254.33.2/32 -j MASQUERADE -A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING -A OUTPUT -j OVN-KUBE-NODEPORT -A OUTPUT -m comment --comment "gcp LB vip DNAT for local clients" -j gcp-vips-local -A gcp-vips -d 35.225.193.67/32 -j REDIRECT -A gcp-vips -d 10.0.0.2/32 -j REDIRECT -A gcp-vips -d 35.194.54.242/32 -j REDIRECT -A gcp-vips-local -d 35.225.193.67/32 -j REDIRECT -A gcp-vips-local -d 35.194.54.242/32 -j REDIRECT -A gcp-vips-local -d 10.0.0.2/32 -j REDIRECT -A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000 -A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000 -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE --random-fully -A OVN-KUBE-NODEPORT -d 35.194.54.242/32 -p tcp -m tcp --dport 443 -j DNAT --to-destination 169.254.33.2:30286 -A OVN-KUBE-NODEPORT -p tcp -m tcp --dport 30286 -j DNAT --to-destination 169.254.33.2:30286 -A OVN-KUBE-NODEPORT -d 35.194.54.242/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 169.254.33.2:31373 -A OVN-KUBE-NODEPORT -p tcp -m tcp --dport 31373 -j DNAT --to-destination 169.254.33.2:31373 -A OVN-KUBE-SNAT-MGMTPORT -o ovn-k8s-mp0 -m comment --comment "OVN SNAT to Management Port" -j SNAT --to-source 10.129.0.2 COMMIT # Completed on Thu Jun 3 03:36:11 2021 # Generated by iptables-save v1.8.4 on Thu Jun 3 03:36:11 2021 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :KUBE-FIREWALL - [0:0] :KUBE-KUBELET-CANARY - [0:0] :OVN-KUBE-NODEPORT - [0:0] -A INPUT -p udp -m udp --dport 6081 -j ACCEPT -A INPUT -i ovn-k8s-gw0 -m comment --comment "from OVN to localhost" -j ACCEPT -A INPUT -j KUBE-FIREWALL -A INPUT -m comment --comment "gcp LB vip existing" -m addrtype ! --dst-type LOCAL -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -j OVN-KUBE-NODEPORT -A FORWARD -o ovn-k8s-gw0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A FORWARD -i ovn-k8s-gw0 -j ACCEPT -A FORWARD -p tcp -m tcp --dport 22624 --tcp-flags FIN,SYN,RST,ACK SYN -j REJECT --reject-with icmp-port-unreachable -A FORWARD -p tcp -m tcp --dport 22623 --tcp-flags FIN,SYN,RST,ACK SYN -j REJECT --reject-with icmp-port-unreachable -A OUTPUT -p tcp -m tcp --dport 22624 --tcp-flags FIN,SYN,RST,ACK SYN -j REJECT --reject-with icmp-port-unreachable -A OUTPUT -p tcp -m tcp --dport 22623 --tcp-flags FIN,SYN,RST,ACK SYN -j REJECT --reject-with icmp-port-unreachable -A OUTPUT -j KUBE-FIREWALL -A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP -A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP -A OVN-KUBE-NODEPORT -d 35.194.54.242/32 -p tcp -m tcp --dport 443 -j ACCEPT -A OVN-KUBE-NODEPORT -p tcp -m tcp --dport 30286 -j ACCEPT -A OVN-KUBE-NODEPORT -d 35.194.54.242/32 -p tcp -m tcp --dport 80 -j ACCEPT -A OVN-KUBE-NODEPORT -p tcp -m tcp --dport 31373 -j ACCEPT COMMIT # Completed on Thu Jun 3 03:36:11 2021 # Generated by iptables-save v1.8.4 on Thu Jun 3 03:36:11 2021 *mangle :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] :KUBE-KUBELET-CANARY - [0:0] COMMIT # Completed on Thu Jun 3 03:36:11 2021 # Generated by iptables-save v1.8.4 on Thu Jun 3 03:36:11 2021 *raw :PREROUTING ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A PREROUTING -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK -A OUTPUT -p udp -m udp --dport 6081 -j NOTRACK COMMIT # Completed on Thu Jun 3 03:36:11 2021
some more details: the ovnkube-node pod that has the ssl key errors has been upgraded to ovn 20.12 and openssl 1.1 in 4.6. in 4.5 it's 20.06 and 1.0 respectively. broken pod: [root@ci-ln-6qxivbb-f76d1-qvpsn-master-0 ~]# rpm -qa | egrep ovn ovn2.13-central-20.12.0-24.el8fdp.x86_64 ovn2.13-vtep-20.12.0-24.el8fdp.x86_64 ovn2.13-20.12.0-24.el8fdp.x86_64 ovn2.13-host-20.12.0-24.el8fdp.x86_64 [root@ci-ln-6qxivbb-f76d1-qvpsn-master-0 ~]# rpm -qa | grep ssl openssl-libs-1.1.1c-18.el8_2.x86_64 openssl-1.1.1c-18.el8_2.x86_64 openssl-pkcs11-0.4.10-2.el8.x86_64 non-upgraded 4.5 pod: [root@ci-ln-6qxivbb-f76d1-qvpsn-master-0 ~]# rpm -qa | egrep ovn ovn2.13-central-20.12.0-24.el8fdp.x86_64 ovn2.13-vtep-20.12.0-24.el8fdp.x86_64 ovn2.13-20.12.0-24.el8fdp.x86_64 ovn2.13-host-20.12.0-24.el8fdp.x86_64 [root@ci-ln-6qxivbb-f76d1-qvpsn-master-0 ~]# rpm -qa | grep ssl openssl-libs-1.1.1c-18.el8_2.x86_64 openssl-1.1.1c-18.el8_2.x86_64 openssl-pkcs11-0.4.10-2.el8.x86_64 researching you can see that ssl 1.1 now requires keys to be at least 2k in size. The keys that seem to be in use ARE 2k in size, however. https://github.com/dask/distributed/issues/2405 [root@ci-ln-6qxivbb-f76d1-qvpsn-master-0 ~]# ps -elf | grep 'ovn-controller.*key' 4 S root 104952 104939 0 80 0 - 66061 x64_sy 16:57 ? 00:00:01 ovn-controller unix:/var/run/openvswitch/db.sock -vfile:off --no-chdir --pidfile=/var/run/ovn/ovn-controller.pid -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt -vconsole:info [root@ci-ln-6qxivbb-f76d1-qvpsn-master-0 ~]# cat /ovn-cert/tls.key -----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEArZyPSHC9sXwghMUicl1ls/zzIjcLXH41hwL8Q/V8JFFMbtPd fyNhhBYsuutykMWoq+ovqh8WsiAtWg4AiDwy/1GKZ425t8bjFk0QuIEfFh9+q9Hj xGKeH+dq3f8zT7SGzweis/F8YqAnEc1IUej8YzDgv1Uw8fpbS6H9xEOEkKQUJnhZ LiLQSaqDr3h+6Hp5z+YO7dtU8x3y+MqCHzbFkxGUE+R27hpvCo16tdgkxlGq7tbm MiI5Qum8+jtutNW9uxAA8pt9xmKfe74V0TbKWhvGmmUDHuJ9Zh0d3caChCiB9itF 2gwS+i+HQBjnXwFHBN3USSezW7hnIPVFROOWgQIDAQABAoIBAFmawXURA0b0mtAL 8yB1xgUkm9Rf9pQxa60YRF1K+VzBPSLCZMK23yDOVR3QYZwI+Gpqf1ldnFgpNV8S dQHbmDKgwj96LY2FsDUVZ3ji48mGTdmeheEZSSWMSmLz0Wbk9OMrvLUFAT7iC+gr PJVSG9mSI518+bsajT43CIPgmS4RN9G+A6bI61Lv98FU3LALgr3Vp6BeY9T+3UKG uYTLdgqn4CRdBoeb7Yaaa3EXK5eyjODOW/2hKNb1pLnhDGhIvksQAog21vcsQkND J9IOP761VbLVEjsGeF0NyAAc7AX7M+8g9wxErbC+BKy6/CB/VPqHrjkMwzRU14ar 0ZjcidUCgYEA1qJqqN5LDHIRQv7M+MNKaPGKYo0D5QEl3RsdNMy0JIJwZxC/dqc9 wZ2c/y6qBPjjFIKrlQpK44w8j/Fhzw4u8QXEwJ/bg/odMuvjrMT1OQet7sogMEw5 WTh7jB7Pr+zIBxIW1r+RT18AdfQqvhLgpnvMloaiuogo8jdg9qn0zCMCgYEAzxIq e75YH6mmHRn02pBDdSl2rH3byZ6ud0phq8d24mELSx4QXtqX07W7ex2RCaDOzMSt 8OTG1vzqMciWu3RjjtK3q9KXiWCYJozziWS61FznMO4Hht+kiWyD/eIyC0d0Ba9I ndYZ9wmhmLRDSKgzYq4h2R4kZc9nra3JX9kuewsCgYAcdbjS33bwFYf4bDTdn3eU ZsLEwpILoZVVtiyvPZ19YdZptCzPHOnxbGbO528f7aiepZz+zDiV55h4NbeUbvsG do9aQ1gmLdoFUIYF+K/TehM0IJUYEmNgUz7+m29KkokLdviUTR6FecFj4pZwUax8 Jto/82ZMeZWEWJeA3ZRaJQKBgQCjz0YmcFT8b1PG79LA2KFU3UFRgZtOq6ZvHjuY DgphWc36VcUzlI/JXpn68k/bowgV+31y9TjLuHZ4fauojZF42f+NAOL6PZz/2j+K sFIQT5JhJdx0eRWL1XmxbbCTNap2GM1Ed6xAvrDEwc38e3tzFDgYr3yxwdrODY4h sgjQMwKBgQC74mirwgxv0bGLLte4/4KLhZDeqCL2zf0BwHdZfU7l69jtKVu6vKFP GTCkF+mDkK2GAGvj2UKnHMpEvCL6QPour0zHknGaba8zUrNxanGL6HkZQPU0K3x1 QMEvfYkEEED+iqO6Q1uLrNZDymO3PbLiJQVsvPaXUCjS3HNQdHryjg== -----END RSA PRIVATE KEY----- then using ssh-keygen locally: sh-4.4# ssh-keygen -l -f /tmp/tls.key 2048 SHA256:zidB0oFgYld5MXCdnaPkcy7MdgDc3x+DyxjPLEleCsc no comment (RSA) There is an openssl client you can use on the ovn-controller container to simulate the connection that seems to be failing with the too small key, but that test passes with the same key. example: openssl s_client -key /ovn-cert/tls.key -cert /ovn-cert/tls.crt -CAfile /ovn-ca/ca-bundle.crt -connect 10.0.0.4:9642 the output looks like this: CONNECTED(00000003) depth=1 CN = openshift-ovn-kubernetes_ovn-ca@1622658657 verify return:1 depth=0 CN = ovn verify return:1 --- Certificate chain 0 s:/CN=ovn i:/CN=openshift-ovn-kubernetes_ovn-ca@1622658657 1 s:/CN=openshift-ovn-kubernetes_ovn-ca@1622658657 i:/CN=openshift-ovn-kubernetes_ovn-ca@1622658657 --- Server certificate -----BEGIN CERTIFICATE----- MIIDVjCCAj6gAwIBAgIIMtYRa1SYBvswDQYJKoZIhvcNAQELBQAwNTEzMDEGA1UE Awwqb3BlbnNoaWZ0LW92bi1rdWJlcm5ldGVzX292bi1jYUAxNjIyNjU4NjU3MB4X DTIxMDYwMjE4MzA1N1oXDTIxMTIwMjA2MzA1OFowDjEMMAoGA1UEAxMDb3ZuMIIB IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvG3ciLTmivDE2oUOd+owLMnc VPTmXvA2djQ4bJuyQh1eyhsosQguLBcm8gvkOV5idlR6d5qZALNPVptp9IdEiJ/L Xd9tgzTSKedznQ4ONGXun1j4cx3C1xWS2skiyMp8L7aHO3V0iH/hOpvDABynVNNy Zdif7imS5poYAgANW4EVtZndvOJgcY0EiPaOCKGyI1iFW/NdKrdAqGptIw2lKNwF /yJkcrnbfHt3pAbAW7lpvmGurkpoeNTl1WVhuSGr4cL+BXjTt46n9lkOib79tx5L zsMUMSXkl17iQQPaGj4Xga2oiws0xRsagP6ipPRywKw5hsLs82R/UMeOMrJ6RQID AQABo4GQMIGNMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYI KwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUwOtsGrYEuVPVTOtATpYm DWtaoMQwHwYDVR0jBBgwFoAUW066+DRUsAQSLTPWZVKUs916ZGAwDgYDVR0RBAcw BYIDb3ZuMA0GCSqGSIb3DQEBCwUAA4IBAQBW1RiCMvpZie5B6GcK7KDkirrz5xtq s2a96WRPS5DefCxyOKTdf85ju6Rb/beBt/Cret91811aHL1suam0SiSe3mPJgBfW 88DxS4GLqRssP7vgmXJRwXNOGk0Gh1Xk6IsF0lSJs6VEvfmlgixoiH98mfQHrTAx TpfjFH4g9APq8qMVOMuhFxrIBJVoqUkhslVKPYzcSKAakO7wAhcqf2sE0AOclK/p tYKmz89qtu5Prki+jCOfse002nHa9WBAkFXy+alCRFKh1I9w5M7u8MoGWmFknefG 8HbeobJZcvkuxgVox8M/1FA7+fScmGGMVWjOe0nJMD+Ka3ZHleVQs2aK -----END CERTIFICATE----- subject=/CN=ovn issuer=/CN=openshift-ovn-kubernetes_ovn-ca@1622658657 --- Acceptable client certificate CA names /CN=openshift-ovn-kubernetes_ovn-ca@1622658657 Client Certificate Types: RSA fixed DH, DSS fixed DH, RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1 Shared Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1 Peer signing digest: SHA512 Server Temp Key: DH, 1024 bits --- SSL handshake has read 3515 bytes and written 2468 bytes --- New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : DHE-RSA-AES256-GCM-SHA384 Session-ID: 1E1E2DC88125B83DDA0A7DE6177BCF81EECCE904A0489E4C48F8B79E3C29F9F2 Session-ID-ctx: Master-Key: 2AF2E91DA56398DC052464A1C0B3638E91AB0FCFE9F11EB15DB508C2FB8A60A761C2825937DA75F640D933398EC11175 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None TLS session ticket lifetime hint: 300 (seconds) TLS session ticket: 0000 - aa bc 67 7a 59 67 8a 28-e7 31 f8 dd c0 8a b5 dd ..gzYg.(.1...... 0010 - 55 bb 3e be b8 d9 4e 87-ed d3 81 93 62 0f d7 1b U.>...N.....b... 0020 - a8 af c4 13 a5 ff 93 5f-41 2b e3 16 59 f4 cd 8b ......._A+..Y... 0030 - 9b 58 3a 77 85 9d e3 30-8e 94 6b 8c 72 fa be bf .X:w...0..k.r... 0040 - 16 2c 1d 83 06 92 c0 70-60 86 e9 5c 22 fb 01 95 .,.....p`..\"... 0050 - 32 63 b7 5a 3a ba a2 6d-5e c1 0a c1 51 6d 7d a1 2c.Z:..m^...Qm}. 0060 - 44 3b 43 18 83 a8 08 89-99 e6 59 43 e2 cf e1 c9 D;C.......YC.... 0070 - 64 fc 3b ae 8d 20 22 86-dc de 1a 8a 13 2d fe b0 d.;.. "......-.. 0080 - 09 6e 72 07 46 a6 50 79-49 5d a8 2d ed 19 7e e0 .nr.F.PyI].-..~. 0090 - d9 92 7e ab a7 58 1b cc-d3 21 e3 5c 93 7c 39 a4 ..~..X...!.\.|9. 00a0 - 30 8c 05 e8 23 81 b0 ba-4c 12 9d ff 00 2b c4 88 0...#...L....+.. 00b0 - 1a 69 09 c8 4f 2c 8a bb-7d 2e 00 a2 c9 86 9b a2 .i..O,..}....... 00c0 - be 55 05 09 0d 31 55 f9-bd 57 08 6d bb 87 78 4e .U...1U..W.m..xN 00d0 - cd 36 9d 96 57 2b f9 8f-bb 37 d5 51 68 61 ba 3b .6..W+...7.Qha.; 00e0 - 00 0b 31 6b 36 43 1c 04-b6 25 20 31 1f ec 77 21 ..1k6C...% 1..w! 00f0 - 8a 00 ec bc 19 2a 81 0e-2d 82 27 d0 fa 97 fb a8 .....*..-.'..... 0100 - fe e5 bb c3 7c 87 c4 8d-7a 56 6b f2 e2 d3 40 5b ....|...zVk...@[ 0110 - 80 3f ed 4b 5d 22 7d 82-5e 4c 3f d9 cf 82 a5 2a .?.K]"}.^L?....* 0120 - 2a 43 1a a9 75 97 96 6c-ae 4b 9e 22 bd b9 8d 28 *C..u..l.K."...( 0130 - fc a8 19 a0 b3 06 a4 35-e5 4d 52 a2 f9 11 23 57 .......5.MR...#W 0140 - 12 4e 85 c2 b3 27 22 17-ed fb 35 aa e6 85 a5 2b .N...'"...5....+ 0150 - f1 8e 28 4e 4a a3 38 8c-cf 7c db ff 8a 72 05 d7 ..(NJ.8..|...r.. 0160 - 8b 71 88 fa 48 89 7d 58-70 cf 1f 02 75 db b3 1d .q..H.}Xp...u... 0170 - ea 5e 0d cf 43 d6 7f 81-a6 a4 48 03 ed 9e bd 28 .^..C.....H....( 0180 - 03 af a7 10 21 60 29 a0-b9 60 56 0b e5 e7 c2 f2 ....!`)..`V..... 0190 - 1e 2f 1d f3 18 0b 33 a4-69 f0 bd 4b 9d 78 98 8e ./....3.i..K.x.. 01a0 - 4c e8 32 94 65 c1 57 9d-3f 3c 34 6f 31 37 a3 ee L.2.e.W.?<4o17.. 01b0 - d2 98 4a b1 34 15 21 76-93 51 6d 91 bc 3f 31 94 ..J.4.!v.Qm..?1. 01c0 - ba 59 c7 4f 90 66 9d c0-ae a3 aa 1c c4 97 3b 90 .Y.O.f........;. 01d0 - c0 ef 2f 22 46 77 28 29-9b 19 3c 56 09 9d 3b a6 ../"Fw()..<V..;. 01e0 - e5 a4 d0 2b 72 2b b5 e1-33 ab 87 53 6c 96 10 46 ...+r+..3..Sl..F 01f0 - a2 6e 1e 3b 60 c3 ce 0c-f2 3c fc 57 6a 59 65 69 .n.;`....<.WjYei 0200 - dd a7 93 f9 e7 32 f6 4d-94 a8 b4 88 67 31 2b 71 .....2.M....g1+q 0210 - ce 63 71 5b 6b 86 a9 8e-3a 8c 34 11 e3 ee 0d 9b .cq[k...:.4..... 0220 - ba 05 23 34 43 de ea 6e-02 8b ce d0 83 3f 3b 43 ..#4C..n.....?;C 0230 - 14 8a a9 97 9b 55 30 16-70 19 4a 66 b7 e4 d7 51 .....U0.p.Jf...Q 0240 - 4e 9f 2d 31 29 03 20 19-84 8d e0 b8 44 3a 68 bd N.-1). .....D:h. 0250 - cc f2 48 91 36 20 93 16-4b 85 f7 ee 7f 1f ec 23 ..H.6 ..K......# 0260 - d2 06 f8 ca 97 1e 50 53-ba ec 4f 7f 4a 6a db 05 ......PS..O.Jj.. 0270 - de ee fb d7 db ed 49 79-8c 4f 10 3d 1b 06 56 1d ......Iy.O.=..V. 0280 - 24 c1 ee ce ac 5b 9f 7f-5a bf 4d 97 2c 5b 4e e9 $....[..Z.M.,[N. 0290 - 2b cf d8 a2 df bf e8 7f-64 d0 26 9e 45 77 1f f8 +.......d.&.Ew.. 02a0 - 26 ac f1 be f7 f3 da 26-d4 cb cc 5f 4c a1 36 ae &......&..._L.6. 02b0 - 29 27 b5 8f b2 41 35 27-7a 3f 1a f9 e7 44 85 f7 )'...A5'z?...D.. 02c0 - ac 95 95 63 d8 13 48 df-02 56 03 66 84 28 8b 3d ...c..H..V.f.(.= 02d0 - e9 a6 18 bb 3c bb f2 9e-3b 6f d8 0c d8 87 6c 4c ....<...;o....lL 02e0 - 52 e6 9e 18 90 b2 e5 34-fe b5 4b 73 b7 8f 05 ad R......4..Ks.... 02f0 - 53 12 70 8b 85 6b 9a 31-25 97 38 1a a6 67 80 03 S.p..k.1%.8..g.. 0300 - 4e 5a 1a 9b e0 52 47 ea-6c 53 38 8b 3a 41 08 d1 NZ...RG.lS8.:A.. 0310 - 47 75 a3 fc 2e 3d 72 15-ea 7e 23 ec 2c 46 4b 09 Gu...=r..~#.,FK. 0320 - af ed 54 8c fb 2d e0 95-2e d9 0d 3c 59 59 ac 27 ..T..-.....<YY.' 0330 - d9 4d 78 4b 31 34 7b a8-4f 3d b6 a7 27 0e 94 91 .MxK14{.O=..'... 0340 - c6 d2 0d c6 19 81 4d ea-02 af fc a2 83 a3 12 44 ......M........D 0350 - 13 68 08 b8 f8 45 5d bf-54 dc 2d 79 28 ac 7d f8 .h...E].T.-y(.}. 0360 - 0a e3 17 4f a3 00 1d 58-72 e9 60 5e 22 d7 84 75 ...O...Xr.`^"..u 0370 - 96 4c 93 6d bd 02 49 43-e3 14 a2 9b df d0 c4 f1 .L.m..IC........ 0380 - 93 d6 d7 cd d1 e9 05 7e-dd c1 ac 4a 04 67 18 c7 .......~...J.g.. 0390 - 24 84 a5 f4 2f 35 94 fd-44 ff e1 18 3e 18 3e 1f $.../5..D...>.>. 03a0 - 3c f5 ff c8 e9 98 07 27-bc e8 e2 d6 33 2b 66 29 <......'....3+f) 03b0 - 5f f1 db bc 21 4d f8 88-86 c0 65 cc 0b bb 5e 5d _...!M....e...^] 03c0 - 09 d1 57 99 fb bb 8b bc-a0 c4 ce ac f0 23 53 07 ..W..........#S. 03d0 - ec f8 12 73 95 ec 3c 0b-2b aa 4b 10 f8 f1 f2 94 ...s..<.+.K..... 03e0 - bc 93 88 3d 75 95 b3 69-ce 62 65 28 5b bf 0d 20 ...=u..i.be([.. 03f0 - ea 30 17 d9 01 26 6d 31-a1 39 44 51 eb e8 1c ba .0...&m1.9DQ.... Start Time: 1622743981 Timeout : 300 (sec) Verify return code: 0 (ok) --- As a shot in the dark, I noticed a PR in the ovn-controller that seemed related to this ssl connection handling that didn't exist in 4.6. I created a backport PR (https://github.com/openshift/ovn-kubernetes/pull/563) and tried to upgrade to an image built with it, and it still failed with the same problem. I did this with the following steps (for posterity): - build release image with PR using cluster-bot - create a 4.5 cluster with cluster-bot - update the cluster pull-secret with the registry auth needed to pull the release image. (extract and modify pull secret, delete cluster pull-secret, add modified pull secret) - oc adm upgrade --to-image=<PR release image> --allow-explicit-upgrade --force
@vpickard, assigning the to you for now. I will be on PTO for 2 weeks soon and maybe there is someone more expert in the SSL world that might have a better chance of guessing what is going on here? You can give it back to me and I can try to take it back up when I'm back, if needed.
I took this over from Jamo while he is on PTO. I managed to reproduce the issue with openssl connect test: ---- kubectl -n openshift-ovn-kubernetes exec -ti ovnkube-node-ktf9q -c ovn-controller -- openssl s_client -key /ovn-cert/tls.key -cert /ovn-cert/tls.crt -CAfile /ovn-ca/ca-bundle.crt -cipher 'DH' -connect 10.0.0.5:9642 CONNECTED(00000003) Can't use SSL_get_servername depth=1 CN = openshift-ovn-kubernetes_ovn-ca@1623680099 verify return:1 depth=0 CN = ovn verify return:1 140017101793088:error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small:ssl/statem/statem_clnt.c:2150: --- Certificate chain 0 s:CN = ovn i:CN = openshift-ovn-kubernetes_ovn-ca@1623680099 1 s:CN = openshift-ovn-kubernetes_ovn-ca@1623680099 i:CN = openshift-ovn-kubernetes_ovn-ca@1623680099 --- Server certificate -----BEGIN CERTIFICATE----- MIIDVjCCAj6gAwIBAgIIKIjTFzKNSw0wDQYJKoZIhvcNAQELBQAwNTEzMDEGA1UE Awwqb3BlbnNoaWZ0LW92bi1rdWJlcm5ldGVzX292bi1jYUAxNjIzNjgwMDk5MB4X DTIxMDYxNDE0MTQ1OFoXDTIxMTIxNDAyMTQ1OVowDjEMMAoGA1UEAxMDb3ZuMIIB IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAy+oHHVBeP27v4/7yyIPwh5F0 Td8rzn8gy/xZR/BA6b2hhCmL3G6v8NKA7KWdefL8B8ermozaPMZzlNc9jOdKqsab swBQckq8+QRVuJ52tsX4kTGbKWbLGQ8FT7M2llhamiIbPD9BCmJfQXIFjsVyBo0S Da77JzQi/Cq+1/Zu3Yqangx/QuIvLZY/nu785Kqn79XaBmkrys+v4E5+5D1aiqxw +e9HqA96KiiJb+EDJU/Q0vI8YhhoA158YOAF6HoW0Fupn/7SjUbyTaJ1EzKYmKtA hV4n7NAz2gI5x52GJ3isFQMmOEy0dK+fT9ZI2XjBD79b92WN0xBx1ZPjaXaKbwID AQABo4GQMIGNMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYI KwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUQLcAvyesssH5P5Qnc6jF LZJ3V7cwHwYDVR0jBBgwFoAUtluvQNwRUwq/ciMKzqNuKqXkI1QwDgYDVR0RBAcw BYIDb3ZuMA0GCSqGSIb3DQEBCwUAA4IBAQCuf3ZFjaUMFi4l8uwqdzC1GFznu6/P AetewNYkMfvUAIOyu53og0xjJ56IqQOl/InvUVDIOXt4EdFy8Q/caSrK0VRoSDl/ Ux3vv61fB2KkvkONrpO34LeSXsODvh8/pyWtkg1Ye9ftRzB8LvHOLUdjIYnY7FYH 3X7w4Lc+WIyltvhGiBVxYHBMaVv7kel/lcPeSz3tGKoL+sbppxWj4EkQOPbAtxcD YaVoBR0IcpJZ+MEhmjKUn2DRrdlhRHYzvGnzXP+sSxawq77VTuRwCmZEF8JJScum ZdHJGUZ7LTQu9PBnPDUqTM79J4t+BM2r/owbW1mElwB62ttUJ9y53izE -----END CERTIFICATE----- subject=CN = ovn issuer=CN = openshift-ovn-kubernetes_ovn-ca@1623680099 --- No client certificate CA names sent --- SSL handshake has read 2310 bytes and written 310 bytes Verification: OK --- New, (NONE), Cipher is (NONE) Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: PSK identity: None PSK identity hint: None SRP username: None Start Time: 1623703779 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no --- command terminated with exit code 1 --- I guess that the test that Jamo did was not attempting the connection from the same origin as the original problem. The issue is not related with the key/certs of the endpoints but with the temporary key generated for the session when using DH based cipher along with stricter security settings enforced by the new openssl version of the upgraded ovn node pod (OpenSSL 1.1.1c FIPS 28 May 2019), specifically in this case the server generating a 1024 bit DH key while the client requires at least a 2048 bit DH key. I believe the root cause of the issue is ovsdb server generating a 1024 DH key @ https://github.com/openvswitch/ovs/blob/2afe31169ae7b9040d1fc78dfc87dc1ad24f6337/lib/stream-ssl.c#L1081. ovsdb server should disregard the keylength requested as it does with all the other parameters, as per https://www.openssl.org/docs/man1.0.2/man3/SSL_set_tmp_dh.html: --- Previous versions of the callback used is_export and keylength parameters to control parameter generation for export and non-export cipher suites. Modern servers that do not support export ciphersuites are advised to either use SSL_CTX_set_tmp_dh() or alternatively, use the callback but ignore keylength and is_export and simply supply at least 2048-bit parameters in the callback. --- A workaround could be to disallow DH based ciphers from negotiation using the --ssl-ciphers argument to ovn-controller. Testing this option right now.
Recap This happens on 4.5 to 4.6 upgrade when ovn-controller from 4.6 ovnkube-node tries to connect to sbdb on 4.5 ovnkube-master. The updated version of openssl in 4.6 requires by default stricter security and rejects the 1024 bit dh params used by ovn/ovs for the key exchange. These are the ciphers supported by ovn/ovs in 4.5: ------ kubectl -n openshift-ovn-kubernetes exec -ti multitool -- nmap --script ssl-enum-ciphers -p 9642 10.0.0.6 Starting Nmap 7.91 ( https://nmap.org ) at 2021-06-16 19:06 UTC Nmap scan report for 10-0-0-6.ovnkube-db.openshift-ovn-kubernetes.svc.cluster.local (10.0.0.6) Host is up (0.0017s latency). PORT STATE SERVICE 9642/tcp open unknown | ssl-enum-ciphers: ... | TLSv1.2: | ciphers: | TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 1024) - A | TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (dh 1024) - A | TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 1024) - A | TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 1024) - A | TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (dh 1024) - A | TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 1024) - A | TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (dh 1024) - A | TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (dh 1024) - A | TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A | compressors: | NULL | cipher preference: client | warnings: | Key exchange (dh 1024) of lower strength than certificate key |_ least strength: A ----- These are the ciphers supported by ovn/ovs in 4.6: ----- kubectl -n openshift-ovn-kubernetes exec -ti multitool -- nmap --script ssl-enum-ciphers -p 9642 10.0.0.3 Starting Nmap 7.91 ( https://nmap.org ) at 2021-06-17 08:46 UTC Nmap scan report for 10.0.0.3 Host is up (0.0018s latency). PORT STATE SERVICE 9642/tcp open unknown | ssl-enum-ciphers: | TLSv1.2: | ciphers: | TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 1024) - A | TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (dh 1024) - A | TLS_DHE_RSA_WITH_AES_128_CCM (dh 1024) - A | TLS_DHE_RSA_WITH_AES_128_CCM_8 (dh 1024) - A | TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 1024) - A | TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 1024) - A | TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (dh 1024) - A | TLS_DHE_RSA_WITH_AES_256_CCM (dh 1024) - A | TLS_DHE_RSA_WITH_AES_256_CCM_8 (dh 1024) - A | TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 1024) - A | TLS_DHE_RSA_WITH_ARIA_128_GCM_SHA256 (dh 1024) - A | TLS_DHE_RSA_WITH_ARIA_256_GCM_SHA384 (dh 1024) - A | TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (dh 1024) - A | TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 (dh 1024) - A | TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (dh 1024) - A | TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256 (dh 1024) - A | TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (dh 1024) - A | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (secp256r1) - A | TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_128_CCM (rsa 2048) - A | TLS_RSA_WITH_AES_128_CCM_8 (rsa 2048) - A | TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_256_CCM (rsa 2048) - A | TLS_RSA_WITH_AES_256_CCM_8 (rsa 2048) - A | TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A | TLS_RSA_WITH_ARIA_128_GCM_SHA256 (rsa 2048) - A | TLS_RSA_WITH_ARIA_256_GCM_SHA384 (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256 (rsa 2048) - A | compressors: | NULL | cipher preference: client | warnings: | Key exchange (dh 1024) of lower strength than certificate key |_ least strength: A ---- This is the cipher list ordered by preference in ovn/ovs 4.5 ---- ❯ kubectl -n openshift-ovn-kubernetes exec -ti ovnkube-master-rtnl2 -- openssl ciphers -V 'HIGH:!aNULL:!MD5' Defaulted container "northd" out of: northd, nbdb, sbdb, ovnkube-master 0xC0,0x30 - ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 0xC0,0x2C - ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESGCM(256) Mac=AEAD 0xC0,0x28 - ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AES(256) Mac=SHA384 0xC0,0x24 - ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AES(256) Mac=SHA384 0x00,0xA5 - DH-DSS-AES256-GCM-SHA384 TLSv1.2 Kx=DH/DSS Au=DH Enc=AESGCM(256) Mac=AEAD 0x00,0xA3 - DHE-DSS-AES256-GCM-SHA384 TLSv1.2 Kx=DH Au=DSS Enc=AESGCM(256) Mac=AEAD 0x00,0xA1 - DH-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH/RSA Au=DH Enc=AESGCM(256) Mac=AEAD 0x00,0x9F - DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH Au=RSA Enc=AESGCM(256) Mac=AEAD 0x00,0x6B - DHE-RSA-AES256-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AES(256) Mac=SHA256 0x00,0x6A - DHE-DSS-AES256-SHA256 TLSv1.2 Kx=DH Au=DSS Enc=AES(256) Mac=SHA256 0x00,0x69 - DH-RSA-AES256-SHA256 TLSv1.2 Kx=DH/RSA Au=DH Enc=AES(256) Mac=SHA256 0x00,0x68 - DH-DSS-AES256-SHA256 TLSv1.2 Kx=DH/DSS Au=DH Enc=AES(256) Mac=SHA256 0xC0,0x32 - ECDH-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(256) Mac=AEAD 0xC0,0x2E - ECDH-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(256) Mac=AEAD 0xC0,0x2A - ECDH-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(256) Mac=SHA384 0xC0,0x26 - ECDH-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256) Mac=SHA384 0x00,0x9D - AES256-GCM-SHA384 TLSv1.2 Kx=RSA Au=RSA Enc=AESGCM(256) Mac=AEAD 0x00,0x3D - AES256-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA256 0xC0,0x2F - ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD 0xC0,0x2B - ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESGCM(128) Mac=AEAD 0xC0,0x27 - ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AES(128) Mac=SHA256 0xC0,0x23 - ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AES(128) Mac=SHA256 0x00,0xA4 - DH-DSS-AES128-GCM-SHA256 TLSv1.2 Kx=DH/DSS Au=DH Enc=AESGCM(128) Mac=AEAD 0x00,0xA2 - DHE-DSS-AES128-GCM-SHA256 TLSv1.2 Kx=DH Au=DSS Enc=AESGCM(128) Mac=AEAD 0x00,0xA0 - DH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH/RSA Au=DH Enc=AESGCM(128) Mac=AEAD 0x00,0x9E - DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AESGCM(128) Mac=AEAD 0x00,0x67 - DHE-RSA-AES128-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AES(128) Mac=SHA256 0x00,0x40 - DHE-DSS-AES128-SHA256 TLSv1.2 Kx=DH Au=DSS Enc=AES(128) Mac=SHA256 0x00,0x3F - DH-RSA-AES128-SHA256 TLSv1.2 Kx=DH/RSA Au=DH Enc=AES(128) Mac=SHA256 0x00,0x3E - DH-DSS-AES128-SHA256 TLSv1.2 Kx=DH/DSS Au=DH Enc=AES(128) Mac=SHA256 0xC0,0x31 - ECDH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(128) Mac=AEAD 0xC0,0x2D - ECDH-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(128) Mac=AEAD 0xC0,0x29 - ECDH-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(128) Mac=SHA256 0xC0,0x25 - ECDH-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128) Mac=SHA256 0x00,0x9C - AES128-GCM-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AESGCM(128) Mac=AEAD 0x00,0x3C - AES128-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA256 ----- This is the cipher list ordered by preference in ovn/ovs 4.6 ----- 0xC0,0x2C - ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESGCM(256) Mac=AEAD 0xC0,0x30 - ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 0x00,0xA3 - DHE-DSS-AES256-GCM-SHA384 TLSv1.2 Kx=DH Au=DSS Enc=AESGCM(256) Mac=AEAD 0x00,0x9F - DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH Au=RSA Enc=AESGCM(256) Mac=AEAD 0xCC,0xA9 - ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH Au=ECDSA Enc=CHACHA20/POLY1305(256) Mac=AEAD 0xCC,0xA8 - ECDHE-RSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH Au=RSA Enc=CHACHA20/POLY1305(256) Mac=AEAD 0xCC,0xAA - DHE-RSA-CHACHA20-POLY1305 TLSv1.2 Kx=DH Au=RSA Enc=CHACHA20/POLY1305(256) Mac=AEAD 0xC0,0xAF - ECDHE-ECDSA-AES256-CCM8 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESCCM8(256) Mac=AEAD 0xC0,0xAD - ECDHE-ECDSA-AES256-CCM TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESCCM(256) Mac=AEAD 0xC0,0xA3 - DHE-RSA-AES256-CCM8 TLSv1.2 Kx=DH Au=RSA Enc=AESCCM8(256) Mac=AEAD 0xC0,0x9F - DHE-RSA-AES256-CCM TLSv1.2 Kx=DH Au=RSA Enc=AESCCM(256) Mac=AEAD 0xC0,0x5D - ECDHE-ECDSA-ARIA256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=ECDSA Enc=ARIAGCM(256) Mac=AEAD 0xC0,0x61 - ECDHE-ARIA256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=ARIAGCM(256) Mac=AEAD 0xC0,0x57 - DHE-DSS-ARIA256-GCM-SHA384 TLSv1.2 Kx=DH Au=DSS Enc=ARIAGCM(256) Mac=AEAD 0xC0,0x53 - DHE-RSA-ARIA256-GCM-SHA384 TLSv1.2 Kx=DH Au=RSA Enc=ARIAGCM(256) Mac=AEAD 0xC0,0x2B - ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESGCM(128) Mac=AEAD 0xC0,0x2F - ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD 0x00,0xA2 - DHE-DSS-AES128-GCM-SHA256 TLSv1.2 Kx=DH Au=DSS Enc=AESGCM(128) Mac=AEAD 0x00,0x9E - DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AESGCM(128) Mac=AEAD 0xC0,0xAE - ECDHE-ECDSA-AES128-CCM8 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESCCM8(128) Mac=AEAD 0xC0,0xAC - ECDHE-ECDSA-AES128-CCM TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESCCM(128) Mac=AEAD 0xC0,0xA2 - DHE-RSA-AES128-CCM8 TLSv1.2 Kx=DH Au=RSA Enc=AESCCM8(128) Mac=AEAD 0xC0,0x9E - DHE-RSA-AES128-CCM TLSv1.2 Kx=DH Au=RSA Enc=AESCCM(128) Mac=AEAD 0xC0,0x5C - ECDHE-ECDSA-ARIA128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=ECDSA Enc=ARIAGCM(128) Mac=AEAD 0xC0,0x60 - ECDHE-ARIA128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=ARIAGCM(128) Mac=AEAD 0xC0,0x56 - DHE-DSS-ARIA128-GCM-SHA256 TLSv1.2 Kx=DH Au=DSS Enc=ARIAGCM(128) Mac=AEAD 0xC0,0x52 - DHE-RSA-ARIA128-GCM-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=ARIAGCM(128) Mac=AEAD 0xC0,0x24 - ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AES(256) Mac=SHA384 0xC0,0x28 - ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AES(256) Mac=SHA384 0x00,0x6B - DHE-RSA-AES256-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AES(256) Mac=SHA256 0x00,0x6A - DHE-DSS-AES256-SHA256 TLSv1.2 Kx=DH Au=DSS Enc=AES(256) Mac=SHA256 0xC0,0x73 - ECDHE-ECDSA-CAMELLIA256-SHA384 TLSv1.2 Kx=ECDH Au=ECDSA Enc=Camellia(256) Mac=SHA384 0xC0,0x77 - ECDHE-RSA-CAMELLIA256-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=Camellia(256) Mac=SHA384 0x00,0xC4 - DHE-RSA-CAMELLIA256-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=Camellia(256) Mac=SHA256 0x00,0xC3 - DHE-DSS-CAMELLIA256-SHA256 TLSv1.2 Kx=DH Au=DSS Enc=Camellia(256) Mac=SHA256 0xC0,0x23 - ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AES(128) Mac=SHA256 0xC0,0x27 - ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AES(128) Mac=SHA256 0x00,0x67 - DHE-RSA-AES128-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AES(128) Mac=SHA256 0x00,0x40 - DHE-DSS-AES128-SHA256 TLSv1.2 Kx=DH Au=DSS Enc=AES(128) Mac=SHA256 0xC0,0x72 - ECDHE-ECDSA-CAMELLIA128-SHA256 TLSv1.2 Kx=ECDH Au=ECDSA Enc=Camellia(128) Mac=SHA256 0xC0,0x76 - ECDHE-RSA-CAMELLIA128-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=Camellia(128) Mac=SHA256 0x00,0xBE - DHE-RSA-CAMELLIA128-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=Camellia(128) Mac=SHA256 0x00,0xBD - DHE-DSS-CAMELLIA128-SHA256 TLSv1.2 Kx=DH Au=DSS Enc=Camellia(128) Mac=SHA256 0x00,0x9D - AES256-GCM-SHA384 TLSv1.2 Kx=RSA Au=RSA Enc=AESGCM(256) Mac=AEAD 0xC0,0xA1 - AES256-CCM8 TLSv1.2 Kx=RSA Au=RSA Enc=AESCCM8(256) Mac=AEAD 0xC0,0x9D - AES256-CCM TLSv1.2 Kx=RSA Au=RSA Enc=AESCCM(256) Mac=AEAD 0xC0,0x51 - ARIA256-GCM-SHA384 TLSv1.2 Kx=RSA Au=RSA Enc=ARIAGCM(256) Mac=AEAD 0x00,0x9C - AES128-GCM-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AESGCM(128) Mac=AEAD 0xC0,0xA0 - AES128-CCM8 TLSv1.2 Kx=RSA Au=RSA Enc=AESCCM8(128) Mac=AEAD 0xC0,0x9C - AES128-CCM TLSv1.2 Kx=RSA Au=RSA Enc=AESCCM(128) Mac=AEAD 0xC0,0x50 - ARIA128-GCM-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=ARIAGCM(128) Mac=AEAD 0x00,0x3D - AES256-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA256 0x00,0xC0 - CAMELLIA256-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=Camellia(256) Mac=SHA256 0x00,0x3C - AES128-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA256 0x00,0xBA - CAMELLIA128-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=Camellia(128) Mac=SHA256 ----- When ovn-controller 4.5 connects to sbdb 4.6 the cipher would be 'TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 1024)': ----- kubectl -n openshift-ovn-kubernetes exec -ti ovnkube-node-ckhqp -c ovn-controller -- openssl s_client -key /ovn-cert/tls.key -cert /ovn-cert/tls.crt -CAfile /ovn-ca/ca-bundle.crt -cipher 'HIGH:!aNULL:!MD5' -connect 10.0.0.6:9642 CONNECTED(00000003) Can't use SSL_get_servername depth=1 CN = openshift-ovn-kubernetes_ovn-ca@1623858690 verify return:1 depth=0 CN = ovn verify return:1 140042392123200:error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small:ssl/statem/statem_clnt.c:2150: --- Certificate chain 0 s:CN = ovn i:CN = openshift-ovn-kubernetes_ovn-ca@1623858690 1 s:CN = openshift-ovn-kubernetes_ovn-ca@1623858690 i:CN = openshift-ovn-kubernetes_ovn-ca@1623858690 --- Server certificate -----BEGIN CERTIFICATE----- MIIDVjCCAj6gAwIBAgIIJkUZ7kADETYwDQYJKoZIhvcNAQELBQAwNTEzMDEGA1UE Awwqb3BlbnNoaWZ0LW92bi1rdWJlcm5ldGVzX292bi1jYUAxNjIzODU4NjkwMB4X DTIxMDYxNjE1NTEyOVoXDTIxMTIxNjAzNTEzMFowDjEMMAoGA1UEAxMDb3ZuMIIB IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwcBN3s0R0HahMhS5tHEqWCNn /4vbxK4ZLC5txfMdcSii3w1Dmeky8WDSL6UWk5WNRW/SqaYeqj2lSoPb+BQu/eHa TxanI/2H+F0ZCZA3jSHoA5+seoFNOyiifSh8BYzhiOVftwNJYYAo0keu8kPj5ZYQ O4OhL8W/+hsNoWQ4a/8LzflgIJqiXsWxGNtA4PwgpXP8ylhYnC5zPxo6/VhM2pgv e9Udot/iwp5Iwh7ao3276sWgtUoWDBPEv00gJ0oRGbE/QrLYqZFeis3TskJrs5ih EGAghshzxbi+SBxThIA1c2lLYETv2RXpWkcrAk+CJ9PI5ctpUOHRGgzWaWa4TwID AQABo4GQMIGNMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYI KwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQU4E3jEJHNw8aUrDxXFq/g 1t2WZEMwHwYDVR0jBBgwFoAUdKG5ZPNys43wuI50Z3JAEsLwMXcwDgYDVR0RBAcw BYIDb3ZuMA0GCSqGSIb3DQEBCwUAA4IBAQA43yKC9yzInYI1ICKpWr3Q0jitGVmD rVo7QaRcf/U23mgitHLcGOp0b0ZRPcZdH18xROcW+jvuDTCMKJGfxOvSr3bpswE9 /wts+jLS6DSzZrbF4Xk5eWa7jLhv5gP+ujzYPYr2S9OFicRHAcsPhoURRLn7vC9P 5myGRdJeBvBXWn5YSWn/iJYnhaVVo7zmahnte3r+wn72kn1IoJBUAHSVsS0G8K/f miQR+hcPKFKXDsoVLYBVBx8Fhl9XntERba/lnBSb22RDlz+iN2KkSeTcsJXmRSId UwmTB6EVJ8RS+ntW7UBbtdOEY2GasLoAVxx/YTdXhOmcGUsqu8hF6/FF -----END CERTIFICATE----- subject=CN = ovn issuer=CN = openshift-ovn-kubernetes_ovn-ca@1623858690 --- No client certificate CA names sent --- SSL handshake has read 2310 bytes and written 386 bytes Verification: OK --- New, (NONE), Cipher is (NONE) Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: PSK identity: None PSK identity hint: None SRP username: None Start Time: 1623868811 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no ----- When ovn-controller >=4.6 connects to sbdb >=4.6, for TLSv1.2 the cipher would be 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1)' which uses X25519 elliptic-curve function instead of fixed dh param so in this case the problem does not happen: ----- ❯ kubectl -n openshift-ovn-kubernetes exec -ti ovnkube-node-686jw -c ovn-controller -- openssl s_client -key /ovn-cert/tls.key -cert /ovn-cert/tls.crt -CAfile /ovn-ca/ca-bundle.crt -tls1_2 -cipher 'HIGH:!aNULL:!MD5' -connect 10.0.0.5:9642 CONNECTED(00000003) Can't use SSL_get_servername depth=1 CN = openshift-ovn-kubernetes_ovn-ca@1623873056 verify return:1 depth=0 CN = ovn verify return:1 --- Certificate chain 0 s:CN = ovn i:CN = openshift-ovn-kubernetes_ovn-ca@1623873056 1 s:CN = openshift-ovn-kubernetes_ovn-ca@1623873056 i:CN = openshift-ovn-kubernetes_ovn-ca@1623873056 --- Server certificate -----BEGIN CERTIFICATE----- MIIDVjCCAj6gAwIBAgIIb6h3wShDeeQwDQYJKoZIhvcNAQELBQAwNTEzMDEGA1UE Awwqb3BlbnNoaWZ0LW92bi1rdWJlcm5ldGVzX292bi1jYUAxNjIzODczMDU2MB4X DTIxMDYxNjE5NTA1NloXDTIxMTIxNjA3NTA1N1owDjEMMAoGA1UEAxMDb3ZuMIIB IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA66st/0wSmsgUBz/P3cr0ZZ/N A8fSQIHTJRKLNu1LVGbXxgnrCdrCDXeNQdwelX94m0FWiPHyD4kwYoac76yEs9ww biVkFdL9DfrcMJL/zJnerLWKQ6m6uCEpB3mryP3HjZTPWtvbleUXQcH3WAcM3C2t S6PRBEi0/iY+EJ5PbvQpwmuqvbsRaXiIXedMDEAIwviwznzstYaxA5DWKpZOAcby AexDzK4rWV+oM7aL9+rHDXITKcX9tiVtlwa2aMMNdhl7+RcGUIVXHOjkaxPZRXfo DG1N76f/wAYUf3oujDtevFvVVJYUSlA9A+fCxN8vtqnSzCTnfoquujkXcxnFiwID AQABo4GQMIGNMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYI KwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQU0kTviYvvjr3n/Ep359V7 bWZGX5owHwYDVR0jBBgwFoAUqe5PZRW1FvKizBFXja88K2JalDwwDgYDVR0RBAcw BYIDb3ZuMA0GCSqGSIb3DQEBCwUAA4IBAQCG1jCy2hyeWJDI+EI6/72QgCSOEURw BhrYtFGgdhbmavbrNTCHK3wo6zRcCgvijjMdXjS52+Hyb1AmEUjEIez/nnLH+79+ dKFMYtNf2jFRZvng6uwqWgt7euRQ44nVjDQuLnch7QiF7lndInq/cbLpZka4M5si v/eN0FzVzxhWFCUEz8+Ynt+fOKNjBD+REEnWTNxzAAvS7YUYlcx4Lg7VLzwkOggt D2qeS2ihNHobGE8KgHKyuc5jzM+MQQnB4gTr402rEuh2ofACKmcQkFxSa1DAqKiR y1dIKmQr9MNe5/c7b94ta/6ZA5pvSFS1BpZiDakmm85XCGyJgUJ7iDB/ -----END CERTIFICATE----- subject=CN = ovn issuer=CN = openshift-ovn-kubernetes_ovn-ca@1623873056 --- Acceptable client certificate CA names CN = openshift-ovn-kubernetes_ovn-ca@1623873056 Client Certificate Types: RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:ECDSA+SHA1:RSA+SHA224:RSA+SHA1:DSA+SHA224:DSA+SHA1:DSA+SHA256:DSA+SHA384:DSA+SHA512 Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:ECDSA+SHA1:RSA+SHA224:RSA+SHA1:DSA+SHA224:DSA+SHA1:DSA+SHA256:DSA+SHA384:DSA+SHA512 Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits --- SSL handshake has read 3314 bytes and written 2364 bytes Verification: OK --- New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: F9199E682DA905586091D39BD6781124BB14031CF8FC7AA6C49FDC02374703CA Session-ID-ctx: Master-Key: 4D2156F8680E9CE80B8C9358EF61DFA687CE30C4318E1A35984B68AFDA8697769852F8F1FC6AAA6588C11CD229EC1CCD PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 7200 (seconds) TLS session ticket: 0000 - 7b 95 0f 88 95 4d 80 f8-0b 02 a3 b6 b8 98 2d 76 {....M........-v 0010 - 8f 6a 9c 3c 29 75 71 15-38 f2 f1 62 9f 85 71 8b .j.<)uq.8..b..q. 0020 - a5 1c 24 0e 74 04 b6 fc-38 99 7a d7 de 2b 88 b5 ..$.t...8.z..+.. 0030 - f8 58 e5 75 65 7a 55 eb-da 5a 70 74 a2 8b af b4 .X.uezU..Zpt.... 0040 - 34 17 3b 0d e8 be 74 55-dd 92 52 10 5d 63 75 9b 4.;...tU..R.]cu. 0050 - 91 e8 ff 25 d5 5f 81 f1-d8 db 48 a4 52 e2 c9 87 ...%._....H.R... 0060 - 44 4e d9 29 35 2e 3f cb-34 bc e1 5f 06 fd 53 b0 DN.)5.?.4.._..S. 0070 - a3 b2 41 36 32 6c 2e e9-ad 12 94 a6 10 be f6 01 ..A62l.......... 0080 - 60 a7 5a 86 8b 55 41 4b-f7 c4 00 64 22 73 70 cf `.Z..UAK...d"sp. 0090 - c7 06 58 db 00 a2 27 39-09 fa 96 47 31 b4 65 24 ..X...'9...G1.e$ 00a0 - 55 70 f1 e5 a1 a7 fb 44-8b 99 32 3f fe b5 d6 7a Up.....D..2?...z 00b0 - a0 e6 e9 2e 2c c4 59 27-7d a2 3a 19 2b 34 61 5c ....,.Y'}.:.+4a\ 00c0 - fb 98 e6 72 ba de 23 6d-79 62 7b 26 7d 7d 07 94 ...r..#myb{&}}.. 00d0 - a2 14 8d cb 43 79 60 41-59 c9 38 f9 a1 82 a2 0e ....Cy`AY.8..... 00e0 - ba 81 a8 60 65 db 87 c6-3e bf 17 53 c1 59 c5 3e ...`e...>..S.Y.> 00f0 - 38 41 58 cd 34 ac 46 25-ee b4 45 27 1a b7 e2 ad 8AX.4.F%..E'.... 0100 - fc 18 a5 09 33 b1 ce 6a-a2 47 fb 2f d8 10 61 ab ....3..j.G./..a. 0110 - 1b 7a f1 e1 52 d0 c4 68-98 40 b7 bb 01 83 b8 b4 .z..R..h.@...... 0120 - f8 16 04 11 2a 7a 23 19-2f c3 26 d4 21 0c 48 aa ....*z#./.&.!.H. 0130 - 54 82 52 46 76 62 9d 0c-7b d3 97 bb ea 86 97 c8 T.RFvb..{....... 0140 - 69 81 d5 a1 8b 5c 46 d3-4c 7a b2 5e 5a 5e e1 57 i....\F.Lz.^Z^.W 0150 - 0d 8e 91 18 2c 30 7e 68-42 96 ed 15 a8 f1 46 d3 ....,0~hB.....F. 0160 - f3 cd 1b 87 a6 8c fe a4-c4 3f af bd fc 99 0e c7 .........?...... 0170 - ce 3b 99 a3 0b 33 dc 4a-b4 e2 bd 83 63 88 a8 3b .;...3.J....c..; 0180 - f9 86 39 d2 cf ce 45 d5-7d d0 64 c3 1a 20 88 56 ..9...E.}.d.. .V 0190 - 0a 59 08 43 85 37 7d ee-c7 c7 3a d4 2f bb f9 a0 .Y.C.7}...:./... 01a0 - cb f7 6f ad ad e7 8f 50-a5 6c 5c ae 79 18 10 92 ..o....P.l\.y... 01b0 - b5 a0 fb a8 ae d1 f1 43-f8 78 33 43 44 63 ad 8f .......C.x3CDc.. 01c0 - 6e 9a 44 dd 57 b3 1b 37-89 f9 91 46 16 f2 b7 80 n.D.W..7...F.... 01d0 - 51 75 44 24 43 e9 11 75-4c e0 8c ab 98 0d 04 ae QuD$C..uL....... 01e0 - 1d 92 ff a2 0f 11 97 5d-05 c1 46 f7 f2 1f 4e 21 .......]..F...N! 01f0 - 10 d1 9f 0e e4 29 5f 88-ed 1a c5 75 50 5a 66 3c .....)_....uPZf< 0200 - d3 91 b5 af 4b 6a 4d 88-3d 1c b8 35 a3 93 23 78 ....KjM.=..5..#x 0210 - a7 ef 2f bf e0 bd cb 3d-f5 7e 01 f1 b7 3b a1 a4 ../....=.~...;.. 0220 - fe 19 cb 35 e4 29 67 23-c1 20 49 92 9e bb ae 1a ...5.)g#. I..... 0230 - 13 a7 59 ae 28 6b e3 53-69 a4 33 19 ea 01 00 1a ..Y.(k.Si.3..... 0240 - 4b 00 2f 9b 1e f1 f6 59-7f 1b 5d 55 54 d1 09 59 K./....Y..]UT..Y 0250 - e5 46 24 10 68 2f af 52-a8 e3 b6 0e 3a 1b 54 c2 .F$.h/.R....:.T. 0260 - 62 57 8f 77 d1 76 3d 5c-f7 5d 37 a6 7f d7 11 fe bW.w.v=\.]7..... 0270 - 4e 83 14 de ec a8 23 f0-42 19 9f 40 9a 06 bd 53 N.....#.B..@...S 0280 - 7e d6 d0 d6 6a 13 87 54-34 1d 9f 9c 0c a2 4b d8 ~...j..T4.....K. 0290 - 69 2b 22 a2 1f 8f f1 17-71 7a 0a 9c 61 87 9d cc i+".....qz..a... 02a0 - 7c 71 4f 2f 84 cf ec e3-d3 4f 53 d6 da 13 f7 c2 |qO/.....OS..... 02b0 - e9 38 c5 14 e9 d6 aa 12-18 0a e4 5a 3e ed 6b 9c .8.........Z>.k. 02c0 - 4b da d1 cf fe b4 12 e5-2d 15 36 7b 43 bc 12 ed K.......-.6{C... 02d0 - 5f 62 2e fe 30 02 ee 2c-9d 5e 01 b8 d3 a7 54 61 _b..0..,.^....Ta 02e0 - da ea 8c 70 91 4a fb 3b-56 c0 76 b9 3d 69 5a 52 ...p.J.;V.v.=iZR 02f0 - 67 fa c4 03 e7 da 91 25-27 dc c0 b3 da 91 98 f1 g......%'....... 0300 - 61 ca 50 00 35 21 37 e8-f0 13 3d 42 95 33 45 0d a.P.5!7...=B.3E. 0310 - 19 6c 1f 0b 3d c5 e9 8d-4d 54 61 1d 35 66 d5 e6 .l..=...MTa.5f.. 0320 - 3f cd 8d 5d 5a 88 b7 de-0f da fd ff 0e 1a 0f 04 ?..]Z........... 0330 - c5 66 30 84 20 73 fd 12-f3 a6 04 44 7c fc 4b 42 .f0. s.....D|.KB 0340 - 72 02 2c 1c 94 82 68 86-ad 41 ef 15 55 1c 21 dc r.,...h..A..U.!. 0350 - 80 1a 9d fc 82 ae e7 c5-f6 1a 3f 68 02 98 a3 f4 ..........?h.... 0360 - 4d 13 4a b5 60 d6 e0 a5-2e e4 06 27 9f 64 cc e5 M.J.`......'.d.. 0370 - fd 7d 0a 46 76 a7 7c f3-6c ad e4 96 5e 98 63 10 .}.Fv.|.l...^.c. 0380 - 66 aa 0a a3 08 d6 ad 2b-f0 5c 0c 90 5d 4d 39 b4 f......+.\..]M9. 0390 - 6b 40 a6 e5 09 c0 eb fd-56 8c 10 6b dd e0 6b 0e k@......V..k..k. 03a0 - 67 c3 d5 21 5a 92 3b db-65 d8 51 51 b4 01 2c 70 g..!Z.;.e.QQ..,p 03b0 - 40 86 d1 db e2 d5 29 15-93 d7 89 44 08 07 be d7 @.....)....D.... 03c0 - 73 d5 0c be 3e 08 ec 20-c7 98 ef 1f de 9d 29 9c s...>.. ......). 03d0 - f3 95 bf b8 6f 7d 2b f9-5c cb ec 13 dd 4a 48 dc ....o}+.\....JH. 03e0 - 9e 21 68 b8 48 60 8a 95-0b 24 3b 62 e9 31 ab 79 .!h.H`...$;b.1.y 03f0 - 6e e6 47 bd 38 c2 b1 a5-c0 74 4f 96 51 a5 b5 50 n.G.8....tO.Q..P Start Time: 1623920491 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: yes ----- When ovn-controller >=4.6 connects to sbdb >=4.6, for TLSv1.3, also uses an X25519 elliptic-curve function: ----- kubectl -n openshift-ovn-kubernetes exec -ti ovnkube-node-686jw -c ovn-controller -- openssl s_client -key /ovn-cert/tls.key -cert /ovn-cert/tls.crt -CAfile /ovn-ca/ca-bundle.crt -cipher 'HIGH:!aNULL:!MD5' -connect 10.0.0.5:9642 CONNECTED(00000003) Can't use SSL_get_servername depth=1 CN = openshift-ovn-kubernetes_ovn-ca@1623873056 verify return:1 depth=0 CN = ovn verify return:1 --- Certificate chain 0 s:CN = ovn i:CN = openshift-ovn-kubernetes_ovn-ca@1623873056 1 s:CN = openshift-ovn-kubernetes_ovn-ca@1623873056 i:CN = openshift-ovn-kubernetes_ovn-ca@1623873056 --- Server certificate -----BEGIN CERTIFICATE----- MIIDVjCCAj6gAwIBAgIIb6h3wShDeeQwDQYJKoZIhvcNAQELBQAwNTEzMDEGA1UE Awwqb3BlbnNoaWZ0LW92bi1rdWJlcm5ldGVzX292bi1jYUAxNjIzODczMDU2MB4X DTIxMDYxNjE5NTA1NloXDTIxMTIxNjA3NTA1N1owDjEMMAoGA1UEAxMDb3ZuMIIB IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA66st/0wSmsgUBz/P3cr0ZZ/N A8fSQIHTJRKLNu1LVGbXxgnrCdrCDXeNQdwelX94m0FWiPHyD4kwYoac76yEs9ww biVkFdL9DfrcMJL/zJnerLWKQ6m6uCEpB3mryP3HjZTPWtvbleUXQcH3WAcM3C2t S6PRBEi0/iY+EJ5PbvQpwmuqvbsRaXiIXedMDEAIwviwznzstYaxA5DWKpZOAcby AexDzK4rWV+oM7aL9+rHDXITKcX9tiVtlwa2aMMNdhl7+RcGUIVXHOjkaxPZRXfo DG1N76f/wAYUf3oujDtevFvVVJYUSlA9A+fCxN8vtqnSzCTnfoquujkXcxnFiwID AQABo4GQMIGNMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYI KwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQU0kTviYvvjr3n/Ep359V7 bWZGX5owHwYDVR0jBBgwFoAUqe5PZRW1FvKizBFXja88K2JalDwwDgYDVR0RBAcw BYIDb3ZuMA0GCSqGSIb3DQEBCwUAA4IBAQCG1jCy2hyeWJDI+EI6/72QgCSOEURw BhrYtFGgdhbmavbrNTCHK3wo6zRcCgvijjMdXjS52+Hyb1AmEUjEIez/nnLH+79+ dKFMYtNf2jFRZvng6uwqWgt7euRQ44nVjDQuLnch7QiF7lndInq/cbLpZka4M5si v/eN0FzVzxhWFCUEz8+Ynt+fOKNjBD+REEnWTNxzAAvS7YUYlcx4Lg7VLzwkOggt D2qeS2ihNHobGE8KgHKyuc5jzM+MQQnB4gTr402rEuh2ofACKmcQkFxSa1DAqKiR y1dIKmQr9MNe5/c7b94ta/6ZA5pvSFS1BpZiDakmm85XCGyJgUJ7iDB/ -----END CERTIFICATE----- subject=CN = ovn issuer=CN = openshift-ovn-kubernetes_ovn-ca@1623873056 --- Acceptable client certificate CA names CN = openshift-ovn-kubernetes_ovn-ca@1623873056 Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:ECDSA+SHA1:RSA+SHA224:RSA+SHA1 Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512 Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits --- SSL handshake has read 2397 bytes and written 2487 bytes Verification: OK --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: 210D9B6D5F95DCBF190AAE402AF106A531C1BD1FCA8A80D3A9306C3952344D57 Session-ID-ctx: Resumption PSK: 95ECD0EF5DC13ED530337F51E168EA7313FC5084BC4A99EA74DCD6AB818658F148792D61BAC39E91A8052F11357B31B3 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 7200 (seconds) TLS session ticket: 0000 - 7b 95 0f 88 95 4d 80 f8-0b 02 a3 b6 b8 98 2d 76 {....M........-v 0010 - db 2d b1 fe ce d4 99 4f-35 d4 df 21 a8 cb df b6 .-.....O5..!.... 0020 - d7 a9 04 ab 50 49 b0 83-34 1a ee 30 d6 91 3e 61 ....PI..4..0..>a 0030 - 4c fa a1 bd 04 c4 49 e6-08 6a 36 bc ca 64 b0 9c L.....I..j6..d.. 0040 - 87 7f 2d d7 b9 74 ce e9-80 b6 4f 4e 60 0e f0 30 ..-..t....ON`..0 0050 - e2 f2 21 62 2a aa 76 e5-6c a1 18 8d 9f c6 b7 8d ..!b*.v.l....... 0060 - e6 1d 09 b7 06 ba ff 45-ae 56 17 8f 57 38 78 c8 .......E.V..W8x. 0070 - 38 66 49 84 ca cf ed be-b1 bc 90 69 4d 71 59 d6 8fI........iMqY. 0080 - f2 0e ce 99 09 54 85 bf-16 85 22 39 0c 1d 18 1a .....T...."9.... 0090 - 44 5d 0f 71 29 2c 11 e4-17 f9 7f b4 7a 5d 42 75 D].q),......z]Bu 00a0 - 5d a3 a7 ad 1f f4 ea 09-39 43 5d 73 a0 ed f0 3d ].......9C]s...= 00b0 - 6b b6 75 86 1c 5d 38 c2-38 53 7c 99 52 f9 09 f5 k.u..]8.8S|.R... 00c0 - 23 4a 01 11 a2 07 04 dc-8c f8 d9 6d 79 0b c5 4d #J.........my..M 00d0 - e7 c9 fe 08 70 9e 7d 00-e1 60 03 b9 f2 b8 03 f5 ....p.}..`...... 00e0 - cc ed b2 04 c0 ae f1 52-79 f3 63 56 ed 44 b8 5b .......Ry.cV.D.[ 00f0 - eb 46 fb 29 33 aa 28 d3-c5 af 8d de 51 ef 61 32 .F.)3.(.....Q.a2 0100 - c6 5d 10 94 07 9b 75 76-f5 75 05 a7 24 59 68 69 .]....uv.u..$Yhi 0110 - 3b d5 0d e7 2b f7 02 5b-4c 65 8d d9 0a 54 82 8a ;...+..[Le...T.. 0120 - 55 91 23 38 3d 13 d1 79-66 8b 16 f1 7f 61 fe 98 U.#8=..yf....a.. 0130 - 3b e7 b6 ba 63 43 f9 8b-94 33 b6 b2 b4 96 94 15 ;...cC...3...... 0140 - 08 88 51 85 ef 72 14 81-ba b8 9f 09 2f ba 53 cd ..Q..r....../.S. 0150 - 1b 0b fd b0 e7 03 a8 5d-98 3a 19 a5 c4 b7 d6 6a .......].:.....j 0160 - 0c 0b b2 8b 56 de 52 69-59 db 4b 96 9b d3 53 69 ....V.RiY.K...Si 0170 - 69 27 7f 45 54 f5 84 ef-a2 0e 5a 8e d3 ed e5 71 i'.ET.....Z....q 0180 - 18 ab 0b 33 ff 91 c3 6b-6e 92 89 73 fa 07 e1 73 ...3...kn..s...s 0190 - cc 8e bf 18 82 a0 72 41-0a b3 c5 4b b9 c4 90 63 ......rA...K...c 01a0 - 40 0f 1b b7 86 e9 6a 4e-12 cf ad 7f 49 ef 00 7a @.....jN....I..z 01b0 - 0a 98 4c 81 c8 25 6a 01-8c 16 dd e0 41 a8 cb 5d ..L..%j.....A..] 01c0 - 18 13 f0 86 74 c7 0c cf-5e 48 ba ab 12 cf 94 9f ....t...^H...... 01d0 - f1 2d b4 8d ff 1b 9b 80-49 bf 66 e7 8d 46 dd c2 .-......I.f..F.. 01e0 - ca 33 06 83 bc bd b6 94-8e 68 65 2a ae e1 cd 8b .3.......he*.... 01f0 - e5 db ac 10 8b d7 02 1d-ee 21 5b d3 09 70 80 4f .........![..p.O 0200 - c0 c1 18 75 a7 19 01 d2-ae 9e e9 fd 63 62 8a 4c ...u........cb.L 0210 - ab e5 5c 18 9b 9d 53 f8-5d a5 af f8 4a c1 9e f5 ..\...S.]...J... 0220 - da 02 68 cc f9 65 3f 08-60 7c 51 96 75 4b 00 07 ..h..e?.`|Q.uK.. 0230 - e5 f4 3b d9 17 4b 5e 12-c2 9b 8b 57 36 13 c9 a1 ..;..K^....W6... 0240 - aa 87 17 e4 f2 a0 f5 db-7c fe d7 9a 4b f6 22 76 ........|...K."v 0250 - 78 2c c5 5e 36 0d 41 b7-28 39 23 8b 7a 7c 44 a0 x,.^6.A.(9#.z|D. 0260 - cf a6 8a 95 f0 a8 1f 4e-3a ad 3e 9e 86 73 05 18 .......N:.>..s.. 0270 - a6 4b c7 f8 42 f9 87 39-ce 68 a9 4f df 16 cb 22 .K..B..9.h.O..." 0280 - b9 57 04 59 22 57 4d b2-24 95 8b e8 1c 98 4b 43 .W.Y"WM.$.....KC 0290 - a2 24 d5 b4 82 f1 55 a8-bc 24 60 fa e7 5e 10 e2 .$....U..$`..^.. 02a0 - ab e9 b8 0b a6 32 80 c7-be 36 46 3c 6d 29 e0 31 .....2...6F<m).1 02b0 - cf 39 e5 44 44 4f f2 71-5b 67 76 21 74 7e 24 f2 .9.DDO.q[gv!t~$. 02c0 - c7 c0 a6 a3 d7 fd c6 b4-35 6b 6f 3e 7e e1 89 5b ........5ko>~..[ 02d0 - 9c 3b 1e 23 46 e6 35 21-50 dc d4 ee ab 8b bb d2 .;.#F.5!P....... 02e0 - 8c 2f 67 5c 4c 66 5b 84-d9 b9 d2 a0 eb d5 10 20 ./g\Lf[........ 02f0 - 1c 9e 7d 64 14 38 38 c6-70 56 b8 87 71 28 0d 31 ..}d.88.pV..q(.1 0300 - 93 17 a0 95 cd 39 fd 3d-d9 26 39 ec 46 05 0c 93 .....9.=.&9.F... 0310 - b2 9f 6c dd 98 c4 a0 83-d6 4e 33 02 22 43 45 ca ..l......N3."CE. 0320 - b4 31 f0 f3 97 b9 c5 dd-cd 06 2b 57 16 13 0f 8b .1........+W.... 0330 - bd 7c 09 dd 0b d2 1d 95-5b b9 6a 72 af 5e 30 45 .|......[.jr.^0E 0340 - 42 40 7f 1a 2b 21 1d c2-eb 9b 1b 85 b1 2f 80 1c B@..+!......./.. 0350 - 05 85 01 80 e4 bc fc dd-7e ef 96 40 ff 2a 1e c1 ........~..@.*.. 0360 - 10 ea 0b 95 0e d2 f8 9b-3d a8 85 01 d6 9d a9 dd ........=....... 0370 - 15 06 d2 7e 3b 5b 6d de-cb 0c 75 e6 e3 57 3e ba ...~;[m...u..W>. 0380 - 6a 21 aa b9 4d 31 d4 c0-9c 43 b5 35 6c fc 87 31 j!..M1...C.5l..1 0390 - 74 53 4f 1a a5 ff de 1b-6a 85 95 1b 90 cd 8d 67 tSO.....j......g 03a0 - da 04 80 bb 99 5d 76 66-72 34 03 b2 3e c8 9e 53 .....]vfr4..>..S 03b0 - e3 8b 01 73 cd 82 fc 8c-9d 2b 0e 16 ac 10 ca e3 ...s.....+...... 03c0 - 5e bc 1a d2 74 8f fd e8-04 b3 ec fb 90 5e 7b 7d ^...t........^{} 03d0 - 0a 4c bb 7c a1 bd 67 41-8f 59 24 b3 ba 43 33 29 .L.|..gA.Y$..C3) 03e0 - 9b 9a 4c ad b9 a9 e5 75-4b d6 a0 32 c6 a2 36 eb ..L....uK..2..6. 03f0 - dc b5 ff f7 a8 33 67 44-da 9f 80 79 73 57 15 06 .....3gD...ysW.. 0400 - c0 7e 07 28 09 e7 ce cb-cb df 3a fd 81 4b bc e8 .~.(......:..K.. 0410 - 14 ab 47 ae 07 62 0d eb-6b 31 b9 84 36 62 28 c3 ..G..b..k1..6b(. Start Time: 1623920448 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no Max Early Data: 0 --- read R BLOCK --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: DC3A9BBA694EBE43B814773B41BFD9D5C24A4181011C34D9BAB9CE9374601E07 Session-ID-ctx: Resumption PSK: F69A632647957543FE2E847B2DB3CA8E3A183DC98E554BD9CD82AFD62C6280664AEE03E1955DEBD30A6B4763818490F4 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 7200 (seconds) TLS session ticket: 0000 - 7b 95 0f 88 95 4d 80 f8-0b 02 a3 b6 b8 98 2d 76 {....M........-v 0010 - 32 a0 0c d6 9d 54 f3 23-da 49 34 47 fc c3 60 4d 2....T.#.I4G..`M 0020 - fc b9 22 3f 6a 7f 5c bc-78 d1 86 4c f9 cb f1 de .."?j.\.x..L.... 0030 - b4 85 06 de bd c2 64 b6-90 27 d9 c6 f5 36 34 fd ......d..'...64. 0040 - 57 e0 50 00 91 64 c9 8d-49 22 b4 ab d6 8d 5a 19 W.P..d..I"....Z. 0050 - 9e 27 b4 15 c6 65 b2 93-04 e1 7f 9c ba 23 18 ff .'...e.......#.. 0060 - 5b 7f a1 26 55 b9 f9 67-06 8a 98 5f 16 ce 35 28 [..&U..g..._..5( 0070 - 3a 60 9b 09 cc ee 98 62-cb 41 9f 16 88 9c 8f 09 :`.....b.A...... 0080 - 38 09 62 ca b0 77 42 aa-cd 3d 23 80 de f8 98 5b 8.b..wB..=#....[ 0090 - 82 37 f4 48 1d cf d0 de-ff 53 85 ed e5 21 66 c1 .7.H.....S...!f. 00a0 - d0 31 52 87 6e e4 02 37-c2 72 7f 0d ce 38 12 9a .1R.n..7.r...8.. 00b0 - 99 a1 20 d1 06 1a 3b 31-e2 c6 6a b9 f4 5d 6c af .. ...;1..j..]l. 00c0 - 5e 78 3e 53 c0 fb 9a 63-d6 92 43 2d c9 3c b6 c0 ^x>S...c..C-.<.. 00d0 - 04 9b 86 13 09 26 de ef-a3 f1 e3 39 74 4a f8 51 .....&.....9tJ.Q 00e0 - 4b fc 6c eb b6 92 0e d1-0a 33 ba a0 95 67 19 81 K.l......3...g.. 00f0 - d5 fa 83 83 11 b9 1f c0-94 47 5c f6 3e 53 64 a7 .........G\.>Sd. 0100 - 51 d3 91 e9 e6 39 b3 3b-ff 1d cb c4 8c e4 06 af Q....9.;........ 0110 - ea 01 e0 f3 e1 1d a2 76-cc d8 63 c0 e9 13 93 39 .......v..c....9 0120 - 89 20 49 a8 4e a2 f1 4f-ee e8 f6 e1 46 c8 30 ae . I.N..O....F.0. 0130 - 4a e6 08 7c 7d 3f f3 71-88 7a 32 d8 2e af 79 4f J..|}?.q.z2...yO 0140 - d0 80 a6 c9 57 0e 54 fd-d4 9d 2b 2b e9 91 53 1a ....W.T...++..S. 0150 - 00 91 65 89 ed b0 db 41-f8 62 0a d1 75 aa 0c 32 ..e....A.b..u..2 0160 - 8f ab 64 72 8b b1 03 d8-11 a1 3f 9b 1f cc 6f 1f ..dr......?...o. 0170 - 14 6c 5f 01 69 82 09 b9-08 bc 1c 68 d6 c0 0a 63 .l_.i......h...c 0180 - 4c 19 16 68 1d 76 d8 6e-61 93 8c 81 00 07 fa 5c L..h.v.na......\ 0190 - fd f2 94 2b da a7 a5 86-dd 30 b6 67 6d 59 bc b3 ...+.....0.gmY.. 01a0 - 95 5b 49 88 cf 95 a9 2c-97 e4 27 44 92 7e b9 98 .[I....,..'D.~.. 01b0 - 6e c5 56 9f 3e 90 c7 77-80 7c 0b b9 51 62 0d 86 n.V.>..w.|..Qb.. 01c0 - 49 c4 28 e7 35 26 95 a7-ba fe 16 08 b5 16 45 90 I.(.5&........E. 01d0 - 8b 5a 2d 86 6e 07 9e 84-69 98 d9 dd c1 84 9e d6 .Z-.n...i....... 01e0 - ee 81 03 67 d0 b9 40 2d-9f 97 84 67 e8 af 54 f0 ...g..@-...g..T. 01f0 - e4 63 66 27 a0 25 68 6f-04 0f 90 5e 55 44 0a 07 .cf'.%ho...^UD.. 0200 - 6b 2a 2a cd 7b d9 4b 52-db 60 58 0b df 39 e3 28 k**.{.KR.`X..9.( 0210 - d6 ac 14 34 86 ce 7b 3f-ea 2b 87 d1 6d 9f 65 cf ...4..{?.+..m.e. 0220 - fd fc 71 33 96 07 67 48-cb 9c 49 58 7b 9c e1 12 ..q3..gH..IX{... 0230 - 35 3e c5 a7 f9 67 f5 ea-7c 27 ef c7 03 2b 7b b4 5>...g..|'...+{. 0240 - b8 16 42 4e c9 79 1f 71-c9 fe 0b af f9 94 dc 07 ..BN.y.q........ 0250 - 13 d2 37 56 cd b7 45 c5-99 f9 89 36 79 bb 09 49 ..7V..E....6y..I 0260 - 11 20 25 be c9 9f 79 e4-c3 7a 88 82 b4 22 cf 03 . %...y..z...".. 0270 - 32 92 ae 13 0e 92 43 e0-0f 87 fb 4d 72 6e 03 cf 2.....C....Mrn.. 0280 - c9 13 06 b8 fe 80 60 df-62 ef 71 0e 4e a4 55 52 ......`.b.q.N.UR 0290 - 1e 49 ea e6 18 3b 40 4b-94 ff 91 e2 c0 28 2e 5e .I...;@K.....(.^ 02a0 - 8c 9b 31 f0 61 54 70 e7-22 ad 19 0a d6 fb 89 10 ..1.aTp."....... 02b0 - 9d 9e ba fd aa 3b 7d 26-a2 a4 80 f0 b4 73 58 e8 .....;}&.....sX. 02c0 - 31 62 6a 25 e4 63 e8 26-ad c6 f4 79 38 f9 3d 2b 1bj%.c.&...y8.=+ 02d0 - 4a 23 55 db 8b 29 8f f7-69 d8 b9 e7 da 93 13 3b J#U..)..i......; 02e0 - c8 84 7d c0 75 ca f6 3e-a9 32 09 25 92 bf 2d fe ..}.u..>.2.%..-. 02f0 - 82 e0 15 6e fb ab f3 f4-ab 12 1c 92 ed 19 14 82 ...n............ 0300 - 44 41 94 65 3b 7c 0e 49-30 91 72 2d f4 45 69 e1 DA.e;|.I0.r-.Ei. 0310 - 95 d4 1c 26 d7 75 b7 c4-7b e0 ff b0 d4 6f b2 e7 ...&.u..{....o.. 0320 - d9 75 43 f6 bc 35 6e bd-50 17 17 ef 42 b3 53 a7 .uC..5n.P...B.S. 0330 - 1f f6 72 fd 61 e4 de 8f-66 cc 7c ec 62 dd f6 e9 ..r.a...f.|.b... 0340 - ba 3d ad 83 90 6e b0 33-e9 66 ac 45 04 67 04 d6 .=...n.3.f.E.g.. 0350 - 9a 6f 80 3f cf 3b 7e b0-c9 5c 37 0c c0 90 60 4a .o.?.;~..\7...`J 0360 - 0d 42 5e d2 fc 8d da e1-4b ac 49 e8 7b a4 5e d9 .B^.....K.I.{.^. 0370 - 0d aa 80 b9 90 d2 79 cf-b8 4d 02 83 b2 bc 32 aa ......y..M....2. 0380 - 61 45 68 cb 83 fe 9a a2-2a 55 a2 9f 7e ca 5b bb aEh.....*U..~.[. 0390 - d2 72 a6 1b 4a 97 39 0b-bf 62 57 c8 a7 3f 75 a7 .r..J.9..bW..?u. 03a0 - 60 6e 1d 69 a7 33 38 10-a9 57 d0 d1 0e 6e e9 d9 `n.i.38..W...n.. 03b0 - 5c 1c 2f 1d 0c 29 fa f2-08 66 9c ea 89 de 31 69 \./..)...f....1i 03c0 - 8d ab 2e 28 dc e2 3c 4d-10 60 55 dc b9 96 9f 8d ...(..<M.`U..... 03d0 - 42 0d ec 89 ef 2b 33 8b-cb 6c 51 3e 99 c1 a4 c3 B....+3..lQ>.... 03e0 - b6 98 a0 ea 20 08 25 a9-b2 49 cb f0 05 05 bb 74 .... .%..I.....t 03f0 - 39 23 5b 3b a2 b7 67 57-99 7b 78 c9 c9 54 6c 05 9#[;..gW.{x..Tl. 0400 - df dd 30 57 48 88 26 41-31 af 92 50 3e 74 48 08 ..0WH.&A1..P>tH. 0410 - 78 10 04 7d f9 85 36 49-4d 4a c2 94 b5 52 fa 44 x..}..6IMJ...R.D Start Time: 1623920448 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no Max Early Data: 0 ----- Submitted patch to openvswitch to stop using 1024 bit dh params: https://mail.openvswitch.org/pipermail/ovs-dev/2021-June/384049.html With that patch now ovn/ovs use 2048 bit dh params: ----- ❯ nmap --script ssl-enum-ciphers -p 46093 127.0.0.1 Starting Nmap 7.80 ( https://nmap.org ) at 2021-06-16 20:22 UTC Nmap scan report for devmaster (127.0.0.1) Host is up (0.000075s latency). PORT STATE SERVICE 46093/tcp open unknown | ssl-enum-ciphers: | TLSv1.2: | ciphers: | TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) - A | TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (dh 2048) - A | TLS_DHE_RSA_WITH_AES_128_CCM (dh 2048) - A | TLS_DHE_RSA_WITH_AES_128_CCM_8 (dh 2048) - A | TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 2048) - A | TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) - A | TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (dh 2048) - A | TLS_DHE_RSA_WITH_AES_256_CCM (dh 2048) - A | TLS_DHE_RSA_WITH_AES_256_CCM_8 (dh 2048) - A | TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 2048) - A | TLS_DHE_RSA_WITH_ARIA_128_GCM_SHA256 (dh 2048) - A | TLS_DHE_RSA_WITH_ARIA_256_GCM_SHA384 (dh 2048) - A | TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (dh 2048) - A | TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 (dh 2048) - A | TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (dh 2048) - A | TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256 (dh 2048) - A | TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (dh 2048) - A | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (secp256r1) - A | TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_128_CCM (rsa 2048) - A | TLS_RSA_WITH_AES_128_CCM_8 (rsa 2048) - A | TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_256_CCM (rsa 2048) - A | TLS_RSA_WITH_AES_256_CCM_8 (rsa 2048) - A | TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A | TLS_RSA_WITH_ARIA_128_GCM_SHA256 (rsa 2048) - A | TLS_RSA_WITH_ARIA_256_GCM_SHA384 (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256 (rsa 2048) - A | compressors: | NULL | cipher preference: client |_ least strength: A ----- Submitted patch to openshift/cluster-network-operator 4.6 to reduce security level so that 1024 bit dh params are allowed. The reasoning behind this is that it is still the most secure cipher out of the options available. https://github.com/openshift/cluster-network-operator/pull/1131
This is a noop for 4.9 as the issue only affects upgrades from 4.5 to 4.6, the fix is only required in 4.6 and not needed in any other release.
*** Bug 1974424 has been marked as a duplicate of this bug. ***
The bug is https://bugzilla.redhat.com/show_bug.cgi?id=1973770 and the PR is https://github.com/openshift/cluster-network-operator/pull/1131.
I'd requested an impact statement in comment 9. I don't think I got one, but comment 27 suggests that the issue doesn't affect 4.7 and later, and the fact that bug 1973770 is still open suggests the issue is either not all that bad or not all that common. I'm dropping UpgradeBlocker, but feel free to add it back if folks think this is serious enough that we'd drop update recommendations because of it.