Bug 2060553
Summary: | service domain can't be resolved when networkpolicy is used in OCP 4.10-rc | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Jiaming Hu <Jiaming.Hu> | ||||||||||
Component: | Networking | Assignee: | Ben Bennett <bbennett> | ||||||||||
Networking sub component: | openshift-sdn | QA Contact: | zhaozhanqi <zzhao> | ||||||||||
Status: | CLOSED ERRATA | Docs Contact: | |||||||||||
Severity: | high | ||||||||||||
Priority: | urgent | CC: | anbhat, anusaxen, aos-bugs, cblecker, danw, ddharwar, dmoessne, jtanenba, kahara, kangell, mifiedle, mmasters, piotr.godowski, rszumski, travi, twbrown, vlaad, wking, xtian, zzhao | ||||||||||
Version: | 4.10-rc3 | Keywords: | FastFix | ||||||||||
Target Milestone: | --- | ||||||||||||
Target Release: | 4.11.0 | ||||||||||||
Hardware: | x86_64 | ||||||||||||
OS: | Linux | ||||||||||||
Whiteboard: | |||||||||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||
Doc Text: | Story Points: | --- | |||||||||||
Clone Of: | |||||||||||||
: | 2060956 (view as bug list) | Environment: | |||||||||||
Last Closed: | 2022-08-10 10:52:11 UTC | Type: | Bug | ||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||
Documentation: | --- | CRM: | |||||||||||
Verified Versions: | Category: | --- | |||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
Embargoed: | |||||||||||||
Bug Depends On: | |||||||||||||
Bug Blocks: | 2060956 | ||||||||||||
Attachments: |
|
Because the DNS query works on OpenShift 4.9 and 4.10 without the NetworkPolicy and only fails on OpenShift 4.10 with the NetworkPolicy, this seems like a NetworkPolicy issue rather than a DNS issue, so I am changing the component to Networking / openshift-sdn for investigation. @Jiaming.Hu is there a default deny policy for that namespace that also has an empty egress clause? In 4.10, we added support for egress and a previously defined deny policy which had an egress clause is now being effective and blocking traffic from the namespace. Can you please confirm what the default deny policy looks like? The solution may just be to add an explicit allow rule for DNS or remove the egress clause from the default deny. From a security standpoint, the former seems desirable. Is this a newly installed cluster or after an upgrade? Which SDN are you using? Are these workers RHEL or CoreOS based? There is no default deny-all egress policy. However, there are other NetworkPolicies with the same namespace, with egress rules, however for other pod selectors. For example: kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: resourceVersion: '6681721' name: iaf-core-operator uid: e38f7051-5023-443c-8e0d-451d9d81d5d9 creationTimestamp: '2022-02-28T10:50:28Z' generation: 1 namespace: acme-iaf ownerReferences: - apiVersion: apps/v1 kind: Deployment name: iaf-core-operator-controller-manager uid: 07624bcf-bdce-4fcd-8577-f67a5f04132d controller: true blockOwnerDeletion: true labels: app.kubernetes.io/component: networkpolicy app.kubernetes.io/instance: iaf-core-operator app.kubernetes.io/managed-by: controller-manager app.kubernetes.io/name: iaf-core-operator name: iaf-core-operator spec: podSelector: matchLabels: app.kubernetes.io/component: iaf-core-operator app.kubernetes.io/instance: iaf-core-operator app.kubernetes.io/managed-by: olm app.kubernetes.io/name: iaf-core-operator name: iaf-core-operator ingress: - ports: - protocol: TCP port: 9443 - protocol: TCP port: 443 egress: - {} policyTypes: - Ingress - Egress and kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: resourceVersion: '6677521' name: iaf-eventprocessing-operator uid: 0d813a0e-1d03-4cc0-96a2-86d261fd7888 creationTimestamp: '2022-02-28T10:47:56Z' generation: 1 namespace: acme-iaf ownerReferences: - apiVersion: apps/v1 kind: Deployment name: iaf-eventprocessing-operator-controller-manager uid: e3ede052-1363-4dc8-b3fb-f4153545f7fc controller: true blockOwnerDeletion: true labels: app.kubernetes.io/component: networkpolicy app.kubernetes.io/instance: iaf-eventprocessing-operator app.kubernetes.io/managed-by: controller-manager app.kubernetes.io/name: eventprocessing-operator name: iaf-eventprocessing-operator spec: podSelector: matchLabels: app.kubernetes.io/component: controller-manager app.kubernetes.io/instance: iaf-eventprocessing-operator app.kubernetes.io/managed-by: olm app.kubernetes.io/name: eventprocessing-operator name: iaf-eventprocessing-operator ingress: - ports: - protocol: TCP port: 9443 - protocol: TCP port: 443 egress: - {} policyTypes: - Ingress - Egress Please review this recorded video showing the problem: https://ibm.webex.com/recordingservice/sites/ibm/recording/playback/97498e9e7dc5103abdb50050568fc64e recording password: 8GrZrRXh And also see the visual representation of the problem attached Created attachment 1864164 [details]
OCP 4.10 NetworkPolicy egress issue
Thanks @piotr.godowski.com. I have some follow-up requests. Can you provide all network policies in the namespace: oc get networkpolicies -o yaml Also, what is the sc.cluster.local domain? Is something special happening with your resolver to make the hostnames resolve? When it is failing, can you also run: nslookup kubernetes.default.svc.cluster.local I want to see if the sc.cluster.local makes any difference. Finally, can we get the pod spec for the pod that is failing? I want to try to reproduce with the same policies and labels. sc.cluster.local is the typo in the bug description. obviously we're speaking *.svc.cluster.local. other stuff will be provided shortly. Failing Pod spec: kind: Pod apiVersion: v1 metadata: generateName: iaf-system-zookeeper- annotations: openshift.io/scc: restricted strimzi.io/cluster-ca-cert-generation: '0' productID: 068a62892a1e4db39641342e592daa25 k8s.v1.cni.cncf.io/network-status: |- [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.254.21.248" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: |- [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.254.21.248" ], "default": true, "dns": {} }] strimzi.io/generation: '0' productName: IBM Cloud Platform Common Services strimzi.io/logging-hash: 0f057cb0003c78f02978b83e4fabad5bd508680c productMetric: FREE resourceVersion: '15394091' name: iaf-system-zookeeper-0 uid: bd666fdc-6867-4094-a2ee-04b7db04b782 creationTimestamp: '2022-03-04T10:55:06Z' managedFields: - manager: kube-controller-manager operation: Update apiVersion: v1 time: '2022-03-04T10:55:06Z' fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:annotations': .: {} 'f:productID': {} 'f:productMetric': {} 'f:productName': {} 'f:strimzi.io/cluster-ca-cert-generation': {} 'f:strimzi.io/generation': {} 'f:strimzi.io/logging-hash': {} 'f:generateName': {} 'f:labels': 'f:statefulset.kubernetes.io/pod-name': {} 'f:controller-revision-hash': {} 'f:app.kubernetes.io/managed-by': {} 'f:ibmevents.ibm.com/name': {} 'f:app.kubernetes.io/name': {} .: {} 'f:app.kubernetes.io/part-of': {} 'f:ibmevents.ibm.com/kind': {} 'f:app.kubernetes.io/instance': {} 'f:ibmevents.ibm.com/cluster': {} 'f:ownerReferences': .: {} 'k:{"uid":"6fcb04dd-ad1a-40f2-b5e2-c76811769b85"}': {} 'f:spec': 'f:volumes': .: {} 'k:{"name":"cluster-ca-certs"}': .: {} 'f:name': {} 'f:secret': .: {} 'f:defaultMode': {} 'f:secretName': {} 'k:{"name":"data"}': .: {} 'f:name': {} 'f:persistentVolumeClaim': .: {} 'f:claimName': {} 'k:{"name":"strimzi-tmp"}': .: {} 'f:emptyDir': .: {} 'f:medium': {} 'f:name': {} 'k:{"name":"zookeeper-metrics-and-logging"}': .: {} 'f:configMap': .: {} 'f:defaultMode': {} 'f:name': {} 'f:name': {} 'k:{"name":"zookeeper-nodes"}': .: {} 'f:name': {} 'f:secret': .: {} 'f:defaultMode': {} 'f:secretName': {} 'f:containers': 'k:{"name":"zookeeper"}': 'f:image': {} 'f:volumeMounts': .: {} 'k:{"mountPath":"/opt/kafka/cluster-ca-certs/"}': .: {} 'f:mountPath': {} 'f:name': {} 'k:{"mountPath":"/opt/kafka/custom-config/"}': .: {} 'f:mountPath': {} 'f:name': {} 'k:{"mountPath":"/opt/kafka/zookeeper-node-certs/"}': .: {} 'f:mountPath': {} 'f:name': {} 'k:{"mountPath":"/tmp"}': .: {} 'f:mountPath': {} 'f:name': {} 'k:{"mountPath":"/var/lib/zookeeper"}': .: {} 'f:mountPath': {} 'f:name': {} 'f:terminationMessagePolicy': {} .: {} 'f:resources': .: {} 'f:limits': .: {} 'f:cpu': {} 'f:memory': {} 'f:requests': .: {} 'f:cpu': {} 'f:memory': {} 'f:command': {} 'f:livenessProbe': .: {} 'f:exec': .: {} 'f:command': {} 'f:failureThreshold': {} 'f:initialDelaySeconds': {} 'f:periodSeconds': {} 'f:successThreshold': {} 'f:timeoutSeconds': {} 'f:env': .: {} 'k:{"name":"DYNAMIC_HEAP_FRACTION"}': .: {} 'f:name': {} 'f:value': {} 'k:{"name":"DYNAMIC_HEAP_MAX"}': .: {} 'f:name': {} 'f:value': {} 'k:{"name":"STRIMZI_KAFKA_GC_LOG_ENABLED"}': .: {} 'f:name': {} 'f:value': {} 'k:{"name":"ZOOKEEPER_CONFIGURATION"}': .: {} 'f:name': {} 'f:value': {} 'k:{"name":"ZOOKEEPER_METRICS_ENABLED"}': .: {} 'f:name': {} 'f:value': {} 'k:{"name":"ZOOKEEPER_SNAPSHOT_CHECK_ENABLED"}': .: {} 'f:name': {} 'f:value': {} 'f:readinessProbe': .: {} 'f:exec': .: {} 'f:command': {} 'f:failureThreshold': {} 'f:initialDelaySeconds': {} 'f:periodSeconds': {} 'f:successThreshold': {} 'f:timeoutSeconds': {} 'f:securityContext': .: {} 'f:allowPrivilegeEscalation': {} 'f:capabilities': .: {} 'f:drop': {} 'f:privileged': {} 'f:readOnlyRootFilesystem': {} 'f:runAsNonRoot': {} 'f:terminationMessagePath': {} 'f:imagePullPolicy': {} 'f:ports': .: {} 'k:{"containerPort":2181,"protocol":"TCP"}': .: {} 'f:containerPort': {} 'f:name': {} 'f:protocol': {} 'k:{"containerPort":2888,"protocol":"TCP"}': .: {} 'f:containerPort': {} 'f:name': {} 'f:protocol': {} 'k:{"containerPort":3888,"protocol":"TCP"}': .: {} 'f:containerPort': {} 'f:name': {} 'f:protocol': {} 'f:name': {} 'f:hostname': {} 'f:dnsPolicy': {} 'f:serviceAccount': {} 'f:restartPolicy': {} 'f:subdomain': {} 'f:schedulerName': {} 'f:terminationGracePeriodSeconds': {} 'f:serviceAccountName': {} 'f:enableServiceLinks': {} 'f:securityContext': .: {} 'f:runAsNonRoot': {} 'f:affinity': .: {} 'f:nodeAffinity': .: {} 'f:requiredDuringSchedulingIgnoredDuringExecution': {} - manager: multus operation: Update apiVersion: v1 time: '2022-03-04T10:55:13Z' fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:annotations': 'f:k8s.v1.cni.cncf.io/network-status': {} 'f:k8s.v1.cni.cncf.io/networks-status': {} subresource: status - manager: Go-http-client operation: Update apiVersion: v1 time: '2022-03-04T10:55:36Z' fieldsType: FieldsV1 fieldsV1: 'f:status': 'f:conditions': 'k:{"type":"ContainersReady"}': .: {} 'f:lastProbeTime': {} 'f:lastTransitionTime': {} 'f:status': {} 'f:type': {} 'k:{"type":"Initialized"}': .: {} 'f:lastProbeTime': {} 'f:lastTransitionTime': {} 'f:status': {} 'f:type': {} 'k:{"type":"Ready"}': .: {} 'f:lastProbeTime': {} 'f:lastTransitionTime': {} 'f:status': {} 'f:type': {} 'f:containerStatuses': {} 'f:hostIP': {} 'f:phase': {} 'f:podIP': {} 'f:podIPs': .: {} 'k:{"ip":"10.254.21.248"}': .: {} 'f:ip': {} 'f:startTime': {} subresource: status namespace: acme-iaf ownerReferences: - apiVersion: apps/v1 kind: StatefulSet name: iaf-system-zookeeper uid: 6fcb04dd-ad1a-40f2-b5e2-c76811769b85 controller: true blockOwnerDeletion: true labels: app.kubernetes.io/part-of: ibmevents-iaf-system app.kubernetes.io/instance: iaf-system statefulset.kubernetes.io/pod-name: iaf-system-zookeeper-0 controller-revision-hash: iaf-system-zookeeper-6797c898fb ibmevents.ibm.com/cluster: iaf-system app.kubernetes.io/managed-by: ibm-events-operator ibmevents.ibm.com/name: iaf-system-zookeeper app.kubernetes.io/name: zookeeper ibmevents.ibm.com/kind: Kafka spec: restartPolicy: Always serviceAccountName: iaf-system-zookeeper imagePullSecrets: - name: iaf-system-zookeeper-dockercfg-gdxvz priority: 0 subdomain: iaf-system-zookeeper-nodes schedulerName: default-scheduler enableServiceLinks: true affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/arch operator: In values: - amd64 - s390x - ppc64le terminationGracePeriodSeconds: 30 preemptionPolicy: PreemptLowerPriority nodeName: worker2.moon1915.cp.fyre.ibm.com securityContext: seLinuxOptions: level: 's0:c27,c14' runAsNonRoot: true fsGroup: 1000730000 containers: - resources: limits: cpu: '1' memory: 2Gi requests: cpu: '1' memory: 2Gi readinessProbe: exec: command: - /opt/kafka/zookeeper_healthcheck.sh initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 terminationMessagePath: /dev/termination-log name: zookeeper command: - /opt/kafka/zookeeper_run.sh livenessProbe: exec: command: - /opt/kafka/zookeeper_healthcheck.sh initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 env: - name: ZOOKEEPER_METRICS_ENABLED value: 'false' - name: ZOOKEEPER_SNAPSHOT_CHECK_ENABLED value: 'true' - name: STRIMZI_KAFKA_GC_LOG_ENABLED value: 'false' - name: DYNAMIC_HEAP_FRACTION value: '0.75' - name: DYNAMIC_HEAP_MAX value: '2147483648' - name: ZOOKEEPER_CONFIGURATION value: | tickTime=2000 initLimit=5 syncLimit=2 autopurge.purgeInterval=1 securityContext: capabilities: drop: - ALL - KILL - MKNOD - SETGID - SETUID privileged: false runAsUser: 1000730000 runAsNonRoot: true readOnlyRootFilesystem: false allowPrivilegeEscalation: false ports: - name: tcp-clustering containerPort: 2888 protocol: TCP - name: tcp-election containerPort: 3888 protocol: TCP - name: tcp-clients containerPort: 2181 protocol: TCP imagePullPolicy: Always volumeMounts: - name: strimzi-tmp mountPath: /tmp - name: data mountPath: /var/lib/zookeeper - name: zookeeper-metrics-and-logging mountPath: /opt/kafka/custom-config/ - name: zookeeper-nodes mountPath: /opt/kafka/zookeeper-node-certs/ - name: cluster-ca-certs mountPath: /opt/kafka/cluster-ca-certs/ - name: kube-api-access-qlvr8 readOnly: true mountPath: /var/run/secrets/kubernetes.io/serviceaccount terminationMessagePolicy: File image: >- quay.io/opencloudio/ibm-events-kafka-2.7.0@sha256:d264cea91edec7c0975b6d11bf08bb47661fb9a8a982ea5752e65cdae9385146 hostname: iaf-system-zookeeper-0 serviceAccount: iaf-system-zookeeper volumes: - name: data persistentVolumeClaim: claimName: data-iaf-system-zookeeper-0 - name: strimzi-tmp emptyDir: medium: Memory - name: zookeeper-metrics-and-logging configMap: name: iaf-system-zookeeper-config defaultMode: 420 - name: zookeeper-nodes secret: secretName: iaf-system-zookeeper-nodes defaultMode: 292 - name: cluster-ca-certs secret: secretName: iaf-system-cluster-ca-cert defaultMode: 292 - name: kube-api-access-qlvr8 projected: sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: name: kube-root-ca.crt items: - key: ca.crt path: ca.crt - downwardAPI: items: - path: namespace fieldRef: apiVersion: v1 fieldPath: metadata.namespace - configMap: name: openshift-service-ca.crt items: - key: service-ca.crt path: service-ca.crt defaultMode: 420 dnsPolicy: ClusterFirst tolerations: - key: node.kubernetes.io/not-ready operator: Exists effect: NoExecute tolerationSeconds: 300 - key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 300 - key: node.kubernetes.io/memory-pressure operator: Exists effect: NoSchedule status: phase: Running conditions: - type: Initialized status: 'True' lastProbeTime: null lastTransitionTime: '2022-03-04T10:55:06Z' - type: Ready status: 'True' lastProbeTime: null lastTransitionTime: '2022-03-04T10:55:36Z' - type: ContainersReady status: 'True' lastProbeTime: null lastTransitionTime: '2022-03-04T10:55:36Z' - type: PodScheduled status: 'True' lastProbeTime: null lastTransitionTime: '2022-03-04T10:55:06Z' hostIP: 10.17.123.228 podIP: 10.254.21.248 podIPs: - ip: 10.254.21.248 startTime: '2022-03-04T10:55:06Z' containerStatuses: - restartCount: 0 started: true ready: true name: zookeeper state: running: startedAt: '2022-03-04T10:55:14Z' imageID: >- quay.io/opencloudio/ibm-events-kafka-2.7.0@sha256:b965631ad6d833439257e9823ac0f5361a8d3d0271cbb90bff5b5e006e564d9c image: >- quay.io/opencloudio/ibm-events-kafka-2.7.0@sha256:d264cea91edec7c0975b6d11bf08bb47661fb9a8a982ea5752e65cdae9385146 lastState: {} containerID: 'cri-o://ce0e9d48046eda508b54ae30e58a29bed243b03407218c332d3c09820e401db7' qosClass: Guaranteed Also, the problem was circumvented by adding the networkpolicy with the pod selector matching failing Pod, with added egress rule. Note: no other network policy was pointing to the failing Pod earlier, thus it's not expected that failing Pod was blocked egress. - apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: creationTimestamp: "2022-03-04T10:02:11Z" generation: 2 labels: app.kubernetes.io/instance: iaf-system app.kubernetes.io/managed-by: ibm-events-operator app.kubernetes.io/name: zookeeper app.kubernetes.io/part-of: ibmevents-iaf-system ibmevents.ibm.com/cluster: iaf-system ibmevents.ibm.com/kind: Kafka ibmevents.ibm.com/name: ibmevents name: iaf-system-network-policy-zookeeper-ocp410 namespace: acme-iaf spec: egress: - {} podSelector: matchLabels: ibmevents.ibm.com/name: iaf-system-zookeeper policyTypes: - Egress Attached yaml dump of all NetworkPolicies Created attachment 1864165 [details]
All Network Policies
Note1: Also, the problem was circumvented by adding the networkpolicy with the pod selector matching failing Pod, with added egress rule. No other network policy was pointing to the failing Pod earlier, thus it's not expected that failing Pod was blocked egress. Note2: We also found another work-around that instead of adding a 'specialized' NetworkPolicy, the same problem is circumvented by adding a general 'allow-all' egress policy: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-all-egress spec: podSelector: {} egress: - {} policyTypes: - Egress Perhaps, given the OCP 4.10 added support for egress policy, the logic for picking up Pods using podSelectors is not working correctly, and the NetworkPolicies for other Pods in same namespace are incorrectly applies to all the other Pods in the same namespace? Or, perhaps it is assumed that if there is a single NetworkPolicy with egress rule in given namespace, then all the other NetworkPolicies must also specify egress rules? As it stands today, we see that OCP 4.10 rc builds are not backwards compatible with OCP 4.9 and the applications which used to be working fine on OCP 4.9 are not working fine any longer. @bbennett Clearing out 'needinfo' flag. Sorry for massive number of posts, new user here. @piotr.godowski.com thank you for the data. It gave us the detail needed to get a simple reproducer. Ok, these two policies break it: kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: egress-all-otherpod spec: podSelector: matchLabels: name: i-don-t-match-anything egress: - {} policyTypes: - Egress kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: allow-all-ingress spec: podSelector: {} ingress: - {} policyTypes: - Ingress Then just make a pod you can `oc debug --as-root` in to and install dig or nslookup and then you can see `dig kubernetes.default` will work until the `allow-all-ingress` rule is installed. Verified on cluster-bot cluster built from this PR using the reproducer in https://bugzilla.redhat.com/show_bug.cgi?id=2060553#c13. Service domain resolvable with the patch. Can we please get an update on which 4.10 RC build the fix will be contained? (In reply to piotr.godowski from comment #16) > Can we please get an update on which 4.10 RC build the fix will be contained? 4.10.2 should contain the fix. (In reply to Xiaoli Tian from comment #17) > > Can we please get an update on which 4.10 RC build the fix will be contained? > 4.10.2 should contain the fix. Thank you. What is the expected date 4.10.2 will be made available? The problem can be still reproduced with OpenShift version: 4.10.2 . See exact detail about the OCP version / channel used below. apiVersion: config.openshift.io/v1 kind: ClusterVersion metadata: name: version spec: channel: candidate-4.10 clusterID: b5a23363-08be-4f3f-80e6-8f358953807a desiredUpdate: image: >- quay.io/openshift-release-dev/ocp-release@sha256:b807908afe1b7b25a8727f092cc11a104bf3996908a80f035d079b014b4b96b6 version: 4.10.2 status: availableUpdates: null conditions: - lastTransitionTime: '2022-03-04T02:19:55Z' message: Done applying 4.10.2 status: 'True' type: Available - lastTransitionTime: '2022-03-05T14:34:44Z' status: 'False' type: Failing - lastTransitionTime: '2022-03-07T09:30:51Z' message: Cluster version is 4.10.2 status: 'False' type: Progressing - lastTransitionTime: '2022-03-06T12:03:32Z' status: 'True' type: RetrievedUpdates desired: channels: - candidate-4.10 image: >- quay.io/openshift-release-dev/ocp-release@sha256:b807908afe1b7b25a8727f092cc11a104bf3996908a80f035d079b014b4b96b6 url: 'https://access.redhat.com/errata/RHSA-2022:0056' version: 4.10.2 history: - completionTime: '2022-03-07T09:30:51Z' image: >- quay.io/openshift-release-dev/ocp-release@sha256:b807908afe1b7b25a8727f092cc11a104bf3996908a80f035d079b014b4b96b6 startedTime: '2022-03-07T09:02:52Z' state: Completed verified: true version: 4.10.2 - completionTime: '2022-03-04T11:32:59Z' image: >- quay.io/openshift-release-dev/ocp-release@sha256:65e8dbc576556e0296e29ac1ae7496743e3494ad65a111e134c6e7202a498d11 startedTime: '2022-03-04T10:38:17Z' state: Completed verified: true version: 4.10.0-rc.8 - completionTime: '2022-03-04T10:05:29Z' image: >- quay.io/openshift-release-dev/ocp-release@sha256:fd96300600f9585e5847f5855ca14e2b3cafbce12aefe3b3f52c5da10c4476eb startedTime: '2022-03-04T09:14:39Z' state: Completed verified: true version: 4.9.21 - completionTime: '2022-03-04T02:19:55Z' image: >- quay.io/openshift-release-dev/ocp-release@sha256:bb1987fb718f81fb30bec4e0e1cd5772945269b77006576b02546cf84c77498e startedTime: '2022-03-04T02:04:56Z' state: Completed verified: false version: 4.9.15 observedGeneration: 7 versionHash: FheEJUFAAVU= Please note: There are NetworkPolicies with egress rules, but the pod selectors ARE matching some Pods. The reproducer uses NetworkPolicy for egress rule, but with pod selector not matching anything. Not sure it's the exact same problem fixed. (In reply to piotr.godowski from comment #19) > The problem can be still reproduced with OpenShift version: 4.10.2 . See > exact detail about the OCP version / channel used below. > > apiVersion: config.openshift.io/v1 > kind: ClusterVersion > metadata: > name: version > spec: > channel: candidate-4.10 > clusterID: b5a23363-08be-4f3f-80e6-8f358953807a > desiredUpdate: > image: >- > > quay.io/openshift-release-dev/ocp-release@sha256: > b807908afe1b7b25a8727f092cc11a104bf3996908a80f035d079b014b4b96b6 > version: 4.10.2 > status: > availableUpdates: null > conditions: > - lastTransitionTime: '2022-03-04T02:19:55Z' > message: Done applying 4.10.2 > status: 'True' > type: Available > - lastTransitionTime: '2022-03-05T14:34:44Z' > status: 'False' > type: Failing > - lastTransitionTime: '2022-03-07T09:30:51Z' > message: Cluster version is 4.10.2 > status: 'False' > type: Progressing > - lastTransitionTime: '2022-03-06T12:03:32Z' > status: 'True' > type: RetrievedUpdates > desired: > channels: > - candidate-4.10 > image: >- > > quay.io/openshift-release-dev/ocp-release@sha256: > b807908afe1b7b25a8727f092cc11a104bf3996908a80f035d079b014b4b96b6 > url: 'https://access.redhat.com/errata/RHSA-2022:0056' > version: 4.10.2 > history: > - completionTime: '2022-03-07T09:30:51Z' > image: >- > > quay.io/openshift-release-dev/ocp-release@sha256: > b807908afe1b7b25a8727f092cc11a104bf3996908a80f035d079b014b4b96b6 > startedTime: '2022-03-07T09:02:52Z' > state: Completed > verified: true > version: 4.10.2 > - completionTime: '2022-03-04T11:32:59Z' > image: >- > > quay.io/openshift-release-dev/ocp-release@sha256: > 65e8dbc576556e0296e29ac1ae7496743e3494ad65a111e134c6e7202a498d11 > startedTime: '2022-03-04T10:38:17Z' > state: Completed > verified: true > version: 4.10.0-rc.8 > - completionTime: '2022-03-04T10:05:29Z' > image: >- > > quay.io/openshift-release-dev/ocp-release@sha256: > fd96300600f9585e5847f5855ca14e2b3cafbce12aefe3b3f52c5da10c4476eb > startedTime: '2022-03-04T09:14:39Z' > state: Completed > verified: true > version: 4.9.21 > - completionTime: '2022-03-04T02:19:55Z' > image: >- > > quay.io/openshift-release-dev/ocp-release@sha256: > bb1987fb718f81fb30bec4e0e1cd5772945269b77006576b02546cf84c77498e > startedTime: '2022-03-04T02:04:56Z' > state: Completed > verified: false > version: 4.9.15 > observedGeneration: 7 > versionHash: FheEJUFAAVU= > > > > Please note: There are NetworkPolicies with egress rules, but the pod > selectors ARE matching some Pods. > The reproducer uses NetworkPolicy for egress rule, but with pod selector not > matching anything. Not sure it's the exact same problem fixed. Could you show your networkpolicy via 'oc get networkpolicy -o yaml -n $namespace'?. I tested this issue with comment 13 and with that networkpolicy, it can be reproduced on 4.10.1 version, but not 4.10.2 version. same networkpolicy in attachment? @zzhao > Could you show your networkpolicy via 'oc get networkpolicy -o yaml -n $namespace'?. I tested this issue with comment 13 and with that networkpolicy, it can be reproduced on 4.10.1 version, but not 4.10.2 version. NetworkPolicies provided earlier in https://bugzilla.redhat.com/show_bug.cgi?id=2060553#c10 and failing Pod spec in https://bugzilla.redhat.com/show_bug.cgi?id=2060553#c9 > Please note: There are NetworkPolicies with egress rules, but the pod
> selectors ARE matching some Pods.
> The reproducer uses NetworkPolicy for egress rule, but with pod selector not
> matching anything. Not sure it's the exact same problem fixed.
It is.
The bug was that if a namespace had any ingress policy that applied to the whole namespace (such as a default-deny-ingress policy), and any egress policy at all (whether or not it matched anything), then it behaved as though there was a default-deny-egress policy in the namespace.
(The reverse would also be true; if there was an egress policy that applied to the whole namespace, and any ingress policy at all, then it would behave as though there was a default-deny-ingress policy. But it's unlikely anyone using openshift-sdn had policies arranged like that [and didn't already have a default-deny-ingress policy].)
Thanks @danw > The bug was that if a namespace had any ingress policy that applied to the whole namespace (such as a default-deny-ingress policy), and any egress policy at all (whether or not it matched anything), then it behaved as though there was a default-deny-egress policy in the namespace. In the problematic namespace there is no default-deny-ingress policy. I feel like we are missing some piece in the puzzles here, unless you point us to the NetworkPolicy which you found as being selected as the one to be applied for the whole namespace, within the archive uploaded in https://bugzilla.redhat.com/show_bug.cgi?id=2060553#c10 . Thanks. can you run ovs-ofctl -O OpenFlow13 dump-flows br0 on the node with the client pod (ie the pod that is being denied egress), and attach that, and indicate the IP of the pod in question In addition to Dan's request above, can you please add the output from 'oc get events' sh-4.4# base64 ovs-ofctl-dump-broken.txt.gz H4sICOlnJmICA292cy1vZmN0bC1kdW1wLWJyb2tlbi50eHQA3F1tjxw3cv6eX6GPNrIes15JCtB9 ujg4ILGDxIccYBjC7Myss7DsXUijy/nfhz0r7bK7yW52T7e6O8bCgEba4VNkvRdZ9cN3//FfP779 7t9++O9X70+P7/549dUP38GOvn711T/uj2/MP/Dr16/u3u1/+fDmp98e3p9+/qdXh4eHX+9P4a/M zavjx/f78/3D72/QG4t2x+o/3Lw672/fhX9w8+r3t4/7w6+n84c3nsEZdL767PaP8+nDGyQF5yj8 2JtXj+/vH97fn/94A8aYm8P57Yfz/nx68835/a8394+v9odqkQ9vDuevPn331yNxEKkHci8wBBx4 x05MhIIDiPvHm/vwew/vz2/w5vf/ffvh/SGg26HwDnkHz5h+eTg/vL2s9JrMSFTArNbEm2ONQWDS CBT1gjLf4uXT44fz509N+BA0BRZlJFjzAjPeM5Q2vAoI4gVFwMHPMI7vHx5Hrm5RbHR8ZNihjbcJ wzbt378AgepPbz887utbcvn0/PLpp+17hvjbw99Pr7//27+//f5vb3/86/dv//Lnn8xuR/DzN3/6 9Ol//su/mp9+vom2FMaePxoA8BgxAFGQC2LvqEHafUxZnQMaZ/1FKcgxRQJxnju/KGIJDCuRxKER BNPBSNhkpEoNNNkoL2/jlQM4r04jpvfogtoKHOLyW42TYsgcL1Qy//lsp5BuIn5ZKWhFl1sNp1gN kFEgsgUIThjr5qg64aT+HLuXlo33zkTniQbJgiVgbax9P+3SIBSdZNA5qhytaHKbSnjrzAnvoqXI K7qdt/Z5JagtxYDoYzvL3tuw4Q0Czx9/f1ZhuAPZSb/wnALzOzRtMAwZMBQExqu+oGHLwRlhsJ1w ZAe+H87RyfFuv+cWHucxh4fAikZHoRT+k/D/Tjx2B9b04nFHH4irqG3i4exhWeBwOhGe4JAEf6Py 17qOy+ygX88cDkfZg23jsV4yeJwL8hHBCXwaJMRSD/dQgVt2ezB7BG1zj2WXO63gImLMPEHVhr0x 3WBw5/rRCO5JbyuV00CjPocGNPivkWAFFrVht7o5h3die8F43Ju7IOdtMJITLETrNHZb1FtChh4w oL5/a45G/fHUFivxPrc1YigOMcAG/ymIeZ+QF6DxQRyCgr5toxHOsY3hYExivgEbPAxC6WYcKBGq O7497UUObTxGc3iCEHGlYT7jcdYzWyXfx8hXBztQ4kyYkSYdS768Cp4+Hh9vzo8X/5Ot88XL6U4c ZpYDDHY7jiaDptC2A/Hst1DCifzkWP7P/o0xr6uf/eu74Ni61wa/ffrk7u755xn1u4f98XUQk8Mt +g7/uPuYdMfIOdJEJdIzKoFVukjjBGmUJY36SWMMFlquoA197tgqhzoSTKNBw3eQJgnSOEsaF5ya t+yPV5AG4jKkYdCCNooVgsJpacCYNk3QJlnapJ82b/dmb0fSps7siE2GNkcQyXcIjw2aDtJsgjTN kqYFHGkPeDpcQVrwJHKkoauRZq12keYSpNksafYLkGaypGFkcro40Sdoclma3Kw0kTMegvnL0lSp jIiuoE9MlwaBZPYAstTdlsjZrT2Ooo6Cew87b3JiFgLvKAxXo+y6SEvZNMgbtUM/afbEYEapRwqx oQ28iFntGB2a6zyxlD2DvEE7FpxYlbvcjyTL2p3Xa0UMUpYM8qbsNDNRanY2rzeMickyXfoQUmYM 8nbsbmYmNMHSeMoQphInHq1AJ2UpVQ9ZXQ8wL2XGBvcAJKc54ii5i6iUroessgecmSgNUSlkHWGO ncUQRHX6imhSpPksaTSriLEnv1POOosEEWk+hFadpKWMGGaNGMi8pIX4ZOes5hyqWplIne2KXTCl FzGrF8HNTJlIBSLPj3GqDrHT88CUZsSsZgQ/L2kc+NE6vdKOYUopYl4p3s5KVKXvd5q3Y8U+B6V0 B+Z1x7w+R1V+3AXicmyoPiLMgsFO0iCVHjBZ0k4zkxYiFjY5tcjiJc6wovddJppSepHyevFuVtrQ BQ9Yo8JM06vyGvOjoc5jSylGyipGxHlJsxVpmItbNC44ob/YtjxlqewAZdMDyDNThrCjbFInsGAU kTlzKR/mSUtpR8pqR5zXTqOKBpcxn0GNs4yCrjPJSCnHkbKOI+rMpFX6WXOkEdu4mi22y6RxSvdT VvejnZcyCZLGlD00oxFpHBySTtJSup+zuh/dzKQxhlhDs2YtFjV12ClqnFL9nFX96GcmDWnnJXdq NjbYRNhpsDmV1uFsWgf381LGVBnsrKTFxWLq9Ic5ldfhbF4HZ3Ud++tLloyLfJHgmIALLNq6JxJV mFr3AnFo0Qjt6KIRWBtXp0MU5pA74HIbLg0uBPXhzReCQJ1yVMADJ1aBNA9Y2oB5cHmnD3C+vBM0 r7DEiL0oe9tyUO/jEk8Lsgyu2nRB7q7aeLUa18HBArNgy2Tcx5WbFmIdnN3vQ5wvxnhVQKkhtuCD 85lH7NqI7eSI8zWWYKeCisT4YqwPZpkQJI/ZtzG7CTH31VA8myBrEWTwYIO5dXnEkLqKDYMrI1nM fZURrG6LR3wRAhJw3AU4oY4BB+cu84D76h3O1Yo5wVBZZNPBFJDQyECDTV4X4s5ShnOqccakumGh jNih3yChk4EnhdxZqKDqImWUCtHA89K1xQmFDDIpU3TWH4Ja8LGq8E6Dl9Fh8yCh3cBOCbi7rEDh /KN4g6qaiXRJXUKzgZsUb2fFIBi8WhQR2CcocCMdflD4gjZkPyEP91UC0KH1cVCnWmXJqMPmYUIb I0yJuTvFD+KB4ljNVBcAO/Am9ATypHg7E/fVreXYDxIOfNIld5hQFChTAu5Ox4PB2Jm3IOo69DAm 1ATaCeH2JNqdJTKR6UABIn+5u5uNlhJih1OKXU8S3Vmv8TMU54N9tl32mSARMZlJEXfmxin4lcF7 jzAHyD5sPAp1oE6oCppSVfRkvZlE4ouaYIgVvXbEIJTQFjSltujJZqMPG2swLs+SD9xsu4L/ROBE Oinm7jw1BIshNadeQ+Bk1XSF1JTQGzSl3uhJQZMXiKM9BSBLHYqOEv4FuUkBdyaWgy13sQfnwTm4 PCDNplkSeo6m1HM9CWP2oPFTq/BHDezd4VxwQs+xmRRxZx44aGGS2qMPoi5LwgkNx5NquO7kbgir OL5+Q/j0UjaLNxGZMk6JtztlW73A0thHDnZajOMuJk6EpjxhaNr7uABjA+JGvV1oJHul5JXl0y58 4bTuNchmzuBeA23mXO010GbOyV4Dbebk6zXQZs6yXgNt5mSq1O8+KPs4eofq5ZsvQDlz/vSaDZw5 U1qDhowY38pVSUXmXzo1etXuzZsCvQravNlOqTtGl3f1L1kXg0GUXAnKeVOcV23gvNlMqb9Wt/Vn rM6J4RKQ8+YvayBt3DnCivoS4Z05WSn1K1y1h8lF8ObNTdYPOQgRxnUi8si2BOS8+cirHOR5c4/1 /cM49jJYAm/ePGNDhmvdOBgwROAlOnDm1GINZHWLwcaJGBIUAi1hw5mTidK4+czxC57gWDBSCch5 04fSeJlu4puV4ETD7xTt5bwZwyZMiKu9lh3Zohhz3gRhXXqc1NS3eC/CJSp85qxgYyttDFIRpAji vGlAqafhvfj4aalUtYSSsH3ezN9VGYV5k3xXQZs3n1d3sykqHitwkYc9d/ZOpuo7UjVX8SEejai3 9cKHl/o7wbBTGjS0r11zo+d9uHTBsxjs65iecX1wuLIO5GN1AGQ0MJZAs2ncc7fNf37/+C7TcSx/ gVV3ljWHIr5ZSaqmXqF4ahj5/vSLefP53Dtt0dV4urrYdQMhjYPBBYFwLapaGIhbydFYXgMQ0NhK LXs0HleyI7QKZg0eKIxBItNvCcYR1YJAKo2mawASuBW/wI6YkqOJNFpXHW+m9TVOJi+yPpqr1wes +RxuGIZ60vDL70GQCuYlz6CmHhahX65fP35mpt7xYB5YdA8qOZBlz8DCgutXxhLNwkIACx+AvV4R xp2WjRkhBHbhM8CFz2BhY0xLy8D1zkDc3Ac0dXG0jwfdwjywpDMQPFO3sDGOXPRF1teFz1/tsue/ qCEO6+uSAUG9groM/bzw+fuFbRAvK3+L6r9L1kYXNgC08Pq49AHgwl4QLyyBujD911tgX5v5YmC4 GSC/NBcubAccL2uHl/UDqgNYOiOiC6sBWZYBF03IhPVl6UBkaTu4aFq6XgdfhAFg4Uh80dJMtf9L l2aWpv96AYy7VqmnEdmohaMhSwsrwYWVMFyfDcDayM1Ux4A+HoBl5ZAXNsR+6aw8L+2JLp2SWdgR wCGDy9jiMAxiPq9/eUw8fPBb2CByY9d8eiY8fFEJ5jF7M1N8/MrcItRuQL4s/vTadtziMHqXL6+P hy5avVJFktHbfHmGOmzRp/enYHOL9r8/jXe6el867MLr5WGpU8j6FlhrtizECJQm/+ml3HRXjqD2 ggEtmlrzt4jui0hfO0yx6LBHXmqmetMOY5xw3H62ejYUhDTROvj5FvE3T7eIk7PJu8d/H85fHR5+ ++3+fPMJDHxdvit15BaMI88m7gXFQkpsa88N0hfPYMCy0JwpjvWZ4sY1739XzYjbw+df3JqHj+fH j+fXOBYEOhGKx0uziqnmskL+IvoMMIiBxdc6ObILUmwarn/YstYV9DYu8y1izChPK1pz8/v+/PVY iFC1TNW4+TyCekzMgG+fVw1QxDxsRm+YWHQcD+hWG5QaWafSnvPevUMRIDsaUDBtcbu6ap4tSqq1 dmN38u8ZZDSWwvcVMOx9hY7Gg9VAaqM11YhWnGFPmgfUt0F+NKC80Y/7Yn4GgnhBEXAkr84DTo0j xH0v29C3OoyXIZJ4hJBhrM0MN5nnLd5PTO5FLkb2BOJRA4bbGgqbqpzKu//wqEnAbQzUxMDlbX54 zMTeNgRuQpDydj7cmqzLtcm67SfUaQzSxKDlfXt4zATcNgRtQhjQn4fHTKptQ7BNCK68Dw8PGhPW Xts11/bljXZ41OTXhIvXcq4Ai/vo8IgJrQkILaUA1NMvJ7sPvVPNEsu39AFwcS+ca1kAWpoApLjb DQ+eTJoA0FIDoMU9bXj4BNEEgpYIgivuV8ODJn0mFm/JIPjijjQ8YiJnAoJvhTflkzN5xOTMhFVu R1hY3FGGh0+4TCBoyQFKcdMYHjGJMgGhJQmoxS1hrtUD2BICdMVNX67XgdjiwQETGHnEBMaEV2Za EKC4ZQuPmZSYwNCSA8Lidiw8YqJhAkJLEEiKm63w8MmDCQQtr4zK+6jwiAmBCQgtaSBX3CWFR0zy S0BoGQbyxS1QePjEvQSClkxy+WQ8HjEZLxGmtGSSobh/CY+YYJeA0BJJxuI+JTx80lwCQcs9ZSpu R8JDJ8Il1m/Hq1yeGWhsgasNLNGS5DzhrTMnvGvk6N3OR8pPGl3ia6NyCk6aTT1X+NvD30+va+1W nruv/PjX79/+5c8/mV1Q1z/ffDid397dn94dX1ffFDhQdqLf/On88fcqq3Pz2amL6DmFGMKhaZPD kCMHTdyKPSg1pQKDAjgRTRJ8xG6ajk6Od/s9t4hyUe6iThTQxTd71ozVpJUChgSdiCgbHH/TSZQ7 +qodt7aJ4izjWa7JGAhCkZhNxn4mHHsnVYfDUfZg21RZL9mjsnETRClR3m4qgmgHnfTcHsweQdvy ZKMwpU4PxTMvyBUYZJpKlIJMum56BPekt1X42KAnuFNZ/RA3YguOli/QeFMdUIiHbSdBHvfmLmjw NkGSU3jBuY1PyJYEcWYyekIE0X1CR6P+eGorO/E+J0E29saDBJV44zyd+u6hyAd7Yglu2xRFUW6T onj4mEWhAq3AU5EE0Kfm7vj2tBc5tEkymvUaoF664hIrSzqdaujWDD2eVp2Ya65AJL5czYjCIxpz ReVcp6WnXk+yzbyV9/FFXKQQMVvfV7MsmjcBP/188+nvKP4c6y38nCmvRDXRBzGNO8yoejHWFaAv mEkRwecr4MdFrDp8cGJqDrtayxZKNr9gbkUEX66AHxfAGvCrO11xCQxQjYDnkv0vmG4REaAjCWhW z+oEeO+Z4rhMVC9EFeAvGIER4bdX4I9Lb038zkl8JSNsvhc2WoDfDsLvrsBvsvirgnwtWSdqVMmJ twUUuEEU+FEUtKt/DREwholia8AihixRAQElkzYiCg5jKGhXDhsa1LpaZxoQUU9cgr5gGEeE/jgK favoaBvecmCWWP8H9zkcWZEAl0zqiPCfRuJvVC0bAsyC8X1KNGCrxtypKd1tAngQAXcjCWhUPhsH oDZu6sqoalwR98gg7gEzEn2jbNqU3mCb4wyxVuUlKtp8Oww/jsPfLLo2dt9S/IiK0Hssk103DD2N RN+o2ja0P6KzNeYn5KcD6SfAD+J9GOW/tWu+Dd0Z3E2pvWJzJFaNL7G/JbNGYgp0HAXNknHLAa3N eAZlcKYE/TDdA34k+ka5uek+YO39GFy8hxL2KRlSEsPfj4PfLFU3N59qAw+RneUS6S2ZYxKjH+c4 tGrdTd+TfM30smFxULtVniVgoPSOs72tanmdABvivVp5UFGr+kSJ9i+ZghITcDeSgEatvaH9wSig r81PwOD+h5isxPukYQoIx1ngVq2+GUIyQhwCY/hANPxSCQXDlBCOs2KtUn/TBQ1BF8TPK0RtIMsU JSF0GAUykoLGVYHmGVjHtTCMhK1H8FhiCmiYNsJxhqx116Dlh2p85YKruMBxEX43DL8dib9xUaHp iXIwXq6mToPrLVLiStAwdYpuHAXNiw5NLnLqLMfuHFbTv7mEAh6mT9GPpKBxT6IlyWJjHnKAxpWI MQ9UpfuR8Bt3LGxjpLaIqT3NC8ElZqbpNPDjMPy34/A3b2g0HApFwrg8acgIcpEt42GxPB5G50Pr ifqGCHANvU5QfHDtiwNUP2RX6W5MPuVrNFJvFiRqO1D8JNQ1LlhB8Jlar/qsI2OxdLDMADyNgRDO FA4dOB8uD+mh9pC+aYtvzo+Xz3wQ/VWBu/Qb+oyuusu3JnSXRiTP4MisGhysGRyuF5yseef0y/Lc x+OKeW4oOFwvOF2dKgH7GRy3r4auBptfGbhLt7iVblwN2xo3zq1449xaN+7Slyryl2C14IKr6Vd2 rPKsgatZPasF580M5qF2mxRBxI2II+AFoqe1BTkxuFUd7qWX3lpltgZuhTLrca07VwO3tp27tO9b c9S/XnSX6YhrTUnUwK3NObnMdVztzsXgrto5mQUcPmth0GscAJnFur6EEqtDd5lkuVp0lzmXL6pu dVsnz+Cq9ysrBsfrBXeVqpN57OuLOnErY7oWOljZwYJOkZmYG9vaTNhldMdKN66GbRVeU3PmSFc+ gEn9pgB7WEF+uz5cCjmRkF9XaNkYTu22BPhSxzU9SZnVIW4xxboRX8BtbItTTIHPsQTCDGysJm4i 6qdijJlRX8cYAdz/K8DOuU0BpvUBTumK9bLEUwKkKykiZmVORS9i3RpiVLJ+bYhlc1whm+MK2RRX PKUjt8QV/Yh1a4jXyBVktsYVZLbGFbQtC/JUHNgSV/Qj1q0hXiNXWNgaV/Qg1q0hXqW3mdJusrkY RDYXg8jmYhDZXAwim4tBZHMxiGwuBpHNxSCyuRhENheDyOZiENlcDCKbi0FkczGIbC4Gkc3FILK5 GEQ2FYPUr19uIQaZ8MLoEog3EYOsnytkc1whm+KKlH+8bq7oR6xbQ7yJGGT1XEFma1xB27IgKf94 3VzRj1i3hngTMcjquaIHsW4N8VZiEDJbi0HIbC0GIbO1GITM1mIQMluLQchsLQYhs7UYhMzWYhAy W4tByGwtBiGztRiEzNZiEDJbi0HIbC0GIbO1GITM1mIQMluLQWhbd7HqjR62EINM2JpiCcSbiEHW zxWyOa6QTXFFyj9eN1f0I9atId5EDLJ6riCzNa6gbVmQlH+8bq7oR6xbQ7yJGGT1XNGDWLeGeCsx iIWtxSARYt0a4q3EICvnCtkcV8imuCLjH6+YK/oR69YQbyUGWTdXkNkaV9C2LEjGP14xV/Qj1q0h 3koMMhVXyBKIdWuIr+MKmaV1Zfu1W62l9BceSxfAmvBV9sRgjnXok8OwLpqBToxGOY3m09bNi6Z3 U5Ze/x8WAdlPBEPGw7D7g93/X3Vn1BrHDcTx934KPybU2c6MZqSRoXkqhkKpC03pQwjm7nZdDG5s nEtpv321d4lP8a5u13ebeBb8YjDS7z8z+mt0PqRnx4hhGerV8yXlerhv8XvfzX5+xu2DTrYZ++K4 4TYeR2uM2xdjeh4HssXIYJzx0+M2tiG3bxeZT3Ynjhtu4wVpjbG3IK1B9hWkwWT3LWwX7TukKcaS Q5qCLDiktWR34rgNrvGKNAfZW5LmKPtq8khI+SqhJNdz93iwBNnyGGdsH/yYA6PaZwxgnDGtbO9m wMj2Gfu+6G7Le/oMcrPYjRukNcY+g7TIqPYZAxhn7DNIi4xsn7HvP2B+FgapMzBInYFB6gwMUmdg kDoDg9QZGKTOwCDVukFuCsC4QVpj7DNIi4xqnzGAccY+g7TIyPYZOwZpzntKBskzMEiegUHyDAyS Z2CQPAOD5BkYJM/AIHkGBhnQvkGaYiwYpDlGtc/YZ5CmGAsGaY6R7TP2GWTAb/+v9tKXx3vejFCz dO03YMk0nTNNx2bpWld8hnPV2FVBlUPTdGSaznBmfeXALF27ZoNpOjVL1zpKME1nO3bRLF3qoBRM +x2bplPTbsym6cQ0nbe7V7Rf3iXbeGwbT2zjedN9SrRM553pTsBbpjPcCbSf9Njuosg0nTNNd9hu ESEkQg5UIMTAGHSHiQzivHMZrTxcvyCrJcUjMJzTAgY5EaYMw/so0IL1YDBFRjmYQyrSUjhcRGzF P1yNkf5cPPdzxLCARTiKA0vx0EA5BjnUL67oyJISA8f6QAqvqcDcE8s3y0RYUbM6aG6nENM2Hkpz CzmOvAPwBCyIVMhEeyHFfo6L899+f3N5/svFnyf3zd3NfycvLs6xci9PXvx7Xacx6OVZgdV5ZqzU F5MlKtl1KgkSlKCfdHu7y8E1Eyr28WnZ2u1X9f3tXTa4o6VCQ1e50uhJqxjCwxTxiymYg3eZWxDE oOIjhtzdYONujxwMKviBdv7/9+0/TSYb3r579frTr2/++PXy55/eQlU5fHf6oVlfXl03N/VZOxJV KJX4V6/XH9+3w59+CiBmuhpBTfHvymIsySJI5ZjrIh+U0iiDutL5dhpdUmHcr6tWqa8WC+4I00gF YU6dePFZbYaIaWWltTQozE8kLKRFDnuFaR2Z1PuuMC4Vog9pD9Ws4hGwlYUah4TRZJUIKft7ha1W tSwwdIWFKAVhiN4FiZkwSXuhhFSRg8J0KmGuwr26litYEPruEguspUoMgMJ5n5FkavQAQ7LcVCss LVXdr0to4fyy7UMe6fKxpCvpEM4WmHMUXdrXh1RNlSuuJOzVFGkBV8nnu5qkZIfJHTC/HywZPEsk HdIEk2lCH/cnqgYf66ZrhRJjKVEuWTpmiUKi1OvR4MY1nb8PiIreQXC47IoSLm1cDhxo2KniEMG5 qMP78VS6EIdc8IqXzUJk1dUFvqQrpC48YO6CguLYswzahZ/OLva7xUBzFo9qzoYGjyMGJ9jegpgF aLMde/l8xaF0jrz0FRg+1t+IwTnJzpAKnM4QvcH+63Z9e7kZISUbxk/XVtzR8+FT5sORHyes1pcf 1ot18+P393c3RwT00YyjFR4wE062QLZ2wtmpHnFU4LILPqduA8ca5UjfGdvNjGzmxh6rxu7NI/uS sbvi2NPR2MPGab4A6SkLniYt0lhB2HWSJK48+OcR39+umzNIPXBdATz8fPc/6DDEVXOBAQA= sh-4.4# base64 ovs-ofctl-dump-working.txt.gz H4sICJNoJmICA292cy1vZmN0bC1kdW1wLXdvcmtpbmcudHh0ANydW4/lNnKA3/Mr5tFG2mdZV5ID zD5tHARI7CDZIAsYxuD0bdPwpRszZ5L1vw91uqebkkiKUktjag1jgekdt76iisW6qfj9t//+n39+ /+2/fv/fbz7cPPz825uvvv8WDvT1m6/+dnf9zvwNv377D2+u7u9/ursJfzIXb64/fTie7u5/fYfe OK8Hj/7jxZvT8fLn8Bcu3vz6/uF49dPN6eM7z2i9d+efXf52uvn4DkkFxFj1ePHm4cPd/Ye702/v wBhzcXV6//F0PN28++b04aeLu4c3x6vuIR/fXZ2+evrdXy/kIHKA6F4wBMGpqAcTUXCAuHu4uAv/ 3f2H0zu8+PX/3n/8cBXoDih8QD7AM9Nf70/3789PektmIRWwkFUbLY4FcMaRRlA0CWX+gOefXn88 ff6pCT8ETcGiLISN3mG8ZihjvA4E8UwROPgZ4/rD/cPCp1t0Bl8IyIhVxzFHWKbjhxcQ6P70/uPD sb8k55+eXn76tHzPiL/c/+/N2+/+8m/vv/vL+z//13fv/+VPP5jDgeDHb/749NP/+Kd/Nj/8eBEt KSx9/2iAvVGK5CLP1lpiGIh2F0vW14DBu/6iEuSUIkGc184vSiwoLlIkROOYS4qEQ0XqzMBQjfL7 bblxAA9EHCmHJ0OEnX3ILzWuypB5vdDt+c/vdo3dHeR8eRIwu9zTcI2nAbIajs8CIlLXP466N5y0 n0vX0jIohu0d6Z4JuugMqJfBs+/WfTQI+Wh9AcnGC2xyi0p46cwN3kaPIm+9OzDZ5ydB71EMbE30 MBFCtkoDAU+ffn02YXgAOcj05rkRcA7NGMZABiZsFhuO+Rcatszqwj9FHDmAn8a5dnJ9ezzyiIcE czwUDnMLLzxK4qxR1iKPPYA1kzzuOjhbTnXMY7Ivy0I4fiKvSFkgHD4TPOF/YdrOXF1dyxHsmAdF MjzOd55hpKlqjWVSKONQhVt2eWWOCDrWnmD/s9pjDEbblYNLIyRchsGDm6YRPJJeMo5oQHI0oGFt ooMrqGhYLfBFGj6InYTxeDS3YZ+PYSC3sYIrbSR2W7Rz8oEmYIKlm16aa6P++ma8rYz43NKIEYiO quBCG8PWT23yChqvYfcQXI5pgHNqEx6NEm1yBhu2lTcTNhBqNtUtX94cRa4GPPbgVXM8Gna5i/TY uWCURYWmFPnVwQ7UOBNm4ZGONb+8C54+XT9cnB7O/idb52c8zjBmHgcINo4mIRx7OnYgnv0WSjiR T47l/xzfGfO2+/f49jY4tu6twT88/uT29vnfZ+qf74/Xb8M2ubpEX/CPy69JDs5xTjTR3pEgYauX ROOEaJQVjaZFY/QM8grZrORemz9nHZ43punO3oJokhCNs6JxxVvzlv31K0RTcBnRMDh5caxgmFzp tWlCNsnKJtOyeXs0R7tUNvaHYLUzsjmCyH4Re3CuIJpNiKZZ0bRCI+0V3ly9QjRlyImGriea9VgS zSVEs1nR7BcQTbKiRTvNlXaZT8jksjK5TWUih2oO6rIyBXc5im5c+BOV3hcksweQle6yZp9d2utF 0pFq8DJIc9sMOHZb1HjGkmipMw3yh9rVtGj2hsEsMo/UeTRBFzFrHWP/pyhW6jyD/IF2XfHGUKw7 LhSL7YHxtVsMUicZ5I+ym42FogCRtxsmOsecYOkUg9QxBvlz7HZbJUTbPZ0ygqn0Uv3ii68sZeoh a+sBNpaM6eCt5CxHHCWXhErZesgae8CNhSI4OJt3hIF6qRouaSKalGg+Kxptu8XA+AOYrLNIELn4 3lLRw8fUIYbZQwxkU9E40B6INOdQWd/LS0DpeMaUXcSsXQS3sWSdNwdZfWQnkT4Okrgj0VKWEbOW Efy2ogVvIgDoK88xTBlFzBvFy02FInUmnKLZcyyuaRhT0kNKGQ/MG4/rjeVSe7A+a+zDBnyRzKIr 6iFBKj9gsqLdbCxaF2hqzi4G20JxitXZYgxNKcNIecN4u61sRmzwVG3WBe75VY6KjhWlLCNlLSPi pqKhD6IZlwtcbFzNC39Xi3stlR6gbH4AeVvJnDcHl83qgI+bVpzx5SxjyjxS1jyibCwaSvAZ8ylU jWuwZIo5HUp5jpT1HFG3Fc16OljMiUZxHA1siz4xp2w/ZW0/2o0lC3G089mXZmwUw3DwoEvHNads P2dtP7qNRQM4aNYHAaWoUqzOQvGtpUw/Z00/buteoQZ9ZMi9taCtcYsRmWI+P5XX4WxeB4/bSia+ O7DzO6230UqOCKcSO5xN7OCmviMEBMO58yx2HItnmaZOac2e0my2lGm6aGYJTbTHJJzqSF1SZND8 EpXNRs2OOLcShnZxJQysk4g3hJbBiHAel8e4NLu6NcWbr24FXTkfNS918K67BCQPLGNgnl2zmgLO 16zAqDoftyZ5CasDVEDWMbLMLkUVkYulKK8O47QQWJAQ3I7csru4HDUi1tkliynifIXJq6p3PWIH nqiw7dyY2K5OnC8chSX2SBB3+3oVsEGb88x+zOxWZJ4qDPmuyyNaZQqqbEQF8sSQ6i+H2eWeLPNU uQfBxcSAIdgd10buehWfMTDOTsjmgaeKOM7buHtOHHStP4WtBwmLDDT7yCsRF+szIbijXhoIsWvR G7Um3fVKNWNkXhW5WH0hwNgiWxVRKOw7SBhkkDWVolxUcWQlbtIzxopwaYET1g3sqsDFWgkx9dxx 9GC44FRAwrKBW5W3WAZRaznOqYMCe3Ji8sjhF4yR/Zo6PFHeCD6W4Sg0YHUaQglTMG6YsMYIKzJP 1C1AfPxdgjpiKRljTNgJ5FV5i9UI58677Fkrup5CNAXHDROGAmVN4HKNAczZ53k2bBDckNL6JswE 2hVxJ6oHznb9vVFaU7DLR5fCD0psO/SrIhcLA84Ffx/jHjQTflJy2ggSEZNZlbiY7yfP4S/4KMbz znknoMWoNGEqaE1TMZHJ7zriNe53NiGIJqCCNlPCWtCa1mIiQ0/GUK/+4K2SRQcF540SgRPpmswT uXdwYWHjD3bCugcP1DOVNmHCbpBdlbqYVicfoo7eaY1gSoc1JfwLcmsCl5PlHPZg/PmX776KBCwQ c8LOkV+VuJgE54AYh6c+hH7ktLD9OGHn2KxKXMxt4+AzKETVkhJzwsLxmhbuKWGdc+qDJ9H78qbz hXxpfRORKeOavI9p6Bxv95GSo94ndi4YC1Na4kRoyiuGpo8ZZqipmEJ3eBQSFZo4PHTFw+Pzxx1k pttDoNdHYWZ9zBG9Pan5yvVxFb5EBlrWIdsi2bzSom2RVvbroG2RPoZ10LbIE+M6aFskhFdC2yLv m0PToMbxyaoYzlZxFZRbpHpXUrstkro5O4LBGYm7osPiaQ3iBlncldRvi2ztWmgbJGY5gybAcYMP CICeQ+1pyg2ysbrSAm6QeM0d/kRWY9MCnli5BnKDVGvurLVx6BbCpqoXvEVe1WYAIU5S2jq8DdKo 2ZccQoW4pGXFeGtrIDdIndqVHOQN0qQ5AxgPNhGCKrwNUqKa3cNELt7DxIA13sEWWdDcHvauq0pG 24TEWvJVi7lF3tPltrKl+Mt0DM6ZitRAbpDpzLla0BWf4gScc9iluGowN0hu+iwmcC8FIGC1SjE3 yGXmfENy4UCJc5nGPE64mabcIIGZo4TeJ0cYLAL5GsQNMpY5RH5MqL582mvDqVSzjlskKVc6ZTbO R74KbdvUYz8A5V63hqUaD3HTROOrEh4bpxVlrYE0j1OAxEWvxQ6m91nCeCoRGwIk8mjGwz0/j0e0 GA7+JcMEp3A4nK3W9D5LAALvjFc7HNz4PIb1Hz88/JwZRZdvbA6xPpgMRWmc4MPFh5u/mnefX/hA NbgXev2OIKRxePo7r4hvAST8WjaNrIjHRnTEciOvhprQEYTg1rehrRiHVrUgcfpjRUOiLYCEbYMr rgiZOOunQLbf41GhLL1IaMZwLE80b3UCyxPIuYS35JnW+6XPfCzOzX8oB+uSdTs0ziaFqM70JkS+ PPyxxrXs4bh4lc81v9kP7cpWcYZl5jKfiz/zHvpU9aHcQ6erPvFKd1Wded7cYzkHcwt9Ht8Yp9Y5 AJv04x/z0+uZXuk11jrpxl0O9ndvcy+aIUkzDf9Cl536bT0GiLnfru/VdJ8VjlrrXq4qePSRkyPZ y1PPr05fXd3/8svd6eIJBr6uX5U+eYirNewSjbtFwx+Di68uvUzxy4AZj4XBKPVe5iOoIPXGrdDT 54rjmfsvsfv9p9PDp9NbXAqBTgUhnuoYIhrvSTUfZm2AQQy26w2MP0d13hhvpPcNUHddwjDAGnP1 riR4vv0iRGq/Hk9fL0Xsxp7HgHK+isMlRt9P3JEQKQ+bxQsmVjzE057Vkglm1LvE3QflFYqA7GIg i5ZM74YJVk19UFw9+l8Ws1RmD2Be9kAX86CGxTk7ci+mkVBZnHWUB5paIL8YKH/qp66dSN1FEt8l gWtzoETLMPV0WL6HKN7Pztj+l7O5ewS8X1nc/mUJ81rxeNFc5bGFwqEpp/qmO140AHnMQEMGru+u 4yWDiscIPESQ+i46Hg0UNr2BwlhjDnsNdU8MWt8ux0sG/44RdIgwoy2OlwzoHSPYIYKrb3/jWdPR xs92w2f7+v42XjTwNuHijZwrwOr2NV4wmDaBMDIKQNXfHvPMAbKJx4/sAXB1C9prVQBGlgCkusmM Zw9kTQCMzABodSsZzx+cmiAYbUFw1W1iPGvAaeLhoz0IvroRjBcMIk0g+FF4Y6rbvHjBwNDEqTyO sLC6kYvnD/ZMEIz2QfYGvHGvFi8YwJlAGO0E1OpOrNfaARxtAnTVvVY8d6Bl4vkjJZwcPJndhFWD JxNumRkhQHWrFC8ZEJlgGG0Ewuo2KF4wyDGBMNoJJNVNTjx/4GKCYOSWUX3/Ei8YjJhAGG0HctXd SbxggGECYXQykK9uPeL5gwYTBKM9yaa6s4gXDARMxCmjPclQ3TfECwb3JRBGW5Kxuj+I5w/YSxCM /FOm6jYgnjsIL/H8ccDKlV0+PG9gXSJCG1kjlfq0xGD548FL2LMES+42fDG80je8cXYXa/wPNvnb ZR87mZ4bm3oXn158vDm9v727+fn6bXRD4jd/7K7ouv54uvjsUVbdjog5cbB39axH1RrHEnAlmc7X LBZlKlyxSBmhgHz8PZG4zqerEEpXEup8V2NRqMI9jVnFsxyrOIRIyVSdd6upX3fjY1Gqwm2Pmn9V 8YuqSiu4tQSiAxTlKdwX6TPy9DxDqvELaa2tdL51sihP4cZJn7UP8bczvmsXnBZprRfU3VxZFKhw a2XO4FHsMgLZmi1kVpMH1JffUPbiy2iC32AH2bjma4WpwvPh9cz3hESFyzMlK5FEOecQ75kKT4bX Eul8B2dRpML9mzbrNdjYMCCPv2BImQZdzzSULcOEpyWr3+EZ/XI1C6qeaMwryva6tjxxMcsOkmZd DSXOGLFVdXaqYFo1YwJ++PHi6f+j+OfY7453pr4MNqQX7XkdGqJ+8a6CvmIORYTPr8CPK2h9fHAC 1Md3LAwV+BWzKiJ8eQV+XH0b4EPQ/d7YXlQkzX0b2hegYqJFJIAuFWBQuhsIYMI/sTEP6hRewHm4 6KQAFXMvIgHsKwSIC399Abz3vbITSQiYQBkr+O0sfvcKfsnyu7D60MtQKXQfpyFIhQRulgR+kQTj 2uNIhYTjw5NZrXW+yoLWjNeIJLhaIsG4bjkwoZ3J6cVtarw3NfQVEzgi+utF9KOSpx24y91Iz9hj JmIZF9+S/BUfmEX8Nwv5BzXTwQZmj7EBQoMiLvhfNTu4ZohHJMDtQgEGddfBC1Af98cxOjFapT0y S3vALKMfFm2Hu5d6Lamd6RRXpz12Hj8u5B+UfAerH3wLG2d1g6hYRe/m0dNC+kHNeGD98TG+fVF+ CqEYS5Xu+1m6D7xMgGHFeWA7wVpycXuoUxcC3qpXUDNgJJZgkQs0LlgPPVDt1UbAstiasxfn2R7w C+kHxe6R+xDPmEVQR7npLgN8mYd/XIY/LJQPF5/7lxsIs1TR23n0yxyHUaV96Hta6H3hEeyHCctf 4zvjzN17s1CAQam+L4AlE7Z2bPyJu76VGvWvGX0SC3C7UIBBoX9g/cG4cL75uO2MUEnq/DeaZ4Bw 2Qk8ahQYxZDIcd8OAoczwEONCaV5RgiXnWKjPoORC2rB9Ab5Qzce2NQcYzXjVGIJZJkEwz4FO7pD io2PHQnpekiRTNVemGeNUBfKMGh0GPqh1sVHQTfs2Ne5EjVzWGJ+u4x/2CUx9ERZKa4KBgUSa9jW nAc0z5yiWyjBoMtiqEUueG/Qi2XQgZGqdNY8e4p+oQSDJo3hTrZxua8bieTB1phSnmlKl7kTowYP Oxj57XrHWTjeLNo6fpzHf7mMf9geMnAolMD2rwxAI67KI+J5sTwu8oiG/SUDdyIOJJlDbFNjfHTe EcawOJPbLzEM9m6va5J0hbKJG9bRTf9OuqCd8ngdWeILyMGsyGEppbcC1V/SusEVRxBUy7nBx5Cs VqR62swMnsEHy85UTkY4XZ0nEUBvEkGkQ+epMxenh/PPeNwt1QybbwyO9KD6Gc7RuGDXEhy0DIft wknLK6dfVuc+XTesc3PhsF04bc6UgG30gOix+QZPL9/wyeo3XTgQ7g1f9dQ1SMzEPGeZnjGNb2oN h3DcEtx5Yt2zSTEOmoXzIU5ubGt4bHXlenCtrdx5JmCrK9eDa1Dn5NkBUO+4WThv2vJOzuMfo9fq mjohziMhW6XrumZdtF1Ns3CteXbh9wI1u3IxXHMr141vetkQzUU68gzXfWjQMBy3C/eaDRG1yqy5 IfAZrrtfoSk6jCOx5ujCi2Vud+26b1le9A4bp4PG3izoCtmJzdl8Y3DnWdKNLlyPrbWF6/bDi6lr 7ZQ4N9Ns6qr3BqeiVaohHAyYBzORhFo/5pZ4CrSK9D8OqeRG/uLcc3VheOfXFwfu9b6QsyupBz5v OYQNUjIEvUvXyduV1GNj7tepR4D7uwJ2zu0KmLYARu7pMnsGu47laFc1HgPyUpDeQmdD/74XNMUI T8zeiHVvxKhkfWvEsjutkN1phexKKx5zK3vSimli3Rtxi1pBZm9aQWZvWkH7OkEeM5170oppYt0b cYtaYWFvWjFBrHsjbtLbTFk32V0MIruLQWR3MYjsLgaR3cUgsrsYRHYXg8juYhDZXQwiu4tBZHcx iOwuBpHdxSCyuxhEdheDyO5iENldDCK7ikH6vWR7iEHW6377XYi30Yquayz6nF55DS+5fd2Q3emG fHndWNlLblsrpol1b8Sv0ops3xv0B3D43gSOpb7yerqxWr/eLGLdG/E2urGyr9y2VkwT696It9GK +MJbax2v4C83rxkTxLo34ubsRcZfJtOsVkwT696It9GKuAmcfX/E73JvuXHNkN1phnx5zVjfW25Y K6aJdW/ELWoFmb1pBZm9aQWZXWlFxk9uWCumiXVvxF/Ct1BYx09uWzMmiHVvxLuIRfpfs+8hFmnw +/s5xE1qhexOK2R3WiG70oqUj9y2VkwT696IdxGHNK8VZPamFbSvEyTlI7etFdPEujfiFrVi5B83 rxUTxLo34r3EIBb2FoNExLo34r3EII1rhexOK2RXWpHxjxvWimli3RvxXmKQtrWCzN60gvZ1gmT8 44a1YppY90a8lxikba2YINa9EbfnbeIBE3qsq1xxsdESi9sT8OP9CIU7E5jU7wrYw5ftqg+wJvwq e8Ngrvvoq2Nw7+Maz54pTfO0dNvSUAQjApxE2Zih4sUgIPt1MJYo9GcMe7yyx98dw9tLe331+72U u8nQl3zqwqW2GNk0zvh0eUvbkI/3QTX/skfreOZuXCFbY0wqZGuQKYVsbiHPo94T14M1xPh4XVnb jJmThnz7J01TjLmTpinIzEnT2sserePj4jaukc1BJlWyOcqUTr4OcklYULGUSImZ8LYlyI6nccYu CbQHRtc+ozWNM4adrbQDRm6fMdW335btSRnI82Zv3EC2xpgykC0yuvYZrWmcMWUgW2Tk9hlTBT3d hYF0OzCQbgcG0u3AQLodGEi3AwPpdmAg3Q4MpGvdQJ4VoHED2RpjykC2yOjaZ7SmccaUgWyRkdtn HBnI5mxPzkDyDgwk78BA8g4MJO/AQPIODCTvwEDyDgwk78BAWmjfQDbFmDGQzTG69hlTBrIpxoyB bI6R22dMGcjWbM+oZ+r8s8b2DK/JuLxlIddAnrhixDVL160dNk1HTdNxs3T/39zV9DiRA9H7/hUG b324ylVIy4kLlz0tJ4RGnXQHrcQSBGF/P+5hmJjpON2TNLuW5jIX13v18arcispjpfwP99OlVUGB sWl01DS6hiOrgaFZdGPNpqbRWbPoRkVJTaNr23feLLo8iRo0rXexaXTWtBrHptFJ0+i03V4x/gia 2oYX24YnbcPTpucUbxmdctOTgLaMruFJYPxi1vYURU2j46bRXdQtyMFcgjPX3k1MkazcBxFBMVH5 upE8rF+Q7Yb8ChjJa44iFk1WwNAEUdxOwojkEeViHLlKYs0d7ITlRgpOLETptDs8ddClq3BQzR+W YgmDmBOl00HxFL2/FEX0oK5PS98iEmlLw/Yi22ykkDtRzbYQeyze9FRCyMmaKpEYF1JcgoNVEQNT NRBiVuYlEmKkeBLF9+0tF+eD5jJ94mfPYy/qP+8/FYczbQwG2pVMPbmFyPZgwn9eCyPAhkc7mahr Tjz86SVVuFOuR+oEAX6no7b/s/93KGjD23fPX97/+9ebP29fv3oLITC+u/kyHG53fw8f+hfjSRRQ gujzl4evH8fjb+4diAWvQdCMYEoLqEaLQMCK91WI1IiVAed45bvrOrwkZFee5dWb9LuuixNiLFwh xhaVuQgYGgAkcIVZYroSsRQwwVli1ntuMKpTYlBLRE3oMEru8TnffFcyBbU5YrRaJkKO/lli220v HaYpMRKtEENUYyyJ5Wx2jyI+S8zWIsYBz/LabKEj1GmJUTHlPMrENNZTOUNEN8sxm1UOXqvCcqna eV5CHesm0oQXSo0XRtZY8OJcbtFltrzWilUMks5ycupgl3V+yglrckgQeWyiDyu31HPnEZrjBKtx QvXzgepBvR+mUggKtUCxWinxjBRTnA9UXE/fZ0jlTgqJcTMlhVJrXAwyoj4+DJY8OovJLK21eCHO qeAuboZOZPuIVwquqcYrEWH5+DKKCGIuw3m50PXk4rxazAxnftVwNne4Lzic4PuWw8JBd+1Y5ccK Q5lcZ+kXYPja/0cY8gWoyBmDhCwnnf1+f9jf3p2Qgw3Lzd3X2nX28Cn2cOGngu3h9suhOwx/PPv8 6cMVDn1kcTHDCyzhagVyLyfFjT1rxRLHFQs81x4DlwrlQt1ZOs0sHOaWXquW9uaFc8nSrrj0drT0 snFTFiA9peBp1SS1IHycJEm4fviPEz/uD8MLyDNwHwAe/n77BnUpJjGpYwEA Pod spec kind: Pod apiVersion: v1 metadata: generateName: iaf-system-zookeeper- annotations: openshift.io/scc: restricted strimzi.io/cluster-ca-cert-generation: '0' productID: 068a62892a1e4db39641342e592daa25 k8s.v1.cni.cncf.io/network-status: |- [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.254.24.29" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: |- [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.254.24.29" ], "default": true, "dns": {} }] strimzi.io/generation: '0' productName: IBM Cloud Platform Common Services strimzi.io/logging-hash: 0f057cb0003c78f02978b83e4fabad5bd508680c productMetric: FREE resourceVersion: '3409155' name: iaf-system-zookeeper-0 uid: c93bd9d8-94d8-4cd0-8c71-ae00f7555e78 creationTimestamp: '2022-03-07T10:57:01Z' namespace: katamari ownerReferences: - apiVersion: apps/v1 kind: StatefulSet name: iaf-system-zookeeper uid: 8ae6e6b5-e895-4983-b90e-dfcc174dce31 controller: true blockOwnerDeletion: true labels: app.kubernetes.io/part-of: ibmevents-iaf-system app.kubernetes.io/instance: iaf-system statefulset.kubernetes.io/pod-name: iaf-system-zookeeper-0 controller-revision-hash: iaf-system-zookeeper-777b6497d5 ibmevents.ibm.com/cluster: iaf-system app.kubernetes.io/managed-by: ibm-events-operator ibmevents.ibm.com/name: iaf-system-zookeeper app.kubernetes.io/name: zookeeper ibmevents.ibm.com/kind: Kafka spec: nodeSelector: type: waiops restartPolicy: Always serviceAccountName: iaf-system-zookeeper imagePullSecrets: - name: iaf-system-zookeeper-dockercfg-gj9wz priority: 0 subdomain: iaf-system-zookeeper-nodes schedulerName: default-scheduler enableServiceLinks: true affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/arch operator: In values: - amd64 - s390x - ppc64le terminationGracePeriodSeconds: 30 preemptionPolicy: PreemptLowerPriority nodeName: worker1.bvt-rtp-04020027.cp.fyre.ibm.com securityContext: seLinuxOptions: level: 's0:c27,c4' runAsNonRoot: true fsGroup: 1000710000 containers: - resources: limits: cpu: '1' memory: 256Mi requests: cpu: 50m memory: 256Mi readinessProbe: exec: command: - /opt/kafka/zookeeper_healthcheck.sh initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 terminationMessagePath: /dev/termination-log name: zookeeper command: - /opt/kafka/zookeeper_run.sh livenessProbe: exec: command: - /opt/kafka/zookeeper_healthcheck.sh initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 env: - name: ZOOKEEPER_METRICS_ENABLED value: 'false' - name: ZOOKEEPER_SNAPSHOT_CHECK_ENABLED value: 'true' - name: STRIMZI_KAFKA_GC_LOG_ENABLED value: 'false' - name: DYNAMIC_HEAP_FRACTION value: '0.75' - name: DYNAMIC_HEAP_MAX value: '2147483648' - name: ZOOKEEPER_CONFIGURATION value: | tickTime=2000 initLimit=5 syncLimit=2 autopurge.purgeInterval=1 securityContext: capabilities: drop: - ALL - KILL - MKNOD - SETGID - SETUID privileged: false runAsUser: 1000710000 runAsNonRoot: true readOnlyRootFilesystem: false allowPrivilegeEscalation: false ports: - name: tcp-clustering containerPort: 2888 protocol: TCP - name: tcp-election containerPort: 3888 protocol: TCP - name: tcp-clients containerPort: 2181 protocol: TCP imagePullPolicy: Always volumeMounts: - name: strimzi-tmp mountPath: /tmp - name: data mountPath: /var/lib/zookeeper - name: zookeeper-metrics-and-logging mountPath: /opt/kafka/custom-config/ - name: zookeeper-nodes mountPath: /opt/kafka/zookeeper-node-certs/ - name: cluster-ca-certs mountPath: /opt/kafka/cluster-ca-certs/ - name: kube-api-access-dnhqv readOnly: true mountPath: /var/run/secrets/kubernetes.io/serviceaccount terminationMessagePolicy: File image: >- quay.io/opencloudio/ibm-events-kafka-2.7.0@sha256:d264cea91edec7c0975b6d11bf08bb47661fb9a8a982ea5752e65cdae9385146 hostname: iaf-system-zookeeper-0 serviceAccount: iaf-system-zookeeper volumes: - name: data persistentVolumeClaim: claimName: data-iaf-system-zookeeper-0 - name: strimzi-tmp emptyDir: medium: Memory - name: zookeeper-metrics-and-logging configMap: name: iaf-system-zookeeper-config defaultMode: 420 - name: zookeeper-nodes secret: secretName: iaf-system-zookeeper-nodes defaultMode: 292 - name: cluster-ca-certs secret: secretName: iaf-system-cluster-ca-cert defaultMode: 292 - name: kube-api-access-dnhqv projected: sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: name: kube-root-ca.crt items: - key: ca.crt path: ca.crt - downwardAPI: items: - path: namespace fieldRef: apiVersion: v1 fieldPath: metadata.namespace - configMap: name: openshift-service-ca.crt items: - key: service-ca.crt path: service-ca.crt defaultMode: 420 dnsPolicy: ClusterFirst tolerations: - key: node.kubernetes.io/not-ready operator: Exists effect: NoExecute tolerationSeconds: 300 - key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 300 - key: node.kubernetes.io/memory-pressure operator: Exists effect: NoSchedule status: phase: Running conditions: - type: Initialized status: 'True' lastProbeTime: null lastTransitionTime: '2022-03-07T10:57:01Z' - type: Ready status: 'True' lastProbeTime: null lastTransitionTime: '2022-03-07T10:57:31Z' - type: ContainersReady status: 'True' lastProbeTime: null lastTransitionTime: '2022-03-07T10:57:31Z' - type: PodScheduled status: 'True' lastProbeTime: null lastTransitionTime: '2022-03-07T10:57:01Z' hostIP: 10.22.7.165 podIP: 10.254.24.29 podIPs: - ip: 10.254.24.29 startTime: '2022-03-07T10:57:01Z' containerStatuses: - restartCount: 0 started: true ready: true name: zookeeper state: running: startedAt: '2022-03-07T10:57:09Z' imageID: >- quay.io/opencloudio/ibm-events-kafka-2.7.0@sha256:b965631ad6d833439257e9823ac0f5361a8d3d0271cbb90bff5b5e006e564d9c image: >- quay.io/opencloudio/ibm-events-kafka-2.7.0@sha256:d264cea91edec7c0975b6d11bf08bb47661fb9a8a982ea5752e65cdae9385146 lastState: {} containerID: 'cri-o://782dd7a7c0899ba473d64398eddd9807ed54af947f5d7445882f30a382fdef3b' qosClass: Burstable Created attachment 1864437 [details]
NetworkPolicy dump for katamari namespace
Moving this back to POST so I can attach the follow-up https://github.com/openshift/sdn/pull/406 So to reproduce you need: 1. at least one policy that selects all pods for ingress (eg, an ingress-default-deny policy) 2. at least one policy that selects _some_ pods for ingress 3. at least one policy that selects _some_ pods for egress The buggy result is that the pods selected by the 2nd policy will be isolated for egress as well. (In reply to Dan Winship from comment #30) > So to reproduce you need: > > 1. at least one policy that selects all pods for ingress (eg, an > ingress-default-deny policy) > 2. at least one policy that selects _some_ pods for ingress > 3. at least one policy that selects _some_ pods for egress > > The buggy result is that the pods selected by the 2nd policy will be > isolated for egress as well. my bad, you don't actually need the first policy I see PRs https://github.com/openshift/sdn/pull/406 and https://github.com/openshift/sdn/pull/407 with the refix. Once merged, can we please ask for the OCP version with the refix and the timeline for it? (In reply to piotr.godowski from comment #32) > I see PRs https://github.com/openshift/sdn/pull/406 and > https://github.com/openshift/sdn/pull/407 with the refix. > Once merged, can we please ask for the OCP version with the refix and the > timeline for it? this fixed merged into 4.10.3 version: https://amd64.ocp.releases.ci.openshift.org/releasestream/4-stable/release/4.10.3 I verified this issue with comment in https://bugzilla.redhat.com/show_bug.cgi?id=2060956#c9 Could you have a try with 4.10.3 version? Will be tested on our side today. The first team on out side ran tests on 4.10.3 confirming fix working. The other team on our side ran entire BVT test scenarios successful on OCP 4.10.3. We'd like to appreciate the quick turn around and the great collaboration ! Awesome! Thank you for your help! Move this to verified Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069 |
Created attachment 1864063 [details] ns lookup result in OCP4.10 Description of problem: We have a zookeeper service that works fine with networkpolicy in the OCP 4.9 and before versions. However, when it is deployed to OCP4.10 rc builds, we observe DNS resolution of *.svc stopped working. OpenShift release version: 4.10-rc6 Actual results: In 4.10 OCP cluster, zookeeper pod fails as it is not able to resolve the zookeeper server names: ``` hostname -f iaf-system-zookeeper-1.iaf-system-zookeeper-nodes.acme-iaf.sc.cluster.local nslookup iaf-system-zookeeper-1.iaf-system-zookeeper-nodes.acme-iaf.sc.cluster.local ;; connection timed out; no servers could be reached ``` Expected results: In OCP 4.9, the version of zookeeper works perfectly Here It is able to resolve all the zookeeper server name as the entries are present in DNS ``` hostname -f iaf-system-zookeeper-0.iaf-system-zookeeper-nodes.acme-iaf.sc.cluster.local nslookup iaf-system-zookeeper-0.iaf-system-zookeeper-nodes.acme-iaf.sc.cluster.local Server: 172.30.0.10 Address: 172.30.0.10#53 ``` Impact of the problem: The problem will cause zookeeper pods crash Additional info: We also notice when we remove the network policy for the zookeeper pods, the DNS resolution works fine on OCP4.10. We are wondering where the network policy impact the DNS resolution on OCP4.10. This is the network policy we are using ``` kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: iaf-system-network-policy-zookeeper namespace: acme-iaf uid: 7122b8ff-4db7-4ba5-b7a3-8d6490a8d950 resourceVersion: '11528510' generation: 1 creationTimestamp: '2022-03-02T12:44:43Z' labels: app.kubernetes.io/instance: iaf-system app.kubernetes.io/managed-by: ibm-events-operator app.kubernetes.io/name: zookeeper app.kubernetes.io/part-of: ibmevents-iaf-system ibmevents.ibm.com/cluster: iaf-system ibmevents.ibm.com/kind: Kafka ibmevents.ibm.com/name: ibmevents managedFields: - manager: Mozilla operation: Update apiVersion: networking.k8s.io/v1 time: '2022-03-02T12:44:43Z' fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:labels': .: {} 'f:app.kubernetes.io/instance': {} 'f:app.kubernetes.io/managed-by': {} 'f:app.kubernetes.io/name': {} 'f:app.kubernetes.io/part-of': {} 'f:ibmevents.ibm.com/cluster': {} 'f:ibmevents.ibm.com/kind': {} 'f:ibmevents.ibm.com/name': {} 'f:spec': 'f:ingress': {} 'f:podSelector': {} 'f:policyTypes': {} spec: podSelector: matchLabels: ibmevents.ibm.com/name: iaf-system-zookeeper ingress: - ports: - protocol: TCP port: 2888 - protocol: TCP port: 3888 from: - podSelector: matchLabels: ibmevents.ibm.com/name: iaf-system-zookeeper - ports: - protocol: TCP port: 2181 from: - podSelector: matchLabels: ibmevents.ibm.com/name: iaf-system-kafka - podSelector: matchLabels: ibmevents.ibm.com/name: iaf-system-zookeeper - podSelector: matchLabels: ibmevents.ibm.com/name: iaf-system-entity-operator - podSelector: matchLabels: ibmevents.ibm.com/kind: cluster-operator namespaceSelector: {} - podSelector: matchLabels: ibmevents.ibm.com/name: iaf-system-cruise-control policyTypes: - Ingress ``` ** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report. You may also mark the bug private if you wish.