Bug 2060553

Summary: service domain can't be resolved when networkpolicy is used in OCP 4.10-rc
Product: OpenShift Container Platform Reporter: Jiaming Hu <Jiaming.Hu>
Component: NetworkingAssignee: Ben Bennett <bbennett>
Networking sub component: openshift-sdn QA Contact: zhaozhanqi <zzhao>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: urgent CC: anbhat, anusaxen, aos-bugs, cblecker, danw, ddharwar, dmoessne, jtanenba, kahara, kangell, mifiedle, mmasters, piotr.godowski, rszumski, travi, twbrown, vlaad, wking, xtian, zzhao
Version: 4.10-rc3Keywords: FastFix
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2060956 (view as bug list) Environment:
Last Closed: 2022-08-10 10:52:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2060956    
Attachments:
Description Flags
ns lookup result in OCP4.10
none
OCP 4.10 NetworkPolicy egress issue
none
All Network Policies
none
NetworkPolicy dump for katamari namespace none

Description Jiaming Hu 2022-03-03 18:44:13 UTC
Created attachment 1864063 [details]
ns lookup result in OCP4.10

Description of problem:

We have a zookeeper service that works fine with networkpolicy in the OCP 4.9 and before versions. However, when it is deployed to OCP4.10 rc builds, we observe DNS resolution of *.svc stopped working.


OpenShift release version:

4.10-rc6


Actual results:


In 4.10 OCP cluster, zookeeper pod fails as it is not able to resolve the zookeeper server names:

```
hostname -f
iaf-system-zookeeper-1.iaf-system-zookeeper-nodes.acme-iaf.sc.cluster.local

nslookup iaf-system-zookeeper-1.iaf-system-zookeeper-nodes.acme-iaf.sc.cluster.local
;; connection timed out; no servers could be reached
```

Expected results:


In OCP 4.9,  the version of zookeeper works perfectly
Here It is able to resolve all the zookeeper server name as the entries are present in DNS

```
hostname -f
iaf-system-zookeeper-0.iaf-system-zookeeper-nodes.acme-iaf.sc.cluster.local

nslookup iaf-system-zookeeper-0.iaf-system-zookeeper-nodes.acme-iaf.sc.cluster.local

Server: 172.30.0.10
Address: 172.30.0.10#53
```


Impact of the problem:

The problem will cause zookeeper pods crash

Additional info:

We also notice when we remove the network policy for the zookeeper pods, the DNS resolution works fine on OCP4.10. We are wondering where the network policy impact the DNS resolution on OCP4.10. 

This is the network policy we are using


```
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: iaf-system-network-policy-zookeeper
  namespace: acme-iaf
  uid: 7122b8ff-4db7-4ba5-b7a3-8d6490a8d950
  resourceVersion: '11528510'
  generation: 1
  creationTimestamp: '2022-03-02T12:44:43Z'
  labels:
    app.kubernetes.io/instance: iaf-system
    app.kubernetes.io/managed-by: ibm-events-operator
    app.kubernetes.io/name: zookeeper
    app.kubernetes.io/part-of: ibmevents-iaf-system
    ibmevents.ibm.com/cluster: iaf-system
    ibmevents.ibm.com/kind: Kafka
    ibmevents.ibm.com/name: ibmevents
  managedFields:
    - manager: Mozilla
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2022-03-02T12:44:43Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            .: {}
            'f:app.kubernetes.io/instance': {}
            'f:app.kubernetes.io/managed-by': {}
            'f:app.kubernetes.io/name': {}
            'f:app.kubernetes.io/part-of': {}
            'f:ibmevents.ibm.com/cluster': {}
            'f:ibmevents.ibm.com/kind': {}
            'f:ibmevents.ibm.com/name': {}
        'f:spec':
          'f:ingress': {}
          'f:podSelector': {}
          'f:policyTypes': {}
spec:
  podSelector:
    matchLabels:
      ibmevents.ibm.com/name: iaf-system-zookeeper
  ingress:
    - ports:
        - protocol: TCP
          port: 2888
        - protocol: TCP
          port: 3888
      from:
        - podSelector:
            matchLabels:
              ibmevents.ibm.com/name: iaf-system-zookeeper
    - ports:
        - protocol: TCP
          port: 2181
      from:
        - podSelector:
            matchLabels:
              ibmevents.ibm.com/name: iaf-system-kafka
        - podSelector:
            matchLabels:
              ibmevents.ibm.com/name: iaf-system-zookeeper
        - podSelector:
            matchLabels:
              ibmevents.ibm.com/name: iaf-system-entity-operator
        - podSelector:
            matchLabels:
              ibmevents.ibm.com/kind: cluster-operator
          namespaceSelector: {}
        - podSelector:
            matchLabels:
              ibmevents.ibm.com/name: iaf-system-cruise-control
  policyTypes:
    - Ingress
```


** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report.  You may also mark the bug private if you wish.

Comment 1 Miciah Dashiel Butler Masters 2022-03-03 22:52:29 UTC
Because the DNS query works on OpenShift 4.9 and 4.10 without the NetworkPolicy and only fails on OpenShift 4.10 with the NetworkPolicy, this seems like a NetworkPolicy issue rather than a DNS issue, so I am changing the component to Networking / openshift-sdn for investigation.

Comment 3 Aniket Bhat 2022-03-04 14:58:50 UTC
@Jiaming.Hu is there a default deny policy for that namespace that also has an empty egress clause? In 4.10, we added support for egress and a previously defined deny policy which had an egress clause is now being effective and blocking traffic from the namespace. Can you please confirm what the default deny policy looks like? The solution may just be to add an explicit allow rule for DNS or remove the egress clause from the default deny. From a security standpoint, the former seems desirable.

Comment 4 Rob Szumski 2022-03-04 15:01:51 UTC
Is this a newly installed cluster or after an upgrade? Which SDN are you using? Are these workers RHEL or CoreOS based?

Comment 5 piotr.godowski 2022-03-04 16:10:48 UTC
There is no default deny-all egress policy.

However, there are other NetworkPolicies with the same namespace, with egress rules, however for other pod selectors. For example:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  resourceVersion: '6681721'
  name: iaf-core-operator
  uid: e38f7051-5023-443c-8e0d-451d9d81d5d9
  creationTimestamp: '2022-02-28T10:50:28Z'
  generation: 1
  namespace: acme-iaf
  ownerReferences:
    - apiVersion: apps/v1
      kind: Deployment
      name: iaf-core-operator-controller-manager
      uid: 07624bcf-bdce-4fcd-8577-f67a5f04132d
      controller: true
      blockOwnerDeletion: true
  labels:
    app.kubernetes.io/component: networkpolicy
    app.kubernetes.io/instance: iaf-core-operator
    app.kubernetes.io/managed-by: controller-manager
    app.kubernetes.io/name: iaf-core-operator
    name: iaf-core-operator
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: iaf-core-operator
      app.kubernetes.io/instance: iaf-core-operator
      app.kubernetes.io/managed-by: olm
      app.kubernetes.io/name: iaf-core-operator
      name: iaf-core-operator
  ingress:
    - ports:
        - protocol: TCP
          port: 9443
        - protocol: TCP
          port: 443
  egress:
    - {}
  policyTypes:
    - Ingress
    - Egress


and

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  resourceVersion: '6677521'
  name: iaf-eventprocessing-operator
  uid: 0d813a0e-1d03-4cc0-96a2-86d261fd7888
  creationTimestamp: '2022-02-28T10:47:56Z'
  generation: 1
  namespace: acme-iaf
  ownerReferences:
    - apiVersion: apps/v1
      kind: Deployment
      name: iaf-eventprocessing-operator-controller-manager
      uid: e3ede052-1363-4dc8-b3fb-f4153545f7fc
      controller: true
      blockOwnerDeletion: true
  labels:
    app.kubernetes.io/component: networkpolicy
    app.kubernetes.io/instance: iaf-eventprocessing-operator
    app.kubernetes.io/managed-by: controller-manager
    app.kubernetes.io/name: eventprocessing-operator
    name: iaf-eventprocessing-operator
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: controller-manager
      app.kubernetes.io/instance: iaf-eventprocessing-operator
      app.kubernetes.io/managed-by: olm
      app.kubernetes.io/name: eventprocessing-operator
      name: iaf-eventprocessing-operator
  ingress:
    - ports:
        - protocol: TCP
          port: 9443
        - protocol: TCP
          port: 443
  egress:
    - {}
  policyTypes:
    - Ingress
    - Egress



Please review this recorded video showing the problem:
https://ibm.webex.com/recordingservice/sites/ibm/recording/playback/97498e9e7dc5103abdb50050568fc64e
recording password: 8GrZrRXh


And also see the visual representation of the problem attached

Comment 6 piotr.godowski 2022-03-04 16:11:29 UTC
Created attachment 1864164 [details]
OCP 4.10 NetworkPolicy egress issue

Comment 7 Ben Bennett 2022-03-04 16:15:56 UTC
Thanks @piotr.godowski.com.  I have some follow-up requests.

Can you provide all network policies in the namespace: oc get networkpolicies -o yaml

Also, what is the sc.cluster.local domain?  Is something special happening with your resolver to make the hostnames resolve?

When it is failing, can you also run: nslookup kubernetes.default.svc.cluster.local

I want to see if the sc.cluster.local makes any difference.

Finally, can we get the pod spec for the pod that is failing?  I want to try to reproduce with the same policies and labels.

Comment 8 piotr.godowski 2022-03-04 16:19:50 UTC
sc.cluster.local is the typo in the bug description. obviously we're speaking *.svc.cluster.local.

other stuff will be provided shortly.

Comment 9 piotr.godowski 2022-03-04 16:30:51 UTC
Failing Pod spec:

kind: Pod
apiVersion: v1
metadata:
  generateName: iaf-system-zookeeper-
  annotations:
    openshift.io/scc: restricted
    strimzi.io/cluster-ca-cert-generation: '0'
    productID: 068a62892a1e4db39641342e592daa25
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "openshift-sdn",
          "interface": "eth0",
          "ips": [
              "10.254.21.248"
          ],
          "default": true,
          "dns": {}
      }]
    k8s.v1.cni.cncf.io/networks-status: |-
      [{
          "name": "openshift-sdn",
          "interface": "eth0",
          "ips": [
              "10.254.21.248"
          ],
          "default": true,
          "dns": {}
      }]
    strimzi.io/generation: '0'
    productName: IBM Cloud Platform Common Services
    strimzi.io/logging-hash: 0f057cb0003c78f02978b83e4fabad5bd508680c
    productMetric: FREE
  resourceVersion: '15394091'
  name: iaf-system-zookeeper-0
  uid: bd666fdc-6867-4094-a2ee-04b7db04b782
  creationTimestamp: '2022-03-04T10:55:06Z'
  managedFields:
    - manager: kube-controller-manager
      operation: Update
      apiVersion: v1
      time: '2022-03-04T10:55:06Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:productID': {}
            'f:productMetric': {}
            'f:productName': {}
            'f:strimzi.io/cluster-ca-cert-generation': {}
            'f:strimzi.io/generation': {}
            'f:strimzi.io/logging-hash': {}
          'f:generateName': {}
          'f:labels':
            'f:statefulset.kubernetes.io/pod-name': {}
            'f:controller-revision-hash': {}
            'f:app.kubernetes.io/managed-by': {}
            'f:ibmevents.ibm.com/name': {}
            'f:app.kubernetes.io/name': {}
            .: {}
            'f:app.kubernetes.io/part-of': {}
            'f:ibmevents.ibm.com/kind': {}
            'f:app.kubernetes.io/instance': {}
            'f:ibmevents.ibm.com/cluster': {}
          'f:ownerReferences':
            .: {}
            'k:{"uid":"6fcb04dd-ad1a-40f2-b5e2-c76811769b85"}': {}
        'f:spec':
          'f:volumes':
            .: {}
            'k:{"name":"cluster-ca-certs"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:secretName': {}
            'k:{"name":"data"}':
              .: {}
              'f:name': {}
              'f:persistentVolumeClaim':
                .: {}
                'f:claimName': {}
            'k:{"name":"strimzi-tmp"}':
              .: {}
              'f:emptyDir':
                .: {}
                'f:medium': {}
              'f:name': {}
            'k:{"name":"zookeeper-metrics-and-logging"}':
              .: {}
              'f:configMap':
                .: {}
                'f:defaultMode': {}
                'f:name': {}
              'f:name': {}
            'k:{"name":"zookeeper-nodes"}':
              .: {}
              'f:name': {}
              'f:secret':
                .: {}
                'f:defaultMode': {}
                'f:secretName': {}
          'f:containers':
            'k:{"name":"zookeeper"}':
              'f:image': {}
              'f:volumeMounts':
                .: {}
                'k:{"mountPath":"/opt/kafka/cluster-ca-certs/"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/opt/kafka/custom-config/"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/opt/kafka/zookeeper-node-certs/"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/tmp"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
                'k:{"mountPath":"/var/lib/zookeeper"}':
                  .: {}
                  'f:mountPath': {}
                  'f:name': {}
              'f:terminationMessagePolicy': {}
              .: {}
              'f:resources':
                .: {}
                'f:limits':
                  .: {}
                  'f:cpu': {}
                  'f:memory': {}
                'f:requests':
                  .: {}
                  'f:cpu': {}
                  'f:memory': {}
              'f:command': {}
              'f:livenessProbe':
                .: {}
                'f:exec':
                  .: {}
                  'f:command': {}
                'f:failureThreshold': {}
                'f:initialDelaySeconds': {}
                'f:periodSeconds': {}
                'f:successThreshold': {}
                'f:timeoutSeconds': {}
              'f:env':
                .: {}
                'k:{"name":"DYNAMIC_HEAP_FRACTION"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"DYNAMIC_HEAP_MAX"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"STRIMZI_KAFKA_GC_LOG_ENABLED"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"ZOOKEEPER_CONFIGURATION"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"ZOOKEEPER_METRICS_ENABLED"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
                'k:{"name":"ZOOKEEPER_SNAPSHOT_CHECK_ENABLED"}':
                  .: {}
                  'f:name': {}
                  'f:value': {}
              'f:readinessProbe':
                .: {}
                'f:exec':
                  .: {}
                  'f:command': {}
                'f:failureThreshold': {}
                'f:initialDelaySeconds': {}
                'f:periodSeconds': {}
                'f:successThreshold': {}
                'f:timeoutSeconds': {}
              'f:securityContext':
                .: {}
                'f:allowPrivilegeEscalation': {}
                'f:capabilities':
                  .: {}
                  'f:drop': {}
                'f:privileged': {}
                'f:readOnlyRootFilesystem': {}
                'f:runAsNonRoot': {}
              'f:terminationMessagePath': {}
              'f:imagePullPolicy': {}
              'f:ports':
                .: {}
                'k:{"containerPort":2181,"protocol":"TCP"}':
                  .: {}
                  'f:containerPort': {}
                  'f:name': {}
                  'f:protocol': {}
                'k:{"containerPort":2888,"protocol":"TCP"}':
                  .: {}
                  'f:containerPort': {}
                  'f:name': {}
                  'f:protocol': {}
                'k:{"containerPort":3888,"protocol":"TCP"}':
                  .: {}
                  'f:containerPort': {}
                  'f:name': {}
                  'f:protocol': {}
              'f:name': {}
          'f:hostname': {}
          'f:dnsPolicy': {}
          'f:serviceAccount': {}
          'f:restartPolicy': {}
          'f:subdomain': {}
          'f:schedulerName': {}
          'f:terminationGracePeriodSeconds': {}
          'f:serviceAccountName': {}
          'f:enableServiceLinks': {}
          'f:securityContext':
            .: {}
            'f:runAsNonRoot': {}
          'f:affinity':
            .: {}
            'f:nodeAffinity':
              .: {}
              'f:requiredDuringSchedulingIgnoredDuringExecution': {}
    - manager: multus
      operation: Update
      apiVersion: v1
      time: '2022-03-04T10:55:13Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:k8s.v1.cni.cncf.io/network-status': {}
            'f:k8s.v1.cni.cncf.io/networks-status': {}
      subresource: status
    - manager: Go-http-client
      operation: Update
      apiVersion: v1
      time: '2022-03-04T10:55:36Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:conditions':
            'k:{"type":"ContainersReady"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Initialized"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
            'k:{"type":"Ready"}':
              .: {}
              'f:lastProbeTime': {}
              'f:lastTransitionTime': {}
              'f:status': {}
              'f:type': {}
          'f:containerStatuses': {}
          'f:hostIP': {}
          'f:phase': {}
          'f:podIP': {}
          'f:podIPs':
            .: {}
            'k:{"ip":"10.254.21.248"}':
              .: {}
              'f:ip': {}
          'f:startTime': {}
      subresource: status
  namespace: acme-iaf
  ownerReferences:
    - apiVersion: apps/v1
      kind: StatefulSet
      name: iaf-system-zookeeper
      uid: 6fcb04dd-ad1a-40f2-b5e2-c76811769b85
      controller: true
      blockOwnerDeletion: true
  labels:
    app.kubernetes.io/part-of: ibmevents-iaf-system
    app.kubernetes.io/instance: iaf-system
    statefulset.kubernetes.io/pod-name: iaf-system-zookeeper-0
    controller-revision-hash: iaf-system-zookeeper-6797c898fb
    ibmevents.ibm.com/cluster: iaf-system
    app.kubernetes.io/managed-by: ibm-events-operator
    ibmevents.ibm.com/name: iaf-system-zookeeper
    app.kubernetes.io/name: zookeeper
    ibmevents.ibm.com/kind: Kafka
spec:
  restartPolicy: Always
  serviceAccountName: iaf-system-zookeeper
  imagePullSecrets:
    - name: iaf-system-zookeeper-dockercfg-gdxvz
  priority: 0
  subdomain: iaf-system-zookeeper-nodes
  schedulerName: default-scheduler
  enableServiceLinks: true
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                  - amd64
                  - s390x
                  - ppc64le
  terminationGracePeriodSeconds: 30
  preemptionPolicy: PreemptLowerPriority
  nodeName: worker2.moon1915.cp.fyre.ibm.com
  securityContext:
    seLinuxOptions:
      level: 's0:c27,c14'
    runAsNonRoot: true
    fsGroup: 1000730000
  containers:
    - resources:
        limits:
          cpu: '1'
          memory: 2Gi
        requests:
          cpu: '1'
          memory: 2Gi
      readinessProbe:
        exec:
          command:
            - /opt/kafka/zookeeper_healthcheck.sh
        initialDelaySeconds: 15
        timeoutSeconds: 5
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      terminationMessagePath: /dev/termination-log
      name: zookeeper
      command:
        - /opt/kafka/zookeeper_run.sh
      livenessProbe:
        exec:
          command:
            - /opt/kafka/zookeeper_healthcheck.sh
        initialDelaySeconds: 15
        timeoutSeconds: 5
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      env:
        - name: ZOOKEEPER_METRICS_ENABLED
          value: 'false'
        - name: ZOOKEEPER_SNAPSHOT_CHECK_ENABLED
          value: 'true'
        - name: STRIMZI_KAFKA_GC_LOG_ENABLED
          value: 'false'
        - name: DYNAMIC_HEAP_FRACTION
          value: '0.75'
        - name: DYNAMIC_HEAP_MAX
          value: '2147483648'
        - name: ZOOKEEPER_CONFIGURATION
          value: |
            tickTime=2000
            initLimit=5
            syncLimit=2
            autopurge.purgeInterval=1
      securityContext:
        capabilities:
          drop:
            - ALL
            - KILL
            - MKNOD
            - SETGID
            - SETUID
        privileged: false
        runAsUser: 1000730000
        runAsNonRoot: true
        readOnlyRootFilesystem: false
        allowPrivilegeEscalation: false
      ports:
        - name: tcp-clustering
          containerPort: 2888
          protocol: TCP
        - name: tcp-election
          containerPort: 3888
          protocol: TCP
        - name: tcp-clients
          containerPort: 2181
          protocol: TCP
      imagePullPolicy: Always
      volumeMounts:
        - name: strimzi-tmp
          mountPath: /tmp
        - name: data
          mountPath: /var/lib/zookeeper
        - name: zookeeper-metrics-and-logging
          mountPath: /opt/kafka/custom-config/
        - name: zookeeper-nodes
          mountPath: /opt/kafka/zookeeper-node-certs/
        - name: cluster-ca-certs
          mountPath: /opt/kafka/cluster-ca-certs/
        - name: kube-api-access-qlvr8
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePolicy: File
      image: >-
        quay.io/opencloudio/ibm-events-kafka-2.7.0@sha256:d264cea91edec7c0975b6d11bf08bb47661fb9a8a982ea5752e65cdae9385146
  hostname: iaf-system-zookeeper-0
  serviceAccount: iaf-system-zookeeper
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-iaf-system-zookeeper-0
    - name: strimzi-tmp
      emptyDir:
        medium: Memory
    - name: zookeeper-metrics-and-logging
      configMap:
        name: iaf-system-zookeeper-config
        defaultMode: 420
    - name: zookeeper-nodes
      secret:
        secretName: iaf-system-zookeeper-nodes
        defaultMode: 292
    - name: cluster-ca-certs
      secret:
        secretName: iaf-system-cluster-ca-cert
        defaultMode: 292
    - name: kube-api-access-qlvr8
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
          - configMap:
              name: openshift-service-ca.crt
              items:
                - key: service-ca.crt
                  path: service-ca.crt
        defaultMode: 420
  dnsPolicy: ClusterFirst
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/memory-pressure
      operator: Exists
      effect: NoSchedule
status:
  phase: Running
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2022-03-04T10:55:06Z'
    - type: Ready
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2022-03-04T10:55:36Z'
    - type: ContainersReady
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2022-03-04T10:55:36Z'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2022-03-04T10:55:06Z'
  hostIP: 10.17.123.228
  podIP: 10.254.21.248
  podIPs:
    - ip: 10.254.21.248
  startTime: '2022-03-04T10:55:06Z'
  containerStatuses:
    - restartCount: 0
      started: true
      ready: true
      name: zookeeper
      state:
        running:
          startedAt: '2022-03-04T10:55:14Z'
      imageID: >-
        quay.io/opencloudio/ibm-events-kafka-2.7.0@sha256:b965631ad6d833439257e9823ac0f5361a8d3d0271cbb90bff5b5e006e564d9c
      image: >-
        quay.io/opencloudio/ibm-events-kafka-2.7.0@sha256:d264cea91edec7c0975b6d11bf08bb47661fb9a8a982ea5752e65cdae9385146
      lastState: {}
      containerID: 'cri-o://ce0e9d48046eda508b54ae30e58a29bed243b03407218c332d3c09820e401db7'
  qosClass: Guaranteed



Also, the problem was circumvented by adding the networkpolicy with the pod selector matching failing Pod, with added egress rule.
Note: no other network policy was pointing to the failing Pod earlier, thus it's not expected that failing Pod was blocked egress.


- apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    creationTimestamp: "2022-03-04T10:02:11Z"
    generation: 2
    labels:
      app.kubernetes.io/instance: iaf-system
      app.kubernetes.io/managed-by: ibm-events-operator
      app.kubernetes.io/name: zookeeper
      app.kubernetes.io/part-of: ibmevents-iaf-system
      ibmevents.ibm.com/cluster: iaf-system
      ibmevents.ibm.com/kind: Kafka
      ibmevents.ibm.com/name: ibmevents
    name: iaf-system-network-policy-zookeeper-ocp410
    namespace: acme-iaf
  spec:
    egress:
    - {}
    podSelector:
      matchLabels:
        ibmevents.ibm.com/name: iaf-system-zookeeper
    policyTypes:
    - Egress


Attached yaml dump of all NetworkPolicies

Comment 10 piotr.godowski 2022-03-04 16:33:38 UTC
Created attachment 1864165 [details]
All Network Policies

Comment 11 piotr.godowski 2022-03-04 16:41:56 UTC
Note1:

Also, the problem was circumvented by adding the networkpolicy with the pod selector matching failing Pod, with added egress rule.
No other network policy was pointing to the failing Pod earlier, thus it's not expected that failing Pod was blocked egress.

Note2:
We also found another work-around that instead of adding a 'specialized' NetworkPolicy, the same problem is circumvented by adding a general 'allow-all' egress policy:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-egress
spec:
  podSelector: {}
  egress:
  - {}
  policyTypes:
  - Egress


Perhaps, given the OCP 4.10 added support for egress policy, the logic for picking up Pods using podSelectors is not working correctly, and the NetworkPolicies for other Pods in same namespace are incorrectly applies to all the other Pods in the same namespace?
Or, perhaps it is assumed that if there is a single NetworkPolicy with egress rule in given namespace, then all the other NetworkPolicies must also specify egress rules?


As it stands today, we see that OCP 4.10 rc builds are not backwards compatible with OCP 4.9 and the applications which used to be working fine on OCP 4.9 are not working fine any longer.

Comment 12 piotr.godowski 2022-03-04 16:51:13 UTC
@bbennett Clearing out 'needinfo' flag. Sorry for massive number of posts, new user here.

Comment 13 Ben Bennett 2022-03-04 16:58:27 UTC
@piotr.godowski.com thank you for the data.  It gave us the detail needed to get a simple reproducer.

Ok, these two policies break it:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: egress-all-otherpod
spec:
  podSelector:
    matchLabels:
      name: i-don-t-match-anything
  egress:
    - {}
  policyTypes:
    - Egress

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-all-ingress
spec:
  podSelector: {}
  ingress:
    - {}
  policyTypes:
    - Ingress


Then just make a pod you can `oc debug --as-root` in to and install dig or nslookup and then you can see `dig kubernetes.default` will work until the `allow-all-ingress` rule is installed.

Comment 14 Mike Fiedler 2022-03-04 21:03:37 UTC

Verified on cluster-bot cluster built from this PR using the reproducer in https://bugzilla.redhat.com/show_bug.cgi?id=2060553#c13.   Service domain resolvable with the patch.

Comment 16 piotr.godowski 2022-03-06 15:40:15 UTC
Can we please get an update on which 4.10 RC build the fix will be contained?

Comment 17 Xiaoli Tian 2022-03-07 00:54:13 UTC
(In reply to piotr.godowski from comment #16)
> Can we please get an update on which 4.10 RC build the fix will be contained?
4.10.2 should contain the fix.

Comment 18 piotr.godowski 2022-03-07 10:27:52 UTC
(In reply to Xiaoli Tian from comment #17)
> > Can we please get an update on which 4.10 RC build the fix will be contained?
> 4.10.2 should contain the fix.

Thank you.
What is the expected date 4.10.2 will be made available?

Comment 19 piotr.godowski 2022-03-07 11:43:38 UTC
The problem can be still reproduced with OpenShift version: 4.10.2 . See exact detail about the OCP version / channel used below.

apiVersion: config.openshift.io/v1
kind: ClusterVersion
metadata:
  name: version
spec:
  channel: candidate-4.10
  clusterID: b5a23363-08be-4f3f-80e6-8f358953807a
  desiredUpdate:
    image: >-
      quay.io/openshift-release-dev/ocp-release@sha256:b807908afe1b7b25a8727f092cc11a104bf3996908a80f035d079b014b4b96b6
    version: 4.10.2
status:
  availableUpdates: null
  conditions:
    - lastTransitionTime: '2022-03-04T02:19:55Z'
      message: Done applying 4.10.2
      status: 'True'
      type: Available
    - lastTransitionTime: '2022-03-05T14:34:44Z'
      status: 'False'
      type: Failing
    - lastTransitionTime: '2022-03-07T09:30:51Z'
      message: Cluster version is 4.10.2
      status: 'False'
      type: Progressing
    - lastTransitionTime: '2022-03-06T12:03:32Z'
      status: 'True'
      type: RetrievedUpdates
  desired:
    channels:
      - candidate-4.10
    image: >-
      quay.io/openshift-release-dev/ocp-release@sha256:b807908afe1b7b25a8727f092cc11a104bf3996908a80f035d079b014b4b96b6
    url: 'https://access.redhat.com/errata/RHSA-2022:0056'
    version: 4.10.2
  history:
    - completionTime: '2022-03-07T09:30:51Z'
      image: >-
        quay.io/openshift-release-dev/ocp-release@sha256:b807908afe1b7b25a8727f092cc11a104bf3996908a80f035d079b014b4b96b6
      startedTime: '2022-03-07T09:02:52Z'
      state: Completed
      verified: true
      version: 4.10.2
    - completionTime: '2022-03-04T11:32:59Z'
      image: >-
        quay.io/openshift-release-dev/ocp-release@sha256:65e8dbc576556e0296e29ac1ae7496743e3494ad65a111e134c6e7202a498d11
      startedTime: '2022-03-04T10:38:17Z'
      state: Completed
      verified: true
      version: 4.10.0-rc.8
    - completionTime: '2022-03-04T10:05:29Z'
      image: >-
        quay.io/openshift-release-dev/ocp-release@sha256:fd96300600f9585e5847f5855ca14e2b3cafbce12aefe3b3f52c5da10c4476eb
      startedTime: '2022-03-04T09:14:39Z'
      state: Completed
      verified: true
      version: 4.9.21
    - completionTime: '2022-03-04T02:19:55Z'
      image: >-
        quay.io/openshift-release-dev/ocp-release@sha256:bb1987fb718f81fb30bec4e0e1cd5772945269b77006576b02546cf84c77498e
      startedTime: '2022-03-04T02:04:56Z'
      state: Completed
      verified: false
      version: 4.9.15
  observedGeneration: 7
  versionHash: FheEJUFAAVU=



Please note: There are NetworkPolicies with egress rules, but the pod selectors ARE matching some Pods.
The reproducer uses NetworkPolicy for egress rule, but with pod selector not matching anything. Not sure it's the exact same problem fixed.

Comment 20 zhaozhanqi 2022-03-07 13:18:44 UTC
(In reply to piotr.godowski from comment #19)
> The problem can be still reproduced with OpenShift version: 4.10.2 . See
> exact detail about the OCP version / channel used below.
> 
> apiVersion: config.openshift.io/v1
> kind: ClusterVersion
> metadata:
>   name: version
> spec:
>   channel: candidate-4.10
>   clusterID: b5a23363-08be-4f3f-80e6-8f358953807a
>   desiredUpdate:
>     image: >-
>      
> quay.io/openshift-release-dev/ocp-release@sha256:
> b807908afe1b7b25a8727f092cc11a104bf3996908a80f035d079b014b4b96b6
>     version: 4.10.2
> status:
>   availableUpdates: null
>   conditions:
>     - lastTransitionTime: '2022-03-04T02:19:55Z'
>       message: Done applying 4.10.2
>       status: 'True'
>       type: Available
>     - lastTransitionTime: '2022-03-05T14:34:44Z'
>       status: 'False'
>       type: Failing
>     - lastTransitionTime: '2022-03-07T09:30:51Z'
>       message: Cluster version is 4.10.2
>       status: 'False'
>       type: Progressing
>     - lastTransitionTime: '2022-03-06T12:03:32Z'
>       status: 'True'
>       type: RetrievedUpdates
>   desired:
>     channels:
>       - candidate-4.10
>     image: >-
>      
> quay.io/openshift-release-dev/ocp-release@sha256:
> b807908afe1b7b25a8727f092cc11a104bf3996908a80f035d079b014b4b96b6
>     url: 'https://access.redhat.com/errata/RHSA-2022:0056'
>     version: 4.10.2
>   history:
>     - completionTime: '2022-03-07T09:30:51Z'
>       image: >-
>        
> quay.io/openshift-release-dev/ocp-release@sha256:
> b807908afe1b7b25a8727f092cc11a104bf3996908a80f035d079b014b4b96b6
>       startedTime: '2022-03-07T09:02:52Z'
>       state: Completed
>       verified: true
>       version: 4.10.2
>     - completionTime: '2022-03-04T11:32:59Z'
>       image: >-
>        
> quay.io/openshift-release-dev/ocp-release@sha256:
> 65e8dbc576556e0296e29ac1ae7496743e3494ad65a111e134c6e7202a498d11
>       startedTime: '2022-03-04T10:38:17Z'
>       state: Completed
>       verified: true
>       version: 4.10.0-rc.8
>     - completionTime: '2022-03-04T10:05:29Z'
>       image: >-
>        
> quay.io/openshift-release-dev/ocp-release@sha256:
> fd96300600f9585e5847f5855ca14e2b3cafbce12aefe3b3f52c5da10c4476eb
>       startedTime: '2022-03-04T09:14:39Z'
>       state: Completed
>       verified: true
>       version: 4.9.21
>     - completionTime: '2022-03-04T02:19:55Z'
>       image: >-
>        
> quay.io/openshift-release-dev/ocp-release@sha256:
> bb1987fb718f81fb30bec4e0e1cd5772945269b77006576b02546cf84c77498e
>       startedTime: '2022-03-04T02:04:56Z'
>       state: Completed
>       verified: false
>       version: 4.9.15
>   observedGeneration: 7
>   versionHash: FheEJUFAAVU=
> 
> 
> 
> Please note: There are NetworkPolicies with egress rules, but the pod
> selectors ARE matching some Pods.
> The reproducer uses NetworkPolicy for egress rule, but with pod selector not
> matching anything. Not sure it's the exact same problem fixed.

Could you show your networkpolicy via 'oc get networkpolicy -o yaml -n $namespace'?.  I tested this issue with comment 13 and with that networkpolicy, it can be reproduced on 4.10.1 version, but not 4.10.2 version.

Comment 21 zhaozhanqi 2022-03-07 13:28:49 UTC
same networkpolicy in attachment?

Comment 22 piotr.godowski 2022-03-07 13:31:38 UTC
@zzhao 
> Could you show your networkpolicy via 'oc get networkpolicy -o yaml -n $namespace'?.  I tested this issue with comment 13 and with that networkpolicy, it can be reproduced on 4.10.1 version, but not 4.10.2 version.

NetworkPolicies provided earlier in https://bugzilla.redhat.com/show_bug.cgi?id=2060553#c10 and failing Pod spec in https://bugzilla.redhat.com/show_bug.cgi?id=2060553#c9

Comment 23 Dan Winship 2022-03-07 13:40:22 UTC
> Please note: There are NetworkPolicies with egress rules, but the pod
> selectors ARE matching some Pods.
> The reproducer uses NetworkPolicy for egress rule, but with pod selector not
> matching anything. Not sure it's the exact same problem fixed.

It is.

The bug was that if a namespace had any ingress policy that applied to the whole namespace (such as a default-deny-ingress policy), and any egress policy at all (whether or not it matched anything), then it behaved as though there was a default-deny-egress policy in the namespace.

(The reverse would also be true; if there was an egress policy that applied to the whole namespace, and any ingress policy at all, then it would behave as though there was a default-deny-ingress policy. But it's unlikely anyone using openshift-sdn had policies arranged like that [and didn't already have a default-deny-ingress policy].)

Comment 24 piotr.godowski 2022-03-07 14:04:27 UTC
Thanks @danw

> The bug was that if a namespace had any ingress policy that applied to the whole namespace (such as a default-deny-ingress policy), and any egress policy at all (whether or not it matched anything), then it behaved as though there was a default-deny-egress policy in the namespace.

In the problematic namespace there is no default-deny-ingress policy. I feel like we are missing some piece in the puzzles here, unless you point us to the NetworkPolicy which you found as being selected as the one to be applied for the whole namespace, within the archive uploaded in https://bugzilla.redhat.com/show_bug.cgi?id=2060553#c10 . Thanks.

Comment 25 Dan Winship 2022-03-07 19:08:24 UTC
can you run

  ovs-ofctl -O OpenFlow13 dump-flows br0

on the node with the client pod (ie the pod that is being denied egress), and attach that, and indicate the IP of the pod in question

Comment 26 Ben Bennett 2022-03-07 19:16:53 UTC
In addition to Dan's request above, can you please add the output from 'oc get events'

Comment 27 piotr.godowski 2022-03-07 20:22:30 UTC
sh-4.4# base64 ovs-ofctl-dump-broken.txt.gz
       
H4sICOlnJmICA292cy1vZmN0bC1kdW1wLWJyb2tlbi50eHQA3F1tjxw3cv6eX6GPNrIes15JCtB9
ujg4ILGDxIccYBjC7Myss7DsXUijy/nfhz0r7bK7yW52T7e6O8bCgEba4VNkvRdZ9cN3//FfP779
7t9++O9X70+P7/549dUP38GOvn711T/uj2/MP/Dr16/u3u1/+fDmp98e3p9+/qdXh4eHX+9P4a/M
zavjx/f78/3D72/QG4t2x+o/3Lw672/fhX9w8+r3t4/7w6+n84c3nsEZdL767PaP8+nDGyQF5yj8
2JtXj+/vH97fn/94A8aYm8P57Yfz/nx68835/a8394+v9odqkQ9vDuevPn331yNxEKkHci8wBBx4
x05MhIIDiPvHm/vwew/vz2/w5vf/ffvh/SGg26HwDnkHz5h+eTg/vL2s9JrMSFTArNbEm2ONQWDS
CBT1gjLf4uXT44fz509N+BA0BRZlJFjzAjPeM5Q2vAoI4gVFwMHPMI7vHx5Hrm5RbHR8ZNihjbcJ
wzbt378AgepPbz887utbcvn0/PLpp+17hvjbw99Pr7//27+//f5vb3/86/dv//Lnn8xuR/DzN3/6
9Ol//su/mp9+vom2FMaePxoA8BgxAFGQC2LvqEHafUxZnQMaZ/1FKcgxRQJxnju/KGIJDCuRxKER
BNPBSNhkpEoNNNkoL2/jlQM4r04jpvfogtoKHOLyW42TYsgcL1Qy//lsp5BuIn5ZKWhFl1sNp1gN
kFEgsgUIThjr5qg64aT+HLuXlo33zkTniQbJgiVgbax9P+3SIBSdZNA5qhytaHKbSnjrzAnvoqXI
K7qdt/Z5JagtxYDoYzvL3tuw4Q0Czx9/f1ZhuAPZSb/wnALzOzRtMAwZMBQExqu+oGHLwRlhsJ1w
ZAe+H87RyfFuv+cWHucxh4fAikZHoRT+k/D/Tjx2B9b04nFHH4irqG3i4exhWeBwOhGe4JAEf6Py
17qOy+ygX88cDkfZg23jsV4yeJwL8hHBCXwaJMRSD/dQgVt2ezB7BG1zj2WXO63gImLMPEHVhr0x
3WBw5/rRCO5JbyuV00CjPocGNPivkWAFFrVht7o5h3die8F43Ju7IOdtMJITLETrNHZb1FtChh4w
oL5/a45G/fHUFivxPrc1YigOMcAG/ymIeZ+QF6DxQRyCgr5toxHOsY3hYExivgEbPAxC6WYcKBGq
O7497UUObTxGc3iCEHGlYT7jcdYzWyXfx8hXBztQ4kyYkSYdS768Cp4+Hh9vzo8X/5Ot88XL6U4c
ZpYDDHY7jiaDptC2A/Hst1DCifzkWP7P/o0xr6uf/eu74Ni61wa/ffrk7u755xn1u4f98XUQk8Mt
+g7/uPuYdMfIOdJEJdIzKoFVukjjBGmUJY36SWMMFlquoA197tgqhzoSTKNBw3eQJgnSOEsaF5ya
t+yPV5AG4jKkYdCCNooVgsJpacCYNk3QJlnapJ82b/dmb0fSps7siE2GNkcQyXcIjw2aDtJsgjTN
kqYFHGkPeDpcQVrwJHKkoauRZq12keYSpNksafYLkGaypGFkcro40Sdoclma3Kw0kTMegvnL0lSp
jIiuoE9MlwaBZPYAstTdlsjZrT2Ooo6Cew87b3JiFgLvKAxXo+y6SEvZNMgbtUM/afbEYEapRwqx
oQ28iFntGB2a6zyxlD2DvEE7FpxYlbvcjyTL2p3Xa0UMUpYM8qbsNDNRanY2rzeMickyXfoQUmYM
8nbsbmYmNMHSeMoQphInHq1AJ2UpVQ9ZXQ8wL2XGBvcAJKc54ii5i6iUroessgecmSgNUSlkHWGO
ncUQRHX6imhSpPksaTSriLEnv1POOosEEWk+hFadpKWMGGaNGMi8pIX4ZOes5hyqWplIne2KXTCl
FzGrF8HNTJlIBSLPj3GqDrHT88CUZsSsZgQ/L2kc+NE6vdKOYUopYl4p3s5KVKXvd5q3Y8U+B6V0
B+Z1x7w+R1V+3AXicmyoPiLMgsFO0iCVHjBZ0k4zkxYiFjY5tcjiJc6wovddJppSepHyevFuVtrQ
BQ9Yo8JM06vyGvOjoc5jSylGyipGxHlJsxVpmItbNC44ob/YtjxlqewAZdMDyDNThrCjbFInsGAU
kTlzKR/mSUtpR8pqR5zXTqOKBpcxn0GNs4yCrjPJSCnHkbKOI+rMpFX6WXOkEdu4mi22y6RxSvdT
VvejnZcyCZLGlD00oxFpHBySTtJSup+zuh/dzKQxhlhDs2YtFjV12ClqnFL9nFX96GcmDWnnJXdq
NjbYRNhpsDmV1uFsWgf381LGVBnsrKTFxWLq9Ic5ldfhbF4HZ3Ud++tLloyLfJHgmIALLNq6JxJV
mFr3AnFo0Qjt6KIRWBtXp0MU5pA74HIbLg0uBPXhzReCQJ1yVMADJ1aBNA9Y2oB5cHmnD3C+vBM0
r7DEiL0oe9tyUO/jEk8Lsgyu2nRB7q7aeLUa18HBArNgy2Tcx5WbFmIdnN3vQ5wvxnhVQKkhtuCD
85lH7NqI7eSI8zWWYKeCisT4YqwPZpkQJI/ZtzG7CTH31VA8myBrEWTwYIO5dXnEkLqKDYMrI1nM
fZURrG6LR3wRAhJw3AU4oY4BB+cu84D76h3O1Yo5wVBZZNPBFJDQyECDTV4X4s5ShnOqccakumGh
jNih3yChk4EnhdxZqKDqImWUCtHA89K1xQmFDDIpU3TWH4Ja8LGq8E6Dl9Fh8yCh3cBOCbi7rEDh
/KN4g6qaiXRJXUKzgZsUb2fFIBi8WhQR2CcocCMdflD4gjZkPyEP91UC0KH1cVCnWmXJqMPmYUIb
I0yJuTvFD+KB4ljNVBcAO/Am9ATypHg7E/fVreXYDxIOfNIld5hQFChTAu5Ox4PB2Jm3IOo69DAm
1ATaCeH2JNqdJTKR6UABIn+5u5uNlhJih1OKXU8S3Vmv8TMU54N9tl32mSARMZlJEXfmxin4lcF7
jzAHyD5sPAp1oE6oCppSVfRkvZlE4ouaYIgVvXbEIJTQFjSltujJZqMPG2swLs+SD9xsu4L/ROBE
Oinm7jw1BIshNadeQ+Bk1XSF1JTQGzSl3uhJQZMXiKM9BSBLHYqOEv4FuUkBdyaWgy13sQfnwTm4
PCDNplkSeo6m1HM9CWP2oPFTq/BHDezd4VxwQs+xmRRxZx44aGGS2qMPoi5LwgkNx5NquO7kbgir
OL5+Q/j0UjaLNxGZMk6JtztlW73A0thHDnZajOMuJk6EpjxhaNr7uABjA+JGvV1oJHul5JXl0y58
4bTuNchmzuBeA23mXO010GbOyV4Dbebk6zXQZs6yXgNt5mSq1O8+KPs4eofq5ZsvQDlz/vSaDZw5
U1qDhowY38pVSUXmXzo1etXuzZsCvQravNlOqTtGl3f1L1kXg0GUXAnKeVOcV23gvNlMqb9Wt/Vn
rM6J4RKQ8+YvayBt3DnCivoS4Z05WSn1K1y1h8lF8ObNTdYPOQgRxnUi8si2BOS8+cirHOR5c4/1
/cM49jJYAm/ePGNDhmvdOBgwROAlOnDm1GINZHWLwcaJGBIUAi1hw5mTidK4+czxC57gWDBSCch5
04fSeJlu4puV4ETD7xTt5bwZwyZMiKu9lh3Zohhz3gRhXXqc1NS3eC/CJSp85qxgYyttDFIRpAji
vGlAqafhvfj4aalUtYSSsH3ezN9VGYV5k3xXQZs3n1d3sykqHitwkYc9d/ZOpuo7UjVX8SEejai3
9cKHl/o7wbBTGjS0r11zo+d9uHTBsxjs65iecX1wuLIO5GN1AGQ0MJZAs2ncc7fNf37/+C7TcSx/
gVV3ljWHIr5ZSaqmXqF4ahj5/vSLefP53Dtt0dV4urrYdQMhjYPBBYFwLapaGIhbydFYXgMQ0NhK
LXs0HleyI7QKZg0eKIxBItNvCcYR1YJAKo2mawASuBW/wI6YkqOJNFpXHW+m9TVOJi+yPpqr1wes
+RxuGIZ60vDL70GQCuYlz6CmHhahX65fP35mpt7xYB5YdA8qOZBlz8DCgutXxhLNwkIACx+AvV4R
xp2WjRkhBHbhM8CFz2BhY0xLy8D1zkDc3Ac0dXG0jwfdwjywpDMQPFO3sDGOXPRF1teFz1/tsue/
qCEO6+uSAUG9groM/bzw+fuFbRAvK3+L6r9L1kYXNgC08Pq49AHgwl4QLyyBujD911tgX5v5YmC4
GSC/NBcubAccL2uHl/UDqgNYOiOiC6sBWZYBF03IhPVl6UBkaTu4aFq6XgdfhAFg4Uh80dJMtf9L
l2aWpv96AYy7VqmnEdmohaMhSwsrwYWVMFyfDcDayM1Ux4A+HoBl5ZAXNsR+6aw8L+2JLp2SWdgR
wCGDy9jiMAxiPq9/eUw8fPBb2CByY9d8eiY8fFEJ5jF7M1N8/MrcItRuQL4s/vTadtziMHqXL6+P
hy5avVJFktHbfHmGOmzRp/enYHOL9r8/jXe6el867MLr5WGpU8j6FlhrtizECJQm/+ml3HRXjqD2
ggEtmlrzt4jui0hfO0yx6LBHXmqmetMOY5xw3H62ejYUhDTROvj5FvE3T7eIk7PJu8d/H85fHR5+
++3+fPMJDHxdvit15BaMI88m7gXFQkpsa88N0hfPYMCy0JwpjvWZ4sY1739XzYjbw+df3JqHj+fH
j+fXOBYEOhGKx0uziqnmskL+IvoMMIiBxdc6ObILUmwarn/YstYV9DYu8y1izChPK1pz8/v+/PVY
iFC1TNW4+TyCekzMgG+fVw1QxDxsRm+YWHQcD+hWG5QaWafSnvPevUMRIDsaUDBtcbu6ap4tSqq1
dmN38u8ZZDSWwvcVMOx9hY7Gg9VAaqM11YhWnGFPmgfUt0F+NKC80Y/7Yn4GgnhBEXAkr84DTo0j
xH0v29C3OoyXIZJ4hJBhrM0MN5nnLd5PTO5FLkb2BOJRA4bbGgqbqpzKu//wqEnAbQzUxMDlbX54
zMTeNgRuQpDydj7cmqzLtcm67SfUaQzSxKDlfXt4zATcNgRtQhjQn4fHTKptQ7BNCK68Dw8PGhPW
Xts11/bljXZ41OTXhIvXcq4Ai/vo8IgJrQkILaUA1NMvJ7sPvVPNEsu39AFwcS+ca1kAWpoApLjb
DQ+eTJoA0FIDoMU9bXj4BNEEgpYIgivuV8ODJn0mFm/JIPjijjQ8YiJnAoJvhTflkzN5xOTMhFVu
R1hY3FGGh0+4TCBoyQFKcdMYHjGJMgGhJQmoxS1hrtUD2BICdMVNX67XgdjiwQETGHnEBMaEV2Za
EKC4ZQuPmZSYwNCSA8Lidiw8YqJhAkJLEEiKm63w8MmDCQQtr4zK+6jwiAmBCQgtaSBX3CWFR0zy
S0BoGQbyxS1QePjEvQSClkxy+WQ8HjEZLxGmtGSSobh/CY+YYJeA0BJJxuI+JTx80lwCQcs9ZSpu
R8JDJ8Il1m/Hq1yeGWhsgasNLNGS5DzhrTMnvGvk6N3OR8pPGl3ia6NyCk6aTT1X+NvD30+va+1W
nruv/PjX79/+5c8/mV1Q1z/ffDid397dn94dX1ffFDhQdqLf/On88fcqq3Pz2amL6DmFGMKhaZPD
kCMHTdyKPSg1pQKDAjgRTRJ8xG6ajk6Od/s9t4hyUe6iThTQxTd71ozVpJUChgSdiCgbHH/TSZQ7
+qodt7aJ4izjWa7JGAhCkZhNxn4mHHsnVYfDUfZg21RZL9mjsnETRClR3m4qgmgHnfTcHsweQdvy
ZKMwpU4PxTMvyBUYZJpKlIJMum56BPekt1X42KAnuFNZ/RA3YguOli/QeFMdUIiHbSdBHvfmLmjw
NkGSU3jBuY1PyJYEcWYyekIE0X1CR6P+eGorO/E+J0E29saDBJV44zyd+u6hyAd7Yglu2xRFUW6T
onj4mEWhAq3AU5EE0Kfm7vj2tBc5tEkymvUaoF664hIrSzqdaujWDD2eVp2Ya65AJL5czYjCIxpz
ReVcp6WnXk+yzbyV9/FFXKQQMVvfV7MsmjcBP/188+nvKP4c6y38nCmvRDXRBzGNO8yoejHWFaAv
mEkRwecr4MdFrDp8cGJqDrtayxZKNr9gbkUEX66AHxfAGvCrO11xCQxQjYDnkv0vmG4REaAjCWhW
z+oEeO+Z4rhMVC9EFeAvGIER4bdX4I9Lb038zkl8JSNsvhc2WoDfDsLvrsBvsvirgnwtWSdqVMmJ
twUUuEEU+FEUtKt/DREwholia8AihixRAQElkzYiCg5jKGhXDhsa1LpaZxoQUU9cgr5gGEeE/jgK
favoaBvecmCWWP8H9zkcWZEAl0zqiPCfRuJvVC0bAsyC8X1KNGCrxtypKd1tAngQAXcjCWhUPhsH
oDZu6sqoalwR98gg7gEzEn2jbNqU3mCb4wyxVuUlKtp8Oww/jsPfLLo2dt9S/IiK0Hssk103DD2N
RN+o2ja0P6KzNeYn5KcD6SfAD+J9GOW/tWu+Dd0Z3E2pvWJzJFaNL7G/JbNGYgp0HAXNknHLAa3N
eAZlcKYE/TDdA34k+ka5uek+YO39GFy8hxL2KRlSEsPfj4PfLFU3N59qAw+RneUS6S2ZYxKjH+c4
tGrdTd+TfM30smFxULtVniVgoPSOs72tanmdABvivVp5UFGr+kSJ9i+ZghITcDeSgEatvaH9wSig
r81PwOD+h5isxPukYQoIx1ngVq2+GUIyQhwCY/hANPxSCQXDlBCOs2KtUn/TBQ1BF8TPK0RtIMsU
JSF0GAUykoLGVYHmGVjHtTCMhK1H8FhiCmiYNsJxhqx116Dlh2p85YKruMBxEX43DL8dib9xUaHp
iXIwXq6mToPrLVLiStAwdYpuHAXNiw5NLnLqLMfuHFbTv7mEAh6mT9GPpKBxT6IlyWJjHnKAxpWI
MQ9UpfuR8Bt3LGxjpLaIqT3NC8ElZqbpNPDjMPy34/A3b2g0HApFwrg8acgIcpEt42GxPB5G50Pr
ifqGCHANvU5QfHDtiwNUP2RX6W5MPuVrNFJvFiRqO1D8JNQ1LlhB8Jlar/qsI2OxdLDMADyNgRDO
FA4dOB8uD+mh9pC+aYtvzo+Xz3wQ/VWBu/Qb+oyuusu3JnSXRiTP4MisGhysGRyuF5yseef0y/Lc
x+OKeW4oOFwvOF2dKgH7GRy3r4auBptfGbhLt7iVblwN2xo3zq1449xaN+7Slyryl2C14IKr6Vd2
rPKsgatZPasF580M5qF2mxRBxI2II+AFoqe1BTkxuFUd7qWX3lpltgZuhTLrca07VwO3tp27tO9b
c9S/XnSX6YhrTUnUwK3NObnMdVztzsXgrto5mQUcPmth0GscAJnFur6EEqtDd5lkuVp0lzmXL6pu
dVsnz+Cq9ysrBsfrBXeVqpN57OuLOnErY7oWOljZwYJOkZmYG9vaTNhldMdKN66GbRVeU3PmSFc+
gEn9pgB7WEF+uz5cCjmRkF9XaNkYTu22BPhSxzU9SZnVIW4xxboRX8BtbItTTIHPsQTCDGysJm4i
6qdijJlRX8cYAdz/K8DOuU0BpvUBTumK9bLEUwKkKykiZmVORS9i3RpiVLJ+bYhlc1whm+MK2RRX
PKUjt8QV/Yh1a4jXyBVktsYVZLbGFbQtC/JUHNgSV/Qj1q0hXiNXWNgaV/Qg1q0hXqW3mdJusrkY
RDYXg8jmYhDZXAwim4tBZHMxiGwuBpHNxSCyuRhENheDyOZiENlcDCKbi0FkczGIbC4Gkc3FILK5
GEQ2FYPUr19uIQaZ8MLoEog3EYOsnytkc1whm+KKlH+8bq7oR6xbQ7yJGGT1XEFma1xB27IgKf94
3VzRj1i3hngTMcjquaIHsW4N8VZiEDJbi0HIbC0GIbO1GITM1mIQMluLQchsLQYhs7UYhMzWYhAy
W4tByGwtBiGztRiEzNZiEDJbi0HIbC0GIbO1GITM1mIQMluLQWhbd7HqjR62EINM2JpiCcSbiEHW
zxWyOa6QTXFFyj9eN1f0I9atId5EDLJ6riCzNa6gbVmQlH+8bq7oR6xbQ7yJGGT1XNGDWLeGeCsx
iIWtxSARYt0a4q3EICvnCtkcV8imuCLjH6+YK/oR69YQbyUGWTdXkNkaV9C2LEjGP14xV/Qj1q0h
3koMMhVXyBKIdWuIr+MKmaV1Zfu1W62l9BceSxfAmvBV9sRgjnXok8OwLpqBToxGOY3m09bNi6Z3
U5Ze/x8WAdlPBEPGw7D7g93/X3Vn1BrHDcTx934KPybU2c6MZqSRoXkqhkKpC03pQwjm7nZdDG5s
nEtpv321d4lP8a5u13ebeBb8YjDS7z8z+mt0PqRnx4hhGerV8yXlerhv8XvfzX5+xu2DTrYZ++K4
4TYeR2uM2xdjeh4HssXIYJzx0+M2tiG3bxeZT3Ynjhtu4wVpjbG3IK1B9hWkwWT3LWwX7TukKcaS
Q5qCLDiktWR34rgNrvGKNAfZW5LmKPtq8khI+SqhJNdz93iwBNnyGGdsH/yYA6PaZwxgnDGtbO9m
wMj2Gfu+6G7Le/oMcrPYjRukNcY+g7TIqPYZAxhn7DNIi4xsn7HvP2B+FgapMzBInYFB6gwMUmdg
kDoDg9QZGKTOwCDVukFuCsC4QVpj7DNIi4xqnzGAccY+g7TIyPYZOwZpzntKBskzMEiegUHyDAyS
Z2CQPAOD5BkYJM/AIHkGBhnQvkGaYiwYpDlGtc/YZ5CmGAsGaY6R7TP2GWTAb/+v9tKXx3vejFCz
dO03YMk0nTNNx2bpWld8hnPV2FVBlUPTdGSaznBmfeXALF27ZoNpOjVL1zpKME1nO3bRLF3qoBRM
+x2bplPTbsym6cQ0nbe7V7Rf3iXbeGwbT2zjedN9SrRM553pTsBbpjPcCbSf9Njuosg0nTNNd9hu
ESEkQg5UIMTAGHSHiQzivHMZrTxcvyCrJcUjMJzTAgY5EaYMw/so0IL1YDBFRjmYQyrSUjhcRGzF
P1yNkf5cPPdzxLCARTiKA0vx0EA5BjnUL67oyJISA8f6QAqvqcDcE8s3y0RYUbM6aG6nENM2Hkpz
CzmOvAPwBCyIVMhEeyHFfo6L899+f3N5/svFnyf3zd3NfycvLs6xci9PXvx7Xacx6OVZgdV5ZqzU
F5MlKtl1KgkSlKCfdHu7y8E1Eyr28WnZ2u1X9f3tXTa4o6VCQ1e50uhJqxjCwxTxiymYg3eZWxDE
oOIjhtzdYONujxwMKviBdv7/9+0/TSYb3r579frTr2/++PXy55/eQlU5fHf6oVlfXl03N/VZOxJV
KJX4V6/XH9+3w59+CiBmuhpBTfHvymIsySJI5ZjrIh+U0iiDutL5dhpdUmHcr6tWqa8WC+4I00gF
YU6dePFZbYaIaWWltTQozE8kLKRFDnuFaR2Z1PuuMC4Vog9pD9Ws4hGwlYUah4TRZJUIKft7ha1W
tSwwdIWFKAVhiN4FiZkwSXuhhFSRg8J0KmGuwr26litYEPruEguspUoMgMJ5n5FkavQAQ7LcVCss
LVXdr0to4fyy7UMe6fKxpCvpEM4WmHMUXdrXh1RNlSuuJOzVFGkBV8nnu5qkZIfJHTC/HywZPEsk
HdIEk2lCH/cnqgYf66ZrhRJjKVEuWTpmiUKi1OvR4MY1nb8PiIreQXC47IoSLm1cDhxo2KniEMG5
qMP78VS6EIdc8IqXzUJk1dUFvqQrpC48YO6CguLYswzahZ/OLva7xUBzFo9qzoYGjyMGJ9jegpgF
aLMde/l8xaF0jrz0FRg+1t+IwTnJzpAKnM4QvcH+63Z9e7kZISUbxk/XVtzR8+FT5sORHyes1pcf
1ot18+P393c3RwT00YyjFR4wE062QLZ2wtmpHnFU4LILPqduA8ca5UjfGdvNjGzmxh6rxu7NI/uS
sbvi2NPR2MPGab4A6SkLniYt0lhB2HWSJK48+OcR39+umzNIPXBdATz8fPc/6DDEVXOBAQA=



sh-4.4# base64 ovs-ofctl-dump-working.txt.gz 
H4sICJNoJmICA292cy1vZmN0bC1kdW1wLXdvcmtpbmcudHh0ANydW4/lNnKA3/Mr5tFG2mdZV5ID
zD5tHARI7CDZIAsYxuD0bdPwpRszZ5L1vw91uqebkkiKUktjag1jgekdt76iisW6qfj9t//+n39+
/+2/fv/fbz7cPPz825uvvv8WDvT1m6/+dnf9zvwNv377D2+u7u9/ursJfzIXb64/fTie7u5/fYfe
OK8Hj/7jxZvT8fLn8Bcu3vz6/uF49dPN6eM7z2i9d+efXf52uvn4DkkFxFj1ePHm4cPd/Ye702/v
wBhzcXV6//F0PN28++b04aeLu4c3x6vuIR/fXZ2+evrdXy/kIHKA6F4wBMGpqAcTUXCAuHu4uAv/
3f2H0zu8+PX/3n/8cBXoDih8QD7AM9Nf70/3789PektmIRWwkFUbLY4FcMaRRlA0CWX+gOefXn88
ff6pCT8ETcGiLISN3mG8ZihjvA4E8UwROPgZ4/rD/cPCp1t0Bl8IyIhVxzFHWKbjhxcQ6P70/uPD
sb8k55+eXn76tHzPiL/c/+/N2+/+8m/vv/vL+z//13fv/+VPP5jDgeDHb/749NP/+Kd/Nj/8eBEt
KSx9/2iAvVGK5CLP1lpiGIh2F0vW14DBu/6iEuSUIkGc184vSiwoLlIkROOYS4qEQ0XqzMBQjfL7
bblxAA9EHCmHJ0OEnX3ILzWuypB5vdDt+c/vdo3dHeR8eRIwu9zTcI2nAbIajs8CIlLXP466N5y0
n0vX0jIohu0d6Z4JuugMqJfBs+/WfTQI+Wh9AcnGC2xyi0p46cwN3kaPIm+9OzDZ5ydB71EMbE30
MBFCtkoDAU+ffn02YXgAOcj05rkRcA7NGMZABiZsFhuO+Rcatszqwj9FHDmAn8a5dnJ9ezzyiIcE
czwUDnMLLzxK4qxR1iKPPYA1kzzuOjhbTnXMY7Ivy0I4fiKvSFkgHD4TPOF/YdrOXF1dyxHsmAdF
MjzOd55hpKlqjWVSKONQhVt2eWWOCDrWnmD/s9pjDEbblYNLIyRchsGDm6YRPJJeMo5oQHI0oGFt
ooMrqGhYLfBFGj6InYTxeDS3YZ+PYSC3sYIrbSR2W7Rz8oEmYIKlm16aa6P++ma8rYz43NKIEYiO
quBCG8PWT23yChqvYfcQXI5pgHNqEx6NEm1yBhu2lTcTNhBqNtUtX94cRa4GPPbgVXM8Gna5i/TY
uWCURYWmFPnVwQ7UOBNm4ZGONb+8C54+XT9cnB7O/idb52c8zjBmHgcINo4mIRx7OnYgnv0WSjiR
T47l/xzfGfO2+/f49jY4tu6twT88/uT29vnfZ+qf74/Xb8M2ubpEX/CPy69JDs5xTjTR3pEgYauX
ROOEaJQVjaZFY/QM8grZrORemz9nHZ43punO3oJokhCNs6JxxVvzlv31K0RTcBnRMDh5caxgmFzp
tWlCNsnKJtOyeXs0R7tUNvaHYLUzsjmCyH4Re3CuIJpNiKZZ0bRCI+0V3ly9QjRlyImGriea9VgS
zSVEs1nR7BcQTbKiRTvNlXaZT8jksjK5TWUih2oO6rIyBXc5im5c+BOV3hcksweQle6yZp9d2utF
0pFq8DJIc9sMOHZb1HjGkmipMw3yh9rVtGj2hsEsMo/UeTRBFzFrHWP/pyhW6jyD/IF2XfHGUKw7
LhSL7YHxtVsMUicZ5I+ym42FogCRtxsmOsecYOkUg9QxBvlz7HZbJUTbPZ0ygqn0Uv3ii68sZeoh
a+sBNpaM6eCt5CxHHCWXhErZesgae8CNhSI4OJt3hIF6qRouaSKalGg+Kxptu8XA+AOYrLNIELn4
3lLRw8fUIYbZQwxkU9E40B6INOdQWd/LS0DpeMaUXcSsXQS3sWSdNwdZfWQnkT4Okrgj0VKWEbOW
Efy2ogVvIgDoK88xTBlFzBvFy02FInUmnKLZcyyuaRhT0kNKGQ/MG4/rjeVSe7A+a+zDBnyRzKIr
6iFBKj9gsqLdbCxaF2hqzi4G20JxitXZYgxNKcNIecN4u61sRmzwVG3WBe75VY6KjhWlLCNlLSPi
pqKhD6IZlwtcbFzNC39Xi3stlR6gbH4AeVvJnDcHl83qgI+bVpzx5SxjyjxS1jyibCwaSvAZ8ylU
jWuwZIo5HUp5jpT1HFG3Fc16OljMiUZxHA1siz4xp2w/ZW0/2o0lC3G089mXZmwUw3DwoEvHNads
P2dtP7qNRQM4aNYHAaWoUqzOQvGtpUw/Z00/buteoQZ9ZMi9taCtcYsRmWI+P5XX4WxeB4/bSia+
O7DzO6230UqOCKcSO5xN7OCmviMEBMO58yx2HItnmaZOac2e0my2lGm6aGYJTbTHJJzqSF1SZND8
EpXNRs2OOLcShnZxJQysk4g3hJbBiHAel8e4NLu6NcWbr24FXTkfNS918K67BCQPLGNgnl2zmgLO
16zAqDoftyZ5CasDVEDWMbLMLkUVkYulKK8O47QQWJAQ3I7csru4HDUi1tkliynifIXJq6p3PWIH
nqiw7dyY2K5OnC8chSX2SBB3+3oVsEGb88x+zOxWZJ4qDPmuyyNaZQqqbEQF8sSQ6i+H2eWeLPNU
uQfBxcSAIdgd10buehWfMTDOTsjmgaeKOM7buHtOHHStP4WtBwmLDDT7yCsRF+szIbijXhoIsWvR
G7Um3fVKNWNkXhW5WH0hwNgiWxVRKOw7SBhkkDWVolxUcWQlbtIzxopwaYET1g3sqsDFWgkx9dxx
9GC44FRAwrKBW5W3WAZRaznOqYMCe3Ji8sjhF4yR/Zo6PFHeCD6W4Sg0YHUaQglTMG6YsMYIKzJP
1C1AfPxdgjpiKRljTNgJ5FV5i9UI58677Fkrup5CNAXHDROGAmVN4HKNAczZ53k2bBDckNL6JswE
2hVxJ6oHznb9vVFaU7DLR5fCD0psO/SrIhcLA84Ffx/jHjQTflJy2ggSEZNZlbiY7yfP4S/4KMbz
znknoMWoNGEqaE1TMZHJ7zriNe53NiGIJqCCNlPCWtCa1mIiQ0/GUK/+4K2SRQcF540SgRPpmswT
uXdwYWHjD3bCugcP1DOVNmHCbpBdlbqYVicfoo7eaY1gSoc1JfwLcmsCl5PlHPZg/PmX776KBCwQ
c8LOkV+VuJgE54AYh6c+hH7ktLD9OGHn2KxKXMxt4+AzKETVkhJzwsLxmhbuKWGdc+qDJ9H78qbz
hXxpfRORKeOavI9p6Bxv95GSo94ndi4YC1Na4kRoyiuGpo8ZZqipmEJ3eBQSFZo4PHTFw+Pzxx1k
pttDoNdHYWZ9zBG9Pan5yvVxFb5EBlrWIdsi2bzSom2RVvbroG2RPoZ10LbIE+M6aFskhFdC2yLv
m0PToMbxyaoYzlZxFZRbpHpXUrstkro5O4LBGYm7osPiaQ3iBlncldRvi2ztWmgbJGY5gybAcYMP
CICeQ+1pyg2ysbrSAm6QeM0d/kRWY9MCnli5BnKDVGvurLVx6BbCpqoXvEVe1WYAIU5S2jq8DdKo
2ZccQoW4pGXFeGtrIDdIndqVHOQN0qQ5AxgPNhGCKrwNUqKa3cNELt7DxIA13sEWWdDcHvauq0pG
24TEWvJVi7lF3tPltrKl+Mt0DM6ZitRAbpDpzLla0BWf4gScc9iluGowN0hu+iwmcC8FIGC1SjE3
yGXmfENy4UCJc5nGPE64mabcIIGZo4TeJ0cYLAL5GsQNMpY5RH5MqL582mvDqVSzjlskKVc6ZTbO
R74KbdvUYz8A5V63hqUaD3HTROOrEh4bpxVlrYE0j1OAxEWvxQ6m91nCeCoRGwIk8mjGwz0/j0e0
GA7+JcMEp3A4nK3W9D5LAALvjFc7HNz4PIb1Hz88/JwZRZdvbA6xPpgMRWmc4MPFh5u/mnefX/hA
NbgXev2OIKRxePo7r4hvAST8WjaNrIjHRnTEciOvhprQEYTg1rehrRiHVrUgcfpjRUOiLYCEbYMr
rgiZOOunQLbf41GhLL1IaMZwLE80b3UCyxPIuYS35JnW+6XPfCzOzX8oB+uSdTs0ziaFqM70JkS+
PPyxxrXs4bh4lc81v9kP7cpWcYZl5jKfiz/zHvpU9aHcQ6erPvFKd1Wded7cYzkHcwt9Ht8Yp9Y5
AJv04x/z0+uZXuk11jrpxl0O9ndvcy+aIUkzDf9Cl536bT0GiLnfru/VdJ8VjlrrXq4qePSRkyPZ
y1PPr05fXd3/8svd6eIJBr6uX5U+eYirNewSjbtFwx+Di68uvUzxy4AZj4XBKPVe5iOoIPXGrdDT
54rjmfsvsfv9p9PDp9NbXAqBTgUhnuoYIhrvSTUfZm2AQQy26w2MP0d13hhvpPcNUHddwjDAGnP1
riR4vv0iRGq/Hk9fL0Xsxp7HgHK+isMlRt9P3JEQKQ+bxQsmVjzE057Vkglm1LvE3QflFYqA7GIg
i5ZM74YJVk19UFw9+l8Ws1RmD2Be9kAX86CGxTk7ci+mkVBZnHWUB5paIL8YKH/qp66dSN1FEt8l
gWtzoETLMPV0WL6HKN7Pztj+l7O5ewS8X1nc/mUJ81rxeNFc5bGFwqEpp/qmO140AHnMQEMGru+u
4yWDiscIPESQ+i46Hg0UNr2BwlhjDnsNdU8MWt8ux0sG/44RdIgwoy2OlwzoHSPYIYKrb3/jWdPR
xs92w2f7+v42XjTwNuHijZwrwOr2NV4wmDaBMDIKQNXfHvPMAbKJx4/sAXB1C9prVQBGlgCkusmM
Zw9kTQCMzABodSsZzx+cmiAYbUFw1W1iPGvAaeLhoz0IvroRjBcMIk0g+FF4Y6rbvHjBwNDEqTyO
sLC6kYvnD/ZMEIz2QfYGvHGvFi8YwJlAGO0E1OpOrNfaARxtAnTVvVY8d6Bl4vkjJZwcPJndhFWD
JxNumRkhQHWrFC8ZEJlgGG0Ewuo2KF4wyDGBMNoJJNVNTjx/4GKCYOSWUX3/Ei8YjJhAGG0HctXd
SbxggGECYXQykK9uPeL5gwYTBKM9yaa6s4gXDARMxCmjPclQ3TfECwb3JRBGW5Kxuj+I5w/YSxCM
/FOm6jYgnjsIL/H8ccDKlV0+PG9gXSJCG1kjlfq0xGD548FL2LMES+42fDG80je8cXYXa/wPNvnb
ZR87mZ4bm3oXn158vDm9v727+fn6bXRD4jd/7K7ouv54uvjsUVbdjog5cbB39axH1RrHEnAlmc7X
LBZlKlyxSBmhgHz8PZG4zqerEEpXEup8V2NRqMI9jVnFsxyrOIRIyVSdd6upX3fjY1Gqwm2Pmn9V
8YuqSiu4tQSiAxTlKdwX6TPy9DxDqvELaa2tdL51sihP4cZJn7UP8bczvmsXnBZprRfU3VxZFKhw
a2XO4FHsMgLZmi1kVpMH1JffUPbiy2iC32AH2bjma4WpwvPh9cz3hESFyzMlK5FEOecQ75kKT4bX
Eul8B2dRpML9mzbrNdjYMCCPv2BImQZdzzSULcOEpyWr3+EZ/XI1C6qeaMwryva6tjxxMcsOkmZd
DSXOGLFVdXaqYFo1YwJ++PHi6f+j+OfY7453pr4MNqQX7XkdGqJ+8a6CvmIORYTPr8CPK2h9fHAC
1Md3LAwV+BWzKiJ8eQV+XH0b4EPQ/d7YXlQkzX0b2hegYqJFJIAuFWBQuhsIYMI/sTEP6hRewHm4
6KQAFXMvIgHsKwSIC399Abz3vbITSQiYQBkr+O0sfvcKfsnyu7D60MtQKXQfpyFIhQRulgR+kQTj
2uNIhYTjw5NZrXW+yoLWjNeIJLhaIsG4bjkwoZ3J6cVtarw3NfQVEzgi+utF9KOSpx24y91Iz9hj
JmIZF9+S/BUfmEX8Nwv5BzXTwQZmj7EBQoMiLvhfNTu4ZohHJMDtQgEGddfBC1Af98cxOjFapT0y
S3vALKMfFm2Hu5d6Lamd6RRXpz12Hj8u5B+UfAerH3wLG2d1g6hYRe/m0dNC+kHNeGD98TG+fVF+
CqEYS5Xu+1m6D7xMgGHFeWA7wVpycXuoUxcC3qpXUDNgJJZgkQs0LlgPPVDt1UbAstiasxfn2R7w
C+kHxe6R+xDPmEVQR7npLgN8mYd/XIY/LJQPF5/7lxsIs1TR23n0yxyHUaV96Hta6H3hEeyHCctf
4zvjzN17s1CAQam+L4AlE7Z2bPyJu76VGvWvGX0SC3C7UIBBoX9g/cG4cL75uO2MUEnq/DeaZ4Bw
2Qk8ahQYxZDIcd8OAoczwEONCaV5RgiXnWKjPoORC2rB9Ab5Qzce2NQcYzXjVGIJZJkEwz4FO7pD
io2PHQnpekiRTNVemGeNUBfKMGh0GPqh1sVHQTfs2Ne5EjVzWGJ+u4x/2CUx9ERZKa4KBgUSa9jW
nAc0z5yiWyjBoMtiqEUueG/Qi2XQgZGqdNY8e4p+oQSDJo3hTrZxua8bieTB1phSnmlKl7kTowYP
Oxj57XrHWTjeLNo6fpzHf7mMf9geMnAolMD2rwxAI67KI+J5sTwu8oiG/SUDdyIOJJlDbFNjfHTe
EcawOJPbLzEM9m6va5J0hbKJG9bRTf9OuqCd8ngdWeILyMGsyGEppbcC1V/SusEVRxBUy7nBx5Cs
VqR62swMnsEHy85UTkY4XZ0nEUBvEkGkQ+epMxenh/PPeNwt1QybbwyO9KD6Gc7RuGDXEhy0DIft
wknLK6dfVuc+XTesc3PhsF04bc6UgG30gOix+QZPL9/wyeo3XTgQ7g1f9dQ1SMzEPGeZnjGNb2oN
h3DcEtx5Yt2zSTEOmoXzIU5ubGt4bHXlenCtrdx5JmCrK9eDa1Dn5NkBUO+4WThv2vJOzuMfo9fq
mjohziMhW6XrumZdtF1Ns3CteXbh9wI1u3IxXHMr141vetkQzUU68gzXfWjQMBy3C/eaDRG1yqy5
IfAZrrtfoSk6jCOx5ujCi2Vud+26b1le9A4bp4PG3izoCtmJzdl8Y3DnWdKNLlyPrbWF6/bDi6lr
7ZQ4N9Ns6qr3BqeiVaohHAyYBzORhFo/5pZ4CrSK9D8OqeRG/uLcc3VheOfXFwfu9b6QsyupBz5v
OYQNUjIEvUvXyduV1GNj7tepR4D7uwJ2zu0KmLYARu7pMnsGu47laFc1HgPyUpDeQmdD/74XNMUI
T8zeiHVvxKhkfWvEsjutkN1phexKKx5zK3vSimli3Rtxi1pBZm9aQWZvWkH7OkEeM5170oppYt0b
cYtaYWFvWjFBrHsjbtLbTFk32V0MIruLQWR3MYjsLgaR3cUgsrsYRHYXg8juYhDZXQwiu4tBZHcx
iOwuBpHdxSCyuxhEdheDyO5iENldDCK7ikH6vWR7iEHW6377XYi30Yquayz6nF55DS+5fd2Q3emG
fHndWNlLblsrpol1b8Sv0ops3xv0B3D43gSOpb7yerqxWr/eLGLdG/E2urGyr9y2VkwT696It9GK
+MJbax2v4C83rxkTxLo34ubsRcZfJtOsVkwT696It9GKuAmcfX/E73JvuXHNkN1phnx5zVjfW25Y
K6aJdW/ELWoFmb1pBZm9aQWZXWlFxk9uWCumiXVvxF/Ct1BYx09uWzMmiHVvxLuIRfpfs+8hFmnw
+/s5xE1qhexOK2R3WiG70oqUj9y2VkwT696IdxGHNK8VZPamFbSvEyTlI7etFdPEujfiFrVi5B83
rxUTxLo34r3EIBb2FoNExLo34r3EII1rhexOK2RXWpHxjxvWimli3RvxXmKQtrWCzN60gvZ1gmT8
44a1YppY90a8lxikba2YINa9EbfnbeIBE3qsq1xxsdESi9sT8OP9CIU7E5jU7wrYw5ftqg+wJvwq
e8Ngrvvoq2Nw7+Maz54pTfO0dNvSUAQjApxE2Zih4sUgIPt1MJYo9GcMe7yyx98dw9tLe331+72U
u8nQl3zqwqW2GNk0zvh0eUvbkI/3QTX/skfreOZuXCFbY0wqZGuQKYVsbiHPo94T14M1xPh4XVnb
jJmThnz7J01TjLmTpinIzEnT2sserePj4jaukc1BJlWyOcqUTr4OcklYULGUSImZ8LYlyI6nccYu
CbQHRtc+ozWNM4adrbQDRm6fMdW335btSRnI82Zv3EC2xpgykC0yuvYZrWmcMWUgW2Tk9hlTBT3d
hYF0OzCQbgcG0u3AQLodGEi3AwPpdmAg3Q4MpGvdQJ4VoHED2RpjykC2yOjaZ7SmccaUgWyRkdtn
HBnI5mxPzkDyDgwk78BA8g4MJO/AQPIODCTvwEDyDgwk78BAWmjfQDbFmDGQzTG69hlTBrIpxoyB
bI6R22dMGcjWbM+oZ+r8s8b2DK/JuLxlIddAnrhixDVL160dNk1HTdNxs3T/39zV9DiRA9H7/hUG
b324ylVIy4kLlz0tJ4RGnXQHrcQSBGF/P+5hmJjpON2TNLuW5jIX13v18arcispjpfwP99OlVUGB
sWl01DS6hiOrgaFZdGPNpqbRWbPoRkVJTaNr23feLLo8iRo0rXexaXTWtBrHptFJ0+i03V4x/gia
2oYX24YnbcPTpucUbxmdctOTgLaMruFJYPxi1vYURU2j46bRXdQtyMFcgjPX3k1MkazcBxFBMVH5
upE8rF+Q7Yb8ChjJa44iFk1WwNAEUdxOwojkEeViHLlKYs0d7ITlRgpOLETptDs8ddClq3BQzR+W
YgmDmBOl00HxFL2/FEX0oK5PS98iEmlLw/Yi22ykkDtRzbYQeyze9FRCyMmaKpEYF1JcgoNVEQNT
NRBiVuYlEmKkeBLF9+0tF+eD5jJ94mfPYy/qP+8/FYczbQwG2pVMPbmFyPZgwn9eCyPAhkc7mahr
Tjz86SVVuFOuR+oEAX6no7b/s/93KGjD23fPX97/+9ebP29fv3oLITC+u/kyHG53fw8f+hfjSRRQ
gujzl4evH8fjb+4diAWvQdCMYEoLqEaLQMCK91WI1IiVAed45bvrOrwkZFee5dWb9LuuixNiLFwh
xhaVuQgYGgAkcIVZYroSsRQwwVli1ntuMKpTYlBLRE3oMEru8TnffFcyBbU5YrRaJkKO/lli220v
HaYpMRKtEENUYyyJ5Wx2jyI+S8zWIsYBz/LabKEj1GmJUTHlPMrENNZTOUNEN8sxm1UOXqvCcqna
eV5CHesm0oQXSo0XRtZY8OJcbtFltrzWilUMks5ycupgl3V+yglrckgQeWyiDyu31HPnEZrjBKtx
QvXzgepBvR+mUggKtUCxWinxjBRTnA9UXE/fZ0jlTgqJcTMlhVJrXAwyoj4+DJY8OovJLK21eCHO
qeAuboZOZPuIVwquqcYrEWH5+DKKCGIuw3m50PXk4rxazAxnftVwNne4Lzic4PuWw8JBd+1Y5ccK
Q5lcZ+kXYPja/0cY8gWoyBmDhCwnnf1+f9jf3p2Qgw3Lzd3X2nX28Cn2cOGngu3h9suhOwx/PPv8
6cMVDn1kcTHDCyzhagVyLyfFjT1rxRLHFQs81x4DlwrlQt1ZOs0sHOaWXquW9uaFc8nSrrj0drT0
snFTFiA9peBp1SS1IHycJEm4fviPEz/uD8MLyDNwHwAe/n77BnUpJjGpYwEA

Pod spec

kind: Pod
apiVersion: v1
metadata:
  generateName: iaf-system-zookeeper-
  annotations:
    openshift.io/scc: restricted
    strimzi.io/cluster-ca-cert-generation: '0'
    productID: 068a62892a1e4db39641342e592daa25
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "openshift-sdn",
          "interface": "eth0",
          "ips": [
              "10.254.24.29"
          ],
          "default": true,
          "dns": {}
      }]
    k8s.v1.cni.cncf.io/networks-status: |-
      [{
          "name": "openshift-sdn",
          "interface": "eth0",
          "ips": [
              "10.254.24.29"
          ],
          "default": true,
          "dns": {}
      }]
    strimzi.io/generation: '0'
    productName: IBM Cloud Platform Common Services
    strimzi.io/logging-hash: 0f057cb0003c78f02978b83e4fabad5bd508680c
    productMetric: FREE
  resourceVersion: '3409155'
  name: iaf-system-zookeeper-0
  uid: c93bd9d8-94d8-4cd0-8c71-ae00f7555e78
  creationTimestamp: '2022-03-07T10:57:01Z'
  namespace: katamari
  ownerReferences:
    - apiVersion: apps/v1
      kind: StatefulSet
      name: iaf-system-zookeeper
      uid: 8ae6e6b5-e895-4983-b90e-dfcc174dce31
      controller: true
      blockOwnerDeletion: true
  labels:
    app.kubernetes.io/part-of: ibmevents-iaf-system
    app.kubernetes.io/instance: iaf-system
    statefulset.kubernetes.io/pod-name: iaf-system-zookeeper-0
    controller-revision-hash: iaf-system-zookeeper-777b6497d5
    ibmevents.ibm.com/cluster: iaf-system
    app.kubernetes.io/managed-by: ibm-events-operator
    ibmevents.ibm.com/name: iaf-system-zookeeper
    app.kubernetes.io/name: zookeeper
    ibmevents.ibm.com/kind: Kafka
spec:
  nodeSelector:
    type: waiops
  restartPolicy: Always
  serviceAccountName: iaf-system-zookeeper
  imagePullSecrets:
    - name: iaf-system-zookeeper-dockercfg-gj9wz
  priority: 0
  subdomain: iaf-system-zookeeper-nodes
  schedulerName: default-scheduler
  enableServiceLinks: true
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                  - amd64
                  - s390x
                  - ppc64le
  terminationGracePeriodSeconds: 30
  preemptionPolicy: PreemptLowerPriority
  nodeName: worker1.bvt-rtp-04020027.cp.fyre.ibm.com
  securityContext:
    seLinuxOptions:
      level: 's0:c27,c4'
    runAsNonRoot: true
    fsGroup: 1000710000
  containers:
    - resources:
        limits:
          cpu: '1'
          memory: 256Mi
        requests:
          cpu: 50m
          memory: 256Mi
      readinessProbe:
        exec:
          command:
            - /opt/kafka/zookeeper_healthcheck.sh
        initialDelaySeconds: 15
        timeoutSeconds: 5
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      terminationMessagePath: /dev/termination-log
      name: zookeeper
      command:
        - /opt/kafka/zookeeper_run.sh
      livenessProbe:
        exec:
          command:
            - /opt/kafka/zookeeper_healthcheck.sh
        initialDelaySeconds: 15
        timeoutSeconds: 5
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      env:
        - name: ZOOKEEPER_METRICS_ENABLED
          value: 'false'
        - name: ZOOKEEPER_SNAPSHOT_CHECK_ENABLED
          value: 'true'
        - name: STRIMZI_KAFKA_GC_LOG_ENABLED
          value: 'false'
        - name: DYNAMIC_HEAP_FRACTION
          value: '0.75'
        - name: DYNAMIC_HEAP_MAX
          value: '2147483648'
        - name: ZOOKEEPER_CONFIGURATION
          value: |
            tickTime=2000
            initLimit=5
            syncLimit=2
            autopurge.purgeInterval=1
      securityContext:
        capabilities:
          drop:
            - ALL
            - KILL
            - MKNOD
            - SETGID
            - SETUID
        privileged: false
        runAsUser: 1000710000
        runAsNonRoot: true
        readOnlyRootFilesystem: false
        allowPrivilegeEscalation: false
      ports:
        - name: tcp-clustering
          containerPort: 2888
          protocol: TCP
        - name: tcp-election
          containerPort: 3888
          protocol: TCP
        - name: tcp-clients
          containerPort: 2181
          protocol: TCP
      imagePullPolicy: Always
      volumeMounts:
        - name: strimzi-tmp
          mountPath: /tmp
        - name: data
          mountPath: /var/lib/zookeeper
        - name: zookeeper-metrics-and-logging
          mountPath: /opt/kafka/custom-config/
        - name: zookeeper-nodes
          mountPath: /opt/kafka/zookeeper-node-certs/
        - name: cluster-ca-certs
          mountPath: /opt/kafka/cluster-ca-certs/
        - name: kube-api-access-dnhqv
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePolicy: File
      image: >-
        quay.io/opencloudio/ibm-events-kafka-2.7.0@sha256:d264cea91edec7c0975b6d11bf08bb47661fb9a8a982ea5752e65cdae9385146
  hostname: iaf-system-zookeeper-0
  serviceAccount: iaf-system-zookeeper
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-iaf-system-zookeeper-0
    - name: strimzi-tmp
      emptyDir:
        medium: Memory
    - name: zookeeper-metrics-and-logging
      configMap:
        name: iaf-system-zookeeper-config
        defaultMode: 420
    - name: zookeeper-nodes
      secret:
        secretName: iaf-system-zookeeper-nodes
        defaultMode: 292
    - name: cluster-ca-certs
      secret:
        secretName: iaf-system-cluster-ca-cert
        defaultMode: 292
    - name: kube-api-access-dnhqv
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
          - configMap:
              name: openshift-service-ca.crt
              items:
                - key: service-ca.crt
                  path: service-ca.crt
        defaultMode: 420
  dnsPolicy: ClusterFirst
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/memory-pressure
      operator: Exists
      effect: NoSchedule
status:
  phase: Running
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2022-03-07T10:57:01Z'
    - type: Ready
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2022-03-07T10:57:31Z'
    - type: ContainersReady
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2022-03-07T10:57:31Z'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2022-03-07T10:57:01Z'
  hostIP: 10.22.7.165
  podIP: 10.254.24.29
  podIPs:
    - ip: 10.254.24.29
  startTime: '2022-03-07T10:57:01Z'
  containerStatuses:
    - restartCount: 0
      started: true
      ready: true
      name: zookeeper
      state:
        running:
          startedAt: '2022-03-07T10:57:09Z'
      imageID: >-
        quay.io/opencloudio/ibm-events-kafka-2.7.0@sha256:b965631ad6d833439257e9823ac0f5361a8d3d0271cbb90bff5b5e006e564d9c
      image: >-
        quay.io/opencloudio/ibm-events-kafka-2.7.0@sha256:d264cea91edec7c0975b6d11bf08bb47661fb9a8a982ea5752e65cdae9385146
      lastState: {}
      containerID: 'cri-o://782dd7a7c0899ba473d64398eddd9807ed54af947f5d7445882f30a382fdef3b'
  qosClass: Burstable

Comment 28 piotr.godowski 2022-03-07 20:30:40 UTC
Created attachment 1864437 [details]
NetworkPolicy dump for katamari namespace

Comment 29 W. Trevor King 2022-03-07 21:18:42 UTC
Moving this back to POST so I can attach the follow-up https://github.com/openshift/sdn/pull/406

Comment 30 Dan Winship 2022-03-07 21:22:17 UTC
So to reproduce you need:

1. at least one policy that selects all pods for ingress (eg, an ingress-default-deny policy)
2. at least one policy that selects _some_ pods for ingress
3. at least one policy that selects _some_ pods for egress

The buggy result is that the pods selected by the 2nd policy will be isolated for egress as well.

Comment 31 Dan Winship 2022-03-07 21:29:20 UTC
(In reply to Dan Winship from comment #30)
> So to reproduce you need:
> 
> 1. at least one policy that selects all pods for ingress (eg, an
> ingress-default-deny policy)
> 2. at least one policy that selects _some_ pods for ingress
> 3. at least one policy that selects _some_ pods for egress
> 
> The buggy result is that the pods selected by the 2nd policy will be
> isolated for egress as well.

my bad, you don't actually need the first policy

Comment 32 piotr.godowski 2022-03-07 21:56:10 UTC
I see PRs https://github.com/openshift/sdn/pull/406 and https://github.com/openshift/sdn/pull/407 with the refix.
Once merged, can we please ask for the OCP version with the refix and the timeline for it?

Comment 33 zhaozhanqi 2022-03-08 05:59:33 UTC
(In reply to piotr.godowski from comment #32)
> I see PRs https://github.com/openshift/sdn/pull/406 and
> https://github.com/openshift/sdn/pull/407 with the refix.
> Once merged, can we please ask for the OCP version with the refix and the
> timeline for it?

this fixed merged into 4.10.3 version: https://amd64.ocp.releases.ci.openshift.org/releasestream/4-stable/release/4.10.3

I verified this issue with comment in https://bugzilla.redhat.com/show_bug.cgi?id=2060956#c9

Could you have a try with 4.10.3 version?

Comment 35 piotr.godowski 2022-03-08 08:10:59 UTC
Will be tested on our side today.

Comment 36 piotr.godowski 2022-03-08 14:19:34 UTC
The first team  on out side ran tests on 4.10.3 confirming fix working.

Comment 37 piotr.godowski 2022-03-08 17:00:14 UTC
The other team on our side ran entire BVT test scenarios successful on OCP 4.10.3.
We'd like to appreciate the quick turn around and the great collaboration !

Comment 38 Ben Bennett 2022-03-08 20:39:50 UTC
Awesome!  Thank you for your help!

Comment 39 zhaozhanqi 2022-03-09 11:17:40 UTC
Move this to verified

Comment 41 errata-xmlrpc 2022-08-10 10:52:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069