+++ This bug was initially created as a clone of Bug #1790440 +++

+++ This bug was initially created as a clone of Bug #1790407 +++

Description of problem:
No new ovs flows add to table 80 after restart sdn pod and create allow-all networkpolicy

Version-Release number of selected component (if applicable):
Versions:

4.4.0-0.nightly-2020-04-07-130324


How reproducible:
Frequently using automation, difficult using manual steps

Steps to Reproduce:

Exact automation steps with timestamps

1. Create a project

[19:13:28] INFO> Shell Commands: oc new-project gpjn8
Now using project "gpjn8" on server "https://api.example.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app ruby~https://github.com/sclorg/ruby-ex.git

to build a new example application in Python. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node




2. Create two pods in this project.


[19:13:29] INFO> Shell Commands: oc create -f verification-tests/features/tierN/testdata/networking/list_for_pods.json
replicationcontroller/test-rc created
service/test-service created

[19:13:30] INFO> Exit Status: 0
[19:13:30] INFO> oc get pods -o wide -l name\=test-pods  -n gpjn8
NAME            READY   STATUS    RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
test-rc-bt7r6   1/1     Running   0          69m   10.131.0.20   worker-westus-9d9nh   <none>           <none>
test-rc-d4mjw   1/1     Running   0          69m   10.129.2.11   worker-westus-74s72   <none>           <none>



3. Create a deny-all networkpolicy in the project.

[19:13:43] INFO> Shell Commands: oc create -f verification-tests/testdata/networking/networkpolicy/defaultdeny-v1-semantic.yaml  -n gpjn8
networkpolicy.networking.k8s.io/default-deny created

[19:13:43] INFO> Exit Status: 0

4. Verify pod traffic is denied
[19:13:44] INFO> Shell Commands: oc exec test-rc-bt7r6   -n gpjn8  -- curl -s --connect-timeout 100 10.129.2.11:8080

STDERR:
command terminated with exit code 28

[19:15:25] INFO> Exit Status: 28


5. Restart SDN pod which pod test-rc-bt7r6 is located, node worker-westus-9d9nh

[19:15:26] INFO> Shell Commands: oc delete pods sdn-w2x27 --wait=false  --namespace=openshift-sdn
pod "sdn-w2x27" deleted

[19:15:26] INFO> Exit Status: 0


6. After new sdn pod running, create a allow-all networkpolicy in the project.

[19:15:39] INFO> Shell Commands: oc create -f verification-tests/features/tierN/testdata/networking/networkpolicy/allow-all.yaml
networkpolicy.networking.k8s.io/allow-all created

[19:15:39] INFO> Exit Status: 0


7.  Try to curl from the pod on the node on which the sdn pod was deleted

[19:15:39] INFO> Shell Commands: oc create -f verification-tests/features/tierN/testdata/networking/networkpolicy/allow-all.yaml
networkpolicy.networking.k8s.io/allow-all created

[19:15:39] INFO> Exit Status: 0

8. Try to curl to the pod on the node on which the sdn pod was deleted


[19:15:40] INFO> Shell Commands: oc exec test-rc-d4mjw   -n gpjn8  -- curl -s --connect-timeout 140 10.131.0.20:8080

STDERR:
command terminated with exit code 7

[19:17:54] INFO> Exit Status: 7



Actual Result:
The curl fails when attempting to connect to the pod on the node on which the sdn pod was deleted



# dump the openflows for the netnamespace for the project
netnamespace=$(printf "%x\n"  $(oc get netnamespaces $(oc get pods  -o jsonpath={.items[0].metadata.namespace}) -o jsonpath={.netid}))

# dump all the matching flows
for n in $(oc get pods -o wide -o jsonpath={.items[*].spec.nodeName})  ; do for f in $(oc get pod -n openshift-sdn -l app=ovs   --field-selector=spec.nodeName=$n  -o jsonpath={.items[0].metadata.name}) ; do echo $f ;  oc -n openshift-sdn exec $f -- ovs-ofctl -O OpenFlow13 dump-flows br0  | grep $netnamespace | tee flows-$netnamespace-$f ; done ; done


flows-node-worker-westus-9d9nh:

cookie=0x0, duration=2831.705s, table=20, n_packets=6, n_bytes=252, priority=100,arp,in_port=21,arp_spa=10.131.0.20,arp_sha=00:00:0a:83:00:14/00:00:ff:ff:ff:ff actions=load:0xc43e71->NXM_NX_REG0[],goto_table:21
cookie=0x0, duration=2831.705s, table=20, n_packets=13, n_bytes=1002, priority=100,ip,in_port=21,nw_src=10.131.0.20 actions=load:0xc43e71->NXM_NX_REG0[],goto_table:21
cookie=0x0, duration=2831.705s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.20 actions=load:0xc43e71->NXM_NX_REG0[],goto_table:30
cookie=0x0, duration=2831.706s, table=70, n_packets=11, n_bytes=924, priority=100,ip,nw_dst=10.131.0.20 actions=load:0xc43e71->NXM_NX_REG1[],load:0x15->NXM_NX_REG2[],goto_table:80



flows-node-worker-westus-74s72

cookie=0x0, duration=2832.078s, table=20, n_packets=6, n_bytes=252, priority=100,arp,in_port=12,arp_spa=10.129.2.11,arp_sha=00:00:0a:81:02:0b/00:00:ff:ff:ff:ff actions=load:0xc43e71->NXM_NX_REG0[],goto_table:21
cookie=0x0, duration=2832.078s, table=20, n_packets=11, n_bytes=924, priority=100,ip,in_port=12,nw_src=10.129.2.11 actions=load:0xc43e71->NXM_NX_REG0[],goto_table:21
cookie=0x0, duration=2832.078s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.129.2.11 actions=load:0xc43e71->NXM_NX_REG0[],goto_table:30
cookie=0x0, duration=2832.078s, table=70, n_packets=13, n_bytes=1002, priority=100,ip,nw_dst=10.129.2.11 actions=load:0xc43e71->NXM_NX_REG1[],load:0xc->NXM_NX_REG2[],goto_table:80
cookie=0x0, duration=2703.262s, table=80, n_packets=3, n_bytes=286, priority=150,reg1=0xc43e71 actions=output:NXM_NX_REG2[]


The failing node is missing the table=80 priority=150 rule actions=output:NXM_NX_REG2[]




Expected results:

The pods should talk to each in that project after add allow-all policy.