Bug 1445694 - [trello:yH9l0ouV] openflow won't be updated after changing dnsName in egressnetworkpolicy
Summary: [trello:yH9l0ouV] openflow won't be updated after changing dnsName in egressn...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.6.z
Assignee: Ravi Sankar
QA Contact: Meng Bo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-04-26 10:16 UTC by Yan Du
Modified: 2017-10-25 13:00 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Locking issue with Egress DNS policy Consequence: Syncing egress policies will not happen and this could lead to invalid egress access from the pods. Fix: Fixed egress locking issue Result: Egress policies are synced and works as expected.
Clone Of:
Environment:
Last Closed: 2017-10-25 13:00:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3049 0 normal SHIPPED_LIVE OpenShift Container Platform 3.6, 3.5, and 3.4 bug fix and enhancement update 2017-10-25 15:57:15 UTC

Description Yan Du 2017-04-26 10:16:34 UTC
Description of problem:
openflow won't be updated after changing dnsName in egressnetworkpolicy, and egressnetwork is not working with the new dnsName


Version-Release number of selected component (if applicable):
openshift v3.6.49
kubernetes v1.5.2+43a9be4
openvswitch-2.6.1-10.git20161206.el7fdp.x86_64


How reproducible:
4/5


Steps to Reproduce:
1. Create a project
oc new-project pro1
2. Create a egressnetworkpolicy by admin
oc create -f policy.json
{
    "kind": "EgressNetworkPolicy",
    "apiVersion": "v1",
    "metadata": {
        "name": "policy2"
    },
    "spec": {
        "egress": [
            {
                "type": "Allow",
                "to": {
                    "dnsName": "www.google.com"
                }
            },
            {
                "type": "Deny",
                "to": {
                     "dnsName": "www.cisco.com"
                }
            }
        ]
    }
}

3. Check the openflow on node
 cookie=0x0, duration=19.897s, table=100, n_packets=0, n_bytes=0, priority=2,ip,reg0=0xe27851,nw_dst=172.217.5.228 actions=output:2
 cookie=0x0, duration=19.882s, table=100, n_packets=0, n_bytes=0, priority=1,ip,reg0=0xe27851,nw_dst=23.214.121.109 actions=drop

4. Change the dnsName in the egressnetworkpolicy
eg: 
                "type": "Allow",
                "to": {
                    "dnsName": "www.baidu.com"
                }
            },
            {
                "type": "Deny",
                "to": {
                     "dnsName": "www.yahoo.com"
5. Wait a few minutes (more than 10 minutes) and check the openflow again
6. Delete the egressnetworkpolicy
# oc delete egressnetworkpolicy policy2
egressnetworkpolicy "policy2" deleted
7. Wait more than 30minutes, and check the openflow again

Actual results:
step5: openflow won't be updated after changing dnsName in egressnetworkpolicy
 cookie=0x0, duration=142.554s, table=100, n_packets=0, n_bytes=0, priority=2,ip,reg0=0xe27851,nw_dst=172.217.5.228 actions=output:2
 cookie=0x0, duration=142.539s, table=100, n_packets=0, n_bytes=0, priority=1,ip,reg0=0xe27851,nw_dst=23.214.121.109 actions=drop

step7: openflow still exist even the egressnetworkpolicy have been deleted.
 cookie=0x0, duration=2313.637s, table=100, n_packets=0, n_bytes=0, priority=2,ip,reg0=0xe27851,nw_dst=172.217.5.228 actions=output:2
 cookie=0x0, duration=2313.622s, table=100, n_packets=0, n_bytes=0, priority=1,ip,reg0=0xe27851,nw_dst=23.214.121.109 actions=drop


Expected results:
openflow will be updated according to the dnsName change


Additional info:
1. openflow could be updated after changing cidrSelector in egressnetworkpolicy
2. old openflow could be updated/deleted after restating openvswitch

Comment 1 Ravi Sankar 2017-04-26 22:01:19 UTC
Followed the reproduction steps on dind cluster(1 master, 2 nodes) and it worked for me.
Updating the egress network policy changed the table=100 openflow rules (on step5) and deleting the egress network policy deleted the table=100 openflow rules (on step7).
You mentioned 'restarting openvswitch' in additional info. Is that a typo? I don't see that in the reproduction steps (and I don't expect that to be done without restarting openshift service)

dind is using fedora 25 image and it has openvswitch-2.5.0 and your test setup has openvswitch-2.6.1. I'm hoping that openvswitch to be backward compatible.

If you are able to reproduce this issue, keep the instance running so that I can take a look.

Comment 3 Yan Du 2017-04-27 09:19:23 UTC
btw, I could reproduce the bug with openvswitch openvswitch-2.5.0-22

Comment 4 Ravi Sankar 2017-04-28 22:47:41 UTC
Fixed in https://github.com/openshift/origin/pull/13965

Comment 6 openshift-github-bot 2017-05-03 20:32:30 UTC
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/1047ae4e8ad6986639344a2c4dd56c5ca00239fd
Bug 1445694 - Fix locking in syncEgressDNSPolicyRules()

Deferred unlocking of egress policy lock is in a loop.
Since unlock is not performed in the first iteration, it won't get the lock in the next iteration.
This change ensures unlock is called once it is done with the current iteration.

Comment 7 Yan Du 2017-05-05 06:41:23 UTC
Test on latest OCP 3.6 env, issue have been fixed, openflow could be updated timely according egressnetworkpolicy changing, and after egressnetworkpolicy deleted, openflow could be cleared too.

oc v3.6.64
kubernetes v1.6.1+5115d708d7

Comment 8 Yan Du 2017-05-24 03:23:56 UTC
Test on openshift v3.6.79 kubernetes v1.6.1+5115d708d7, issue have been fixed.

Comment 10 errata-xmlrpc 2017-10-25 13:00:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3049


Note You need to log in before you can comment on or make changes to this bug.