Bug 1461709 - Pod connectivity is broken after upgrade 3.5 to 3.6 with openshift-ovs-networkpolicy plugin
Pod connectivity is broken after upgrade 3.5 to 3.6 with openshift-ovs-networ...
Status: CLOSED WORKSFORME
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking (Show other bugs)
3.6.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Dan Winship
Meng Bo
: Reopened, UpcomingRelease
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-15 05:06 EDT by Yan Du
Modified: 2017-07-11 16:31 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-11 16:31:03 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
openflow (18.29 KB, text/plain)
2017-06-15 05:08 EDT, Yan Du
no flags Details

  None (edit)
Description Yan Du 2017-06-15 05:06:42 EDT
Description of problem:
Pod connectivity is broken after upgrade 3.5 to 3.6 with openshift-ovs-networkpolicy plugin


Version-Release number of selected component (if applicable):
upgrade from
openshift v3.5.5.26 
kubernetes v1.5.2+43a9be4 

to 
openshift v3.6.106
kubernetes v1.6.1+5115d708d7


How reproducible:
Always

Steps to Reproduce:
1. Setup an OCP 3.5 env with openshift-ovs-networkpolicy plugin
2. Create some pods in some project (pod could be connect each other)
3. Upgrade the env to OCP 3.6
4. Check the pod connectivity
5. Create some new pod in a new project
6. Check the pod connectivity again


Actual results:
pods can not connect to each other 

# oc get pod -o wide
NAME            READY     STATUS    RESTARTS   AGE       IP            NODE
test-rc-b3p0s   1/1       Running   0          56m       10.128.0.25   ip-172-18-13-27.ec2.internal
test-rc-mmrzs   1/1       Running   0          56m       10.128.0.24   ip-172-18-13-27.ec2.internal
# oc rsh test-rc-b3p0s
/ $ curl --connect-timeout 5 10.128.0.24:8080
curl: (28) Connection timed out after 5001 milliseconds


Expected results:
Pods could be connect to each other successfully, since the network between projects are flat without specific rules defined


Additional info:
Attached the openflow log
Comment 1 Yan Du 2017-06-15 05:08 EDT
Created attachment 1287964 [details]
openflow
Comment 2 Dan Winship 2017-06-16 16:25:19 EDT
openvswitch gets restarted as part of the upgrade, right? If so, this is a dup of bug 1453113
Comment 3 Ben Bennett 2017-06-19 11:34:23 EDT

*** This bug has been marked as a duplicate of bug 1453113 ***
Comment 4 Yan Du 2017-06-26 05:17:49 EDT
We just finish a round of upgrade testing today with multi-tenant plugin, and we didn't meet this issue. I think it may related the networkpolicy plugin and may not have the same root as #Bug 1453113. So move it to Assign in cases missing it.
Comment 5 Dan Winship 2017-06-26 10:04:06 EDT
After doing an upgrade, when connectivity is broken, can you do:

systemctl status openvswitch
systemctl status atomic-openshift-node

ovs-vsctl show
Comment 6 Yan Du 2017-06-29 03:59:55 EDT
Hi, Dan
I tried to upgrade OCP v3.5.5.28 to v3.6.126.1 with networkpolicy plugin, seems the issue could not be reproduced now, will let you know if I have more clues. Thanks.

Note You need to log in before you can comment on or make changes to this bug.