Bug 1765461
| Summary: | ovs pod with many error "connection dropped (Connection reset by peer)" after upgrade which caused the other pods cannot be running | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | zhaozhanqi <zzhao> |
| Component: | Networking | Assignee: | Alexander Constantinescu <aconstan> |
| Networking sub component: | openshift-sdn | QA Contact: | zhaozhanqi <zzhao> |
| Status: | CLOSED WORKSFORME | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | aconole, aconstan, anusaxen, bbennett, hasha, imm |
| Version: | 4.1.z | Keywords: | Reopened |
| Target Milestone: | --- | ||
| Target Release: | 4.5.0 | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | SDN-CI-IMPACT | ||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-05-08 20:41:42 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
zhaozhanqi
2019-10-25 07:33:25 UTC
Aaron,
I'm confused - can you take a look?
Jumped on the node, set ovsdb and ovs-vswitchd to verbose. ovsdb is complaining about resets:
2019-10-25T10:57:12.124Z|10552|jsonrpc|DBG|unix#33212: send reply, result={"Interface":{"5a3768ef-68b2-4790-be63-44732f22c699":{"initial":{"name":"br0","ofport":65534}},"dc60e281-6c4d-4ed7-8bf3-389b1188a713":{"initial":{"name":"tun0","ofport":2}},"159daa7a-322f-4323-b3ba-02deb1e2e0f0":{"initial":{"name":"vxlan0","ofport":1}}},"Open_vSwitch":{"26838199-2efd-4e9d-91ef-edce5864ad62":{"initial":{"cur_cfg":169}}}}, id=2
2019-10-25T10:57:12.125Z|10553|poll_loop|DBG|wakeup due to 0-ms timeout at unix#33212 (0% CPU usage)
2019-10-25T10:57:12.125Z|10554|jsonrpc|DBG|unix#33212: received request, method="set_db_change_aware", params=[true], id=3
2019-10-25T10:57:12.125Z|10555|jsonrpc|DBG|unix#33212: send reply, result={}, id=3
2019-10-25T10:57:12.125Z|10556|poll_loop|DBG|wakeup due to [POLLIN][POLLHUP] on fd 17 (/var/run/openvswitch/db.sock<->) at ../lib/stream-fd.c:157 (0% CPU usage)
2019-10-25T10:57:12.125Z|10557|poll_loop|DBG|wakeup due to 0-ms timeout at unix#33212 (0% CPU usage)
2019-10-25T10:57:12.125Z|10558|reconnect|DBG|unix#33212: connection closed by peer
2019-10-25T10:57:12.125Z|10559|reconnect|DBG|unix#33212: entering VOID
there is nothing obvious in vswitchd logs at the same timeframe:
2019-10-25T10:57:11.675Z|00761|poll_loop(revalidator14)|DBG|wakeup due to [POLLIN] on fd 30 (FIFO pipe:[38862]) at ../lib/ovs-thread.c:311 (0% CPU usage)
2019-10-25T10:57:12.175Z|03066|poll_loop(revalidator17)|DBG|wakeup due to 501-ms timeout at ../ofproto/ofproto-dpif-upcall.c:982 (0% CPU usage)
2019-10-25T10:57:12.175Z|03067|netlink_socket(revalidator17)|DBG|Dropped 33 log messages in last 1 seconds (most recently, 0 seconds ago) due to excessive rate
2019-10-25T10:57:12.175Z|03068|netlink_socket(revalidator17)|DBG|nl_sock_transact_multiple__ (Success): nl(len:24, type=29(ovs_datapath), flags=9[REQUEST][ECHO], seq=aba9, pid=4648,genl(cmd=3,version=2)
2019-10-25T10:57:12.175Z|03069|dpif(revalidator17)|DBG|Dropped 4 log messages in last 0 seconds (most recently, 0 seconds ago) due to excessive rate
2019-10-25T10:57:12.175Z|03070|dpif(revalidator17)|DBG|system@ovs-system: get_stats success
2019-10-25T10:57:12.175Z|03071|dpif(revalidator17)|DBG|system@ovs-system: flow_dump ufid:960dcbec-8ff1-427e-91be-1e07137c347b <empty>, packets:228210, bytes:28319721, used:0.769s, flags:SFPR.
2019-10-25T10:57:12.175Z|03072|dpif(revalidator17)|DBG|system@ovs-system: flow_dump ufid:19c9dec0-3ce5-4f7f-b520-e32c6d248ba6 <empty>, packets:10610, bytes:2185051, used:2.151s, flags:P
2019-10-25T10:57:12.175Z|03073|dpif(revalidator17)|DBG|system@ovs-system: flow_dump ufid:800c953c-c589-4e78-87cd-32270cd9f0b4 <empty>, packets:163518, bytes:21894562, used:0.416s, flags:SFPR.
2019-10-25T10:57:12.175Z|03074|dpif(revalidator17)|DBG|system@ovs-system: flow_dump ufid:dcd4be9a-0e4a-4425-ae9e-a66e915ca3e4 <empty>, packets:139036, bytes:133298860, used:0.416s, flags:SFP.
2019-10-25T10:57:12.176Z|00762|poll_loop(revalidator14)|DBG|wakeup due to [POLLIN] on fd 30 (FIFO pipe:[38862]) at ../lib/ovs-thread.c:311 (0% CPU usage)
2019-10-25T10:57:12.176Z|03075|poll_loop(revalidator17)|DBG|wakeup due to [POLLIN] on fd 35 (FIFO pipe:[40887]) at ../lib/ovs-thread.c:311 (0% CPU usage)
2019-10-25T10:57:12.176Z|00760|poll_loop|DBG|wakeup due to [POLLIN] on fd 46 (FIFO pipe:[37649]) at ../vswitchd/bridge.c:384 (0% CPU usage)
2019-10-25T10:57:12.176Z|00763|poll_loop(revalidator14)|DBG|wakeup due to [POLLIN] on fd 30 (FIFO pipe:[38862]) at ../lib/ovs-thread.c:311 (0% CPU usage)
2019-10-25T10:57:12.491Z|00761|poll_loop|DBG|wakeup due to 316-ms timeout at ../vswitchd/bridge.c:2828 (0% CPU usage)
2019-10-25T10:57:12.491Z|00762|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock: send request, method="transact", params=["Open_vSwitch",{"lock":"ovs_vswitchd","op":"assert"},{"where":[["_u
uid","==",["uuid","dc60e281-6c4d-4ed7-8bf3-389b1188a713"]]],"row":{"statistics":["map",[["collisions",0],["rx_bytes",323710096],["rx_crc_err",0],["rx_dropped",0],["rx_errors",0],["rx_fra
me_err",0],["rx_over_err",0],["rx_packets",388802],["tx_bytes",53793089],["tx_dropped",0],["tx_errors",0],["tx_packets",425766]]]},"op":"update","table":"Interface"},{"where":[["_uuid","
==",["uuid","159daa7a-322f-4323-b3ba-02deb1e2e0f0"]]],"row":{"statistics":["map",[["rx_bytes",329154739],["rx_packets",388837],["tx_bytes",53790572],["tx_packets",425723]]]},"op":"update
","table":"Interface"}], id=3237
2019-10-25T10:57:12.492Z|00763|poll_loop|DBG|wakeup due to [POLLIN] on fd 15 (<->/var/run/openvswitch/db.sock) at ../lib/stream-fd.c:157 (0% CPU usage)
2019-10-25T10:57:12.492Z|00764|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock: received notification, method="update2", params=[["monid","Open_vSwitch"],{"Interface":{"dc60e281-6c4d-4ed7-8bf3-389b118
8a713":{"modify":{"statistics":["map",[["rx_bytes",323710096],["rx_packets",388802],["tx_bytes",53793089],["tx_packets",425766]]]}},"159daa7a-322f-4323-b3ba-02deb1e2e0f0":{"modify":{"statistics":["m
ap",[["rx_bytes",329154739],["rx_packets",388837],["tx_bytes",53790572],["tx_packets",425723]]]}}}}]
2019-10-25T10:57:12.492Z|00765|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock: received reply, result=[{},{"count":1},{"count":1}], id=3237
this issue not reproduce on today's upgrade testing 4.1.20->4.1.21 @Ben Could you please confirm if we support upgrades between minor versions in 4.1? /Alex Closing this since it seems to be unreproducible now. (And Alex, we do support upgrades between minor versions of 4.1, as long as 4.1 is supported) I do not think it's NOT a bug since it's really happen even it's not always every time. seems in 4.4 we met same issue https://bugzilla.redhat.com/show_bug.cgi?id=1802481 Hi So, I think Zhao is right. This definately seems to be the same as bug 1802481 (judging by it's difficulty to reproduce and description), however there seems to be a bug opened for that problem against 4.1: https://bugzilla.redhat.com/show_bug.cgi?id=1767178 I will keep this one open for the time being, until confirming that it's the same issue. @Zhao, if you manage to reproduce on 4.1 please provide me with a kubeconfig so that I can verify that it's the same problem Thanks again, -Alex sure. I will try to reproduce this issue on 4.1. Hi Zhao Any luck reproducing it? -Alex Hi, Alexander still not reproduce this issue recently. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |