Bug 1908616
| Summary: | sdn pod crashed on rhel worker | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | zhaozhanqi <zzhao> |
| Component: | Networking | Assignee: | Antonio Ojea <aojeagar> |
| Networking sub component: | openshift-sdn | QA Contact: | zhaozhanqi <zzhao> |
| Status: | CLOSED NOTABUG | Docs Contact: | |
| Severity: | urgent | ||
| Priority: | high | CC: | aconstan, huirwang |
| Version: | 4.7 | Keywords: | TestBlocker |
| Target Milestone: | --- | ||
| Target Release: | 4.7.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-12-18 09:44:00 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Yes, Antonio, When setup the cluster the master and rhcos worker are using kube 1.19. but when scale up rhel workers, the rhel worker show 1.20. it's strange and no idea why this happen. and not sure if the version mismatch issue cause the sdn pod crash. I also think it should not be supported for different kube version. will have a try with next accepted build with 1.20 version. setup another cluster with kube 1.20, this issue did not be happened. I will close this bug |
Description of problem: setup cluster with 2 rhel worker. found the sdn pod for rhel worker in crash status Version-Release number of selected component (if applicable): 4.7.0-0.nightly-2020-12-14-165231 How reproducible: Steps to Reproduce: 1. setup cluster and scale up rhel worker (UPI_Bare Metal_Restricted_RHEL 7.9 &RHCOS 4.7_Real Time Kernel ) 2.check the logs of sdn which is in crash 3. Check the ovs logs Actual results: #oc logs sdn-gj5l8 -n openshift-sdn -c sdn I1217 07:45:07.002845 222904 cmd.go:121] Reading proxy configuration from /config/kube-proxy-config.yaml I1217 07:45:07.004513 222904 feature_gate.go:243] feature gates: &{map[EndpointSlice:false EndpointSliceProxying:false]} I1217 07:45:07.004565 222904 feature_gate.go:243] feature gates: &{map[EndpointSlice:false EndpointSliceProxying:false]} I1217 07:45:07.004599 222904 cmd.go:218] Watching config file /config/kube-proxy-config.yaml for changes I1217 07:45:07.004625 222904 cmd.go:218] Watching config file /config/..2020_12_17_03_44_20.784771945/kube-proxy-config.yaml for changes I1217 07:45:07.049525 222904 node.go:152] Initializing SDN node "piqin-1217-fjnpq-rhel-1" (10.0.98.215) of type "redhat/openshift-ovs-networkpolicy" I1217 07:45:07.049829 222904 cmd.go:159] Starting node networking (v0.0.0-alpha.0-258-ge48f05e4) I1217 07:45:07.049846 222904 node.go:340] Starting openshift-sdn network plugin I1217 07:45:07.205071 222904 sdn_controller.go:139] [SDN setup] full SDN setup required (Link not found) I1217 07:45:37.238388 222904 ovs.go:158] Error executing ovs-vsctl: 2020-12-17T07:45:37Z|00002|fatal_signal|WARN|terminating with signal 14 (Alarm clock) I1217 07:46:07.763834 222904 ovs.go:158] Error executing ovs-vsctl: 2020-12-17T07:46:07Z|00002|fatal_signal|WARN|terminating with signal 14 (Alarm clock) I1217 07:46:08.278434 222904 ovs.go:158] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused) I1217 07:46:08.784885 222904 ovs.go:158] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused) I1217 07:46:09.416273 222904 ovs.go:158] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused) I1217 07:46:10.204059 222904 ovs.go:158] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused) I1217 07:46:11.186794 222904 ovs.go:158] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused) I1217 07:46:12.414438 222904 ovs.go:158] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused) I1217 07:46:13.947346 222904 ovs.go:158] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused) I1217 07:46:15.862405 222904 ovs.go:158] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused) I1217 07:46:18.254166 222904 ovs.go:158] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused) I1217 07:46:21.240631 222904 ovs.go:158] Error executing ovs-ofctl: ovs-ofctl: /var/run/openvswitch/br0.mgmt: failed to open socket (Connection refused) F1217 07:46:21.240681 222904 cmd.go:111] Failed to start sdn: node SDN setup failed: timed out waiting for the condition goroutine 1 [running]: k8s.io/klog/v2.stacks(0xc000134001, 0xc0003da480, 0x7a, 0xbf) k8s.io/klog/v2.0/klog.go:1026 +0xb9 k8s.io/klog/v2.(*loggingT).output(0x2e264c0, 0xc000000003, 0x0, 0x0, 0xc00020f960, 0x2d6c415, 0x6, 0x6f, 0x0) k8s.io/klog/v2.0/klog.go:975 +0x19b k8s.io/klog/v2.(*loggingT).printf(0x2e264c0, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x1eac4c7, 0x17, 0xc000504240, 0x1, ...) k8s.io/klog/v2.0/klog.go:750 +0x191 k8s.io/klog/v2.Fatalf(...) k8s.io/klog/v2.0/klog.go:1502 github.com/openshift/sdn/pkg/openshift-sdn.(*OpenShiftSDN).Run(0xc00020f8f0, 0xc0008b0780, 0x2129940, 0xc000134010, 0xc00010e120) github.com/openshift/sdn/pkg/openshift-sdn/cmd.go:111 +0x6d5 github.com/openshift/sdn/pkg/openshift-sdn.NewOpenShiftSDNCommand.func1.2(0xc000000010, 0x1fb4300) github.com/openshift/sdn/pkg/openshift-sdn/cmd.go:63 +0x4e k8s.io/kubernetes/pkg/util/interrupt.(*Handler).Run(0xc0008ec660, 0xc000907d40, 0x0, 0x0) k8s.io/kubernetes.0-rc.0/pkg/util/interrupt/interrupt.go:103 +0xf2 github.com/openshift/sdn/pkg/openshift-sdn.NewOpenShiftSDNCommand.func1(0xc0008b0780, 0xc0000a4980, 0x0, 0x8) github.com/openshift/sdn/pkg/openshift-sdn/cmd.go:62 +0x147 github.com/spf13/cobra.(*Command).execute(0xc0008b0780, 0xc000138010, 0x8, 0x8, 0xc0008b0780, 0xc000138010) github.com/spf13/cobra.1/command.go:766 +0x2c2 github.com/spf13/cobra.(*Command).ExecuteC(0xc0008b0780, 0xd, 0x2129940, 0xc000134010) github.com/spf13/cobra.1/command.go:850 +0x30b github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/cobra.1/command.go:800 main.main() github.com/openshift/sdn/cmd/openshift-sdn/openshift-sdn.go:28 +0x185 goroutine 19 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0x2e264c0) k8s.io/klog/v2.0/klog.go:1169 +0x8b created by k8s.io/klog/v2.init.0 k8s.io/klog/v2.0/klog.go:417 +0xdf goroutine 20 [chan receive]: k8s.io/klog.(*loggingT).flushDaemon(0x2e262e0) k8s.io/klog.0/klog.go:1010 +0x8b created by k8s.io/klog.init.0 k8s.io/klog.0/klog.go:411 +0xd8 goroutine 7 [select]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x1fb32a8, 0x2128c60, 0xc000670060, 0xc00010e601, 0xc00010e0c0) k8s.io/apimachinery.6/pkg/util/wait/wait.go:167 +0x149 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x1fb32a8, 0x12a05f200, 0x0, 0x7d1001, 0xc00010e0c0) k8s.io/apimachinery.6/pkg/util/wait/wait.go:133 +0x98 k8s.io/apimachinery/pkg/util/wait.Until(...) k8s.io/apimachinery.6/pkg/util/wait/wait.go:90 k8s.io/apimachinery/pkg/util/wait.Forever(0x1fb32a8, 0x12a05f200) k8s.io/apimachinery.6/pkg/util/wait/wait.go:81 +0x4f created by k8s.io/component-base/logs.InitLogs k8s.io/component-base.6/logs/logs.go:58 +0x8a goroutine 9 [syscall, 1 minutes]: os/signal.signal_recv(0x0) runtime/sigqueue.go:147 +0x9d os/signal.loop() os/signal/signal_unix.go:23 +0x25 created by os/signal.Notify.func1.1 2. Check the ovs pod is running as container other than systemd for rhel worker openvswitch is running in container Starting ovsdb-server. Configuring Open vSwitch system IDs. Enabling remote OVSDB managers. Starting ovs-vswitchd. Enabling remote OVSDB managers. 2020-12-17 03:44:25 info: Loading previous flows ... 2020-12-17 03:44:25 info: Remove other config ... 2020-12-17 03:44:25 info: Removed other config ... 2020-12-17T03:44:25.746Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log 2020-12-17T03:44:25.757Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.13.2 2020-12-17T03:44:25.917Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2020-12-17T03:44:25.928Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on NUMA node 0 2020-12-17T03:44:25.928Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU cores 2020-12-17T03:44:25.928Z|00004|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2020-12-17T03:44:25.928Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2020-12-17T03:44:25.930Z|00006|dpdk|INFO|DPDK Disabled - Use other_config:dpdk-init to enable 2020-12-17T03:44:25.934Z|00007|dpif_netlink|INFO|The kernel module does not support meters. 2020-12-17T03:44:25.939Z|00008|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.13.2 2020-12-17T03:44:35.764Z|00003|memory|INFO|3492 kB peak resident set size after 10.0 seconds 2020-12-17T03:44:35.764Z|00004|memory|INFO|cells:39 monitors:2 sessions:1 2020-12-17T03:44:39.693Z|00009|memory|INFO|35692 kB peak resident set size after 13.8 seconds 2020-12-17T03:44:39.704Z|00010|ofproto_dpif|INFO|system@ovs-system: Datapath supports recirculation 2020-12-17T03:44:39.704Z|00011|ofproto_dpif|INFO|system@ovs-system: VLAN header stack length probed as 2 2020-12-17T03:44:39.704Z|00012|ofproto_dpif|INFO|system@ovs-system: MPLS label stack length probed as 1 2020-12-17T03:44:39.704Z|00013|ofproto_dpif|INFO|system@ovs-system: Datapath supports truncate action 2020-12-17T03:44:39.704Z|00014|ofproto_dpif|INFO|system@ovs-system: Datapath supports unique flow ids 2020-12-17T03:44:39.704Z|00015|ofproto_dpif|INFO|system@ovs-system: Datapath does not support clone action 2020-12-17T03:44:39.704Z|00016|ofproto_dpif|INFO|system@ovs-system: Max sample nesting level probed as 10 2020-12-17T03:44:39.704Z|00017|ofproto_dpif|INFO|system@ovs-system: Datapath supports eventmask in conntrack action 2020-12-17T03:44:39.704Z|00018|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_clear action 2020-12-17T03:44:39.704Z|00019|ofproto_dpif|INFO|system@ovs-system: Max dp_hash algorithm probed to be 0 2020-12-17T03:44:39.704Z|00020|ofproto_dpif|INFO|system@ovs-system: Datapath does not support check_pkt_len action 2020-12-17T03:44:39.704Z|00021|ofproto_dpif|INFO|system@ovs-system: Datapath does not support timeout policy in conntrack action 2020-12-17T03:44:39.704Z|00022|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_state 2020-12-17T03:44:39.704Z|00023|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_zone 2020-12-17T03:44:39.704Z|00024|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_mark 2020-12-17T03:44:39.704Z|00025|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_label 2020-12-17T03:44:39.704Z|00026|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_state_nat 2020-12-17T03:44:39.704Z|00027|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_orig_tuple 2020-12-17T03:44:39.704Z|00028|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_orig_tuple6 2020-12-17T03:44:39.704Z|00029|ofproto_dpif|INFO|system@ovs-system: Datapath does not support IPv6 ND Extensions 2020-12-17T03:44:39.716Z|00030|dpif_netlink|INFO|dpif_netlink_meter_transact OVS_METER_CMD_SET failed 2020-12-17T03:44:39.716Z|00031|dpif_netlink|INFO|dpif_netlink_meter_transact OVS_METER_CMD_SET failed 2020-12-17T03:44:39.716Z|00032|dpif_netlink|INFO|dpif_netlink_meter_transact get failed 2020-12-17T03:44:39.716Z|00033|dpif_netlink|INFO|The kernel module has a broken meter implementation. 2020-12-17T03:44:39.720Z|00034|bridge|INFO|bridge br0: added interface br0 on port 65534 2020-12-17T03:44:39.720Z|00035|bridge|INFO|bridge br0: using datapath ID 000086a3bd75ec44 2020-12-17T03:44:39.721Z|00036|connmgr|INFO|br0: added service controller "punix:/var/run/openvswitch/br0.mgmt" 2020-12-17T03:44:39.776Z|00037|bridge|INFO|bridge br0: added interface vxlan0 on port 1 2020-12-17T03:44:39.810Z|00038|netdev|WARN|failed to set MTU for network device tun0: No such device Check the logs of ovs-vswitched sh-4.2# journalctl -u ovs-vswitchd.service -- Logs begin at Wed 2020-12-16 22:43:49 EST, end at Thu 2020-12-17 02:37:04 EST. -- Dec 16 22:43:54 piqin-1217-fjnpq-rhel-1 systemd[1]: Starting Open vSwitch Forwarding Unit... Dec 16 22:43:54 piqin-1217-fjnpq-rhel-1 ovs-ctl[869]: Inserting openvswitch module [ OK ] Dec 16 22:43:54 piqin-1217-fjnpq-rhel-1 ovs-ctl[869]: Starting ovs-vswitchd [ OK ] Dec 16 22:43:54 piqin-1217-fjnpq-rhel-1 ovs-ctl[869]: Enabling remote OVSDB managers [ OK ] Dec 16 22:43:54 piqin-1217-fjnpq-rhel-1 ovs-vsctl[926]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . external-ids:hostname=piqin-1217-fjnpq-rhel-1 Dec 16 22:43:54 piqin-1217-fjnpq-rhel-1 systemd[1]: Started Open vSwitch Forwarding Unit. Dec 16 22:44:32 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00036|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: No such file or directory Dec 16 22:44:32 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00038|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: No such file or directory Dec 16 22:44:32 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00039|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: No such file or directory Dec 16 22:44:32 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00041|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: No such file or directory Dec 16 22:44:32 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00043|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: No such file or directory Dec 16 22:44:32 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00044|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: No such file or directory Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00050|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: No such file or directory Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00052|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: No such file or directory Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00053|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: No such file or directory Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00055|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: No such file or directory Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00057|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: No such file or directory Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00058|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: No such file or directory Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00060|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: No such file or directory Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00062|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: No such file or directory Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00063|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: No such file or directory Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00066|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00069|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00070|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00073|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00076|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00077|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00080|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00083|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00084|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00087|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00090|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00091|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00094|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00097|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00098|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00101|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00104|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00105|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00108|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00111|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00112|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00115|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00118|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00119|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00122|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00125|stream_unix|ERR|/var/run/openvswitch/br0.snoop: binding failed: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00126|connmgr|ERR|failed to listen on punix:/var/run/openvswitch/br0.snoop: Permission denied Dec 16 22:44:45 piqin-1217-fjnpq-rhel-1 ovs-vswitchd[913]: ovs|00129|stream_unix|ERR|/var/run/openvswitch/br0.mgmt: binding failed: Permiss Expected results: Additional info: master and other rhcos worker is using kuber1.19. However it is 1.20 for rhel worker oc get node NAME STATUS ROLES AGE VERSION piqin-1217-fjnpq-compute-0 Ready worker 5h29m v1.19.2+e386040 piqin-1217-fjnpq-compute-1 Ready worker 5h28m v1.19.2+e386040 piqin-1217-fjnpq-control-plane-0 Ready master 5h40m v1.19.2+e386040 piqin-1217-fjnpq-control-plane-1 Ready master 5h40m v1.19.2+e386040 piqin-1217-fjnpq-control-plane-2 Ready master 5h40m v1.19.2+e386040 piqin-1217-fjnpq-rhel-0 Ready worker 3h59m v1.20.0+87544c5 piqin-1217-fjnpq-rhel-1 Ready worker 3h59m v1.20.0+87544c5