Bug 2078945
| Summary: | Ensure only one apiserver-watcher process is active on a node. | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Luis Sanchez <sanchezl> |
| Component: | kube-apiserver | Assignee: | Luis Sanchez <sanchezl> |
| Status: | CLOSED ERRATA | QA Contact: | Rahul Gangwar <rgangwar> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.9 | CC: | aos-bugs, mfojtik, rgangwar, xxia |
| Target Milestone: | --- | ||
| Target Release: | 4.11.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-08-10 11:08:24 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1948551, 2079097 | ||
|
Description
Luis Sanchez
2022-04-26 14:38:34 UTC
Checking gcp cluster which have not fix. oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.nightly-2022-04-26-181148 True False 64m Cluster version is 4.11.0-0.nightly-2022-04-26-181148 rahulgangwar@rgangwar-mac openshift-tests-private % oc debug node/geliu11283-jtrm5-master-0.c.openshift-qe.internal Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), hostPath volumes (volume "host"), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") Starting pod/geliu11283-jtrm5-master-0copenshift-qeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.0.4 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host Not see any apiserver_watcher lock file. sh-4.4# lslocks COMMAND PID TYPE SIZE MODE M START END PATH (unknown) 46254 FLOCK WRITE 0 0 0 /var/lib/kubelet/pods/370d1983-9cbc-416f-94a1-6475bee6c535/volume-subpaths/etc/tuned/5... ovsdb-server 1230 POSIX 5B WRITE 0 0 0 /run/openvswitch/ovsdb-server.pid ovsdb-server 1230 POSIX 0B WRITE 0 0 0 /etc/openvswitch/.conf.db.~lock~ ovs-vswitchd 1301 POSIX 5B WRITE 0 0 0 /run/openvswitch/ovs-vswitchd.pid (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/370d1983-9cbc-416f-94a1-6475bee6c535/volume-subpaths/etc/tuned/5... (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/370d1983-9cbc-416f-94a1-6475bee6c535/volume-subpaths/etc/tuned/5... etcd 52787 FLOCK 68.4M WRITE 0 0 0 /var/lib/etcd/member/snap/db (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/370d1983-9cbc-416f-94a1-6475bee6c535/volume-subpaths/etc/tuned/5... rpcbind 1466 FLOCK 0B WRITE 0 0 0 /run/rpcbind/rpcbind.lock sssd_nss 1193 POSIX 8.8M WRITE 0 0 0 /var/lib/sss/mc/passwd sssd_nss 1193 POSIX 6.6M WRITE 0 0 0 /var/lib/sss/mc/group sssd_nss 1193 POSIX 11M WRITE 0 0 0 /var/lib/sss/mc/initgroups (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/370d1983-9cbc-416f-94a1-6475bee6c535/volume-subpaths/etc/tuned/5... (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/370d1983-9cbc-416f-94a1-6475bee6c535/volume-subpaths/etc/tuned/5... Checking gcp cluster which have fix. oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.nightly-2022-04-27-234931 True False 47m Cluster version is 4.11.0-0.nightly-2022-04-27-234931 oc debug node/rgangwar-28de4-2mqxn-master-0.c.openshift-qe.internal Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), hostPath volumes (volume "host"), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") Starting pod/rgangwar-28de4-2mqxn-master-0copenshift-qeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.0.5 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host Sees apiserver_watcher lock file. sh-4.4# lslocks COMMAND PID TYPE SIZE MODE M START END PATH ovsdb-server 1238 POSIX 5B WRITE 0 0 0 /run/openvswitch/ovsdb-server.pid ovsdb-server 1238 POSIX 0B WRITE 0 0 0 /etc/openvswitch/.conf.db.~lock~ (unknown) 56912 FLOCK WRITE 0 0 0 /var/lib/kubelet/pods/b5afe8d2-080a-4906-ac12-32711205f4c7/volume-subpaths/etc/tuned/5... (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/b5afe8d2-080a-4906-ac12-32711205f4c7/volume-subpaths/etc/tuned/5... (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/b5afe8d2-080a-4906-ac12-32711205f4c7/volume-subpaths/etc/tuned/5... (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/b5afe8d2-080a-4906-ac12-32711205f4c7/volume-subpaths/etc/tuned/5... sssd_nss 1186 POSIX 6.6M WRITE 0 0 0 /var/lib/sss/mc/group sssd_nss 1186 POSIX 11M WRITE 0 0 0 /var/lib/sss/mc/initgroups etcd 62495 FLOCK 103M WRITE 0 0 0 /var/lib/etcd/member/snap/db (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/b5afe8d2-080a-4906-ac12-32711205f4c7/volume-subpaths/etc/tuned/5... sssd_nss 1186 POSIX 8.8M WRITE 0 0 0 /var/lib/sss/mc/passwd ovs-vswitchd 1309 POSIX 5B WRITE 0 0 0 /run/openvswitch/ovs-vswitchd.pid flock 1679 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock rpcbind 1475 FLOCK 0B WRITE 0 0 0 /run/rpcbind/rpcbind.lock (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/b5afe8d2-080a-4906-ac12-32711205f4c7/volume-subpaths/etc/tuned/5... Checking azure cluster which have fix oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.nightly-2022-04-27-234931 True False 121m Cluster version is 4.11.0-0.nightly-2022-04-27-234931 oc debug node/rgangwar-28de5-b7nwk-master-0 Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), hostPath volumes (volume "host"), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") Starting pod/rgangwar-28de5-b7nwk-master-0-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.0.7 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host Sees apiserver_watcher lock file. sh-4.4# lslocks COMMAND PID TYPE SIZE MODE M START END PATH sssd_nss 1368 POSIX 8.8M WRITE 0 0 0 /var/lib/sss/mc/passwd sssd_nss 1368 POSIX 6.6M WRITE 0 0 0 /var/lib/sss/mc/group sssd_nss 1368 POSIX 11M WRITE 0 0 0 /var/lib/sss/mc/initgroups (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/62ba4619-99f5-4acd-baf6-6c7557296f21/volume-subpaths/etc/tuned/5... (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/62ba4619-99f5-4acd-baf6-6c7557296f21/volume-subpaths/etc/tuned/5... (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/62ba4619-99f5-4acd-baf6-6c7557296f21/volume-subpaths/etc/tuned/5... ovsdb-server 1382 POSIX 5B WRITE 0 0 0 /run/openvswitch/ovsdb-server.pid ovsdb-server 1382 POSIX 0B WRITE 0 0 0 /etc/openvswitch/.conf.db.~lock~ rpcbind 1618 FLOCK 0B WRITE 0 0 0 /run/rpcbind/rpcbind.lock (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/62ba4619-99f5-4acd-baf6-6c7557296f21/volume-subpaths/etc/tuned/5... flock 1774 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/62ba4619-99f5-4acd-baf6-6c7557296f21/volume-subpaths/etc/tuned/5... ovs-vswitchd 1464 POSIX 5B WRITE 0 0 0 /run/openvswitch/ovs-vswitchd.pid (unknown) 42331 FLOCK WRITE 0 0 0 /var/lib/kubelet/pods/62ba4619-99f5-4acd-baf6-6c7557296f21/volume-subpaths/etc/tuned/5... etcd 37274 FLOCK 69.5M WRITE 0 0 0 /var/lib/etcd/member/snap/db (undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/62ba4619-99f5-4acd-baf6-6c7557296f21/volume-subpaths/etc/tuned/5... Checking azure cluster which have not fix.
oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.0-0.nightly-2022-04-26-181148 True False 56m Cluster version is 4.11.0-0.nightly-2022-04-26-181148
oc debug node/rgangwar-28de9-djckn-master-0
Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), hostPath volumes (volume "host"), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Starting pod/rgangwar-28de9-djckn-master-0-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.0.7
If you don't see a command prompt, try pressing enter.
Not see any apiserver_watcher lock file.
sh-4.4# chroot /host
sh-4.4# lslocks
COMMAND PID TYPE SIZE MODE M START END PATH
rpcbind 1627 FLOCK 0B WRITE 0 0 0 /run/rpcbind/rpcbind.lock
etcd 42095 FLOCK 109.6M WRITE 0 0 0 /var/lib/etcd/member/snap/db
(undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/59f7847c-acef-4e62-9dd0-5f7ca91b708f/volume-subpaths/etc/tuned/5...
(undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/59f7847c-acef-4e62-9dd0-5f7ca91b708f/volume-subpaths/etc/tuned/5...
ovs-vswitchd 1473 POSIX 5B WRITE 0 0 0 /run/openvswitch/ovs-vswitchd.pid
sssd_nss 1378 POSIX 8.8M WRITE 0 0 0 /var/lib/sss/mc/passwd
sssd_nss 1378 POSIX 6.6M WRITE 0 0 0 /var/lib/sss/mc/group
sssd_nss 1378 POSIX 11M WRITE 0 0 0 /var/lib/sss/mc/initgroups
(unknown) 33719 FLOCK WRITE 0 0 0 /var/lib/kubelet/pods/59f7847c-acef-4e62-9dd0-5f7ca91b708f/volume-subpaths/etc/tuned/5...
(undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/59f7847c-acef-4e62-9dd0-5f7ca91b708f/volume-subpaths/etc/tuned/5...
(undefined) -1 OFDLCK WRITE 0 0 0 /var/lib/kubelet/pods/59f7847c-acef-4e62-9dd0-5f7ca91b708f/volume-subpaths/etc/tuned/5...
ovsdb-server 1392 POSIX 5B WRITE 0 0 0 /run/openvswitch/ovsdb-server.pid
ovsdb-server 1392 POSIX 0B WRITE 0 0 0 /etc/openvswitch/.conf.db.~lock~
Same for Alibaba cloud.
for i in `oc get node|grep -i master|awk '{print $1}'`; do oc debug node/$i -- chroot /host bash -c "lslocks|grep apiserver-watcher";done
Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), hostPath volumes (volume "host"), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Starting pod/rgangwar-28de6-mljxl-master-0-debug ...
To use host binaries, run `chroot /host`
flock 1547 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock
Removing debug pod ...
Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), hostPath volumes (volume "host"), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Starting pod/rgangwar-28de6-mljxl-master-1-debug ...
To use host binaries, run `chroot /host`
flock 1551 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock
Removing debug pod ...
Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), hostPath volumes (volume "host"), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Starting pod/rgangwar-28de6-mljxl-master-2-debug ...
To use host binaries, run `chroot /host`
flock 1559 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock
Removing debug pod ...
Checking for all master nodes on gcp. oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.nightly-2022-04-27-234931 True False 121m Cluster version is 4.11.0-0.nightly-2022-04-27-234931 rgangwar-28de4-2mqxn-master-0.c.openshift-qe.internal flock 1679 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock rgangwar-28de4-2mqxn-master-1.c.openshift-qe.internal flock 1689 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock rgangwar-28de4-2mqxn-master-2.c.openshift-qe.internal flock 1684 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock Checking for all master nodes on azure. rgangwar-28de5-b7nwk-master-0 flock 1774 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock rgangwar-28de5-b7nwk-master-1 flock 1772 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock rgangwar-28de5-b7nwk-master-2 flock 1775 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock Checking for all master nodes on Alibaba. rgangwar-28de6-mljxl-master-0 flock 1547 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock rgangwar-28de6-mljxl-master-1 flock 1551 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock rgangwar-28de6-mljxl-master-2 flock 1559 FLOCK 0B WRITE 0 0 0 /rootfs/run/cloud-routes/apiserver-watcher.lock Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069 |