Bug 1477298
| Summary: | spurious FailedSync and FailedMount events 2 minutes after pod terminates | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Seth Jennings <sjenning> |
| Component: | Node | Assignee: | Seth Jennings <sjenning> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Xiaoli Tian <xtian> |
| Severity: | low | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.6.0 | CC: | aos-bugs, eparis, jokerman, mmccomas |
| Target Milestone: | --- | ||
| Target Release: | 3.6.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-07-22 20:05:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This has since been fixed. |
Description of problem: Running hack/local-up-cluster.sh with `KEEP_TERMINATED_POD_VOLUMES=false`, start a busybox pod that terminates after 10s and does not restart: ``` apiVersion: v1 kind: Pod metadata: name: busybox spec: containers: - name: busybox image: busybox command: - sleep - "10" terminationGracePeriodSeconds: 0 restartPolicy: Never ``` In ~10s, the pod goes to Complete. 2 minutes later two events occur: ``` 2017-07-26 16:32:54 -0500 CDT 2017-07-26 16:32:54 -0500 CDT 1 busybox Pod Warning FailedMount kubelet, 127.0.0.1 Unable to mount volumes for pod "busybox_default(aaa4c2ce-7249-11e7-8f15-7085c20cf2ab)": timeout expired waiting for volumes to attach/mount for pod "default"/"busybox". list of unattached/unmounted volumes=[default-token-3dgs5] 2017-07-26 16:32:54 -0500 CDT 2017-07-26 16:32:54 -0500 CDT 1 busybox Pod Warning FailedSync kubelet, 127.0.0.1 Error syncing pod ``` The occurs because a `syncPod()` is called from a pod worker after the volumes have already been unmounted, because the pod is terminated, and `kl.volumeManager.WaitForAttachAndMount(pod)` hangs for 2 minutes then times out as the volume reconciler will not remount these volumes. Version-Release number of selected component (if applicable): 3.6.0 How reproducible: Always Steps to Reproduce: 1. see above 2. 3. Actual results: 2 spurious events 2 minutes after pod termination Expected results: No spurious events from terminated pods Additional info: This is already fixed in kube upstream: https://github.com/kubernetes/kubernetes/pull/49724 And origin master: https://github.com/openshift/origin/pull/15533 This bug is track a backport to OCP 3.6.z