Bug 1477298 - spurious FailedSync and FailedMount events 2 minutes after pod terminates
Summary: spurious FailedSync and FailedMount events 2 minutes after pod terminates
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 3.6.z
Assignee: Seth Jennings
QA Contact: Xiaoli Tian
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-01 16:42 UTC by Seth Jennings
Modified: 2019-07-22 20:05 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-22 20:05:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Seth Jennings 2017-08-01 16:42:05 UTC
Description of problem:
Running hack/local-up-cluster.sh with `KEEP_TERMINATED_POD_VOLUMES=false`, start a busybox pod that terminates after 10s and does not restart:

```
apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  containers:
  - name: busybox
    image: busybox
    command:
    - sleep
    - "10"
  terminationGracePeriodSeconds: 0
  restartPolicy: Never
```

In ~10s, the pod goes to Complete.  2 minutes later two events occur:

```
2017-07-26 16:32:54 -0500 CDT   2017-07-26 16:32:54 -0500 CDT   1         busybox   Pod                 Warning   FailedMount   kubelet, 127.0.0.1   Unable to mount volumes for pod "busybox_default(aaa4c2ce-7249-11e7-8f15-7085c20cf2ab)": timeout expired waiting for volumes to attach/mount for pod "default"/"busybox". list of unattached/unmounted volumes=[default-token-3dgs5]
2017-07-26 16:32:54 -0500 CDT   2017-07-26 16:32:54 -0500 CDT   1         busybox   Pod                 Warning   FailedSync   kubelet, 127.0.0.1   Error syncing pod
```

The occurs because a `syncPod()` is called from a pod worker after the volumes have already been unmounted, because the pod is terminated, and `kl.volumeManager.WaitForAttachAndMount(pod)` hangs for 2 minutes then times out as the volume reconciler will not remount these volumes.


Version-Release number of selected component (if applicable):
3.6.0

How reproducible:
Always

Steps to Reproduce:
1. see above
2.
3.

Actual results:
2 spurious events 2 minutes after pod termination

Expected results:
No spurious events from terminated pods

Additional info:

This is already fixed in kube upstream:
https://github.com/kubernetes/kubernetes/pull/49724

And origin master:
https://github.com/openshift/origin/pull/15533

This bug is track a backport to OCP 3.6.z

Comment 1 Seth Jennings 2017-08-01 16:52:19 UTC
OSE PR:
https://github.com/openshift/ose/pull/821

Comment 3 Seth Jennings 2019-07-22 20:05:33 UTC
This has since been fixed.


Note You need to log in before you can comment on or make changes to this bug.