Bug 1546156

Summary:	[3.7]Fix bug on kubelet failure to umount mount points.
Product:	OpenShift Container Platform	Reporter:	Jaspreet Kaur <jkaur>
Component:	Node	Assignee:	Seth Jennings <sjenning>
Status:	CLOSED ERRATA	QA Contact:	Wenqi He <wehe>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	3.7.0	CC:	aos-bugs, dcain, dmoessne, eparis, erjones, gchakkar, hchiramm, jcall, jkaur, jokerman, mfojtik, mmccomas
Target Milestone:	---
Target Release:	3.7.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Fixes an issue where a "transport endpoint is not connected" error is encountered when deleted projects with pods using CNS backed PVs.	Story Points:	---
Clone Of:
Clones:	1546159 (view as bug list)		Environment:
Last Closed:	2018-04-05 09:38:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1546159

Description Jaspreet Kaur 2018-02-16 13:04:41 UTC

Description of problem: UnmountVolume.TearDown tries to stat the gluster mount point and errors with "transport endpoint is not connected" due to the endpoint being deleted.

https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/pkg/volume/util/util.go#L84

This blocks the umount later in that function causing the overall volume detach to fail and hang the pod termination.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results: Fails every time.


Expected results: should not have an issue. 


Additional info: Required fix in 3.7 for below bugzilla to be backported 

https://bugzilla.redhat.com/show_bug.cgi?id=1489082

Comment 5 Humble Chirammal 2018-02-21 11:25:09 UTC

I had filed a 3.7 PR and its merged https://github.com/openshift/ose/pull/1068

Comment 6 Seth Jennings 2018-02-21 15:15:15 UTC

Ah ok, I searched for a PR with the upstream description but looks like there were two upstream commits in this one PR and the PR description was that of the other commit.  Hence why I didn't find it.  And it looks like it merged around the same time I was opening my PR :-/

I'll close mine and move this to modified.

Comment 8 Wenqi He 2018-03-01 14:23:45 UTC

Tested on below version:
openshift v3.7.35
kubernetes v1.7.6+a08f5eeb62

[root@host-172-16-120-121 ~]# oc get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS        AGE
mypv      Bound     pvc-44ace2ba-1d5b-11e8-8a10-fa163e91297f   1Gi        RWO           glusterfs-storage   22s
[root@host-172-16-120-121 ~]# oc create -f pod.yaml
pod "busybox-pod" created
[root@host-172-16-120-121 ~]# oc get pods
NAME          READY     STATUS              RESTARTS   AGE
busybox-pod   0/1       ContainerCreating   0          3s
[root@host-172-16-120-121 ~]# oc get pods
NAME          READY     STATUS    RESTARTS   AGE
busybox-pod   1/1       Running   0          1m
[root@host-172-16-120-121 ~]# oc delete project test
project "test" deleted
[root@host-172-16-120-121 ~]# oc get pods
NAME          READY     STATUS    RESTARTS   AGE
busybox-pod   1/1       Running   0          1m

[root@host-172-16-120-121 ~]# oc get project
test                                               Terminating

[root@host-172-16-120-121 ~]# oc get project

Projects has been deleted

Comment 12 errata-xmlrpc 2018-04-05 09:38:31 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0636