Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1279335

Summary:	PV could not be available when policy is recycle for host path volume
Product:	OpenShift Container Platform	Reporter:	Chao Yang <chaoyang>
Component:	Storage	Assignee:	Mark Turansky <mturansk>
Status:	CLOSED NOTABUG	QA Contact:	Liang Xia <lxia>
Severity:	low	Docs Contact:
Priority:	medium
Version:	3.1.0	CC:	aos-bugs, bchilds, bleanhar, chaoyang, dma, jkrieger, jliggitt, jokerman, lxia, pruan, sdodson, xtian
Target Milestone:	---	Keywords:	Reopened
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	atomic-openshift-3.1.0.4-1.git.8.4cc134c.el7aos	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-02-15 16:14:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Chao Yang 2015-11-09 08:27:21 UTC

Description of problem:
PV could not be available when policy is recycle for host path volume

Version-Release number of selected component (if applicable):
openshift v3.1.0.2
kubernetes v1.1.0-origin-1107-g4c8e6f4

How reproducible:
always

Steps to Reproduce:
1.create a local pv
oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/persistent-volumes/hostpath/pv-rwx-recycle.yaml
2.create a pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: localc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

3.create a pod
oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/persistent-volumes/hostpath/pod.yaml
4.after the pod is running ,delete the pod and pvc

5. check the pv status
[root@openshift-140 ~]# oc get pv
NAME              LABELS       CAPACITY   ACCESSMODES   STATUS      CLAIM                    REASON    AGE
local             type=local   5Gi        RWX           Failed      chao-project/localc                11m



Actual results:
the pv status is failed

Expected results:
the pv status should be available

Additional info:
Nov  9 14:46:37 openshift-140 atomic-openshift-master: I1109 14:46:37.792524    6706 factory.go:220] About to try and schedule pod pv-recycler-hostpath-9ftjv
Nov  9 14:46:37 openshift-140 atomic-openshift-master: I1109 14:46:37.793056    6706 factory.go:330] Attempting to bind pv-recycler-hostpath-9ftjv to openshift-154.lab.eng.nay.redhat.com
Nov  9 14:46:39 openshift-140 docker: time="2015-11-09T14:46:39.547111700+08:00" level=info msg="GET /version"
Nov  9 14:46:40 openshift-140 atomic-openshift-master: E1109 14:46:40.732240    6706 persistentvolume_recycler_controller.go:143] PersistentVolume[local] failed recycling: Pod failed, pod.Status.Message unknown.

Comment 1 Mark Turansky 2015-11-10 17:09:35 UTC

Recycling with UID:GID will be fixed by https://github.com/openshift/origin/pull/5792

Comment 2 Mark Turansky 2015-11-10 18:27:35 UTC

*** Bug 1279683 has been marked as a duplicate of this bug. ***

Comment 3 Jordan Liggitt 2015-11-12 06:19:40 UTC

https://github.com/openshift/origin/pull/5792 is superceded by https://github.com/openshift/origin/pull/5847 

ForkAMI available at https://ci.openshift.redhat.com/jenkins/job/fork_ami/132/

Comment 4 Jordan Liggitt 2015-11-12 21:43:32 UTC

Fixed in origin in https://github.com/openshift/origin/pull/5847

Comment 5 Chao Yang 2015-11-13 10:15:40 UTC

Re-test this on the load devenv-rhel7_2695, this issue is still exist
[root@ip-172-18-6-161 /]# oc get pv
NAME            LABELS         CAPACITY   ACCESSMODES   STATUS     CLAIM                     REASON    AGE
local           type=local     5Gi        RWX           Failed     default/localc                      8m

load version is
oc v1.0.8-40-g42ad235
kubernetes v1.1.0-origin-1107-g4c8e6f4

Comment 6 Mark Turansky 2015-11-13 14:42:22 UTC

can you look at the docker container that performed the recycler operation and show the logs?

"docker ps -a" and near the top -- after the failed recycling run -- will be the container that ran the recycler.  "docker logs <id>" of the container.

Comment 7 Mark Turansky 2015-11-13 19:45:50 UTC

I see this is a HostPath PV.  HostPath is not an officially supported PersistentVolume.

Comment 8 Chao Yang 2015-11-16 05:16:18 UTC

I re-test on the devenv-rhel7_2712, this time the recycle pv is become available.
[root@ip-172-18-15-144 ~]# oc get pv
NAME      LABELS       CAPACITY   ACCESSMODES   STATUS      CLAIM     REASON    AGE
local     type=local   5Gi        RWX           Available                       2m

[root@ip-172-18-15-144 ~]# docker logs 5ea09fa29460
Scrub OK

Comment 9 Chao Yang 2015-11-16 08:17:13 UTC

I test this on the ose env, it still exist.
[root@openshift-140 ~]# oc version
oc v3.1.0.4-9-g72d3991
kubernetes v1.1.0-origin-1107-g4c8e6f4

from the /var/log/message 
Nov 16 15:06:11 openshift-149 atomic-openshift-node: I1116 15:06:11.008370    5090 kubelet.go:2115] SyncLoop (REMOVE, "api"): "pv-recycler-hostpath-yhq6k_default"
Nov 16 15:06:11 openshift-149 atomic-openshift-node: I1116 15:06:11.008458    5090 kubelet.go:1926] Killing unwanted pod "pv-recycler-hostpath-yhq6k"

but there is no output for docker logs

Comment 10 Mark Turansky 2015-11-16 15:07:03 UTC

HostPath PV is not officially supported because there is no guarantee your pod ends up on the correct node with the HostPath.  Are you running in a multi-node cluster?

I tried HostPath PV and encountered permission issues.  This might be affecting you, too.

It would be interesting to figure out why HostPath is or isn't working for this BZ, but I think the priority is low because HostPath is not a supported PV.  There are no docs for it and we do not tell anyone to use it as a PV.

Comment 11 Chao Yang 2015-11-17 02:29:59 UTC

yes, I test it on multi-node cluster env.

Comment 18 DeShuai Ma 2015-12-02 10:12:34 UTC

For this bug, if we disable selinux, the recycler can scrub hostpath dir successfully. when enable selinux, it always failed with error like:
Scrub failed: /scrub (drwxrwxrwx), reading dir names as 1000: open /scrub: permission denied

Comment 19 Bradley Childs 2016-01-26 16:02:49 UTC

Closing per comments above.  Please create feature request if required.

Comment 20 Chao Yang 2016-01-27 07:02:01 UTC

(In reply to Bradley Childs from comment #19)
> Closing per comments above.  Please create feature request if required.
It should be a bug and will fix in the UpcomingRelease (see the keywords).
So I re-opened it。

Comment 21 Mark Turansky 2016-01-29 13:24:10 UTC

@chaoyang This is not a bug.  Perhaps the "UpcomingRelease" keyword should be removed.  We do not support HostPath as a PV and there is no supported local storage at this time.