Bug 1281726 - NFS Recycler Fails in containerized Kubernetes
NFS Recycler Fails in containerized Kubernetes
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage (Show other bugs)
3.1.0
Unspecified Unspecified
urgent Severity medium
: ---
: ---
Assigned To: Mark Turansky
Liang Xia
: Regression
Depends On: 1277608
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-13 05:03 EST by Liang Xia
Modified: 2015-12-08 12:03 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1277608
Environment:
Last Closed: 2015-12-08 12:03:24 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Comment 1 Liang Xia 2015-11-13 05:06:44 EST
# openshift version
openshift v1.0.8-40-g42ad235
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2

$ docker inspect openshift/origin:latest
[
{
    "Id": "6f81e7b0d3a0a3ea481286125107a0f15a32bb7701f38eb7ee29dc7133c15a5c",
    "RepoTags": [
        "openshift/origin:latest"
    ],
    "RepoDigests": [],
    "Parent": "1042ad0ba8725fb5e6e5371cddb51885f732be6e88f4aa218a198b7f51278ab6",
    "Comment": "",
    "Created": "2015-11-13T07:48:57.157192063Z",
    "Container": "9739e63f23bc1679e697a0e5368e8501679345798f21a9364bcc60f36c58ac79",
    "ContainerConfig": {
        "Hostname": "7aa5783a47d5",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "ExposedPorts": {
            "53/tcp": {},
            "8443/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "HOME=/root",
            "OPENSHIFT_CONTAINERIZED=true",
            "KUBECONFIG=/var/lib/openshift/openshift.local.config/master/admin.kubeconfig"
        ],
        "Cmd": [
            "/bin/sh",
            "-c",
            "#(nop) ENTRYPOINT \u0026{[\"/usr/bin/openshift\"]}"
        ],
        "Image": "90ffa6a3709dc6c77de52db77f6ccd4b7c5efc6aeb3bb31dfa4838ec696a46cb",
        "Volumes": null,
        "WorkingDir": "/var/lib/openshift",
        "Entrypoint": [
            "/usr/bin/openshift"
        ],
        "OnBuild": [],
        "Labels": {
            "License": "GPLv2",
            "Vendor": "CentOS"
        }
    },
    "DockerVersion": "1.8.2",
    "Author": "",
    "Config": {
        "Hostname": "7aa5783a47d5",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "ExposedPorts": {
            "53/tcp": {},
            "8443/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "HOME=/root",
            "OPENSHIFT_CONTAINERIZED=true",
            "KUBECONFIG=/var/lib/openshift/openshift.local.config/master/admin.kubeconfig"
        ],
        "Cmd": null,
        "Image": "90ffa6a3709dc6c77de52db77f6ccd4b7c5efc6aeb3bb31dfa4838ec696a46cb",
        "Volumes": null,
        "WorkingDir": "/var/lib/openshift",
        "Entrypoint": [
            "/usr/bin/openshift"
        ],
        "OnBuild": [],
        "Labels": {
            "License": "GPLv2",
            "Vendor": "CentOS"
        }
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Size": 0,
    "VirtualSize": 449917458,
    "GraphDriver": {
        "Name": "devicemapper",
        "Data": {
            "DeviceId": "588",
            "DeviceName": "docker-253:0-659024-6f81e7b0d3a0a3ea481286125107a0f15a32bb7701f38eb7ee29dc7133c15a5c",
            "DeviceSize": "10737418240"
        }
    }
}
]
Comment 2 Liang Xia 2015-11-13 05:17:48 EST
# oc describe pv nfs
Name:		nfs
Labels:		<none>
Status:		Failed
Claim:		default/nfsc
Reclaim Policy:	Recycle
Access Modes:	RWO
Capacity:	5Gi
Message:	Recycling error: Pod was active on the node longer than specified deadline
Source:
    Type:	NFS (an NFS mount that lasts the lifetime of a pod)
    Server:	*(Server info hide here)
    Path:	/jhou
    ReadOnly:	false


Mark the bug as regression since we suspect it is caused by https://github.com/openshift/origin/pull/5847
Comment 3 Mark Turansky 2015-11-13 09:41:15 EST
@liang Xia,

What is the timeout on the pod?

when the PV is being recycled, you can view the recycler pod with "oc get pod".  If the pod is timing out, there should be plenty of time to see it in the API and describe it.

The pod *should* have a timeout that is incremented per Gi of size.  We can increase this number if it is too small currently.
Comment 4 Mark Turansky 2015-11-13 15:28:33 EST
what are the permissions on the NFS export?
Comment 5 Liang Xia 2015-11-16 00:24:26 EST
PV can be recycled when NFS exported with 777.
Comment 7 errata-xmlrpc 2015-12-08 12:03:24 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2580

Note You need to log in before you can comment on or make changes to this bug.