Bug 1281726 - NFS Recycler Fails in containerized Kubernetes
Summary: NFS Recycler Fails in containerized Kubernetes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.1.0
Hardware: Unspecified
OS: Unspecified
urgent
medium
Target Milestone: ---
: ---
Assignee: Mark Turansky
QA Contact: Liang Xia
URL:
Whiteboard:
Depends On: 1277608
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-11-13 10:03 UTC by Liang Xia
Modified: 2015-12-08 17:03 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1277608
Environment:
Last Closed: 2015-12-08 17:03:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2580 0 normal SHIPPED_LIVE Red Hat OpenShift Enterprise bug fix update 2015-12-08 22:02:52 UTC

Comment 1 Liang Xia 2015-11-13 10:06:44 UTC
# openshift version
openshift v1.0.8-40-g42ad235
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2

$ docker inspect openshift/origin:latest
[
{
    "Id": "6f81e7b0d3a0a3ea481286125107a0f15a32bb7701f38eb7ee29dc7133c15a5c",
    "RepoTags": [
        "openshift/origin:latest"
    ],
    "RepoDigests": [],
    "Parent": "1042ad0ba8725fb5e6e5371cddb51885f732be6e88f4aa218a198b7f51278ab6",
    "Comment": "",
    "Created": "2015-11-13T07:48:57.157192063Z",
    "Container": "9739e63f23bc1679e697a0e5368e8501679345798f21a9364bcc60f36c58ac79",
    "ContainerConfig": {
        "Hostname": "7aa5783a47d5",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "ExposedPorts": {
            "53/tcp": {},
            "8443/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "HOME=/root",
            "OPENSHIFT_CONTAINERIZED=true",
            "KUBECONFIG=/var/lib/openshift/openshift.local.config/master/admin.kubeconfig"
        ],
        "Cmd": [
            "/bin/sh",
            "-c",
            "#(nop) ENTRYPOINT \u0026{[\"/usr/bin/openshift\"]}"
        ],
        "Image": "90ffa6a3709dc6c77de52db77f6ccd4b7c5efc6aeb3bb31dfa4838ec696a46cb",
        "Volumes": null,
        "WorkingDir": "/var/lib/openshift",
        "Entrypoint": [
            "/usr/bin/openshift"
        ],
        "OnBuild": [],
        "Labels": {
            "License": "GPLv2",
            "Vendor": "CentOS"
        }
    },
    "DockerVersion": "1.8.2",
    "Author": "",
    "Config": {
        "Hostname": "7aa5783a47d5",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "ExposedPorts": {
            "53/tcp": {},
            "8443/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "HOME=/root",
            "OPENSHIFT_CONTAINERIZED=true",
            "KUBECONFIG=/var/lib/openshift/openshift.local.config/master/admin.kubeconfig"
        ],
        "Cmd": null,
        "Image": "90ffa6a3709dc6c77de52db77f6ccd4b7c5efc6aeb3bb31dfa4838ec696a46cb",
        "Volumes": null,
        "WorkingDir": "/var/lib/openshift",
        "Entrypoint": [
            "/usr/bin/openshift"
        ],
        "OnBuild": [],
        "Labels": {
            "License": "GPLv2",
            "Vendor": "CentOS"
        }
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Size": 0,
    "VirtualSize": 449917458,
    "GraphDriver": {
        "Name": "devicemapper",
        "Data": {
            "DeviceId": "588",
            "DeviceName": "docker-253:0-659024-6f81e7b0d3a0a3ea481286125107a0f15a32bb7701f38eb7ee29dc7133c15a5c",
            "DeviceSize": "10737418240"
        }
    }
}
]

Comment 2 Liang Xia 2015-11-13 10:17:48 UTC
# oc describe pv nfs
Name:		nfs
Labels:		<none>
Status:		Failed
Claim:		default/nfsc
Reclaim Policy:	Recycle
Access Modes:	RWO
Capacity:	5Gi
Message:	Recycling error: Pod was active on the node longer than specified deadline
Source:
    Type:	NFS (an NFS mount that lasts the lifetime of a pod)
    Server:	*(Server info hide here)
    Path:	/jhou
    ReadOnly:	false


Mark the bug as regression since we suspect it is caused by https://github.com/openshift/origin/pull/5847

Comment 3 Mark Turansky 2015-11-13 14:41:15 UTC
@liang Xia,

What is the timeout on the pod?

when the PV is being recycled, you can view the recycler pod with "oc get pod".  If the pod is timing out, there should be plenty of time to see it in the API and describe it.

The pod *should* have a timeout that is incremented per Gi of size.  We can increase this number if it is too small currently.

Comment 4 Mark Turansky 2015-11-13 20:28:33 UTC
what are the permissions on the NFS export?

Comment 5 Liang Xia 2015-11-16 05:24:26 UTC
PV can be recycled when NFS exported with 777.

Comment 7 errata-xmlrpc 2015-12-08 17:03:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2580


Note You need to log in before you can comment on or make changes to this bug.