Bug 1418977
| Summary: | Failed to create dynamic persist volume of nfs-provisioner on Azure and Openstack | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Wenqi He <wehe> |
| Component: | Storage | Assignee: | Matthew Wong <mawong> |
| Status: | CLOSED NOTABUG | QA Contact: | Wenqi He <wehe> |
| Severity: | high | Docs Contact: | |
| Priority: | low | ||
| Version: | 3.5.0 | CC: | aos-bugs, bchilds, eparis, mawong, wehe |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | Flags: | mawong:
needinfo-
|
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-04-13 15:16:44 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Wenqi He
2017-02-03 10:31:12 UTC
What is the output of `oc logs nfs-provisioner`? In step 3, is the <user> you add the SCC to the serviceaccount running the pod? Here is what should work 0. Create a serviceaccount "nfs-provisioner" $ cat > /tmp/serviceaccount.yaml <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: nfs-provisioner EOF $ oc create -f /tmp/serviceaccount.yaml serviceaccount "nfs-provisioner" created ... 3. Add serviceaccount user to this scc $ oadm policy add-scc-to-user nfs-provisioner system:serviceaccount:$PROJECT:nfs-provisioner 4. Create a nfs-provisioner pod with the serviceaccount "nfs-provisioner" $ oc create -f https://raw.githubusercontent.com/kubernetes-incubator/nfs-provisioner/master/deploy/kube-config/pod-sa.yaml Sorry I neglected to address openshift authentication, I will review the test cases again reducing priority to low because the NFS provisioner is not a supported OCP piece of code. Also see: https://trello.com/c/ihX3mRPJ/383-5-documentation-nfs-provisioner-external-provisioners#comment-5894cece9bf9b6417610d679 If you know what serviceaccount the pods will be running as (default?) you don't need to create one and you can use pod.yaml, not pod-sa.yaml (In reply to Matthew Wong from comment #3) > Also see: > https://trello.com/c/ihX3mRPJ/383-5-documentation-nfs-provisioner-external- > provisioners#comment-5894cece9bf9b6417610d679 > > If you know what serviceaccount the pods will be running as (default?) you > don't need to create one and you can use pod.yaml, not pod-sa.yaml I got this worked today with add normal user to "storage-admin" 1. Create a scc as you mentioned below 2. Add serviceaccount user to this sccc $ oadm policy add-scc-to-user nfs-provisioner system:serviceaccount:wehe:default 3. Add user to storage admin role: $ oadm policy add-cluster-role-to-user storage-admin wehe Working per last comment. I'd like to re-open this from today's testing, get another error with deployment nfs-provisioner 1. Create NFS provisioner deployment oc create -f https://raw.githubusercontent.com/kubernetes-incubator/nfs-provisioner/master/demo/deployment.yaml 2. Create a storage class oc create -f https://raw.githubusercontent.com/kubernetes-incubator/nfs-provisioner/master/demo/class.yaml 3. Create a pvc oc create -f https://raw.githubusercontent.com/kubernetes-incubator/nfs-provisioner/master/demo/claim.yaml $ oc get pods NAME READY STATUS RESTARTS AGE nfs-provisioner-770926304-pnfk2 1/1 Running 0 11m $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESSMODES AGE nfs Pending 11m $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESSMODES AGE nfs Pending 11m [wehe@dhcp-136-45 octest]$ oc describe pvc nfs Name: nfs Namespace: wehe StorageClass: example-nfs Status: Pending Volume: Labels: <none> Capacity: Access Modes: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 11m 9m 10 {example.com/nfs nfs-provisioner-770926304-pnfk2 5e96b006-ede6-11e6-bb75-562ab5031b7f } Warning ProvisioningFailed Failed to provision volume with StorageClass "example-nfs": error creating export for volume: error exporting export block EXPORT { Export_Id = 1; Path = /export/pvc-79eb980e-ede6-11e6-94a4-000d3a179c12; Pseudo = /export/pvc-79eb980e-ede6-11e6-94a4-000d3a179c12; Access_Type = RW; Squash = no_root_squash; SecType = sys; Filesystem_id = 1.1; FSAL { Name = VFS; } } : error getting dbus session bus: dial unix /var/run/dbus/system_bus_socket: connect: permission denied 11m 3s 59 {persistentvolume-controller } Normal ExternalProvisioning cannot find provisioner "example.com/nfs", expecting that a volume for the claim is provisioned either manually or via external software $ oc logs nfs-provisioner-770926304-pnfk2 I0208 10:07:16.349710 1 main.go:58] Provisioner example.com/nfs specified I0208 10:07:16.349812 1 main.go:71] Starting NFS server! I0208 10:07:16.658412 1 controller.go:256] Starting provisioner controller 5e96b006-ede6-11e6-bb75-562ab5031b7f! I0208 10:08:02.436869 1 controller.go:841] scheduleOperation[lock-provision-wehe/nfs[79eb980e-ede6-11e6-94a4-000d3a179c12]] I0208 10:08:02.452287 1 controller.go:841] scheduleOperation[lock-provision-wehe/nfs[79eb980e-ede6-11e6-94a4-000d3a179c12]] I0208 10:08:02.496389 1 leaderelection.go:157] attempting to acquire leader lease... I0208 10:08:02.561434 1 leaderelection.go:179] sucessfully acquired lease to provision for pvc wehe/nfs I0208 10:08:02.561557 1 controller.go:841] scheduleOperation[provision-wehe/nfs[79eb980e-ede6-11e6-94a4-000d3a179c12]] I0208 10:08:02.601652 1 provision.go:363] using service SERVICE_NAME=nfs-provisioner cluster IP 172.30.98.7 as NFS server IP E0208 10:08:02.612964 1 controller.go:572] Failed to provision volume for claim "wehe/nfs" with StorageClass "example-nfs": error creating export for volume: error exporting export block EXPORT { Export_Id = 1; Path = /export/pvc-79eb980e-ede6-11e6-94a4-000d3a179c12; Pseudo = /export/pvc-79eb980e-ede6-11e6-94a4-000d3a179c12; Access_Type = RW; Squash = no_root_squash; SecType = sys; Filesystem_id = 1.1; FSAL { Name = VFS; } } Can you provide also the output of `oc get pod -o yaml $nfs-provisioner-pod` and `docker inspect $nfs-provisioner-pod-container`. Thanks Also the output of `ls -lZ /run/dbus/system_bus_socket` inside the container? e.g. mine is srwxrwxrwx. 1 root root system_u:object_r:container_share_t:s0 0 Feb 8 18:45 /run/dbus/system_bus_socket $ oc get pods nfs-provisioner-770926304-pnfk2 -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/created-by: |
{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"wehe","name":"nfs-provisioner-770926304","uid":"e5456b61-edca-11e6-94a4-000d3a179c12","apiVersion":"extensions","resourceVersion":"7644"}}
openshift.io/scc: nfs-provisioner
creationTimestamp: 2017-02-08T10:07:15Z
generateName: nfs-provisioner-770926304-
labels:
app: nfs-provisioner
pod-template-hash: "770926304"
name: nfs-provisioner-770926304-pnfk2
namespace: wehe
resourceVersion: "8908"
selfLink: /api/v1/namespaces/wehe/pods/nfs-provisioner-770926304-pnfk2
uid: 5dc12dbf-ede6-11e6-94a4-000d3a179c12
spec:
containers:
- args:
- -provisioner=example.com/nfs
- -grace-period=10
env:
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: SERVICE_NAME
value: nfs-provisioner
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/kubernetes_incubator/nfs-provisioner:v1.0.3
imagePullPolicy: IfNotPresent
name: nfs-provisioner
ports:
- containerPort: 2049
name: nfs
protocol: TCP
- containerPort: 20048
name: mountd
protocol: TCP
- containerPort: 111
name: rpcbind
protocol: TCP
- containerPort: 111
name: rpcbind-udp
protocol: UDP
resources: {}
securityContext:
capabilities:
add:
- DAC_READ_SEARCH
drop:
- KILL
- MKNOD
- SYS_CHROOT
privileged: false
seLinuxOptions:
level: s0:c8,c2
terminationMessagePath: /dev/termination-log
volumeMounts:
- mountPath: /export
name: export-volume
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-6lff3
readOnly: true
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: default-dockercfg-6mjr0
nodeName: wehe-node-1.eastus.cloudapp.azure.com
restartPolicy: Always
securityContext:
fsGroup: 1000060000
seLinuxOptions:
level: s0:c8,c2
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /tmp/nfs-provisioner
name: export-volume
- name: default-token-6lff3
secret:
defaultMode: 420
secretName: default-token-6lff3
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2017-02-08T10:07:15Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2017-02-08T10:07:16Z
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: 2017-02-08T10:07:15Z
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://1b68467495792876d7e07f9e3c017c6e88a72e92657cae4d410af367f2a24f44
image: quay.io/kubernetes_incubator/nfs-provisioner:v1.0.3
imageID: docker-pullable://quay.io/kubernetes_incubator/nfs-provisioner@sha256:ee2900e758c36214aad5bd4d3a7974bf43bbf3d21174a54d62567ca99c69d9e4
lastState: {}
name: nfs-provisioner
ready: true
restartCount: 0
state:
running:
startedAt: 2017-02-08T10:07:16Z
hostIP: 172.27.17.5
phase: Running
podIP: 10.129.0.18
startTime: 2017-02-08T10:07:15Z
=====================================================================
# docker inspect 1b6846749579
[
{
"Id": "1b68467495792876d7e07f9e3c017c6e88a72e92657cae4d410af367f2a24f44",
"Created": "2017-02-08T10:07:16.096598773Z",
"Path": "/nfs-provisioner",
"Args": [
"-provisioner=example.com/nfs",
"-grace-period=10"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 11538,
"ExitCode": 0,
"Error": "",
"StartedAt": "2017-02-08T10:07:16.269364942Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:c12625ede8fd43271ae0117ec714b0ab2203e6a3177c4f5136f9aa791098d2ea",
"ResolvConfPath": "/var/lib/docker/containers/609f212dc85d7c40899441595553911a94df5d3924ec4165e81ca1aedb291653/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/609f212dc85d7c40899441595553911a94df5d3924ec4165e81ca1aedb291653/hostname",
"HostsPath": "/var/lib/origin/openshift.local.volumes/pods/5dc12dbf-ede6-11e6-94a4-000d3a179c12/etc-hosts",
"LogPath": "",
"Name": "/k8s_nfs-provisioner.d375cb8_nfs-provisioner-770926304-pnfk2_wehe_5dc12dbf-ede6-11e6-94a4-000d3a179c12_b8244be6",
"RestartCount": 0,
"Driver": "overlay",
"MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c8,c2",
"ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c8,c2",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/tmp/nfs-provisioner:/export",
"/var/lib/origin/openshift.local.volumes/pods/5dc12dbf-ede6-11e6-94a4-000d3a179c12/volumes/kubernetes.io~secret/default-token-6lff3:/var/run/secrets/kubernetes.io/serviceaccount:ro,Z",
"/var/lib/origin/openshift.local.volumes/pods/5dc12dbf-ede6-11e6-94a4-000d3a179c12/etc-hosts:/etc/hosts:Z",
"/var/lib/origin/openshift.local.volumes/pods/5dc12dbf-ede6-11e6-94a4-000d3a179c12/containers/nfs-provisioner/b8244be6:/dev/termination-log:Z"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "journald",
"Config": {}
},
"NetworkMode": "container:609f212dc85d7c40899441595553911a94df5d3924ec4165e81ca1aedb291653",
"PortBindings": null,
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": [
"DAC_READ_SEARCH"
],
"CapDrop": [
"KILL",
"MKNOD",
"SYS_CHROOT"
],
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": null,
"GroupAdd": [
"1000060000"
],
"IpcMode": "container:609f212dc85d7c40899441595553911a94df5d3924ec4165e81ca1aedb291653",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 1000,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"label:level:s0:c8,c2",
"label=user:system_u",
"label=role:system_r",
"label=type:svirt_lxc_net_t",
"label=level:s0:c8,c2"
],
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "docker-runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 2,
"Memory": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": -1,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Name": "overlay",
"Data": {
"LowerDir": "/var/lib/docker/overlay/a0e956c6c67742911e5a7de615e17cf02b46ced26ba911efb792da078cae63ff/root",
"MergedDir": "/var/lib/docker/overlay/f0d49402a5a0710d99dcf34f471cfeaedf03217f126d234aec5dd13948a631e3/merged",
"UpperDir": "/var/lib/docker/overlay/f0d49402a5a0710d99dcf34f471cfeaedf03217f126d234aec5dd13948a631e3/upper",
"WorkDir": "/var/lib/docker/overlay/f0d49402a5a0710d99dcf34f471cfeaedf03217f126d234aec5dd13948a631e3/work"
}
},
"Mounts": [
{
"Source": "/tmp/nfs-provisioner",
"Destination": "/export",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Source": "/var/lib/origin/openshift.local.volumes/pods/5dc12dbf-ede6-11e6-94a4-000d3a179c12/volumes/kubernetes.io~secret/default-token-6lff3",
"Destination": "/var/run/secrets/kubernetes.io/serviceaccount",
"Mode": "ro,Z",
"RW": false,
"Propagation": "rprivate"
},
{
"Source": "/var/lib/origin/openshift.local.volumes/pods/5dc12dbf-ede6-11e6-94a4-000d3a179c12/etc-hosts",
"Destination": "/etc/hosts",
"Mode": "Z",
"RW": true,
"Propagation": "rprivate"
},
{
"Source": "/var/lib/origin/openshift.local.volumes/pods/5dc12dbf-ede6-11e6-94a4-000d3a179c12/containers/nfs-provisioner/b8244be6",
"Destination": "/dev/termination-log",
"Mode": "Z",
"RW": true,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "nfs-provisioner-770926304-pnfk2",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"111/tcp": {},
"111/udp": {},
"20048/tcp": {},
"2049/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"POD_IP=10.129.0.18",
"SERVICE_NAME=nfs-provisioner",
"POD_NAMESPACE=wehe",
"NFS_PROVISIONER_PORT_20048_TCP_PORT=20048",
"NFS_PROVISIONER_PORT_111_UDP_ADDR=172.30.98.7",
"KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443",
"KUBERNETES_PORT_443_TCP_PROTO=tcp",
"NFS_PROVISIONER_SERVICE_PORT_MOUNTD=20048",
"KUBERNETES_SERVICE_HOST=172.30.0.1",
"KUBERNETES_PORT_53_UDP=udp://172.30.0.1:53",
"NFS_PROVISIONER_PORT_111_TCP_ADDR=172.30.98.7",
"NFS_PROVISIONER_PORT_111_UDP=udp://172.30.98.7:111",
"KUBERNETES_SERVICE_PORT_DNS=53",
"KUBERNETES_PORT=tcp://172.30.0.1:443",
"NFS_PROVISIONER_SERVICE_PORT_NFS=2049",
"NFS_PROVISIONER_SERVICE_PORT_RPCBIND=111",
"NFS_PROVISIONER_PORT_2049_TCP_PORT=2049",
"NFS_PROVISIONER_PORT_20048_TCP=tcp://172.30.98.7:20048",
"KUBERNETES_PORT_53_UDP_PORT=53",
"NFS_PROVISIONER_PORT_111_UDP_PORT=111",
"KUBERNETES_SERVICE_PORT_HTTPS=443",
"KUBERNETES_PORT_53_TCP_PORT=53",
"KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53",
"KUBERNETES_PORT_53_TCP_ADDR=172.30.0.1",
"NFS_PROVISIONER_PORT=tcp://172.30.98.7:2049",
"NFS_PROVISIONER_PORT_2049_TCP_ADDR=172.30.98.7",
"NFS_PROVISIONER_PORT_20048_TCP_PROTO=tcp",
"NFS_PROVISIONER_PORT_111_TCP=tcp://172.30.98.7:111",
"NFS_PROVISIONER_SERVICE_PORT=2049",
"NFS_PROVISIONER_PORT_2049_TCP_PROTO=tcp",
"NFS_PROVISIONER_PORT_20048_TCP_ADDR=172.30.98.7",
"KUBERNETES_PORT_443_TCP_PORT=443",
"KUBERNETES_PORT_53_UDP_PROTO=udp",
"NFS_PROVISIONER_SERVICE_HOST=172.30.98.7",
"NFS_PROVISIONER_SERVICE_PORT_RPCBIND_UDP=111",
"NFS_PROVISIONER_PORT_111_TCP_PROTO=tcp",
"KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1",
"KUBERNETES_SERVICE_PORT_DNS_TCP=53",
"KUBERNETES_PORT_53_UDP_ADDR=172.30.0.1",
"KUBERNETES_PORT_53_TCP_PROTO=tcp",
"NFS_PROVISIONER_PORT_2049_TCP=tcp://172.30.98.7:2049",
"NFS_PROVISIONER_PORT_111_TCP_PORT=111",
"NFS_PROVISIONER_PORT_111_UDP_PROTO=udp",
"KUBERNETES_SERVICE_PORT=443",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"DISTTAG=f24docker",
"FGC=f24"
],
"Cmd": [
"-provisioner=example.com/nfs",
"-grace-period=10"
],
"Image": "quay.io/kubernetes_incubator/nfs-provisioner:v1.0.3",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/nfs-provisioner"
],
"OnBuild": null,
"Labels": {
"io.kubernetes.container.hash": "d375cb8",
"io.kubernetes.container.name": "nfs-provisioner",
"io.kubernetes.container.ports": "[{\"name\":\"nfs\",\"containerPort\":2049,\"protocol\":\"TCP\"},{\"name\":\"mountd\",\"containerPort\":20048,\"protocol\":\"TCP\"},{\"name\":\"rpcbind\",\"containerPort\":111,\"protocol\":\"TCP\"},{\"name\":\"rpcbind-udp\",\"containerPort\":111,\"protocol\":\"UDP\"}]",
"io.kubernetes.container.restartCount": "0",
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"io.kubernetes.pod.name": "nfs-provisioner-770926304-pnfk2",
"io.kubernetes.pod.namespace": "wehe",
"io.kubernetes.pod.terminationGracePeriod": "30",
"io.kubernetes.pod.uid": "5dc12dbf-ede6-11e6-94a4-000d3a179c12"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": null,
"SandboxKey": "",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": null
}
}
]
===========================================================================
[root@nfs-provisioner-770926304-pnfk2 /]# ls -lZ /run/dbus/system_bus_socket
srwxrwxrwx. 1 root root system_u:object_r:svirt_sandbox_file_t:s0:c2,c8 0 Feb 8 10:07 /run/dbus/system_bus_socket
Please contact me if you need more info, thanks.
Got this worked again with today's build... Not sure what's wrong with my last testing. Sorry about this.. $ oc version openshift v3.5.0.18+9a5d1aa kubernetes v1.5.2+43a9be4 $ oc get pods NAME READY STATUS RESTARTS AGE nfs-provisioner-770926304-sz6kt 1/1 Running 0 1m $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESSMODES AGE nfs Bound pvc-2a190044-eea0-11e6-9a4b-42010af00018 1Mi RWX 4s OK, no problem, thank you for reporting. It had to have been an SELinux issue: but my SCC has the same seLinuxContext setting as the default restricted SCC and the categories (c2,c8) look fine to me in the output you provided. So it seems there was an selinux-related issue in that openshift build that has been fixed. I'd like to re-open this bug since I found two different issue on Azure and Openstack, and for AWS and GCE, it works well:
$ oc version
openshift v3.5.5
kubernetes v1.5.2+43a9be4
On Openstack:
[wehe@dhcp-136-45 octest]$ oc get pods
NAME READY STATUS RESTARTS AGE
nfs-provisioner-toqd2 1/1 Running 0 1m
[wehe@dhcp-136-45 octest]$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
nfsdynpvc Pending 1m
[wehe@dhcp-136-45 octest]$ oc logs nfs-provisioner-toqd2
I0331 08:36:11.167448 1 main.go:58] Provisioner example.com/nfs specified
I0331 08:36:11.167579 1 main.go:71] Starting NFS server!
I0331 08:36:11.377529 1 controller.go:256] Starting provisioner controller 1818361a-15ed-11e7-b411-529c47630055!
I0331 08:36:19.641133 1 controller.go:841] scheduleOperation[lock-provision-pf93k/nfsdynpvc[1cf74c96-15ed-11e7-9cc6-fa163ece4ef6]]
I0331 08:36:19.652247 1 controller.go:841] scheduleOperation[lock-provision-pf93k/nfsdynpvc[1cf74c96-15ed-11e7-9cc6-fa163ece4ef6]]
I0331 08:36:19.657827 1 controller.go:641] cannot start watcher for PVC pf93k/nfsdynpvc: User "system:serviceaccount:pf93k:nfs-provisioner" cannot list events in project "pf93k"
E0331 08:36:19.657850 1 controller.go:493] Error watching for provisioning success, can't provision for claim "pf93k/nfsdynpvc": User "system:serviceaccount:pf93k:nfs-provisioner" cannot list events in project "pf93k"
I0331 08:36:19.657858 1 leaderelection.go:157] attempting to acquire leader lease...
I0331 08:36:19.667265 1 leaderelection.go:179] sucessfully acquired lease to provision for pvc pf93k/nfsdynpvc
I0331 08:36:19.667326 1 controller.go:841] scheduleOperation[provision-pf93k/nfsdynpvc[1cf74c96-15ed-11e7-9cc6-fa163ece4ef6]]
I0331 08:36:19.672231 1 provision.go:312] using potentially unstable pod IP POD_IP=10.129.0.20 as NFS server IP (because neither service env SERVICE_NAME nor node env NODE_NAME are set)
E0331 08:36:19.694625 1 controller.go:572] Failed to provision volume for claim "pf93k/nfsdynpvc" with StorageClass "nfs-provisioner-pf93k": error creating export for volume: error exporting export block
EXPORT
{
Export_Id = 1;
Path = /export/pvc-1cf74c96-15ed-11e7-9cc6-fa163ece4ef6;
Pseudo = /export/pvc-1cf74c96-15ed-11e7-9cc6-fa163ece4ef6;
Access_Type = RW;
Squash = no_root_squash;
SecType = sys;
Filesystem_id = 1.1;
FSAL {
Name = VFS;
}
}
: error calling org.ganesha.nfsd.exportmgr.AddExport: 0 export entries in /export/vfs.conf added because (invalid param value) errors. Details:
[root@nfs-provisioner-toqd2 /]# ls -lZd /export/
drwxr-xr-x. 1 root root system_u:object_r:container_share_t:s0 82 Mar 31 08:38 /export/
[root@nfs-provisioner-toqd2 /]# ls /export/
nfs-provisioner.identity v4old v4recov vfs.conf
[root@host-8-175-74 ~]# docker inspect 9517589f7f59
[
{
"Id": "9517589f7f59486a70318372100f582b17b81c8ab060234d676c74c3ae78c414",
"Created": "2017-03-31T08:36:11.015596579Z",
"Path": "/nfs-provisioner",
"Args": [
"-provisioner=example.com/nfs",
"-grace-period=0"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 14960,
"ExitCode": 0,
"Error": "",
"StartedAt": "2017-03-31T08:36:11.10522218Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:c12625ede8fd43271ae0117ec714b0ab2203e6a3177c4f5136f9aa791098d2ea",
"ResolvConfPath": "/var/lib/docker/containers/46b6654a478a924310b51ade0cb3c63596b5766f64ed0970124b0a5de32a7674/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/46b6654a478a924310b51ade0cb3c63596b5766f64ed0970124b0a5de32a7674/hostname",
"HostsPath": "/var/lib/origin/openshift.local.volumes/pods/1765cde9-15ed-11e7-9cc6-fa163ece4ef6/etc-hosts",
"LogPath": "",
"Name": "/k8s_nfs-provisioner.944baba3_nfs-provisioner-toqd2_pf93k_1765cde9-15ed-11e7-9cc6-fa163ece4ef6_150e0c14",
"RestartCount": 0,
"Driver": "overlay",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/var/lib/origin/openshift.local.volumes/pods/1765cde9-15ed-11e7-9cc6-fa163ece4ef6/volumes/kubernetes.io~secret/nfs-provisioner-token-rj0c0:/var/run/secrets/kubernetes.io/serviceaccount:ro,Z",
"/var/lib/origin/openshift.local.volumes/pods/1765cde9-15ed-11e7-9cc6-fa163ece4ef6/etc-hosts:/etc/hosts:Z",
"/var/lib/origin/openshift.local.volumes/pods/1765cde9-15ed-11e7-9cc6-fa163ece4ef6/containers/nfs-provisioner/150e0c14:/dev/termination-log:Z"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "journald",
"Config": {}
},
"NetworkMode": "container:46b6654a478a924310b51ade0cb3c63596b5766f64ed0970124b0a5de32a7674",
"PortBindings": null,
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": [
"DAC_READ_SEARCH"
],
"CapDrop": [
"KILL",
"MKNOD",
"SYS_CHROOT"
],
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": null,
"GroupAdd": [
"1000120000"
],
"IpcMode": "container:46b6654a478a924310b51ade0cb3c63596b5766f64ed0970124b0a5de32a7674",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 1000,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"label=level:s0:c11,c5"
],
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "docker-runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 2,
"Memory": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": -1,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Name": "overlay",
"Data": {
"LowerDir": "/var/lib/docker/overlay/07c1a11104cdea0ad606e0e525fc8af1dff548a2531a495746879044eec30efa/root",
"MergedDir": "/var/lib/docker/overlay/d5b875f17bbfcd8582c68c1c4e60ea248f0e19f1cca687aca380e109c5f29e5c/merged",
"UpperDir": "/var/lib/docker/overlay/d5b875f17bbfcd8582c68c1c4e60ea248f0e19f1cca687aca380e109c5f29e5c/upper",
"WorkDir": "/var/lib/docker/overlay/d5b875f17bbfcd8582c68c1c4e60ea248f0e19f1cca687aca380e109c5f29e5c/work"
}
},
"Mounts": [
{
"Source": "/var/lib/origin/openshift.local.volumes/pods/1765cde9-15ed-11e7-9cc6-fa163ece4ef6/etc-hosts",
"Destination": "/etc/hosts",
"Mode": "Z",
"RW": true,
"Propagation": "rprivate"
},
{
"Source": "/var/lib/origin/openshift.local.volumes/pods/1765cde9-15ed-11e7-9cc6-fa163ece4ef6/containers/nfs-provisioner/150e0c14",
"Destination": "/dev/termination-log",
"Mode": "Z",
"RW": true,
"Propagation": "rprivate"
},
{
"Source": "/var/lib/origin/openshift.local.volumes/pods/1765cde9-15ed-11e7-9cc6-fa163ece4ef6/volumes/kubernetes.io~secret/nfs-provisioner-token-rj0c0",
"Destination": "/var/run/secrets/kubernetes.io/serviceaccount",
"Mode": "ro,Z",
"RW": false,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "nfs-provisioner-toqd2",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"111/tcp": {},
"111/udp": {},
"20048/tcp": {},
"2049/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"POD_IP=10.129.0.20",
"KUBERNETES_SERVICE_HOST=172.30.0.1",
"KUBERNETES_PORT_443_TCP_PROTO=tcp",
"KUBERNETES_PORT_53_UDP_PORT=53",
"KUBERNETES_SERVICE_PORT_DNS=53",
"KUBERNETES_SERVICE_PORT_DNS_TCP=53",
"KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443",
"KUBERNETES_PORT_53_UDP_ADDR=172.30.0.1",
"KUBERNETES_PORT_53_TCP_ADDR=172.30.0.1",
"KUBERNETES_SERVICE_PORT_HTTPS=443",
"KUBERNETES_PORT_443_TCP_PORT=443",
"KUBERNETES_PORT_53_UDP=udp://172.30.0.1:53",
"KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53",
"KUBERNETES_PORT_53_TCP_PROTO=tcp",
"KUBERNETES_PORT_53_TCP_PORT=53",
"KUBERNETES_SERVICE_PORT=443",
"KUBERNETES_PORT=tcp://172.30.0.1:443",
"KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1",
"KUBERNETES_PORT_53_UDP_PROTO=udp",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"DISTTAG=f24docker",
"FGC=f24"
],
"Cmd": [
"-provisioner=example.com/nfs",
"-grace-period=0"
],
"Image": "quay.io/kubernetes_incubator/nfs-provisioner:v1.0.3",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/nfs-provisioner"
],
"OnBuild": null,
"Labels": {
"io.kubernetes.container.hash": "944baba3",
"io.kubernetes.container.name": "nfs-provisioner",
"io.kubernetes.container.ports": "[{\"name\":\"nfs\",\"containerPort\":2049,\"protocol\":\"TCP\"},{\"name\":\"mountd\",\"containerPort\":20048,\"protocol\":\"TCP\"},{\"name\":\"rpcbind\",\"containerPort\":111,\"protocol\":\"TCP\"},{\"name\":\"rpcbind-udp\",\"containerPort\":111,\"protocol\":\"UDP\"}]",
"io.kubernetes.container.restartCount": "0",
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"io.kubernetes.pod.name": "nfs-provisioner-toqd2",
"io.kubernetes.pod.namespace": "pf93k",
"io.kubernetes.pod.terminationGracePeriod": "30",
"io.kubernetes.pod.uid": "1765cde9-15ed-11e7-9cc6-fa163ece4ef6"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": null,
"SandboxKey": "",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": null
}
}
]
And for Azure it is same issue on Comment 6 and Comment 9.
I don't see a mount to /export in this docker inspect, it should look like this
{
"Source": "/tmp/nfs-provisioner",
"Destination": "/export",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
since https://raw.githubusercontent.com/kubernetes-incubator/nfs-provisioner/master/demo/deployment.yaml mounts /tmp/nfs-provisioner to /export. I think here, https://github.com/kubernetes-incubator/external-storage/blob/master/nfs/deploy/kubernetes/pod.yaml or another pod.yaml was used instead, is that correct?
So this issue is because https://github.com/kubernetes-incubator/external-storage/blob/master/nfs/deploy/kubernetes/pod.yaml does not mount anything to /export AND RHEL 7.3 now defaults to the overlay docker storage driver: exporting does not work from an overlay fs, this is a known issue.
So IMO the next course of action is to remove https://github.com/kubernetes-incubator/external-storage/blob/master/nfs/deploy/kubernetes/pod.yaml (documentation pointing to it is already removed) and use https://github.com/kubernetes-incubator/external-storage/blob/master/nfs/deploy/kubernetes/pod_emptydir.yaml instead in all test cases to accommodate the new RHEL default
Correction: I didn't remove https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs/deploy/kubernetes/pod.yaml, i renamed pod_emptydir.yaml -> pod.yaml So if we replace links to https://raw.githubusercontent.com/kubernetes-incubator/nfs-provisioner/master/nfs/deploy/kubernetes/pod.yaml to https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs/deploy/kubernetes/pod.yaml, it should work. (In reply to Matthew Wong from comment #14) > Correction: I didn't remove > https://raw.githubusercontent.com/kubernetes-incubator/external-storage/ > master/nfs/deploy/kubernetes/pod.yaml, i renamed pod_emptydir.yaml -> > pod.yaml > > So if we replace links to > https://raw.githubusercontent.com/kubernetes-incubator/nfs-provisioner/ > master/nfs/deploy/kubernetes/pod.yaml to > https://raw.githubusercontent.com/kubernetes-incubator/external-storage/ > master/nfs/deploy/kubernetes/pod.yaml, it should work. So we are not going to maintain the https://github.com/kubernetes-incubator/nfs-provisioner/tree/master/deploy/kube-config and move them all to "external-storage", right? You are right, to use the pod under /external-storage/ on an overlay OCP, the nfs-provisioner pod works well on openstack. But it still has the problem with Comment 6 and Comment 9 on Azure, I think it might be also caused by "overlay", will try a new env with "devicemapper" to see whether it is repro I have tried to run on Azure with a "devicemapper" docker, the nfs-provisioner works well. So the issue in Comment 6 and Comment 9 should caused by "overlay". Yamls have been amended to use emptyDir only, this is a known issue that cannot be otherwise fixed, closing. |