Bug 1260388

Reassigning to Huamin, resident Ceph expert and author of the plugin.

Can you add kering: '' to your yaml like the following?

{
    "apiVersion": "v1",
    "id": "cephfs",
    "kind": "Pod",
    "metadata": {
        "name": "cephfs"
    },
    "spec": {
        "containers": [
            {
                "name": "cephfs-rw",
                "image": "jhou/hello-openshift",
                "volumeMounts": [
                    {
                        "mountPath": "/mnt/cephfs",
                        "name": "cephfs"
                    }
                ]
            }
        ],
        "volumes": [
            {
                "name": "cephfs",
                "cephfs": {
                    "monitors": [
                                "192.168.0.130:6789",
                                "192.168.0.131:6789",
                                "192.168.0.132:6789",
                                "192.168.0.147:6789"
                     ],
                    "user": "admin",
                    "secretRef": {
                          "name": "ceph-secret"
                     },
                     keyring: ''
                    "readOnly": true
                }
            }
        ]
    }
}

hi,
I think kubernetes api does not support 'keyring' for ceph volume

error validating "/root/pod1.json": error validating data: [found invalid field id for v1.Pod, found invalid field keyring for v1.CephFSVolumeSource]; if you choose to ignore these errors, turn validation off with --validate=false

https://github.com/kubernetes/kubernetes/blob/9ed2d842bc3c87db0799a40226320550f2759e24/pkg/api/types.go
admin.secret

If I using "secretFile": "/etc/ceph/admin.secret" , pod can be created successfully

I see, I was thinking of rbd. Let me look at ceph fs. Thanks.

secret overrides secretFile at this line https://github.com/kubernetes/kubernetes/blob/9ed2d842bc3c87db0799a40226320550f2759e24/pkg/volume/cephfs/cephfs.go#L237.

I tested your pod and got this from kubelet log, so secret was used in the mount:


E0916 14:50:54.062950   20356 mount_linux.go:103] Mount failed: exit status 5
Mounting arguments: 192.168.0.130:6789,192.168.0.131:6789,192.168.0.132:6789,192.168.0.147:6789:/ /var/lib/kubelet/pods/b7f272fb-5ca3-11e5-be49-d4bed9b38fad/volumes/kubernetes.io~cephfs/cephfs ceph [ro name=admin,secret=AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==]
Output: mount error 5 = Input/output error


Can I login to your kube host?

the kube env is on beijing openstack , I don't know if you can access it or not.

and we have a card in trello for this bug
https://trello.com/c/A2Ba5OyY/161-secretref-does-not-overwrite-secretfile-for-ceph-volume-bugzilla

I still cannot reproduce this problem on our OSE setup

[root@host02-rack08 hchen]# oc version
oc v3.0.1.0-528-g8c2fe51
kubernetes v1.0.0


[root@host02-rack08 hchen]# cat cephfs.yaml 
{
    "apiVersion": "v1",
    "id": "cephfs",
    "kind": "Pod",
    "metadata": {
        "name": "cephfs"
    },
    "spec": {
        "containers": [
            {
                "name": "cephfs-rw",
                "image": "tutum/mysql",
                "volumeMounts": [
                    {
                        "mountPath": "/mnt/cephfs",
                        "name": "cephfs"
                    }
                ]
            }
        ],
        "volumes": [
            {
                "name": "cephfs",
                "cephfs": {
                    "monitors": [
                                "192.168.0.130:6789",
                                "192.168.0.131:6789",
                                "192.168.0.132:6789",
                                "192.168.0.147:6789"
                     ],
                    "user": "admin",
                    "secretRef": {
                          "name": "ceph-secret"
                     },
                    "readOnly": true
                }
            }
        ]
    }
}

[root@host02-rack08 hchen]# oc get pod
NAME                                    READY     STATUS                                        RESTARTS   AGE
cephfs                                  0/1       Image: tutum/mysql is not ready on the node   0          3m
[root@host02-rack08 hchen]# oc describe pod cephfs
Name:				cephfs
Namespace:			default
Image(s):			tutum/mysql
Node:				host14-rack08.scale.openstack.engineering.redhat.com/10.1.4.118
Labels:				<none>
Status:				Pending
Reason:				
Message:			
IP:				
Replication Controllers:	<none>
Containers:
  cephfs-rw:
    Image:		tutum/mysql
    State:		Waiting
      Reason:		Image: tutum/mysql is not ready on the node
    Ready:		False
    Restart Count:	0
Conditions:
  Type		Status
  Ready 	False 
Events:
  FirstSeen				LastSeen			Count	From								SubobjectPath	Reason		Message
  Tue, 22 Sep 2015 19:27:13 +0000	Tue, 22 Sep 2015 19:27:13 +0000	1	{scheduler }									scheduled	Successfully assigned cephfs to host14-rack08.scale.openstack.engineering.redhat.com
  Tue, 22 Sep 2015 19:28:13 +0000	Tue, 22 Sep 2015 19:30:13 +0000	3	{kubelet host14-rack08.scale.openstack.engineering.redhat.com}			failedMountUnable to mount volumes for pod "cephfs_default": CephFS: mount failed: exit status 5
  Tue, 22 Sep 2015 19:28:13 +0000	Tue, 22 Sep 2015 19:30:13 +0000	3	{kubelet host14-rack08.scale.openstack.engineering.redhat.com}			failedSync	Error syncing pod, skipping: CephFS: mount failed: exit status 5


On Kubelet node, secret is there:
Sep 22 19:28:13 host14-rack08 openshift-node: Mounting arguments: 192.168.0.130:6789,192.168.0.131:6789,192.168.0.132:6789,192.168.0.147:6789:/ /var/lib/openshift/openshift.local.volumes/pods/ed6bff23-615f-11e5-b8c5-b8ca3a627d6c/volumes/kubernetes.io~cephfs/cephfs ceph [ro name=admin,secret=AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==

Jianwei, your secret doesn't look right to me, is the secret in your yaml base64 encoded? If not, get base64 encoded secret using this command 

echo AQBT/+tVmLVpNBAASNoemkLGMsIwx6moYpeGzQ== |base64

correction:
echo -n AQBT/+tVmLVpNBAASNoemkLGMsIwx6moYpeGzQ== |base64

@hchen Thank you very much for pointing it out! You are right, the secretRef has to have the key base64 encoded. The problem is reproducible when key is not encoded: 'AQBT/+tVmLVpNBAASNoemkLGMsIwx6moYpeGzQ=='.

I have updated the secret with the base64 encoded secret, now the problem is gone!

This problem appears to come from non-base64 encoded secret.