Description of problem: Ceph RBD PV/volume provisioning failed with error 'Failed to provision volume with StorageClass "cephrbdprovisioner": rbd: create volume failed, err: exit status 1'. /var/log/messages logged possible cause: ``` Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: E1025 03:28:27.544541 41587 rbd.go:435] failed to get secret from ["default"/"cephrbd-secret"] Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: E1025 03:28:27.544563 41587 rbd.go:286] failed to get admin secret from ["default"/"cephrbd-secret"] ``` But secret was present: # oc get secrets cephrbd-secret -n default NAME TYPE DATA AGE cephrbd-secret Opaque 1 20m Version-Release number of selected component (if applicable): openshift v3.4.0.15+9c963ec kubernetes v1.4.0+776c994 etcd 3.1.0-alpha.1 How reproducible: Always Steps to Reproduce: 1. Setup Ceph cluster 2. Create StorageClass, secret. The secret is created under corresponding namespaces specified by StorageClass 3. Create PVC to provision PV/volume via StorageClass Actual results: PV/Volume were not created ``` Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 17s 9s 2 {persistentvolume-controller } Warning ProvisioningFailed Failed to provision volume with StorageClass "cephrbdprovisioner": rbd: create volume failed, err: exit status 1 ``` Master logs: ``` Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.534574 41587 pv_controller_base.go:555] storeObjectUpdate: adding claim "jhou/cephrbdprovisioner", version 9978 Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.534621 41587 pv_controller.go:201] synchronizing PersistentVolumeClaim[jhou/cephrbdprovisioner]: phase: Pending, bound to: "", bindCompleted: false, boundByController: false Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.534645 41587 pv_controller.go:225] synchronizing unbound PersistentVolumeClaim[jhou/cephrbdprovisioner]: no volume found Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.534652 41587 pv_controller.go:1189] provisionClaim[jhou/cephrbdprovisioner]: started Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.534659 41587 pv_controller.go:1363] scheduleOperation[provision-jhou/cephrbdprovisioner[9f00107f-9a84-11e6-9420-0e4f3633a564]] Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.534678 41587 pv_controller.go:1208] provisionClaimOperation [jhou/cephrbdprovisioner] started, class: "cephrbdprovisioner" Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: E1025 03:28:27.544541 41587 rbd.go:435] failed to get secret from ["default"/"cephrbd-secret"] Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: E1025 03:28:27.544563 41587 rbd.go:286] failed to get admin secret from ["default"/"cephrbd-secret"] Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.544582 41587 rbd_util.go:329] rbd: create kubernetes-dynamic-pvc-9f01f7b4-9a84-11e6-9420-0e4f3633a564 size 5120 using mon ip-172-18-8-151.ec2.internal:6789, pool rbd id admin key Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: W1025 03:28:27.628872 41587 rbd_util.go:336] failed to create rbd image, output 2016-10-25 03:28:27.574693 7f86ecac67c0 -1 did not load config file, using default settings. Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: 2016-10-25 03:28:27.627682 7f86ecac67c0 -1 monclient(hunting): authenticate NOTE: no keyring found; disabled cephx authentication Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: 2016-10-25 03:28:27.627686 7f86ecac67c0 0 librados: client.admin authentication error (95) Operation not supported Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: rbd: couldn't connect to the cluster! Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: E1025 03:28:27.628916 41587 rbd_util.go:341] rbd: Error creating rbd image: exit status 1 Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: E1025 03:28:27.628944 41587 rbd.go:310] rbd: create volume failed, err: exit status 1 Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.628972 41587 pv_controller.go:1279] failed to provision volume for claim "jhou/cephrbdprovisioner" with StorageClass "cephrbdprovisioner": rbd: create volume failed, err: exit status ``` Expected results: Should successfully provision volume and PV. Additional info: GlusterFS provisioner has similar issue(https://bugzilla.redhat.com/show_bug.cgi?id=1388316), they are possibly same cause, opened this to track different features separately
It's the same permission problem as #1388316 *** This bug has been marked as a duplicate of bug 1388316 ***