Bug 1388368 - Failed to provision CephRBD volumes because master couldn't get secrets
Summary: Failed to provision CephRBD volumes because master couldn't get secrets
Keywords:
Status: CLOSED DUPLICATE of bug 1388316
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Bradley Childs
QA Contact: Jianwei Hou
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-25 08:05 UTC by Jianwei Hou
Modified: 2016-10-26 11:25 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-10-26 11:25:03 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Jianwei Hou 2016-10-25 08:05:45 UTC
Description of problem:
Ceph RBD PV/volume provisioning failed with error 'Failed to provision volume with StorageClass "cephrbdprovisioner": rbd: create volume failed, err: exit status 1'.  /var/log/messages logged possible cause:

```
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: E1025 03:28:27.544541   41587 rbd.go:435] failed to get secret from ["default"/"cephrbd-secret"]
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: E1025 03:28:27.544563   41587 rbd.go:286] failed to get admin secret from ["default"/"cephrbd-secret"]
```

But secret was present:
# oc get secrets cephrbd-secret -n default
NAME             TYPE      DATA      AGE
cephrbd-secret   Opaque    1         20m

Version-Release number of selected component (if applicable):
openshift v3.4.0.15+9c963ec
kubernetes v1.4.0+776c994
etcd 3.1.0-alpha.1

How reproducible:
Always

Steps to Reproduce:
1. Setup Ceph cluster
2. Create StorageClass, secret. The secret is created under corresponding namespaces specified by StorageClass
3. Create PVC to provision PV/volume via StorageClass

Actual results:
PV/Volume were not created

```
Events:
  FirstSeen	LastSeen	Count	From				SubobjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  17s		9s		2	{persistentvolume-controller }			Warning		ProvisioningFailed	Failed to provision volume with StorageClass
"cephrbdprovisioner": rbd: create volume failed, err: exit status 1
```

Master logs:
```
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.534574   41587 pv_controller_base.go:555] storeObjectUpdate: adding claim "jhou/cephrbdprovisioner", version 9978
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.534621   41587 pv_controller.go:201] synchronizing PersistentVolumeClaim[jhou/cephrbdprovisioner]: phase: Pending, bound to: "", bindCompleted: false, boundByController: false
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.534645   41587 pv_controller.go:225] synchronizing unbound PersistentVolumeClaim[jhou/cephrbdprovisioner]: no volume found
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.534652   41587 pv_controller.go:1189] provisionClaim[jhou/cephrbdprovisioner]: started
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.534659   41587 pv_controller.go:1363] scheduleOperation[provision-jhou/cephrbdprovisioner[9f00107f-9a84-11e6-9420-0e4f3633a564]]
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.534678   41587 pv_controller.go:1208] provisionClaimOperation [jhou/cephrbdprovisioner] started, class: "cephrbdprovisioner"
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: E1025 03:28:27.544541   41587 rbd.go:435] failed to get secret from ["default"/"cephrbd-secret"]
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: E1025 03:28:27.544563   41587 rbd.go:286] failed to get admin secret from ["default"/"cephrbd-secret"]
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.544582   41587 rbd_util.go:329] rbd: create kubernetes-dynamic-pvc-9f01f7b4-9a84-11e6-9420-0e4f3633a564 size 5120 using mon ip-172-18-8-151.ec2.internal:6789, pool rbd id admin key
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: W1025 03:28:27.628872   41587 rbd_util.go:336] failed to create rbd image, output 2016-10-25 03:28:27.574693 7f86ecac67c0 -1 did not load config file, using default settings.
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: 2016-10-25 03:28:27.627682 7f86ecac67c0 -1 monclient(hunting): authenticate NOTE: no keyring found; disabled cephx authentication
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: 2016-10-25 03:28:27.627686 7f86ecac67c0  0 librados: client.admin authentication error (95) Operation not supported
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: rbd: couldn't connect to the cluster!
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: E1025 03:28:27.628916   41587 rbd_util.go:341] rbd: Error creating rbd image: exit status 1
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: E1025 03:28:27.628944   41587 rbd.go:310] rbd: create volume failed, err: exit status 1
Oct 25 03:28:27 ip-172-18-13-116 atomic-openshift-master: I1025 03:28:27.628972   41587 pv_controller.go:1279] failed to provision volume for claim "jhou/cephrbdprovisioner" with StorageClass "cephrbdprovisioner": rbd: create volume failed, err: exit status 
```

Expected results:
Should successfully provision volume and PV.

Additional info:
GlusterFS provisioner has similar issue(https://bugzilla.redhat.com/show_bug.cgi?id=1388316), they are possibly same cause, opened this to track different features separately

Comment 1 Jan Safranek 2016-10-26 11:25:03 UTC
It's the same permission problem as #1388316

*** This bug has been marked as a duplicate of bug 1388316 ***


Note You need to log in before you can comment on or make changes to this bug.