Bug 1411244 - Unable to provision rbd volumes in containerized OpenShift, "err: executable file not found in $PATH"
Summary: Unable to provision rbd volumes in containerized OpenShift, "err: executable ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.4.z
Assignee: hchen
QA Contact: Jianwei Hou
URL:
Whiteboard:
Depends On:
Blocks: 1420698
TreeView+ depends on / blocked
 
Reported: 2017-01-09 09:03 UTC by Jianwei Hou
Modified: 2017-02-09 10:31 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The ceph-common client tools were missing from the containerized node image which prevented containerized environments from mounting ceph volumes. The package has been added enabling containerized environments to mount ceph volumes.
Clone Of:
: 1420698 (view as bug list)
Environment:
Last Closed: 2017-01-31 20:19:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0218 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.4.1.2 bug fix update 2017-02-01 01:18:20 UTC

Description Jianwei Hou 2017-01-09 09:03:16 UTC
Description of problem:
In containerzied OpenShift cluster, rbd volumes could not be dynamically provisioned.  PVC events logged failure reasion "rbd: create volume failed, err: executable file not found in $PATH"

Version-Release number of selected component (if applicable):
openshift v3.4.0.39
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

How reproducible:
Always

Steps to Reproduce:
1. Setup containerized OpenShift cluster, setup ceph rbd storage server
2. Create a StorageClass for Ceph RBD provisioner
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  creationTimestamp: 2017-01-09T06:54:58Z
  name: cephrbdprovisioner
  resourceVersion: "8131"
  selfLink: /apis/storage.k8s.io/v1beta1/storageclasses/cephrbdprovisioner
  uid: 892a27c2-d638-11e6-800d-0eb7d2142d46
parameters:
  adminId: admin
  adminSecretName: cephrbd-secret
  adminSecretNamespace: default
  monitors: xxx:6789
  pool: rbd
  userId: admin
  userSecretName: cephrbd-user-secret
provisioner: kubernetes.io/rbd

3. Create a PVC to dynamically provision a PV
{
   "kind": "PersistentVolumeClaim",
   "apiVersion": "v1",
   "metadata": {
     "name": "rbdc",
     "annotations": {
     "volume.beta.kubernetes.io/storage-class": "cephrbdprovisioner"
     }
   },
   "spec": {
     "accessModes": [
       "ReadWriteOnce"
     ],
     "resources": {
       "requests": {
         "storage": "5Gi"
       }
     }
   }
}


Actual results:
After step 3: PVC is Pending, PV could not be provisioned.
 oc describe pvc rbdc
Name:		rbdc
Namespace:	jhou1
StorageClass:	cephrbdprovisioner
Status:		Pending
Volume:		
Labels:		<none>
Capacity:	
Access Modes:	
Events:
  FirstSeen	LastSeen	Count	From				SubobjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  18s		5s		2	{persistentvolume-controller }			Warning		ProvisioningFailed	Failed to provision volume with StorageClass "cephrbdprovisioner": rbd: create volume failed, err: executable file not found in $PATH


Expected results:
Should provision successfully.

Additional info:

Comment 1 Bradley Childs 2017-01-09 16:07:54 UTC
This sounds like missing RBD packages on the nodes.  

Huamin - can you figure out the packages needed on each node?

Comment 2 Eric Paris 2017-01-09 16:15:42 UTC
Does this call actually nsenter to the node? Or is it looking inside the (master) container for the binaries?

Comment 4 hchen 2017-01-09 16:19:13 UTC
@eparis: this call happens on master (provisioner), so rbd binary (in ceph-common) has to be in master image.

Comment 11 Jianwei Hou 2017-01-25 08:08:57 UTC
Verified this is fixed in openshift v3.4.1.2

Comment 14 errata-xmlrpc 2017-01-31 20:19:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0218


Note You need to log in before you can comment on or make changes to this bug.