Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1411244 - Unable to provision rbd volumes in containerized OpenShift, "err: executable file not found in $PATH"
Unable to provision rbd volumes in containerized OpenShift, "err: executable ...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage (Show other bugs)
3.4.0
Unspecified Unspecified
high Severity high
: ---
: 3.4.z
Assigned To: hchen
Jianwei Hou
:
Depends On:
Blocks: 1420698
  Show dependency treegraph
 
Reported: 2017-01-09 04:03 EST by Jianwei Hou
Modified: 2017-02-09 05:31 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The ceph-common client tools were missing from the containerized node image which prevented containerized environments from mounting ceph volumes. The package has been added enabling containerized environments to mount ceph volumes.
Story Points: ---
Clone Of:
: 1420698 (view as bug list)
Environment:
Last Closed: 2017-01-31 15:19:41 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0218 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.4.1.2 bug fix update 2017-01-31 20:18:20 EST

  None (edit)
Description Jianwei Hou 2017-01-09 04:03:16 EST
Description of problem:
In containerzied OpenShift cluster, rbd volumes could not be dynamically provisioned.  PVC events logged failure reasion "rbd: create volume failed, err: executable file not found in $PATH"

Version-Release number of selected component (if applicable):
openshift v3.4.0.39
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

How reproducible:
Always

Steps to Reproduce:
1. Setup containerized OpenShift cluster, setup ceph rbd storage server
2. Create a StorageClass for Ceph RBD provisioner
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  creationTimestamp: 2017-01-09T06:54:58Z
  name: cephrbdprovisioner
  resourceVersion: "8131"
  selfLink: /apis/storage.k8s.io/v1beta1/storageclasses/cephrbdprovisioner
  uid: 892a27c2-d638-11e6-800d-0eb7d2142d46
parameters:
  adminId: admin
  adminSecretName: cephrbd-secret
  adminSecretNamespace: default
  monitors: xxx:6789
  pool: rbd
  userId: admin
  userSecretName: cephrbd-user-secret
provisioner: kubernetes.io/rbd

3. Create a PVC to dynamically provision a PV
{
   "kind": "PersistentVolumeClaim",
   "apiVersion": "v1",
   "metadata": {
     "name": "rbdc",
     "annotations": {
     "volume.beta.kubernetes.io/storage-class": "cephrbdprovisioner"
     }
   },
   "spec": {
     "accessModes": [
       "ReadWriteOnce"
     ],
     "resources": {
       "requests": {
         "storage": "5Gi"
       }
     }
   }
}


Actual results:
After step 3: PVC is Pending, PV could not be provisioned.
 oc describe pvc rbdc
Name:		rbdc
Namespace:	jhou1
StorageClass:	cephrbdprovisioner
Status:		Pending
Volume:		
Labels:		<none>
Capacity:	
Access Modes:	
Events:
  FirstSeen	LastSeen	Count	From				SubobjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  18s		5s		2	{persistentvolume-controller }			Warning		ProvisioningFailed	Failed to provision volume with StorageClass "cephrbdprovisioner": rbd: create volume failed, err: executable file not found in $PATH


Expected results:
Should provision successfully.

Additional info:
Comment 1 Bradley Childs 2017-01-09 11:07:54 EST
This sounds like missing RBD packages on the nodes.  

Huamin - can you figure out the packages needed on each node?
Comment 2 Eric Paris 2017-01-09 11:15:42 EST
Does this call actually nsenter to the node? Or is it looking inside the (master) container for the binaries?
Comment 4 hchen 2017-01-09 11:19:13 EST
@eparis: this call happens on master (provisioner), so rbd binary (in ceph-common) has to be in master image.
Comment 11 Jianwei Hou 2017-01-25 03:08:57 EST
Verified this is fixed in openshift v3.4.1.2
Comment 14 errata-xmlrpc 2017-01-31 15:19:41 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0218

Note You need to log in before you can comment on or make changes to this bug.