Bug 1395547

Summary: Provisioned PV logged an event for volume creation failure
Product: OpenShift Container Platform Reporter: Jianwei Hou <jhou>
Component: StorageAssignee: Jan Safranek <jsafrane>
Status: CLOSED ERRATA QA Contact: Jianwei Hou <jhou>
Severity: low Docs Contact:
Priority: low    
Version: 3.4.0CC: aos-bugs, bchilds, jsafrane, smunilla
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-10 05:15:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jianwei Hou 2016-11-16 07:47:27 UTC
Description of problem:
Sometimes, the provisioned PV logs an event for failed volume creation. PV and volume are proivisioned successfully, but an event for the error is left there.

Version-Release number of selected component (if applicable):
openshift v3.4.0.26+f7e109e
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

How reproducible:
Sometimes

Steps to Reproduce:
1. Create StorageClass for Ceph RBD  provisioner
2. Create PVC
3. After PV is provisioned, run 'oc describe pv'

Actual results:
Name:           cephrbdprovisioner
Namespace:      default
StorageClass:   cephrbdprovisioner
Status:         Bound
Volume:         pvc-e40020d8-abc9-11e6-9cd7-0eeaaf62bca4
Labels:         <none>
Capacity:       5Gi
Access Modes:   RWO
Events:
  FirstSeen     LastSeen        Count   From                            SubobjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                            -------------   --------        ------                  -------
  33m           33m             1       {persistentvolume-controller }                  Warning         ProvisioningFailed      Failed to provision volume with StorageClass "cephrbdprovisioner": rbd: create volume failed, err: exit status 1


Expected results:
It seems reasonable that the provisioner failed at first try but later succeeded so an failure event was logged, not sure why sometimes it failed at provisioning. Reported this bug just in case customer get confused seeing this.

Additional info:

Comment 1 Jan Safranek 2016-11-23 13:08:30 UTC
jhou, do you have any logs from the kube-controller-manager (openshift-master) or ceph why it failed? We should log something better than "exit status 1".

Also, adding an event "provisioned volume xxx using volume plugin yyy" would not harm either so the users can see that we recovered from the error.

Comment 2 Jan Safranek 2016-11-23 13:32:57 UTC
(In reply to Jan Safranek from comment #1)
> Also, adding an event "provisioned volume xxx using volume plugin yyy" would
> not harm either so the users can see that we recovered from the error.

Filled https://github.com/kubernetes/kubernetes/issues/37377 and https://trello.com/c/E1uljgPb/369-internal-provisioners-should-report-success-as-an-event


Now the ambiguous "exit status 1" from RBD. We should make it nicer.

Comment 3 Jan Safranek 2016-11-23 14:41:02 UTC
(In reply to Jan Safranek from comment #1)
> We should log something better than "exit status 1".

Pushed PR https://github.com/kubernetes/kubernetes/pull/37380, scheduled for 1.6. Please raise the priority if you want it earlier.

Comment 5 Jan Safranek 2017-02-20 16:01:26 UTC
Almost missed this one, Origin PR: https://github.com/openshift/origin/pull/13017

Comment 7 Jianwei Hou 2017-06-13 06:25:21 UTC
This is not reproduced in latest ocp. Mark as verified.

Comment 9 errata-xmlrpc 2017-08-10 05:15:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716