Bug 1554379

Summary:	[vSphere] Sometimes PVC dynamic provision failed with error: NotAuthenticated
Product:	OpenShift Container Platform	Reporter:	Jianwei Hou <jhou>
Component:	Storage	Assignee:	Hemant Kumar <hekumar>
Status:	CLOSED ERRATA	QA Contact:	Jianwei Hou <jhou>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	3.9.0	CC:	aos-bugs, aos-storage-staff, bchilds, bleanhar, guilherme.camposo, hekumar, jhou, vcorrea
Target Milestone:	---
Target Release:	3.9.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-05-17 06:42:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jianwei Hou 2018-03-12 15:03:33 UTC

Description of problem:
Sometimes vSphere PVC dynamic provision could fail with error 'NotAuthenitcated', restarting controller brings things back to normal. No clear reproduce step yet. 

Googled this problem and found upstream issue and fix:
https://github.com/kubernetes/kubernetes-anywhere/issues/502
https://github.com/kubernetes/kubernetes/pull/58124

Version-Release number of selected component (if applicable):
openshift v3.9.7
kubernetes v1.9.1+a0ce1bc657

How reproducible:
Sometimes

Steps to Reproduce:
1. Create PVC for dynamic vSphere PV

Actual results:
Provision failed with error 'NotAuthenticated'.

Expected results:
PV provisioned.

Comment 1 Jianwei Hou 2018-03-13 05:51:50 UTC

oc describe pvc vspherec1                              
Name:          vspherec1
Namespace:     jhou
StorageClass:  vspheredefault
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/vsphere-volume
Finalizers:    []
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Events:
  Type     Reason              Age   From                         Message
  ----     ------              ----  ----                         -------
  Warning  ProvisioningFailed  3s    persistentvolume-controller  Failed to provision volume with StorageClass "vspheredefault": NotAuthenticated

Comment 2 Hemant Kumar 2018-03-13 15:02:32 UTC

Thank you for identifying the root cause and the fix. We will backport it. :-)

Comment 7 Jianwei Hou 2018-04-17 08:15:09 UTC

Verified on 3.9.20, this is not reproducible now after many tries.

Comment 9 Jianwei Hou 2018-04-24 03:00:45 UTC

Verified on 3.9.24

Comment 10 guilherme.camposo 2018-05-02 20:52:14 UTC

(In reply to Hemant Kumar from comment #2)
> Thank you for identifying the root cause and the fix. We will backport it.
> :-)

Hi Hermant, 

I have a customer that has the same problem. Here are the results from "oc version": 

oc v3.9.14
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://haproxy.rhocp.buy4.io:8443
openshift v3.9.14
kubernetes v1.9.1+a0ce1bc657

Do you have any advice on how we can fix it on client's installation? 

Thanks

Comment 15 errata-xmlrpc 2018-05-17 06:42:42 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1566