1554379 – [vSphere] Sometimes PVC dynamic provision failed with error: NotAuthenticated

Bug 1554379 - [vSphere] Sometimes PVC dynamic provision failed with error: NotAuthenticated

Summary: [vSphere] Sometimes PVC dynamic provision failed with error: NotAuthenticated

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Storage
Sub Component:
Version:	3.9.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	3.9.z
Assignee:	Hemant Kumar
QA Contact:	Jianwei Hou
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-03-12 15:03 UTC by Jianwei Hou
Modified:	2018-05-17 06:43 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-05-17 06:42:42 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2018:1566	0	None	None	None	2018-05-17 06:43:17 UTC

Description Jianwei Hou 2018-03-12 15:03:33 UTC

Description of problem:
Sometimes vSphere PVC dynamic provision could fail with error 'NotAuthenitcated', restarting controller brings things back to normal. No clear reproduce step yet. 

Googled this problem and found upstream issue and fix:
https://github.com/kubernetes/kubernetes-anywhere/issues/502
https://github.com/kubernetes/kubernetes/pull/58124

Version-Release number of selected component (if applicable):
openshift v3.9.7
kubernetes v1.9.1+a0ce1bc657

How reproducible:
Sometimes

Steps to Reproduce:
1. Create PVC for dynamic vSphere PV

Actual results:
Provision failed with error 'NotAuthenticated'.

Expected results:
PV provisioned.

Comment 1 Jianwei Hou 2018-03-13 05:51:50 UTC

oc describe pvc vspherec1                              
Name:          vspherec1
Namespace:     jhou
StorageClass:  vspheredefault
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/vsphere-volume
Finalizers:    []
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Events:
  Type     Reason              Age   From                         Message
  ----     ------              ----  ----                         -------
  Warning  ProvisioningFailed  3s    persistentvolume-controller  Failed to provision volume with StorageClass "vspheredefault": NotAuthenticated

Comment 2 Hemant Kumar 2018-03-13 15:02:32 UTC

Thank you for identifying the root cause and the fix. We will backport it. :-)

Comment 7 Jianwei Hou 2018-04-17 08:15:09 UTC

Verified on 3.9.20, this is not reproducible now after many tries.

Comment 9 Jianwei Hou 2018-04-24 03:00:45 UTC

Verified on 3.9.24

Comment 10 guilherme.camposo 2018-05-02 20:52:14 UTC

(In reply to Hemant Kumar from comment #2)
> Thank you for identifying the root cause and the fix. We will backport it.
> :-)

Hi Hermant, 

I have a customer that has the same problem. Here are the results from "oc version": 

oc v3.9.14
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://haproxy.rhocp.buy4.io:8443
openshift v3.9.14
kubernetes v1.9.1+a0ce1bc657

Do you have any advice on how we can fix it on client's installation? 

Thanks

Comment 15 errata-xmlrpc 2018-05-17 06:42:42 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1566

Note You need to log in before you can comment on or make changes to this bug.