Bug 1554379 - [vSphere] Sometimes PVC dynamic provision failed with error: NotAuthenticated
Summary: [vSphere] Sometimes PVC dynamic provision failed with error: NotAuthenticated
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.9.z
Assignee: Hemant Kumar
QA Contact: Jianwei Hou
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-12 15:03 UTC by Jianwei Hou
Modified: 2018-05-17 06:43 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-17 06:42:42 UTC
Target Upstream Version:

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1566 None None None 2018-05-17 06:43:17 UTC

Description Jianwei Hou 2018-03-12 15:03:33 UTC 
Description of problem:
Sometimes vSphere PVC dynamic provision could fail with error 'NotAuthenitcated', restarting controller brings things back to normal. No clear reproduce step yet. 

Googled this problem and found upstream issue and fix:
https://github.com/kubernetes/kubernetes-anywhere/issues/502
https://github.com/kubernetes/kubernetes/pull/58124

Version-Release number of selected component (if applicable):
openshift v3.9.7
kubernetes v1.9.1+a0ce1bc657

How reproducible:
Sometimes

Steps to Reproduce:
1. Create PVC for dynamic vSphere PV

Actual results:
Provision failed with error 'NotAuthenticated'.

Expected results:
PV provisioned.
Comment 1 Jianwei Hou 2018-03-13 05:51:50 UTC 
oc describe pvc vspherec1                              
Name:          vspherec1
Namespace:     jhou
StorageClass:  vspheredefault
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/vsphere-volume
Finalizers:    []
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Events:
  Type     Reason              Age   From                         Message
  ----     ------              ----  ----                         -------
  Warning  ProvisioningFailed  3s    persistentvolume-controller  Failed to provision volume with StorageClass "vspheredefault": NotAuthenticated
Comment 2 Hemant Kumar 2018-03-13 15:02:32 UTC 
Thank you for identifying the root cause and the fix. We will backport it. :-)
Comment 7 Jianwei Hou 2018-04-17 08:15:09 UTC 
Verified on 3.9.20, this is not reproducible now after many tries.
Comment 9 Jianwei Hou 2018-04-24 03:00:45 UTC 
Verified on 3.9.24
Comment 10 guilherme.camposo 2018-05-02 20:52:14 UTC 
(In reply to Hemant Kumar from comment #2)
> Thank you for identifying the root cause and the fix. We will backport it.
> :-)

Hi Hermant, 

I have a customer that has the same problem. Here are the results from "oc version": 

oc v3.9.14
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://haproxy.rhocp.buy4.io:8443
openshift v3.9.14
kubernetes v1.9.1+a0ce1bc657

Do you have any advice on how we can fix it on client's installation? 

Thanks
Comment 15 errata-xmlrpc 2018-05-17 06:42:42 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1566
Note You need to log in before you can comment on or make changes to this bug.