Bug 1821945

Summary: vCenter hostname must be all lower case
Product: OpenShift Container Platform Reporter: rvanderp
Component: Cloud ComputeAssignee: Danil Grigorev <dgrigore>
Cloud Compute sub component: Other Providers QA Contact: Jianwei Hou <jhou>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: unspecified CC: agarcial, aos-bugs, mfojtik
Version: 4.5   
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-13 17:26:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description rvanderp 2020-04-07 22:17:09 UTC
Description of problem:
kube-controller-manager is unable to locate the credentials of the vCenter host, even though they are defined and cased identically to the hostname in the cloud-provider-config.

Version-Release number of selected component (if applicable):
4.2.20 UPI on VMWare

How reproducible:

Steps to Reproduce:
1. Attempt to provision storage with the "thin" storage class
2. PVC will stay suck in pending

Actual results:
vCenter hostname is case-sensitive in a way that is opaque to administrators

Expected results:
vCenter hostname should not be case-sensitive, or should be documented that it is.

Additional info:

Log Messages:
E0318 18:58:01.956856       1 connection.go:65] Failed to create govmomi client. err: ServerFaultCode: Cannot complete login due to an incorrect user name or password.
E0318 18:58:01.956888       1 credentialmanager.go:79] credentials not found for server "VCENTER.host.local"
E0318 18:58:01.956899       1 nodemanager.go:404] Failed to get credentials from Secret Credential Manager with err: Credentials not found
E0318 18:58:01.956907       1 vsphere.go:616] failed connecting to vcServer "VCENTER.host.local" with error Credentials not found
I0318 18:58:01.956947       1 pv_controller.go:1492] failed to provision volume for claim "openshift-monitoring/openshift-monitoring-alertman-pvc-alertmanager-main-2" with StorageClass "thin": Credentials not found
E0318 18:58:01.956980       1 goroutinemap.go:150] Operation for "provision-openshift-monitoring/openshift-monitoring-alertman-pvc-alertmanager-main-2[9c8615ed-6916-11ea-a483-005056184020]" failed. No retries permitted until 2020-03-18 19:00:03.956966958 +0000 UTC m=+113504.770261005 (durationBeforeRetry 2m2s). Error: "Credentials not found"

This appears to be induced by the way the vCenter server name is calculated when the secret containing the credentials is parsed[https://github.com/kubernetes/kubernetes/blob/release-1.15/staging/src/k8s.io/legacy-cloud-providers/vsphere/credentialmanager.go#L138].  The credentialKey is converted to lower case and is then parsed to lookup the server name.

Comment 1 Maciej Szulik 2020-04-08 09:12:28 UTC
These bits are handled by the cloud team, moving accordingly.

Comment 2 Danil Grigorev 2020-05-06 11:36:35 UTC
PR to fix: https://github.com/kubernetes/kubernetes/pull/90796

Comment 6 Jianwei Hou 2020-05-20 06:26:52 UTC
QE do not have a vSphere with vCenter hostname in uppercase, therefore unit tested this.

--- PASS: TestParseSecretConfig (0.00s)
    credentialmanager_test.go:372: Executing Testcase: Valid username and password
    credentialmanager_test.go:372: Executing Testcase: Invalid username key with valid password key
    credentialmanager_test.go:372: Executing Testcase: Missing username
    credentialmanager_test.go:372: Executing Testcase: Missing password
    credentialmanager_test.go:372: Executing Testcase: FQIN stored as lowercase
    credentialmanager_test.go:372: Executing Testcase: IP with unknown key
ok  	k8s.io/legacy-cloud-providers/vsphere	0.404s

Comment 7 errata-xmlrpc 2020-07-13 17:26:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.