Description of problem: https://github.com/kubernetes/kubernetes/issues/88046 Kubelet is quering different providers than the one that it's already configured, as should not be needed. The issue seems to be covered at upstream level and the bug should server as tracker as well. Customer felt there was security issue as there was connections from vsphere to AWS for example, even though the connection was blacklisted at proxy level. This part was clarified but still should be better to cover the upstream issue to improve this behaviour. Actual results: As an example, trying to connect to AWS EC2 endpoint innecessarily from vSphere cluster: Sep 09 14:14:54 master1.xxx hyperkube[1532]: E0909 14:14:54.057702 1532 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated. # cat /var/log/squid/access.log.5 | grep -v 200 238 x.x.x.84 TCP_DENIED/403 4038 PUT http://169.254.169.254/latest/api/token - HIER_NONE/- text/html 0 x.x.x.84 TCP_DENIED/403 4038 PUT http://169.254.169.254/latest/api/token - HIER_NONE/- text/html 0 x.x.x.84 TCP_DENIED/403 4038 PUT http://169.254.169.254/latest/api/token - HIER_NONE/- text/html 0 x.x.x.84 TCP_DENIED/403 4040 GET http://169.254.169.254/latest/meta-data/iam/security-credentials/ - HIER_NONE/- text/html 0 x.x.x.84 TCP_DENIED/403 4040 GET http://169.254.169.254/latest/meta-data/iam/security-credentials/ - HIER_NONE/- text/html 0 x.x.x.84 TCP_DENIED/403 4040 GET http://169.254.169.254/latest/meta-data/iam/security-credentials/ - HIER_NONE/- text/html Expected results: No connection should be tried.
Why has this been moved to the cloud-cred-operator component? CCO is responsible for taking a CredentialsRequest CR and creating (or copying depending on the underlying platform capabilities) IAM Users/credentials and storing those details in a Secret (as specified in the CredentialsRequest CR). CCO has no involvement in what codepaths kubelet takes during startup, and IIUC kubelet does not even rely on CredentialsRequest CRs for obtaining credentials to speak to the underlying platform/cloud.
The fix for this bug was already merged into the K8s 1.22 release which 4.9 is based upon. Therefore the fix is already in the nightly builds for 4.10 and can be moved straight to QA verification. For QE, please check that on a platform other than AWS, no kubelet starts up trying to fetch AWS credentials. Prior to this bug (eg 4.8) there should be a long delay during startup and then some AWS credential error, which should no longer happen.
verified checked 4.7.33-x86_64 must-gather, could found the connections info. $ grep -ir "NoCredentialProviders" . ./host_service_logs/masters/kubelet_service.log:Oct 09 08:51:31.445793 zhsun109osp1-89pfw-master-0 hyperkube[1822]: E1009 08:51:31.445647 1822 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated. ./host_service_logs/masters/kubelet_service.log:Oct 09 08:54:13.335526 zhsun109osp1-89pfw-master-2 hyperkube[1849]: E1009 08:54:13.335409 1849 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated. ./host_service_logs/masters/kubelet_service.log:Oct 09 08:54:53.636892 zhsun109osp1-89pfw-master-1 hyperkube[1844]: E1009 08:54:53.636617 1844 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated. 4.7.33-x86_64 checked 4.10.0-0.nightly-2021-10-28-211203 must-gather on gcp, no connection info $ grep -ir "NoCredentialProviders" .
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056