Bug 1914305
| Summary: | When having auto-added values with spaces in proxy.no_proxy install-config.yaml setting, machine-config operator fails to deploy | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Robert Heinzmann <rheinzma> |
| Component: | Installer | Assignee: | Patrick Dillon <padillon> |
| Installer sub component: | openshift-installer | QA Contact: | Gaoyun Pei <gpei> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | low | ||
| Priority: | low | CC: | bleanhar, kgarriso, mstaeble |
| Version: | 4.6 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-02-04 18:58:40 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This looks like a dupe of: https://bugzilla.redhat.com/show_bug.cgi?id=1909502 moving this over to general installer team to let them mark as dupe if that's the case Also a dupe of the closed: https://bugzilla.redhat.com/show_bug.cgi?id=1901034 it seems? We aren't going to allow spaces going forward as we work to improve validation in Bug #1873649 *** This bug has been marked as a duplicate of bug 1873649 *** |
Description of problem: When deploying OCP 4.6.9 IPI on OpenStack/AWS, the machine-config operator fails to successfully role out with: "machineconfig.machineconfiguration.openshift.io "rendered-master-d54c47d38b866846f2dd3a335ad01a4f" not found ~~~ [stack@osp16amd ocp-test1]$ oc get clusteroperator NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.6.9 True False False 21s cloud-credential 4.6.9 True False False 32m cluster-autoscaler 4.6.9 True False False 30m config-operator 4.6.9 True False False 31m console 4.6.9 True False False 5s csi-snapshot-controller 4.6.9 True False False 31m dns 4.6.9 True False False 30m etcd 4.6.9 True False False 30m image-registry 4.6.9 True False False 9m58s ingress 4.6.9 True False False 9m44s insights 4.6.9 True False False 31m kube-apiserver 4.6.9 True False False 29m kube-controller-manager 4.6.9 True False False 29m kube-scheduler 4.6.9 True False False 28m kube-storage-version-migrator 4.6.9 True False False 20m machine-api 4.6.9 True False False 27m machine-approver 4.6.9 True False False 30m machine-config False True True 31m marketplace 4.6.9 True False False 30m monitoring 4.6.9 True False False 4m13s network 4.6.9 True False False 31m node-tuning 4.6.9 True False False 31m openshift-apiserver 4.6.9 True False False 10m openshift-controller-manager 4.6.9 True False False 29m openshift-samples 4.6.9 True False False 28m operator-lifecycle-manager 4.6.9 True False False 30m operator-lifecycle-manager-catalog 4.6.9 True False False 30m operator-lifecycle-manager-packageserver 4.6.9 True False False 10m service-ca 4.6.9 True False False 31m storage 4.6.9 True False False 31m ~~~ MachineconfigPool degrated: ~~~ [stack@osp16amd ocp-test1]$ oc get machineconfigpool NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master False True True 3 0 0 3 32m worker rendered-worker-4db3e52768096fb16af1e6304d8410f8 True False False 2 2 2 0 32m ~~~ MachineConfigs generated by machineConfigOperator: ~~~ [stack@osp16amd ocp-test1]$ oc get machineconfig NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master eb9778355a9020673e8ce9aee092cb98d80cde5e 3.1.0 3m51s 00-worker eb9778355a9020673e8ce9aee092cb98d80cde5e 3.1.0 3m51s 01-master-container-runtime eb9778355a9020673e8ce9aee092cb98d80cde5e 3.1.0 3m51s 01-master-kubelet eb9778355a9020673e8ce9aee092cb98d80cde5e 3.1.0 3m51s 01-worker-container-runtime eb9778355a9020673e8ce9aee092cb98d80cde5e 3.1.0 3m51s 01-worker-kubelet eb9778355a9020673e8ce9aee092cb98d80cde5e 3.1.0 3m51s 99-master-generated-registries eb9778355a9020673e8ce9aee092cb98d80cde5e 3.1.0 3m51s 99-master-ssh 3.1.0 6m57s 99-worker-generated-registries eb9778355a9020673e8ce9aee092cb98d80cde5e 3.1.0 3m51s 99-worker-ssh 3.1.0 6m56s rendered-master-7f75f4f73559d4db30e35e6db2142ca9 eb9778355a9020673e8ce9aee092cb98d80cde5e 3.1.0 3m47s rendered-worker-4db3e52768096fb16af1e6304d8410f8 eb9778355a9020673e8ce9aee092cb98d80cde5e 3.1.0 3m47s ~~~ Master Node is labeled with a different MachineConfig by the installer ~~~ [stack@osp16amd ocp-test1]$ oc get node ocp-6jh95-master-0 -o json | jq -r .metadata.annotations { "machine.openshift.io/machine": "openshift-machine-api/ocp-6jh95-master-0", "machineconfiguration.openshift.io/currentConfig": "rendered-master-d54c47d38b866846f2dd3a335ad01a4f", "machineconfiguration.openshift.io/desiredConfig": "rendered-master-d54c47d38b866846f2dd3a335ad01a4f", "machineconfiguration.openshift.io/reason": "machineconfig.machineconfiguration.openshift.io \"rendered-master-d54c47d38b866846f2dd3a335ad01a4f\" not found", "machineconfiguration.openshift.io/state": "Degraded", "volumes.kubernetes.io/controller-managed-attach-detach": "true" } ~~~ Version-Release number of selected component (if applicable): OpenShift 4.6.9 IPI on Stack How reproducible: Always when the install-config.yaml proxy.no_proxy section contains records that are normally auto-added by the installer/cluster network operator (e.g. service network, metadata url and others) AND those records contain spaces Example: # Proxy NOTE: Check the spaces in front of 169.254.169.254 proxy: httpProxy: http://192.168.100.73:3128 httpsProxy: http://192.168.100.73:3128 noProxy: "x.x.x.x,apps.ocp.ocp-test1.osp16amd.x.x.x.x.nip.io, 169.254.169.254, 172.30.0.0/16,localhost" The auto-added records need to contain "spaces". Without spaces, the problem does not occur. Steps to Reproduce: 1. Prepare install-config.yaml with normally auto-added records and spaces 2. Deploy on a cloud provider with IPI 3. Verify that machine-config operator is not running Actual results: Machine config operator is degrated Expected results: Machine config operator is OK Additional info: Reason seems to be that Installer trims the spaces: https://github.com/openshift/installer/blob/release-4.6/pkg/asset/manifests/proxy.go#L195 Cluster Network Operator does not trim spaces: https://github.com/openshift/cluster-network-operator/blob/release-4.6/pkg/util/proxyconfig/no_proxy.go Mitigation is to remove the spaces.