Bug 1859360
| Summary: | [behind proxy] When installing without direct egress access mcd fails pulling its image | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | ewolinet |
| Component: | Machine Config Operator | Assignee: | Yu Qi Zhang <jerzhang> |
| Status: | CLOSED NOTABUG | QA Contact: | Michael Nguyen <mnguyen> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 4.6 | CC: | jerzhang, wking |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-07-23 17:02:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Description of problem: When trying to install a cluster in a subnet that does not have direct egress access, we fail during bootstrapping and find the following output from the machine-config-daemon-pull-service: ● machine-config-daemon-pull.service - Machine Config Daemon Pull Loaded: loaded (/etc/systemd/system/machine-config-daemon-pull.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2020-07-20 23:18:30 UTC; 26min ago Process: 1628 ExecStart=/bin/sh -c /usr/bin/podman pull --authfile=/var/lib/kubelet/config.json --quiet 'registry.svc.ci.openshift.org/ci-op-dlxy481c/stable@sha256:b96429f4c9b57756eee93aa76c23c3e7ba41fe009122ac298703cbcc65d331f0' (code=exited, status=125) Process: 1614 ExecStart=/bin/sh -c /bin/mkdir -p /run/bin && chcon --reference=/usr/bin /run/bin (code=exited, status=0/SUCCESS) Main PID: 1628 (code=exited, status=125) CPU: 270ms Jul 20 23:17:30 ip-10-0-51-54 systemd[1]: Starting Machine Config Daemon Pull... Jul 20 23:18:30 ip-10-0-51-54 sh[1628]: Error: error pulling image "registry.svc.ci.openshift.org/ci-op-dlxy481c/stable@sha256:b96429f4c9b57756eee93aa76c23c3e7ba41fe009122ac298703cbcc65d331f0": unable to pull registry.svc.ci.openshift.org/ci-op-dlxy481c/stable@sha256:b96429f4c9b57756eee93aa76c23c3e7ba41fe009122ac298703cbcc65d331f0: unable to pull image: Error initializing source docker://registry.svc.ci.openshift.org/ci-op-dlxy481c/stable@sha256:b96429f4c9b57756eee93aa76c23c3e7ba41fe009122ac298703cbcc65d331f0: error pinging docker registry registry.svc.ci.openshift.org: Get https://registry.svc.ci.openshift.org/v2/: proxyconnect tcp: dial tcp 52.53.162.131:3128: i/o timeout Jul 20 23:18:30 ip-10-0-51-54 systemd[1]: machine-config-daemon-pull.service: Main process exited, code=exited, status=125/n/a Jul 20 23:18:30 ip-10-0-51-54 systemd[1]: machine-config-daemon-pull.service: Failed with result 'exit-code'. Jul 20 23:18:30 ip-10-0-51-54 systemd[1]: Failed to start Machine Config Daemon Pull. Jul 20 23:18:30 ip-10-0-51-54 systemd[1]: machine-config-daemon-pull.service: Consumed 270ms CPU time Version-Release number of selected component (if applicable): 4.6 How reproducible: Currently always Steps to Reproduce: 1. Configure squid proxy 2. Using step registry blackhole subnet list [1] set install config using these subnets 3. Install Actual results: Time out during bootstrap when control planes are being stood up Expected results: Install to complete Additional info: This was successful when installing behind the same proxy and not on a private subnet that prevented direct egress. Discovered as part of https://github.com/openshift/release/pull/10355 [1] https://github.com/openshift/release/blob/324d057d2934caddf9a90793efeb94c75b7afad4/ci-operator/step-registry/ipi/conf/aws/blackholenetwork/ipi-conf-aws-blackholenetwork-commands.sh#L17-L20