Bug 1761677
| Summary: | Cannot pull images from dockerhub in proxy testing environment | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | xiyuan |
| Component: | Networking | Assignee: | Ben Bennett <bbennett> |
| Networking sub component: | openshift-sdn | QA Contact: | zhaozhanqi <zzhao> |
| Status: | CLOSED WORKSFORME | Docs Contact: | |
| Severity: | low | ||
| Priority: | low | CC: | aos-bugs, bbennett, bparees, dmace, dmoessne, mfojtik, slaznick, xtian, yinzhou, zzhao |
| Version: | 4.2.0 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.4.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-04-28 15:46:50 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
xiyuan
2019-10-15 05:05:14 UTC
"Oct 14 13:43:09.242: INFO: At 2019-10-14 13:39:58 +0000 UTC - event for openldap-server-df855dfcb-qpk8p: {kubelet ip-10-0-132-1.ec2.internal} Failed: Failed to pull image "docker.io/mrogers950/origin-openldap-test:fedora29": rpc error: code = Unknown desc = Get https://auth.docker.io/token?scope=repository%3Amrogers950%2Forigin-openldap-test%3Apull&service=registry.docker.io: proxyconnect tcp: dial tcp 35.196.128.173:3128: connect: no route to host"
The above means that the test couldn't pull an image from DockerHub in proxy environment. I have no idea who to assign it to, so moving to routing who lead this feature.
I am able to pull and run the image using payload 4.3.0-0.okd-2019-10-14-190618: $ oc run -it openldap --image=mrogers950/origin-openldap-test:fedora29 kubectl run --generator=deploymentconfig/v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. If you don't see a command prompt, try pressing enter. 5da5ef11 conn=1000 fd=14 ACCEPT from IP=[::1]:48344 (IP=[::]:389) 5da5ef11 conn=1000 op=0 SRCH base="" scope=0 deref=0 filter="(objectClass=*)" 5da5ef11 conn=1000 op=0 SRCH attr=supportedSASLMechanisms 5da5ef11 conn=1000 op=0 SEARCH RESULT tag=101 err=0 nentries=1 text= 5da5ef11 conn=1000 op=1 UNBIND <SNIP> The "proxyconnect tcp: dial tcp 35.196.128.173:3128: connect: no route to host" error indicates that kubelet does not have a route to proxy 35.196.128.173:3128. Check the routing table on node ip-10-0-132-1.ec2.internal and make sure the default route is associated to the node's correct network interface and that the node can connect to the proxy. Also reproduced in: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-proxy-4.2/83 Caching blobs under "/var/cache/blobs". Pulling image busybox:latest ... Warning: Pull failed, retrying in 5s ... Warning: Pull failed, retrying in 5s ... Warning: Pull failed, retrying in 5s ... error: build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: while pulling "docker://busybox:latest" as "busybox:latest": Error initializing source docker://busybox:latest: pinging docker registry returned: Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp 35.196.128.173:3128: connect: no route to host <----end of log for "multi-stage-build"/"docker-build" *** Bug 1762164 has been marked as a duplicate of this bug. *** Eric Wolinetz increased the resource capacity of the proxy that gets created by the e2e-aws-proxy job. Please retest and update the bug based on your findings. The dns-operator manages /etc/resolv.conf on the nodes, which is the only way I can see that dns-operator could be relevant in this context. I see the IP is resolved, but the client can't open a socket to the registry IP. And, this connection doesn't involve ingress. I am thus reassigning to SDN. Please retest based on https://bugzilla.redhat.com/show_bug.cgi?id=1761677#c5 The proxy CI job is completely broken; I can't see if this is still an issue. not seeing lad test failures in current runs: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.2-informing#release-openshift-ocp-installer-e2e-aws-proxy-4.2&sort-by-failures= I do see the error message: https://search.svc.ci.openshift.org/?search=replica+for+openldap-server+not+avaiable&maxAge=336h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job but it only occurs in runs where many many things were going wrong. Closing this as working now. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |