Bug 1786024 - [OSP13withKuryr]Cannot pull image from image registry for no image registry internal address in /etc/hosts
Summary: [OSP13withKuryr]Cannot pull image from image registry for no image registry i...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
: 4.4.0
Assignee: Michał Dulko
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On:
Blocks: 1798869
TreeView+ depends on / blocked
 
Reported: 2019-12-23 06:29 UTC by Wenjing Zheng
Modified: 2020-05-04 11:21 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1798869 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:20:55 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-dns-operator pull 153 None closed Bug 1786024: Try TCP DNS in dns-node-resolver 2020-06-18 16:27:24 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:21:35 UTC

Description Wenjing Zheng 2019-12-23 06:29:05 UTC
Description of problem:
Meet "lookup image-registry.openshift-image-registry.svc on 10.196.0.13:53: no such host" when build in an OSP13 cluster which is enabled Kuryr:
Events:
  Type     Reason     Age                From                                Message
  ----     ------     ----               ----                                -------
  Normal   Scheduled  <unknown>          default-scheduler                   Successfully assigned wzheng2/ruby-ex-3-4qkkq to ostest-2sf2t-worker-5r9z9
  Normal   Pulling    12s (x2 over 23s)  kubelet, ostest-2sf2t-worker-5r9z9  Pulling image "image-registry.openshift-image-registry.svc:5000/wzheng2/ruby-ex@sha256:beee8628f25b9a5e9d1dd54caee69c1e1ffe8cb29ba5cf8d3f402d88650d7903"
  Warning  Failed     12s (x2 over 23s)  kubelet, ostest-2sf2t-worker-5r9z9  Failed to pull image "image-registry.openshift-image-registry.svc:5000/wzheng2/ruby-ex@sha256:beee8628f25b9a5e9d1dd54caee69c1e1ffe8cb29ba5cf8d3f402d88650d7903": rpc error: code = Unknown desc = error pinging docker registry image-registry.openshift-image-registry.svc:5000: Get https://image-registry.openshift-image-registry.svc:5000/v2/: dial tcp: lookup image-registry.openshift-image-registry.svc on 10.196.0.29:53: no such host
  Warning  Failed     12s (x2 over 23s)  kubelet, ostest-2sf2t-worker-5r9z9  Error: ErrImagePull
  Normal   BackOff    0s (x3 over 23s)   kubelet, ostest-2sf2t-worker-5r9z9  Back-off pulling image "image-registry.openshift-image-registry.svc:5000/wzheng2/ruby-ex@sha256:beee8628f25b9a5e9d1dd54caee69c1e1ffe8cb29ba5cf8d3f402d88650d7903"
  Warning  Failed     0s (x3 over 23s)   kubelet, ostest-2sf2t-worker-5r9z9  Error: ImagePullBackOff


Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-12-22-054650

How reproducible:
Always

Steps to Reproduce:
1.Set up a OSP 13 cluster and enable kuryr
2.Create a build
3.Watch pod status

Actual results:
$ oc get pods
NAME               READY   STATUS             RESTARTS   AGE
ruby-ex-1-build    0/1     Completed          0          24m
ruby-ex-1-4qkkq    0/1     ImagePullBackOff   0          17s
ruby-ex-1-deploy   1/1     Running            0          33s

Expected results:
Pod should be running

Additional info:
There is a workaround to fix the issue by adding image registry internal address to /etc/hosts in each master/workder:
$ oc debug nodes/ostest-2sf2t-master-2
Starting pod/ostest-2sf2t-master-2-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.196.0.30
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# echo "172.30.179.121 image-registry.openshift-image-registry.svc image-registry.openshift-image-registry.svc.cluster.local # openshift-generated-node-resolver" >> /etc/hosts
sh-4.4# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.30.97.23 image-registry.openshift-image-registry.svc image-registry.openshift-image-registry.svc.cluster.local # openshift-generated-node-resolver

Comment 5 Jon Uriarte 2020-02-13 16:30:05 UTC
Verified in 4.4.0-0.nightly-2020-02-10-234204 on top of OSP 13 2020-01-15.3 puddle.

Build creation and image pull from internal registry works.

$ oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git

$ oc get pods                                                                                                                                                                   
NAME               READY   STATUS      RESTARTS   AGE
ruby-ex-1-build    0/1     Completed   0          6m22s
ruby-ex-1-cqw5h    1/1     Running     0          4m55s
ruby-ex-1-deploy   0/1     Completed   0          5m2s

$ oc describe pod ruby-ex-1-cqw5h
...
 Type    Reason     Age        From                                Message
  ----    ------     ----       ----                                -------
  Normal  Scheduled  <unknown>  default-scheduler                   Successfully assigned test2/ruby-ex-1-cqw5h to ostest-rgz26-worker-9sqkc
  Normal  Pulling    2m33s      kubelet, ostest-rgz26-worker-9sqkc  Pulling image "image-registry.openshift-image-registry.svc:5000/test2/ruby-ex@sha256:47f28e7b3b6031847f1ac44722e8b2688f7ebf503e0107e0537e11c36d2d7afd"
  Normal  Pulled     2m17s      kubelet, ostest-rgz26-worker-9sqkc  Successfully pulled image "image-registry.openshift-image-registry.svc:5000/test2/ruby-ex@sha256:47f28e7b3b6031847f1ac44722e8b2688f7ebf503e0107e0537e11c36d2d7afd"
  Normal  Created    2m16s      kubelet, ostest-rgz26-worker-9sqkc  Created container ruby-ex
  Normal  Started    2m16s      kubelet, ostest-rgz26-worker-9sqkc  Started container ruby-ex

Comment 7 errata-xmlrpc 2020-05-04 11:20:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.