Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1786024

Summary:	[OSP13withKuryr]Cannot pull image from image registry for no image registry internal address in /etc/hosts
Product:	OpenShift Container Platform	Reporter:	Wenjing Zheng <wzheng>
Component:	Networking	Assignee:	Michał Dulko <mdulko>
Networking sub component:	kuryr	QA Contact:	Jon Uriarte <juriarte>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	low	CC:	aos-bugs, juriarte, m.andre, scuppett
Version:	4.3.0
Target Milestone:	---
Target Release:	4.4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	1798869 (view as bug list)		Environment:
Last Closed:	2020-05-04 11:20:55 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1798869

Description Wenjing Zheng 2019-12-23 06:29:05 UTC

Description of problem:
Meet "lookup image-registry.openshift-image-registry.svc on 10.196.0.13:53: no such host" when build in an OSP13 cluster which is enabled Kuryr:
Events:
  Type     Reason     Age                From                                Message
  ----     ------     ----               ----                                -------
  Normal   Scheduled  <unknown>          default-scheduler                   Successfully assigned wzheng2/ruby-ex-3-4qkkq to ostest-2sf2t-worker-5r9z9
  Normal   Pulling    12s (x2 over 23s)  kubelet, ostest-2sf2t-worker-5r9z9  Pulling image "image-registry.openshift-image-registry.svc:5000/wzheng2/ruby-ex@sha256:beee8628f25b9a5e9d1dd54caee69c1e1ffe8cb29ba5cf8d3f402d88650d7903"
  Warning  Failed     12s (x2 over 23s)  kubelet, ostest-2sf2t-worker-5r9z9  Failed to pull image "image-registry.openshift-image-registry.svc:5000/wzheng2/ruby-ex@sha256:beee8628f25b9a5e9d1dd54caee69c1e1ffe8cb29ba5cf8d3f402d88650d7903": rpc error: code = Unknown desc = error pinging docker registry image-registry.openshift-image-registry.svc:5000: Get https://image-registry.openshift-image-registry.svc:5000/v2/: dial tcp: lookup image-registry.openshift-image-registry.svc on 10.196.0.29:53: no such host
  Warning  Failed     12s (x2 over 23s)  kubelet, ostest-2sf2t-worker-5r9z9  Error: ErrImagePull
  Normal   BackOff    0s (x3 over 23s)   kubelet, ostest-2sf2t-worker-5r9z9  Back-off pulling image "image-registry.openshift-image-registry.svc:5000/wzheng2/ruby-ex@sha256:beee8628f25b9a5e9d1dd54caee69c1e1ffe8cb29ba5cf8d3f402d88650d7903"
  Warning  Failed     0s (x3 over 23s)   kubelet, ostest-2sf2t-worker-5r9z9  Error: ImagePullBackOff


Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-12-22-054650

How reproducible:
Always

Steps to Reproduce:
1.Set up a OSP 13 cluster and enable kuryr
2.Create a build
3.Watch pod status

Actual results:
$ oc get pods
NAME               READY   STATUS             RESTARTS   AGE
ruby-ex-1-build    0/1     Completed          0          24m
ruby-ex-1-4qkkq    0/1     ImagePullBackOff   0          17s
ruby-ex-1-deploy   1/1     Running            0          33s

Expected results:
Pod should be running

Additional info:
There is a workaround to fix the issue by adding image registry internal address to /etc/hosts in each master/workder：
$ oc debug nodes/ostest-2sf2t-master-2
Starting pod/ostest-2sf2t-master-2-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.196.0.30
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# echo "172.30.179.121 image-registry.openshift-image-registry.svc image-registry.openshift-image-registry.svc.cluster.local # openshift-generated-node-resolver" >> /etc/hosts
sh-4.4# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.30.97.23 image-registry.openshift-image-registry.svc image-registry.openshift-image-registry.svc.cluster.local # openshift-generated-node-resolver

Comment 5 Jon Uriarte 2020-02-13 16:30:05 UTC

Verified in 4.4.0-0.nightly-2020-02-10-234204 on top of OSP 13 2020-01-15.3 puddle.

Build creation and image pull from internal registry works.

$ oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git

$ oc get pods                                                                                                                                                                   
NAME               READY   STATUS      RESTARTS   AGE
ruby-ex-1-build    0/1     Completed   0          6m22s
ruby-ex-1-cqw5h    1/1     Running     0          4m55s
ruby-ex-1-deploy   0/1     Completed   0          5m2s

$ oc describe pod ruby-ex-1-cqw5h
...
 Type    Reason     Age        From                                Message
  ----    ------     ----       ----                                -------
  Normal  Scheduled  <unknown>  default-scheduler                   Successfully assigned test2/ruby-ex-1-cqw5h to ostest-rgz26-worker-9sqkc
  Normal  Pulling    2m33s      kubelet, ostest-rgz26-worker-9sqkc  Pulling image "image-registry.openshift-image-registry.svc:5000/test2/ruby-ex@sha256:47f28e7b3b6031847f1ac44722e8b2688f7ebf503e0107e0537e11c36d2d7afd"
  Normal  Pulled     2m17s      kubelet, ostest-rgz26-worker-9sqkc  Successfully pulled image "image-registry.openshift-image-registry.svc:5000/test2/ruby-ex@sha256:47f28e7b3b6031847f1ac44722e8b2688f7ebf503e0107e0537e11c36d2d7afd"
  Normal  Created    2m16s      kubelet, ostest-rgz26-worker-9sqkc  Created container ruby-ex
  Normal  Started    2m16s      kubelet, ostest-rgz26-worker-9sqkc  Started container ruby-ex

Comment 7 errata-xmlrpc 2020-05-04 11:20:55 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581