Bug 1463499 - app's dc is pulling image from registry by IP but not by DNS.
app's dc is pulling image from registry by IP but not by DNS.
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry (Show other bugs)
Unspecified Unspecified
high Severity high
: ---
: 3.7.0
Assigned To: Michal Fojtik
ge liu
: 1466583 (view as bug list)
Depends On:
Blocks: 1466784
  Show dependency treegraph
Reported: 2017-06-21 02:59 EDT by Johnny Liu
Modified: 2017-11-28 16:58 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1466784 (view as bug list)
Last Closed: 2017-11-28 16:58:09 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-28 21:34:54 EST

  None (edit)
Description Johnny Liu 2017-06-21 02:59:20 EDT
Description of problem:
Found this issue when testing https://trello.com/c/TK6FwQ6k/314-2-push-to-registry-by-dns, one goal of this user story is allow user does not have to restart master service once registry svc IP is changed. While in my testing, app's dc is trying pulling image from registry svc orignal IP but not registry svc DNS.

Version-Release number of selected component (if applicable):
openshift v3.6.121
kubernetes v1.6.1+5115d708d7
etcd 3.2.0

How reproducible:

Steps to Reproduce:
1. Trigger a fresh installation with single master.
2. After installation, trigger a sti build.
# oc new-app nodejs-mongodb-example -n install-test
3. check build log, sti build is pushed to registry by DNS successfully.
4. delete docker-registry svc, and re-expose it to make its svc IP is changed.
# oc get svc docker-registry
docker-registry   <none>        5000/TCP   1h

# oc delete svc docker-registry
service "docker-registry" deleted

# oc expose dc docker-registry
service "docker-registry" exposed

# oc get svc docker-registry
NAME              CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
docker-registry   <none>        5000/TCP   5s

5. scale down replica to 0 for the app.
# oc scale --replicas=0 dc nodejs-mongodb-example -n install-test
6. Go to node, delete the pulled sti build image locally.
# docker images
REPOSITORY                                                             TAG                 IMAGE ID            CREATED             SIZE
docker-registry.default.svc:5000/install-test/nodejs-mongodb-example   latest              5d3c16cde11d        About an hour ago   460.6 MB                  <none>              5d3c16cde11d        About an hour ago   460.6 MB

# docker rmi 5d3c16cde11d -f
Untagged: docker-registry.default.svc:5000/install-test/nodejs-mongodb-example:latest
Untagged: docker-registry.default.svc:5000/install-test/nodejs-mongodb-example@sha256:933471ee23af23a8f2890880b1be2e26e0d9c3d2373f9e1c7eac1c958d28ea67
Deleted: sha256:5d3c16cde11d6c48e65e19c59481a3c2d2d60336088db7309cd4665ca4c75f78

7. scale up replica to 1 for the app again.

Actual results:
Failed to be scaled up.
# oc get po -n install-test
NAME                             READY     STATUS         RESTARTS   AGE
mongodb-1-2kk0n                  1/1       Running        0          1h
nodejs-mongodb-example-1-build   0/1       Completed      0          1h
nodejs-mongodb-example-1-vg5qf   0/1       ErrImagePull   0          16s

# oc describe po nodejs-mongodb-example-1-vg5qf -n install-test
  FirstSeen	LastSeen	Count	From						SubObjectPath				Type		Reason			Message
  ---------	--------	-----	----						-------------				--------	------			-------
  33s		33s		1	default-scheduler									Normal		Scheduled		Successfully assigned nodejs-mongodb-example-1-vg5qf to qe-36-smoke-master-registry-router-1
  27s		27s		1	kubelet, qe-36-smoke-master-registry-router-1	spec.containers{nodejs-mongodb-example}	Normal		BackOff			Back-off pulling image ""
  33s		16s		4	kubelet, qe-36-smoke-master-registry-router-1						Warning		DNSSearchForming	Found and omitted duplicated dns domain in host search line: 'cluster.local' during merging with cluster dns domains
  30s		16s		2	kubelet, qe-36-smoke-master-registry-router-1	spec.containers{nodejs-mongodb-example}	Normal		Pulling			pulling image ""
  27s		13s		2	kubelet, qe-36-smoke-master-registry-router-1	spec.containers{nodejs-mongodb-example}	Warning		Failed			Failed to pull image "": rpc error: code = 2 desc = Get dial tcp getsockopt: no route to host
  27s		13s		3	kubelet, qe-36-smoke-master-registry-router-1						Warning		FailedSync		Error syncing pod

# oc get dc nodejs-mongodb-example -n install-test -o yaml
apiVersion: v1
kind: DeploymentConfig
    description: Defines how to deploy the application server
    openshift.io/generated-by: OpenShiftNewApp
  creationTimestamp: 2017-06-21T04:07:59Z
  generation: 5
    app: nodejs-mongodb-example
    template: nodejs-mongodb-example
  name: nodejs-mongodb-example
  namespace: install-test
  resourceVersion: "4246"
  selfLink: /oapi/v1/namespaces/install-test/deploymentconfigs/nodejs-mongodb-example
  uid: 36be4ea4-5637-11e7-8b48-42010af00013
  replicas: 1
    name: nodejs-mongodb-example
    activeDeadlineSeconds: 21600
      timeoutSeconds: 600
    resources: {}
    type: Recreate
        openshift.io/generated-by: OpenShiftNewApp
      creationTimestamp: null
        app: nodejs-mongodb-example
        name: nodejs-mongodb-example
      name: nodejs-mongodb-example
      - env:
          value: mongodb
        - name: MONGODB_USER
              key: database-user
              name: nodejs-mongodb-example
        - name: MONGODB_PASSWORD
              key: database-password
              name: nodejs-mongodb-example
        - name: MONGODB_DATABASE
          value: sampledb
              key: database-admin-password
              name: nodejs-mongodb-example
        imagePullPolicy: IfNotPresent
          failureThreshold: 3
            path: /pagecount
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        name: nodejs-mongodb-example
        - containerPort: 8080
          protocol: TCP
          failureThreshold: 3
            path: /pagecount
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 3
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
            memory: 512Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  test: false
  - imageChangeParams:
      automatic: true
      - nodejs-mongodb-example
        kind: ImageStreamTag
        name: nodejs-mongodb-example:latest
        namespace: install-test
    type: ImageChange
  - type: ConfigChange
  availableReplicas: 0
  - lastTransitionTime: 2017-06-21T04:11:08Z
    lastUpdateTime: 2017-06-21T04:11:28Z
    message: replication controller "nodejs-mongodb-example-1" successfully rolled
    reason: NewReplicationControllerAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: 2017-06-21T06:01:44Z
    lastUpdateTime: 2017-06-21T06:01:44Z
    message: Deployment config does not have minimum availability.
    status: "False"
    type: Available
    - imageTrigger:
          kind: ImageStreamTag
          name: nodejs-mongodb-example:latest
          namespace: install-test
      type: ImageChange
    message: image change
  latestVersion: 1
  observedGeneration: 5
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

In the above app dc yaml file, app's image is pointed to registry svc IP but not registry svc DNS.

Expected results:
In app's dc, image is be pointed to registry svc DNS but not IP.

Additional info:
Comment 1 Johnny Liu 2017-06-21 03:47:13 EDT
Once the above issue happened, restart master service, re-trigger sti build, then image in app's dc will be pointed to registry svc DNS. That means app's dc image is pointed to registry svc IP only happened in the initial moment when app's dc is created in the first time.
Comment 2 Scott Dodson 2017-06-21 09:46:30 EDT
If this is an HA environment maybe this is a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1463498 ? I'll investigate today.
Comment 3 Johnny Liu 2017-06-21 22:27:45 EDT
(In reply to Scott Dodson from comment #2)
> If this is an HA environment maybe this is a dupe of
> https://bugzilla.redhat.com/show_bug.cgi?id=1463498 ? I'll investigate today.

No, this is a single master env. Totally different bug with BZ#1463498. Pls refer to reproduce steps, in a multiple master HA env, step 3 will fail, but in this single master env, step 3 is working well.
Comment 4 Scott Dodson 2017-06-26 09:42:21 EDT
Any idea what's going on here?
Comment 5 Michal Fojtik 2017-06-26 09:58:09 EDT
I guess the image stream still had the IP address in the pull spec and by restarting this and rebuilding the image stream pull spec was updated to point to the registry by DNS. Was this cluster live before setting the DEFAULT_REGISTRY variable or this happens on a clean cluster with the registry set?
Comment 6 Michal Fojtik 2017-06-26 10:03:52 EDT
Can we see the image stream when this breaks? If the image stream has recorded the the IP address in pull spec... The image stream is the source of truth for deployment configs.
Comment 7 Michal Fojtik 2017-06-26 10:50:16 EDT
PR: https://github.com/openshift/origin/pull/14882

After this, you have to set the OPENSHIFT_DEFAULT_REGISTRY variable for the docker-registry DC in order for the integrated registry using the DNS name over the IP address (which is the default).
Currently this is hard-coded into registry image entrypoint and it should not.
I guess what this cause is the registry everytime it updates the image stream it replaces the DNS with the IP address. However with the OPENSHIFT_DEFAULT_REGISTRY set for master, when the master API updates the image stream (oc tag or build?) it will revert the IP address back to the DNS.

This should be tested more to make sure we don't break image streams (otherwise we will have to run ugly scripts replacing the broken IP's).
Comment 8 Scott Dodson 2017-06-26 11:28:59 EDT
The master should have OPENSHIFT_DEFAULT_REGISTRY=docker-registry.default.svc:5000 set before that image stream is even created.

I'll watch that PR.
Comment 9 Scott Dodson 2017-06-28 14:29:31 EDT
Since I don't think there's any additional work required of the installer here I'm going to move this to Image Registry component.
Comment 11 Michal Fojtik 2017-07-03 04:34:10 EDT
*** Bug 1466583 has been marked as a duplicate of this bug. ***
Comment 14 Scott Dodson 2017-07-12 08:46:02 EDT
Yes the variable should be set in the registry DC now.

Comment 17 Johnny Liu 2017-07-17 04:46:37 EDT
According to comment 15, move this bug to verified state.
Comment 21 errata-xmlrpc 2017-11-28 16:58:09 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.