Bug 1673989 - dns daemonset is not upgraded on cluster upgrade
Summary: dns daemonset is not upgraded on cluster upgrade
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.1.0
Assignee: Dan Mace
QA Contact: Hongan Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-08 16:30 UTC by Seth Jennings
Modified: 2019-06-04 10:43 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: coredns image update is not handled by the operator Consequence: After Cluster upgrade, openshift still has old coredns image. Fix: Update DNS operator to handle coredns image update. Result: Cluster upgrade will move the coredns image to the new version as expected.
Clone Of:
Environment:
Last Closed: 2019-06-04 10:42:43 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:43:51 UTC
Github openshift cluster-dns-operator pull 81 None None None 2019-03-07 22:16:39 UTC

Description Seth Jennings 2019-02-08 16:30:44 UTC
After upgrade, dns DS is not using the new image

$ oc get clusterversion
NAME      VERSION                           AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.alpha-2019-02-08-113402   True        False         31m     Cluster version is 4.0.0-0.alpha-2019-02-08-113402

$ oc get pod -oyaml dns-operator-7cd8dc9b-qrht5 
spec:
  containers:
  - command:
    - cluster-dns-operator
    env:
    - name: IMAGE
      value: registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-08-113402@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8
    - name: OPENSHIFT_CLI_IMAGE
      value: registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-08-113402@sha256:f9ef6603d89ba087703200e591d005df4fd0952bb0a93dc0e8ce6f9cb98221cd
    image: registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-08-113402@sha256:c26420542bb21106db26c3deaa1a44993c15e2f3d2c316ae079258dee4b47b5d



$ oc get ds -oyaml dns-default 
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  creationTimestamp: 2019-02-08T15:09:41Z
  generation: 1
  labels:
    openshift-app: dns
  name: dns-default
  namespace: openshift-dns
  resourceVersion: "37729"
  selfLink: /apis/extensions/v1beta1/namespaces/openshift-dns/daemonsets/dns-default
  uid: 8f0f4635-2bb3-11e9-8323-064ca5078f4a
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      dns: dns-default
      openshift-app: dns
  template:
    metadata:
      creationTimestamp: null
      labels:
        dns: dns-default
        openshift-app: dns
    spec:
      containers:
..
        image: registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-08-055616@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8  <--- still using old image
...
        image: registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-08-055616@sha256:f9ef6603d89ba087703200e591d005df4fd0952bb0a93dc0e8ce6f9cb98221cd  <--- still using old image

Comment 3 Hongan Li 2019-03-15 03:06:45 UTC
before upgrade:
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-13-233958   True        False         35m     Cluster version is 4.0.0-0.nightly-2019-03-13-233958

$ oc get deployment -o yaml -n openshift-dns-operator | grep image
          image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f4266bc3d567a91a4b14ae61fb63daf761c48ac537100470e77d0821db452501

$ oc get ds -o yaml -n openshift-dns | grep image
          image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4fcfa8e44f8049fd22a1d6392ac2b03a83e15b0904374a7c9f3b7a8c64581618

          image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b11ded775f893f9998d453b1da4351da39bb3ea9d9f4e698fb432fc5b6413d2

after upgrade:
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-14-040908   True        False         16h     Cluster version is 4.0.0-0.nightly-2019-03-14-040908

$ oc get deployment -o yaml -n openshift-dns-operator | grep image
          image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f4266bc3d567a91a4b14ae61fb63daf761c48ac537100470e77d0821db452501

$ oc get ds -o yaml -n openshift-dns | grep image
          image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4fcfa8e44f8049fd22a1d6392ac2b03a83e15b0904374a7c9f3b7a8c64581618

          image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b11ded775f893f9998d453b1da4351da39bb3ea9d9f4e698fb432fc5b6413d2

Test result: after upgrade, dns operator and DS is not using the new image.

Comment 4 Hongan Li 2019-03-15 03:17:59 UTC
And another issue is the operator's .stauts.conditions is not updated after upgrade.

$ oc get clusteroperator dns 
NAME   VERSION                             AVAILABLE   PROGRESSING   FAILING   SINCE
dns    4.0.0-0.nightly-2019-03-14-040908   True        False         False     23h
$ oc get clusteroperator dns -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-03-14T03:43:53Z
  generation: 1
  name: dns
  resourceVersion: "210756"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/dns
  uid: 6310aea5-460b-11e9-b25f-0645167dc34a
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-03-14T03:43:59Z
    status: "False"
    type: Failing
  - lastTransitionTime: 2019-03-14T03:43:59Z
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-03-14T03:44:21Z
    status: "True"
    type: Available
  extension: null
  relatedObjects:
  - group: ""
    name: openshift-dns-operator
    resource: namespaces
  - group: ""
    name: openshift-dns
    resource: namespaces
  versions:
  - name: operator
    version: 4.0.0-0.nightly-2019-03-14-040908
  - name: coredns
    version: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4fcfa8e44f8049fd22a1d6392ac2b03a83e15b0904374a7c9f3b7a8c64581618

Comment 5 Hongan Li 2019-03-15 09:39:49 UTC
verified with upgrading from 4.0.0-0.nightly-2019-03-13-233958 to 4.0.0-0.nightly-2019-03-15-043409.

the images is updated after upgrade

$ oc get deployment -n openshift-dns-operator -o yaml | grep quay -C 1
          - name: IMAGE
            value: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b9e3107ae30e589be00c2c28fab543e0c05a1dff5188de479cf37693834eabe1
          - name: OPENSHIFT_CLI_IMAGE
            value: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf1f556c9abacfa8ed8a2eacb10d47c5c369c4217d866e8a88cdccf2be33d17a
          image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8855628583840d826092c00fdbab15aa1129bd127c5545a988da5ce0ca67487
          imagePullPolicy: IfNotPresent

$ oc get ds -n openshift-dns -o yaml | grep quay 
          image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b9e3107ae30e589be00c2c28fab543e0c05a1dff5188de479cf37693834eabe1
          image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b11ded775f893f9998d453b1da4351da39bb3ea9d9f4e698fb432fc5b6413d2

the dns node resolver container is not updated but it has been tracked by https://bugzilla.redhat.com/show_bug.cgi?id=1687976, so this can be verified

Comment 8 errata-xmlrpc 2019-06-04 10:42:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.