Bug 1673989
| Summary: | dns daemonset is not upgraded on cluster upgrade | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Seth Jennings <sjenning> |
| Component: | Networking | Assignee: | Dan Mace <dmace> |
| Networking sub component: | router | QA Contact: | Hongan Li <hongli> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | urgent | ||
| Priority: | urgent | CC: | aos-bugs, dmace |
| Version: | 4.1.0 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.1.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: coredns image update is not handled by the operator
Consequence: After Cluster upgrade, openshift still has old coredns image.
Fix: Update DNS operator to handle coredns image update.
Result: Cluster upgrade will move the coredns image to the new version as expected.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-06-04 10:42:43 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
before upgrade:
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.0.0-0.nightly-2019-03-13-233958 True False 35m Cluster version is 4.0.0-0.nightly-2019-03-13-233958
$ oc get deployment -o yaml -n openshift-dns-operator | grep image
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f4266bc3d567a91a4b14ae61fb63daf761c48ac537100470e77d0821db452501
$ oc get ds -o yaml -n openshift-dns | grep image
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4fcfa8e44f8049fd22a1d6392ac2b03a83e15b0904374a7c9f3b7a8c64581618
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b11ded775f893f9998d453b1da4351da39bb3ea9d9f4e698fb432fc5b6413d2
after upgrade:
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.0.0-0.nightly-2019-03-14-040908 True False 16h Cluster version is 4.0.0-0.nightly-2019-03-14-040908
$ oc get deployment -o yaml -n openshift-dns-operator | grep image
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f4266bc3d567a91a4b14ae61fb63daf761c48ac537100470e77d0821db452501
$ oc get ds -o yaml -n openshift-dns | grep image
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4fcfa8e44f8049fd22a1d6392ac2b03a83e15b0904374a7c9f3b7a8c64581618
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b11ded775f893f9998d453b1da4351da39bb3ea9d9f4e698fb432fc5b6413d2
Test result: after upgrade, dns operator and DS is not using the new image.
And another issue is the operator's .stauts.conditions is not updated after upgrade.
$ oc get clusteroperator dns
NAME VERSION AVAILABLE PROGRESSING FAILING SINCE
dns 4.0.0-0.nightly-2019-03-14-040908 True False False 23h
$ oc get clusteroperator dns -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
creationTimestamp: 2019-03-14T03:43:53Z
generation: 1
name: dns
resourceVersion: "210756"
selfLink: /apis/config.openshift.io/v1/clusteroperators/dns
uid: 6310aea5-460b-11e9-b25f-0645167dc34a
spec: {}
status:
conditions:
- lastTransitionTime: 2019-03-14T03:43:59Z
status: "False"
type: Failing
- lastTransitionTime: 2019-03-14T03:43:59Z
status: "False"
type: Progressing
- lastTransitionTime: 2019-03-14T03:44:21Z
status: "True"
type: Available
extension: null
relatedObjects:
- group: ""
name: openshift-dns-operator
resource: namespaces
- group: ""
name: openshift-dns
resource: namespaces
versions:
- name: operator
version: 4.0.0-0.nightly-2019-03-14-040908
- name: coredns
version: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4fcfa8e44f8049fd22a1d6392ac2b03a83e15b0904374a7c9f3b7a8c64581618
verified with upgrading from 4.0.0-0.nightly-2019-03-13-233958 to 4.0.0-0.nightly-2019-03-15-043409.
the images is updated after upgrade
$ oc get deployment -n openshift-dns-operator -o yaml | grep quay -C 1
- name: IMAGE
value: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b9e3107ae30e589be00c2c28fab543e0c05a1dff5188de479cf37693834eabe1
- name: OPENSHIFT_CLI_IMAGE
value: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf1f556c9abacfa8ed8a2eacb10d47c5c369c4217d866e8a88cdccf2be33d17a
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8855628583840d826092c00fdbab15aa1129bd127c5545a988da5ce0ca67487
imagePullPolicy: IfNotPresent
$ oc get ds -n openshift-dns -o yaml | grep quay
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b9e3107ae30e589be00c2c28fab543e0c05a1dff5188de479cf37693834eabe1
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b11ded775f893f9998d453b1da4351da39bb3ea9d9f4e698fb432fc5b6413d2
the dns node resolver container is not updated but it has been tracked by https://bugzilla.redhat.com/show_bug.cgi?id=1687976, so this can be verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758 |
After upgrade, dns DS is not using the new image $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.alpha-2019-02-08-113402 True False 31m Cluster version is 4.0.0-0.alpha-2019-02-08-113402 $ oc get pod -oyaml dns-operator-7cd8dc9b-qrht5 spec: containers: - command: - cluster-dns-operator env: - name: IMAGE value: registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-08-113402@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8 - name: OPENSHIFT_CLI_IMAGE value: registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-08-113402@sha256:f9ef6603d89ba087703200e591d005df4fd0952bb0a93dc0e8ce6f9cb98221cd image: registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-08-113402@sha256:c26420542bb21106db26c3deaa1a44993c15e2f3d2c316ae079258dee4b47b5d $ oc get ds -oyaml dns-default apiVersion: extensions/v1beta1 kind: DaemonSet metadata: creationTimestamp: 2019-02-08T15:09:41Z generation: 1 labels: openshift-app: dns name: dns-default namespace: openshift-dns resourceVersion: "37729" selfLink: /apis/extensions/v1beta1/namespaces/openshift-dns/daemonsets/dns-default uid: 8f0f4635-2bb3-11e9-8323-064ca5078f4a spec: revisionHistoryLimit: 10 selector: matchLabels: dns: dns-default openshift-app: dns template: metadata: creationTimestamp: null labels: dns: dns-default openshift-app: dns spec: containers: .. image: registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-08-055616@sha256:e4936a702d7d466a64a6a9359f35c7ad528bba7c35fe5c582a90e46f9051d8b8 <--- still using old image ... image: registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-08-055616@sha256:f9ef6603d89ba087703200e591d005df4fd0952bb0a93dc0e8ce6f9cb98221cd <--- still using old image