Bug 1857162
| Summary: | Fails to update behind proxy | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Sunil Choudhary <schoudha> |
| Component: | Machine Config Operator | Assignee: | Yu Qi Zhang <jerzhang> |
| Status: | CLOSED ERRATA | QA Contact: | Michael Nguyen <mnguyen> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 4.6 | CC: | amurdaca, mkrejci, walters, wking, xxia, yanyang |
| Target Milestone: | --- | Keywords: | TestBlocker |
| Target Release: | 4.6.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-10-27 16:14:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1867908 | ||
|
Comment 1
Antonio Murdaca
2020-07-15 10:09:44 UTC
Possibly a regression from https://github.com/openshift/machine-config-operator/pull/1766 though we do have the proxy config stuff in the unit: {{if .Proxy -}} (In reply to Colin Walters from comment #6) > Possibly a regression from > https://github.com/openshift/machine-config-operator/pull/1766 > though we do have the proxy config stuff in the unit: > {{if .Proxy -}} I think changing the unit name here https://github.com/openshift/machine-config-operator/pull/1766/files#diff-95e83e4216073d5ba6d128c764d05756R323 is what makes it _not_ use the proxy variables. Could that be? I don't see anything that specifies proxy for "mco-pivot" > I think changing the unit name here https://github.com/openshift/machine-config-operator/pull/1766/files#diff-95e83e4216073d5ba6d128c764d05756R323 is what makes it _not_ use the proxy variables. Could that be? I don't see anything that specifies proxy for "mco-pivot"
Ah yes, sounds like the bug indeed! Hmm. I guess we could ship a static unit for this instead of using `systemd-run`. Or we could inject the proxy variables.
The annoying thing is that `systemd-run` has clearer semantics here than blocking on `systemctl start` on a statically defined unit (e.g. what happens if the MCD pod dies in the middle of an update).
In the end we probably want to use the systemd DBus API via bindings, it's just a lot more code.
Verified on 4.6.0-0.nightly-2020-09-17-113547. Created 4.5.1 cluster with same profile (UPI on Azure with http_proxy, fips, etcd_encryption) with which I was able to reproduce this issue. It got successfully upgraded to 4.6.0-0.nightly-2020-09-17-113547
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.5.1 True False 59m Cluster version is 4.5.1
$ oc get nodes
NAME STATUS ROLES AGE VERSION
sunilc-bz-09171420-master-0 Ready master 84m v1.18.3+6025c28
sunilc-bz-09171420-master-1 Ready master 83m v1.18.3+6025c28
sunilc-bz-09171420-master-2 Ready master 84m v1.18.3+6025c28
sunilc-bz-09171420-worker-centralus-1 Ready worker 68m v1.18.3+6025c28
sunilc-bz-09171420-worker-centralus-2 Ready worker 67m v1.18.3+6025c28
sunilc-bz-09171420-worker-centralus-3 Ready worker 68m v1.18.3+6025c28
$ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-17-113547 --force --allow-explicit-upgrade
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to preceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to release image registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-17-113547
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.6.0-0.nightly-2020-09-17-113547 True False 2m49s Cluster version is 4.6.0-0.nightly-2020-09-17-113547
$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
sunilc-bz-09171420-master-0 Ready master 159m v1.19.0+b4ffb45 10.0.0.6 <none> Red Hat Enterprise Linux CoreOS 46.82.202009170240-0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.19.0-18.rhaos4.6.gitd802e19.el8
sunilc-bz-09171420-master-1 Ready master 159m v1.19.0+b4ffb45 10.0.0.8 <none> Red Hat Enterprise Linux CoreOS 46.82.202009170240-0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.19.0-18.rhaos4.6.gitd802e19.el8
sunilc-bz-09171420-master-2 Ready master 159m v1.19.0+b4ffb45 10.0.0.7 <none> Red Hat Enterprise Linux CoreOS 46.82.202009170240-0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.19.0-18.rhaos4.6.gitd802e19.el8
sunilc-bz-09171420-worker-centralus-1 Ready worker 143m v1.19.0+b4ffb45 10.0.1.5 <none> Red Hat Enterprise Linux CoreOS 46.82.202009170240-0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.19.0-18.rhaos4.6.gitd802e19.el8
sunilc-bz-09171420-worker-centralus-2 Ready worker 142m v1.19.0+b4ffb45 10.0.1.6 <none> Red Hat Enterprise Linux CoreOS 46.82.202009170240-0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.19.0-18.rhaos4.6.gitd802e19.el8
sunilc-bz-09171420-worker-centralus-3 Ready worker 143m v1.19.0+b4ffb45 10.0.1.4 <none> Red Hat Enterprise Linux CoreOS 46.82.202009170240-0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.19.0-18.rhaos4.6.gitd802e19.el8
$ oc describe clusterversion
Name: version
Namespace:
Labels: <none>
Annotations: <none>
API Version: config.openshift.io/v1
Kind: ClusterVersion
Metadata:
Creation Timestamp: 2020-09-17T14:46:48Z
Generation: 2
Managed Fields:
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:channel:
f:clusterID:
f:upstream:
Manager: cluster-bootstrap
Operation: Update
Time: 2020-09-17T14:46:48Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:desiredUpdate:
.:
f:force:
f:image:
f:version:
Manager: oc
Operation: Update
Time: 2020-09-17T16:16:36Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:availableUpdates:
f:conditions:
f:desired:
.:
f:force:
f:image:
f:version:
f:history:
f:observedGeneration:
f:versionHash:
Manager: cluster-version-operator
Operation: Update
Time: 2020-09-17T17:26:08Z
Resource Version: 146814
Self Link: /apis/config.openshift.io/v1/clusterversions/version
UID: fa1053d2-b077-43ce-acfc-f67b65428b90
Spec:
Channel: stable-4.5
Cluster ID: 00548598-816a-4846-860b-257c6a811102
Desired Update:
Force: true
Image: registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-17-113547
Version:
Upstream: https://api.openshift.com/api/upgrades_info/v1/graph
Status:
Available Updates: <nil>
Conditions:
Last Transition Time: 2020-09-17T15:15:30Z
Message: Done applying 4.6.0-0.nightly-2020-09-17-113547
Status: True
Type: Available
Last Transition Time: 2020-09-17T17:21:22Z
Status: False
Type: Failing
Last Transition Time: 2020-09-17T17:26:08Z
Message: Cluster version is 4.6.0-0.nightly-2020-09-17-113547
Status: False
Type: Progressing
Last Transition Time: 2020-09-17T16:16:59Z
Message: Unable to retrieve available updates: currently reconciling cluster version 4.6.0-0.nightly-2020-09-17-113547 not found in the "stable-4.5" channel
Reason: VersionNotFound
Status: False
Type: RetrievedUpdates
Last Transition Time: 2020-09-17T15:33:06Z
Message: Cluster operator marketplace cannot be upgraded between minor versions: The cluster has custom OperatorSource, which is deprecated in future versions. Please visit this link for further details: https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated
Reason: DeprecatedAPIsInUse
Status: False
Type: Upgradeable
Desired:
Image: registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-17-113547
Version: 4.6.0-0.nightly-2020-09-17-113547
History:
Completion Time: 2020-09-17T17:26:08Z
Image: registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-17-113547
Started Time: 2020-09-17T16:16:45Z
State: Completed
Verified: false
Version: 4.6.0-0.nightly-2020-09-17-113547
Completion Time: 2020-09-17T15:15:30Z
Image: quay.io/openshift-release-dev/ocp-release@sha256:a656048696e79a30f0536f5acd5a1e8ec5ae331d4c7d21ca62bc8de412c79dc4
Started Time: 2020-09-17T14:46:52Z
State: Completed
Verified: false
Version: 4.5.1
Observed Generation: 2
Version Hash: HIbzcE_ZWqc=
Events: <none>
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1]. If you feel like this bug still needs to be a suspect, please add keyword again. [1]: https://github.com/openshift/enhancements/pull/475 |