Bug 1857162 - Fails to update behind proxy
Summary: Fails to update behind proxy
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.6
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.6.0
Assignee: Yu Qi Zhang
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On:
Blocks: 1867908
TreeView+ depends on / blocked
 
Reported: 2020-07-15 09:55 UTC by Sunil Choudhary
Modified: 2020-09-17 17:31 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2037 None closed Bug 1857162: daemon: inject proxy vars into MCD container 2020-09-14 01:37:09 UTC

Comment 1 Antonio Murdaca 2020-07-15 10:09:44 UTC
This is a regression, we'll work on it.

Comment 6 Colin Walters 2020-07-21 19:32:43 UTC
Possibly a regression from https://github.com/openshift/machine-config-operator/pull/1766
though we do have the proxy config stuff in the unit:
  {{if .Proxy -}}

Comment 8 Antonio Murdaca 2020-07-27 08:40:23 UTC
(In reply to Colin Walters from comment #6)
> Possibly a regression from
> https://github.com/openshift/machine-config-operator/pull/1766
> though we do have the proxy config stuff in the unit:
>   {{if .Proxy -}}

I think changing the unit name here https://github.com/openshift/machine-config-operator/pull/1766/files#diff-95e83e4216073d5ba6d128c764d05756R323 is what makes it _not_ use the proxy variables. Could that be? I don't see anything that specifies proxy for "mco-pivot"

Comment 9 Colin Walters 2020-07-27 13:49:30 UTC
> I think changing the unit name here https://github.com/openshift/machine-config-operator/pull/1766/files#diff-95e83e4216073d5ba6d128c764d05756R323 is what makes it _not_ use the proxy variables. Could that be? I don't see anything that specifies proxy for "mco-pivot"

Ah yes, sounds like the bug indeed!  Hmm.  I guess we could ship a static unit for this instead of using `systemd-run`.  Or we could inject the proxy variables.

The annoying thing is that `systemd-run` has clearer semantics here than blocking on `systemctl start` on a statically defined unit (e.g. what happens if the MCD pod dies in the middle of an update).
In the end we probably want to use the systemd DBus API via bindings, it's just a lot more code.

Comment 20 Sunil Choudhary 2020-09-17 17:31:50 UTC
Verified on 4.6.0-0.nightly-2020-09-17-113547. Created 4.5.1 cluster with same profile (UPI on Azure with http_proxy, fips, etcd_encryption) with which I was able to reproduce this issue. It got successfully upgraded to 4.6.0-0.nightly-2020-09-17-113547

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.1     True        False         59m     Cluster version is 4.5.1

$ oc get nodes
NAME                                    STATUS   ROLES    AGE   VERSION
sunilc-bz-09171420-master-0             Ready    master   84m   v1.18.3+6025c28
sunilc-bz-09171420-master-1             Ready    master   83m   v1.18.3+6025c28
sunilc-bz-09171420-master-2             Ready    master   84m   v1.18.3+6025c28
sunilc-bz-09171420-worker-centralus-1   Ready    worker   68m   v1.18.3+6025c28
sunilc-bz-09171420-worker-centralus-2   Ready    worker   67m   v1.18.3+6025c28
sunilc-bz-09171420-worker-centralus-3   Ready    worker   68m   v1.18.3+6025c28

$ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-17-113547 --force --allow-explicit-upgrade
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to preceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to release image registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-17-113547

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-09-17-113547   True        False         2m49s   Cluster version is 4.6.0-0.nightly-2020-09-17-113547

$ oc get nodes -o wide
NAME                                    STATUS   ROLES    AGE    VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
sunilc-bz-09171420-master-0             Ready    master   159m   v1.19.0+b4ffb45   10.0.0.6      <none>        Red Hat Enterprise Linux CoreOS 46.82.202009170240-0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.19.0-18.rhaos4.6.gitd802e19.el8
sunilc-bz-09171420-master-1             Ready    master   159m   v1.19.0+b4ffb45   10.0.0.8      <none>        Red Hat Enterprise Linux CoreOS 46.82.202009170240-0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.19.0-18.rhaos4.6.gitd802e19.el8
sunilc-bz-09171420-master-2             Ready    master   159m   v1.19.0+b4ffb45   10.0.0.7      <none>        Red Hat Enterprise Linux CoreOS 46.82.202009170240-0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.19.0-18.rhaos4.6.gitd802e19.el8
sunilc-bz-09171420-worker-centralus-1   Ready    worker   143m   v1.19.0+b4ffb45   10.0.1.5      <none>        Red Hat Enterprise Linux CoreOS 46.82.202009170240-0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.19.0-18.rhaos4.6.gitd802e19.el8
sunilc-bz-09171420-worker-centralus-2   Ready    worker   142m   v1.19.0+b4ffb45   10.0.1.6      <none>        Red Hat Enterprise Linux CoreOS 46.82.202009170240-0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.19.0-18.rhaos4.6.gitd802e19.el8
sunilc-bz-09171420-worker-centralus-3   Ready    worker   143m   v1.19.0+b4ffb45   10.0.1.4      <none>        Red Hat Enterprise Linux CoreOS 46.82.202009170240-0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.19.0-18.rhaos4.6.gitd802e19.el8

$ oc describe clusterversion
Name:         version
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  config.openshift.io/v1
Kind:         ClusterVersion
Metadata:
  Creation Timestamp:  2020-09-17T14:46:48Z
  Generation:          2
  Managed Fields:
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:channel:
        f:clusterID:
        f:upstream:
    Manager:      cluster-bootstrap
    Operation:    Update
    Time:         2020-09-17T14:46:48Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:desiredUpdate:
          .:
          f:force:
          f:image:
          f:version:
    Manager:      oc
    Operation:    Update
    Time:         2020-09-17T16:16:36Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:availableUpdates:
        f:conditions:
        f:desired:
          .:
          f:force:
          f:image:
          f:version:
        f:history:
        f:observedGeneration:
        f:versionHash:
    Manager:         cluster-version-operator
    Operation:       Update
    Time:            2020-09-17T17:26:08Z
  Resource Version:  146814
  Self Link:         /apis/config.openshift.io/v1/clusterversions/version
  UID:               fa1053d2-b077-43ce-acfc-f67b65428b90
Spec:
  Channel:     stable-4.5
  Cluster ID:  00548598-816a-4846-860b-257c6a811102
  Desired Update:
    Force:    true
    Image:    registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-17-113547
    Version:  
  Upstream:   https://api.openshift.com/api/upgrades_info/v1/graph
Status:
  Available Updates:  <nil>
  Conditions:
    Last Transition Time:  2020-09-17T15:15:30Z
    Message:               Done applying 4.6.0-0.nightly-2020-09-17-113547
    Status:                True
    Type:                  Available
    Last Transition Time:  2020-09-17T17:21:22Z
    Status:                False
    Type:                  Failing
    Last Transition Time:  2020-09-17T17:26:08Z
    Message:               Cluster version is 4.6.0-0.nightly-2020-09-17-113547
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2020-09-17T16:16:59Z
    Message:               Unable to retrieve available updates: currently reconciling cluster version 4.6.0-0.nightly-2020-09-17-113547 not found in the "stable-4.5" channel
    Reason:                VersionNotFound
    Status:                False
    Type:                  RetrievedUpdates
    Last Transition Time:  2020-09-17T15:33:06Z
    Message:               Cluster operator marketplace cannot be upgraded between minor versions: The cluster has custom OperatorSource, which is deprecated in future versions. Please visit this link for further details: https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated
    Reason:                DeprecatedAPIsInUse
    Status:                False
    Type:                  Upgradeable
  Desired:
    Image:    registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-17-113547
    Version:  4.6.0-0.nightly-2020-09-17-113547
  History:
    Completion Time:    2020-09-17T17:26:08Z
    Image:              registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-17-113547
    Started Time:       2020-09-17T16:16:45Z
    State:              Completed
    Verified:           false
    Version:            4.6.0-0.nightly-2020-09-17-113547
    Completion Time:    2020-09-17T15:15:30Z
    Image:              quay.io/openshift-release-dev/ocp-release@sha256:a656048696e79a30f0536f5acd5a1e8ec5ae331d4c7d21ca62bc8de412c79dc4
    Started Time:       2020-09-17T14:46:52Z
    State:              Completed
    Verified:           false
    Version:            4.5.1
  Observed Generation:  2
  Version Hash:         HIbzcE_ZWqc=
Events:                 <none>


Note You need to log in before you can comment on or make changes to this bug.