1840460 – Applying "ctrcfg" causes cri-o to fail to upgrade a cluster

Bug 1840460 - Applying "ctrcfg" causes cri-o to fail to upgrade a cluster

Summary: Applying "ctrcfg" causes cri-o to fail to upgrade a cluster

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Node
Sub Component:
Version:	4.3.0
Hardware:	x86_64
OS:	Linux
Priority:	low
Severity:	low
Target Milestone:	---
Target Release:	4.3.z
Assignee:	Ryan Phillips
QA Contact:	Sunil Choudhary
Docs Contact:
URL:
Whiteboard:
Depends On:	1838606 1840458
Blocks:	1840461
TreeView+	depends on / blocked

Reported:	2020-05-27 01:25 UTC by Urvashi Mohnani
Modified:	2020-06-17 20:28 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1840458
Clones:	1840461 (view as bug list)
Environment:
Last Closed:	2020-06-17 20:28:32 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2020:2436	0	None	None	None	2020-06-17 20:28:49 UTC

Comment 2 Sunil Choudhary 2020-05-28 13:55:09 UTC

Upgraded from 4.3 to 4.4 with custom ContainerRuntimeConfig.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.0-0.nightly-2020-05-27-224458   True        False         40m     Cluster version is 4.3.0-0.nightly-2020-05-27-224458

$ oc label machineconfigpool worker custom=pids-limit
machineconfigpool.machineconfiguration.openshift.io/worker labeled

$ oc create -f ctr.yaml 
containerruntimeconfig.machineconfiguration.openshift.io/custom-ctrcfg created

$ oc get mc
NAME                                                              GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
00-master                                                         dc885ffbf8bc475591c955fb4f9504b4d6f2ffb2   2.2.0             51m
00-worker                                                         dc885ffbf8bc475591c955fb4f9504b4d6f2ffb2   2.2.0             51m
01-master-container-runtime                                       dc885ffbf8bc475591c955fb4f9504b4d6f2ffb2   2.2.0             51m
01-master-kubelet                                                 dc885ffbf8bc475591c955fb4f9504b4d6f2ffb2   2.2.0             51m
01-worker-container-runtime                                       dc885ffbf8bc475591c955fb4f9504b4d6f2ffb2   2.2.0             51m
01-worker-kubelet                                                 dc885ffbf8bc475591c955fb4f9504b4d6f2ffb2   2.2.0             51m
99-master-236615e2-0c64-4f56-bfa3-53268d22f16a-registries         dc885ffbf8bc475591c955fb4f9504b4d6f2ffb2   2.2.0             51m
99-master-ssh                                                                                                2.2.0             52m
99-worker-477f1374-daff-4810-ad95-2cf58a2af37e-containerruntime   dc885ffbf8bc475591c955fb4f9504b4d6f2ffb2   2.2.0             5s
99-worker-477f1374-daff-4810-ad95-2cf58a2af37e-registries         dc885ffbf8bc475591c955fb4f9504b4d6f2ffb2   2.2.0             51m
99-worker-ssh                                                                                                2.2.0             52m
rendered-master-65275f7cf2c2d13a1967e00bf98d0c8c                  dc885ffbf8bc475591c955fb4f9504b4d6f2ffb2   2.2.0             51m
rendered-worker-0eef6d250c1b6251659ccb17a922c164                  dc885ffbf8bc475591c955fb4f9504b4d6f2ffb2   2.2.0             5s
rendered-worker-243b34f8a0b721598d53ee427dd14c67                  dc885ffbf8bc475591c955fb4f9504b4d6f2ffb2   2.2.0             51m

$ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-05-28-003808 --allow-explicit-upgrade=true --force
Updating to release image registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-05-28-003808

$  oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-05-28-003808   True        False         2m31s   Cluster version is 4.4.0-0.nightly-2020-05-28-003808

$ oc get clusterversion -o yaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2020-05-28T06:53:43Z"
    generation: 2
    name: version
    resourceVersion: "61001"
    selfLink: /apis/config.openshift.io/v1/clusterversions/version
    uid: 57623dd0-55a6-42af-83a3-4bcff84fe108
  spec:
    channel: stable-4.3
    clusterID: d9a2183d-8ca5-48d1-afdc-659da48b1ecf
    desiredUpdate:
      force: true
      image: registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-05-28-003808
      version: ""
    upstream: https://api.openshift.com/api/upgrades_info/v1/graph
  status:
    availableUpdates: null
    conditions:
    - lastTransitionTime: "2020-05-28T07:09:36Z"
      message: Done applying 4.4.0-0.nightly-2020-05-28-003808
      status: "True"
      type: Available
    - lastTransitionTime: "2020-05-28T08:42:16Z"
      status: "False"
      type: Failing
    - lastTransitionTime: "2020-05-28T08:43:36Z"
      message: Cluster version is 4.4.0-0.nightly-2020-05-28-003808
      status: "False"
      type: Progressing
    - lastTransitionTime: "2020-05-28T06:53:50Z"
      message: 'Unable to retrieve available updates: currently installed version
        4.4.0-0.nightly-2020-05-28-003808 not found in the "stable-4.3" channel'
      reason: VersionNotFound
      status: "False"
      type: RetrievedUpdates
    - lastTransitionTime: "2020-05-28T08:11:57Z"
      message: 'Cluster operator marketplace cannot be upgraded: The cluster has custom
        OperatorSource/CatalogSourceConfig, which are deprecated in future versions.
        Please visit this link for further deatils: https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated'
      reason: DprecatedAPIsInUse
      status: "False"
      type: Upgradeable
    desired:
      force: true
      image: registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-05-28-003808
      version: 4.4.0-0.nightly-2020-05-28-003808
    history:
    - completionTime: "2020-05-28T08:43:36Z"
      image: registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-05-28-003808
      startedTime: "2020-05-28T07:58:34Z"
      state: Completed
      verified: false
      version: 4.4.0-0.nightly-2020-05-28-003808
    - completionTime: "2020-05-28T07:09:36Z"
      image: registry.svc.ci.openshift.org/ocp/release@sha256:6c5511bf47e22638511439514e4a965c31730368032513b1a5545c4b7da1f8fa
      startedTime: "2020-05-28T06:53:50Z"
      state: Completed
      verified: false
      version: 4.3.0-0.nightly-2020-05-27-224458
    observedGeneration: 2
    versionHash: iYnFebCUr2Y=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

$ oc get containerruntimeconfig -o yaml
apiVersion: v1
items:
- apiVersion: machineconfiguration.openshift.io/v1
  kind: ContainerRuntimeConfig
  metadata:
    creationTimestamp: "2020-05-28T07:50:34Z"
    finalizers:
    - 99-worker-477f1374-daff-4810-ad95-2cf58a2af37e-containerruntime
    generation: 1
    name: custom-ctrcfg
    resourceVersion: "27976"
    selfLink: /apis/machineconfiguration.openshift.io/v1/containerruntimeconfigs/custom-ctrcfg
    uid: f7d4c642-e6d5-4fa0-8183-00467db61e5a
  spec:
    containerRuntimeConfig:
      pidsLimit: 8192
    machineConfigPoolSelector:
      matchLabels:
        custom: pids-limit
  status:
    conditions:
    - lastTransitionTime: "2020-05-28T07:50:34Z"
      message: Success
      status: "True"
      type: Success
    observedGeneration: 1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

sh-4.4# cat /etc/crio/crio.conf | grep -v "#" | grep -i log_dir
log_dir = "/var/log/crio/pods"

Comment 7 errata-xmlrpc 2020-06-17 20:28:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2436

Note You need to log in before you can comment on or make changes to this bug.