1838606 – Applying "ctrcfg" causes cri-o to fail to upgrade a cluster

Bug 1838606 - Applying "ctrcfg" causes cri-o to fail to upgrade a cluster

Summary: Applying "ctrcfg" causes cri-o to fail to upgrade a cluster

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Node
Sub Component:
Version:	4.3.z
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.5.0
Assignee:	Urvashi Mohnani
QA Contact:	Sunil Choudhary
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1840458 1840460
TreeView+	depends on / blocked

Reported:	2020-05-21 12:22 UTC by Jose Ortiz Padilla
Modified:	2023-10-06 20:11 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1840458 (view as bug list)
Environment:
Last Closed:	2020-07-13 17:40:55 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2020:2409	0	None	None	None	2020-07-13 17:41:14 UTC

Description Jose Ortiz Padilla 2020-05-21 12:22:18 UTC

Description of problem:

When using ctrcfg in mcp on 4.2, if you try to upgrade the rendered config for crio.conf prevent the node to start.

Version-Release number of selected component (if applicable):
4.3

How reproducible:
100%

Steps to Reproduce:
1. In 4.2 set pids_limit in a ctrcfg in mcp
2. Upgrade
3. Wait for upgrade the nodes

Actual results:
The logs_dir variable in crio.conf is set to "" instead of "/var/logs/crio/pods"

Expected results:
The logs_dir variable in crio.conf is set to "/var/logs/crio/pods"

Additional info:
They are using the following ctrcfg:

apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
 name: b-default
spec:
 containerRuntimeConfig:
   pidsLimit: 8192
 machineConfigPoolSelector:
   matchLabels:
     b-default: 'true'

Comment 3 Sunil Choudhary 2020-05-29 10:10:12 UTC

Upgraded successfully from 4.4.6 to 4.5.0-0.nightly-2020-05-27-151956 with custom ContainerRuntimeConfig.

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.6     True        False         26m     Cluster version is 4.4.6

$ cat ctrcfg.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
  name: custom-ctrcfg
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom: pids-limit
  containerRuntimeConfig:
    pidsLimit: 8192

$ oc label machineconfigpool worker custom=pids-limit
machineconfigpool.machineconfiguration.openshift.io/worker labeled

$ oc create -f ctrcfg.yaml 
containerruntimeconfig.machineconfiguration.openshift.io/custom-ctrcfg created

$ oc get mc
NAME                                                              GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                                         8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             38m
00-worker                                                         8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             38m
01-master-container-runtime                                       8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             38m
01-master-kubelet                                                 8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             38m
01-worker-container-runtime                                       8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             38m
01-worker-kubelet                                                 8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             38m
99-master-1edf8648-ece4-4f23-a0af-f6ceb259ff4c-registries         8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             38m
99-master-ssh                                                                                                2.2.0             39m
99-worker-f8381e75-3393-4ab2-a028-b061302f5c98-containerruntime   8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             3s
99-worker-f8381e75-3393-4ab2-a028-b061302f5c98-registries         8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             38m
99-worker-ssh                                                                                                2.2.0             39m
rendered-master-310dd33f673f9df9d19d651c31a8c2bb                  8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             38m
rendered-worker-73a890011f34df29e1d153f1202f32fd                  8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             38m

$ oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-128-198.us-east-2.compute.internal   Ready    worker   34m   v1.17.1
ip-10-0-129-246.us-east-2.compute.internal   Ready    master   43m   v1.17.1
ip-10-0-147-127.us-east-2.compute.internal   Ready    worker   33m   v1.17.1
ip-10-0-152-152.us-east-2.compute.internal   Ready    master   43m   v1.17.1
ip-10-0-164-147.us-east-2.compute.internal   Ready    worker   35m   v1.17.1
ip-10-0-167-166.us-east-2.compute.internal   Ready    master   43m   v1.17.1

$ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-27-151956 --allow-explicit-upgrade=true --force
Updating to release image registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-27-151956

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-05-27-151956   True        False         37m     Cluster version is 4.5.0-0.nightly-2020-05-27-151956

$ oc get clusterversion -o yaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2020-05-29T07:44:20Z"
    generation: 2
    name: version
    resourceVersion: "66907"
    selfLink: /apis/config.openshift.io/v1/clusterversions/version
    uid: a9e895a6-7160-432d-aa35-7efc41b38dbc
  spec:
    channel: stable-4.4
    clusterID: 8edec9ca-4d53-4fa4-bedd-1124433d2896
    desiredUpdate:
      force: true
      image: registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-27-151956
      version: ""
    upstream: https://api.openshift.com/api/upgrades_info/v1/graph
  status:
    availableUpdates: null
    conditions:
    - lastTransitionTime: "2020-05-29T08:02:24Z"
      message: Done applying 4.5.0-0.nightly-2020-05-27-151956
      status: "True"
      type: Available
    - lastTransitionTime: "2020-05-29T09:22:04Z"
      status: "False"
      type: Failing
    - lastTransitionTime: "2020-05-29T09:30:50Z"
      message: Cluster version is 4.5.0-0.nightly-2020-05-27-151956
      status: "False"
      type: Progressing
    - lastTransitionTime: "2020-05-29T07:44:34Z"
      message: 'Unable to retrieve available updates: currently installed version
        4.5.0-0.nightly-2020-05-27-151956 not found in the "stable-4.4" channel'
      reason: VersionNotFound
      status: "False"
      type: RetrievedUpdates
    - lastTransitionTime: "2020-05-29T08:04:34Z"
      message: 'Cluster operator marketplace cannot be upgraded between minor versions:
        The cluster has custom OperatorSource, which is deprecated in future versions.
        Please visit this link for further details: https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated'
      reason: DeprecatedAPIsInUse
      status: "False"
      type: Upgradeable
    desired:
      force: true
      image: registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-27-151956
      version: 4.5.0-0.nightly-2020-05-27-151956
    history:
    - completionTime: "2020-05-29T09:31:50Z"
      image: registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-27-151956
      startedTime: "2020-05-29T08:41:24Z"
      state: Completed
      verified: false
      version: 4.5.0-0.nightly-2020-05-27-151956
    - completionTime: "2020-05-29T08:02:24Z"
      image: quay.io/openshift-release-dev/ocp-release@sha256:7613d8f7db639147b91b16b54b24cfa351c3cbde6aa7b7bf1b9c80c260efad06
      startedTime: "2020-05-29T07:44:34Z"
      state: Completed
      verified: false
      version: 4.4.6
    observedGeneration: 2
    versionHash: uP4qmctmMU0=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

$ oc describe containerruntimeconfig
Name:         custom-ctrcfg
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1
Kind:         ContainerRuntimeConfig
Metadata:
  Creation Timestamp:  2020-05-29T08:29:49Z
  Finalizers:
    99-worker-f8381e75-3393-4ab2-a028-b061302f5c98-containerruntime
    99-worker-f8381e75-3393-4ab2-a028-b061302f5c98-containerruntime
  Generation:        1
  Resource Version:  52893
  Self Link:         /apis/machineconfiguration.openshift.io/v1/containerruntimeconfigs/custom-ctrcfg
  UID:               8d2c4722-18f5-4dfc-a66a-8346b67f6eb8
Spec:
  Container Runtime Config:
    Pids Limit:  8192
  Machine Config Pool Selector:
    Match Labels:
      Custom:  pids-limit
Status:
  Conditions:
    Last Transition Time:  2020-05-29T08:29:49Z
    Message:               Success
    Status:                True
    Type:                  Success
  Observed Generation:     1
Events:                    <none>

Comment 5 errata-xmlrpc 2020-07-13 17:40:55 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409

Note You need to log in before you can comment on or make changes to this bug.