Bug 1838606
| Summary: | Applying "ctrcfg" causes cri-o to fail to upgrade a cluster | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jose Ortiz Padilla <jortizpa> | |
| Component: | Node | Assignee: | Urvashi Mohnani <umohnani> | |
| Status: | CLOSED ERRATA | QA Contact: | Sunil Choudhary <schoudha> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 4.3.z | CC: | aos-bugs, jokerman, rphillips, umohnani | |
| Target Milestone: | --- | |||
| Target Release: | 4.5.0 | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1840458 (view as bug list) | Environment: | ||
| Last Closed: | 2020-07-13 17:40:55 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1840458, 1840460 | |||
Upgraded successfully from 4.4.6 to 4.5.0-0.nightly-2020-05-27-151956 with custom ContainerRuntimeConfig.
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.4.6 True False 26m Cluster version is 4.4.6
$ cat ctrcfg.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
name: custom-ctrcfg
spec:
machineConfigPoolSelector:
matchLabels:
custom: pids-limit
containerRuntimeConfig:
pidsLimit: 8192
$ oc label machineconfigpool worker custom=pids-limit
machineconfigpool.machineconfiguration.openshift.io/worker labeled
$ oc create -f ctrcfg.yaml
containerruntimeconfig.machineconfiguration.openshift.io/custom-ctrcfg created
$ oc get mc
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE
00-master 8af4f709c4ba9c0afff3408ecc99c8fce61dd314 2.2.0 38m
00-worker 8af4f709c4ba9c0afff3408ecc99c8fce61dd314 2.2.0 38m
01-master-container-runtime 8af4f709c4ba9c0afff3408ecc99c8fce61dd314 2.2.0 38m
01-master-kubelet 8af4f709c4ba9c0afff3408ecc99c8fce61dd314 2.2.0 38m
01-worker-container-runtime 8af4f709c4ba9c0afff3408ecc99c8fce61dd314 2.2.0 38m
01-worker-kubelet 8af4f709c4ba9c0afff3408ecc99c8fce61dd314 2.2.0 38m
99-master-1edf8648-ece4-4f23-a0af-f6ceb259ff4c-registries 8af4f709c4ba9c0afff3408ecc99c8fce61dd314 2.2.0 38m
99-master-ssh 2.2.0 39m
99-worker-f8381e75-3393-4ab2-a028-b061302f5c98-containerruntime 8af4f709c4ba9c0afff3408ecc99c8fce61dd314 2.2.0 3s
99-worker-f8381e75-3393-4ab2-a028-b061302f5c98-registries 8af4f709c4ba9c0afff3408ecc99c8fce61dd314 2.2.0 38m
99-worker-ssh 2.2.0 39m
rendered-master-310dd33f673f9df9d19d651c31a8c2bb 8af4f709c4ba9c0afff3408ecc99c8fce61dd314 2.2.0 38m
rendered-worker-73a890011f34df29e1d153f1202f32fd 8af4f709c4ba9c0afff3408ecc99c8fce61dd314 2.2.0 38m
$ oc get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-128-198.us-east-2.compute.internal Ready worker 34m v1.17.1
ip-10-0-129-246.us-east-2.compute.internal Ready master 43m v1.17.1
ip-10-0-147-127.us-east-2.compute.internal Ready worker 33m v1.17.1
ip-10-0-152-152.us-east-2.compute.internal Ready master 43m v1.17.1
ip-10-0-164-147.us-east-2.compute.internal Ready worker 35m v1.17.1
ip-10-0-167-166.us-east-2.compute.internal Ready master 43m v1.17.1
$ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-27-151956 --allow-explicit-upgrade=true --force
Updating to release image registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-27-151956
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.5.0-0.nightly-2020-05-27-151956 True False 37m Cluster version is 4.5.0-0.nightly-2020-05-27-151956
$ oc get clusterversion -o yaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
kind: ClusterVersion
metadata:
creationTimestamp: "2020-05-29T07:44:20Z"
generation: 2
name: version
resourceVersion: "66907"
selfLink: /apis/config.openshift.io/v1/clusterversions/version
uid: a9e895a6-7160-432d-aa35-7efc41b38dbc
spec:
channel: stable-4.4
clusterID: 8edec9ca-4d53-4fa4-bedd-1124433d2896
desiredUpdate:
force: true
image: registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-27-151956
version: ""
upstream: https://api.openshift.com/api/upgrades_info/v1/graph
status:
availableUpdates: null
conditions:
- lastTransitionTime: "2020-05-29T08:02:24Z"
message: Done applying 4.5.0-0.nightly-2020-05-27-151956
status: "True"
type: Available
- lastTransitionTime: "2020-05-29T09:22:04Z"
status: "False"
type: Failing
- lastTransitionTime: "2020-05-29T09:30:50Z"
message: Cluster version is 4.5.0-0.nightly-2020-05-27-151956
status: "False"
type: Progressing
- lastTransitionTime: "2020-05-29T07:44:34Z"
message: 'Unable to retrieve available updates: currently installed version
4.5.0-0.nightly-2020-05-27-151956 not found in the "stable-4.4" channel'
reason: VersionNotFound
status: "False"
type: RetrievedUpdates
- lastTransitionTime: "2020-05-29T08:04:34Z"
message: 'Cluster operator marketplace cannot be upgraded between minor versions:
The cluster has custom OperatorSource, which is deprecated in future versions.
Please visit this link for further details: https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated'
reason: DeprecatedAPIsInUse
status: "False"
type: Upgradeable
desired:
force: true
image: registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-27-151956
version: 4.5.0-0.nightly-2020-05-27-151956
history:
- completionTime: "2020-05-29T09:31:50Z"
image: registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-27-151956
startedTime: "2020-05-29T08:41:24Z"
state: Completed
verified: false
version: 4.5.0-0.nightly-2020-05-27-151956
- completionTime: "2020-05-29T08:02:24Z"
image: quay.io/openshift-release-dev/ocp-release@sha256:7613d8f7db639147b91b16b54b24cfa351c3cbde6aa7b7bf1b9c80c260efad06
startedTime: "2020-05-29T07:44:34Z"
state: Completed
verified: false
version: 4.4.6
observedGeneration: 2
versionHash: uP4qmctmMU0=
kind: List
metadata:
resourceVersion: ""
selfLink: ""
$ oc describe containerruntimeconfig
Name: custom-ctrcfg
Namespace:
Labels: <none>
Annotations: <none>
API Version: machineconfiguration.openshift.io/v1
Kind: ContainerRuntimeConfig
Metadata:
Creation Timestamp: 2020-05-29T08:29:49Z
Finalizers:
99-worker-f8381e75-3393-4ab2-a028-b061302f5c98-containerruntime
99-worker-f8381e75-3393-4ab2-a028-b061302f5c98-containerruntime
Generation: 1
Resource Version: 52893
Self Link: /apis/machineconfiguration.openshift.io/v1/containerruntimeconfigs/custom-ctrcfg
UID: 8d2c4722-18f5-4dfc-a66a-8346b67f6eb8
Spec:
Container Runtime Config:
Pids Limit: 8192
Machine Config Pool Selector:
Match Labels:
Custom: pids-limit
Status:
Conditions:
Last Transition Time: 2020-05-29T08:29:49Z
Message: Success
Status: True
Type: Success
Observed Generation: 1
Events: <none>
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409 |
Description of problem: When using ctrcfg in mcp on 4.2, if you try to upgrade the rendered config for crio.conf prevent the node to start. Version-Release number of selected component (if applicable): 4.3 How reproducible: 100% Steps to Reproduce: 1. In 4.2 set pids_limit in a ctrcfg in mcp 2. Upgrade 3. Wait for upgrade the nodes Actual results: The logs_dir variable in crio.conf is set to "" instead of "/var/logs/crio/pods" Expected results: The logs_dir variable in crio.conf is set to "/var/logs/crio/pods" Additional info: They are using the following ctrcfg: apiVersion: machineconfiguration.openshift.io/v1 kind: ContainerRuntimeConfig metadata: name: b-default spec: containerRuntimeConfig: pidsLimit: 8192 machineConfigPoolSelector: matchLabels: b-default: 'true'