Bug 1820401

Summary:	blocking gcp-op failure: Failed to execute rpm-ostree
Product:	OpenShift Container Platform	Reporter:	Kirsten Garrison <kgarriso>
Component:	Machine Config Operator	Assignee:	Sinny Kumari <skumari>
Status:	CLOSED ERRATA	QA Contact:	Michael Nguyen <mnguyen>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	4.5	CC:	amurdaca, skumari, walters
Target Milestone:	---
Target Release:	4.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-07-13 17:25:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Kirsten Garrison 2020-04-03 01:11:31 UTC

Description of problem:
For all of Apr2, 2020, the gcp-op job was failing. Looking thru the mcd logs i see a few tests taking a long time and the node never seems to finish its update and reboot finally giving up and erroring:

Marking Degraded due to: Failed to execute rpm-ostree ["override" "reset" "kernel" "kernel-core" "kernel-modules" "kernel-modules-extra" "--uninstall" "kernel-rt-core" "--uninstall" "kernel-rt-modules" "--uninstall" "kernel-rt-modules-extra"] : exit status 1

All runs:
https://prow.svc.ci.openshift.org/job-history/origin-ci-test/pr-logs/directory/pull-ci-openshift-machine-config-operator-master-e2e-gcp-op

Not sure what happened, but need to dig on this as this is blocking PRs from merging.

Examples: 
https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/1601/pull-ci-openshift-machine-config-operator-master-e2e-gcp-op/1666/artifacts/e2e-gcp-op/pods/openshift-machine-config-operator_machine-config-daemon-rlwzn_machine-config-daemon.log
https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/1602/pull-ci-openshift-machine-config-operator-master-e2e-gcp-op/1665/artifacts/e2e-gcp-op/pods/openshift-machine-config-operator_machine-config-daemon-pnvn7_machine-config-daemon.log
https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/1602/pull-ci-openshift-machine-config-operator-master-e2e-gcp-op/1645/artifacts/e2e-gcp-op/pods/openshift-machine-config-operator_machine-config-daemon-tm8n7_machine-config-daemon.log

Comment 1 Colin Walters 2020-04-03 01:54:58 UTC

Hum....this might be fallout from https://gitlab.cee.redhat.com/coreos/redhat-coreos/merge_requests/877

Comment 2 Sinny Kumari 2020-04-03 07:23:27 UTC

Right, this is happening because of additional kernel-rt packages being shipped now in machine-os-content.

Currently during rt-kernel switch, we are installing all kernel-rt packages being available in machine-os-content (https://github.com/openshift/machine-config-operator/blob/db561314c7afae1d77c16cfdb95f0f0ce6b8977d/pkg/daemon/update.go#L721 ). As a result, switching to kernel-rt packages works fine and installs all kernel-rt specific packages. But during rollback to traditional kernel, we are relying on the specific list of kernel-rt to be uninstalled which doesn't take into consideration the additional packages (https://github.com/openshift/machine-config-operator/blob/db561314c7afae1d77c16cfdb95f0f0ce6b8977d/pkg/daemon/update.go#L663). This causes rollback to fail and node goes to degraded state.

We will need better way to know layered kernel-rt packages, maybe it is time to parse `rpm-ostree status --json` result.

Comment 5 Michael Nguyen 2020-04-07 18:40:38 UTC

The https://prow.svc.ci.openshift.org/job-history/origin-ci-test/pr-logs/directory/pull-ci-openshift-machine-config-operator-master-e2e-gcp-op tests are passing again.  Therefore considering this verified.

Comment 7 errata-xmlrpc 2020-07-13 17:25:20 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409