Bug 1899603

Summary: workers-rhel7 CI jobs failing: Failed to remove rollback: error running rpm-ostree cleanup
Product: OpenShift Container Platform Reporter: Russell Teague <rteague>
Component: Machine Config OperatorAssignee: Yu Qi Zhang <jerzhang>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: high Docs Contact:
Priority: high    
Version: 4.7CC: jerzhang
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:35:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Russell Teague 2020-11-19 16:28:15 UTC
Description of problem:
CI fails waiting for the machineconfigpool/worker to update after installing RHEL7 workers.

  - lastTransitionTime: "2020-11-19T15:09:25Z"
    message: 'Node ip-10-0-219-88.us-east-2.compute.internal is reporting: "Failed to remove rollback: error running rpm-ostree cleanup -r: : exec: \"rpm-ostree\": executable file not found in $PATH", Node ip-10-0-131-226.us-east-2.compute.internal is reporting: "Failed to remove rollback: error running rpm-ostree cleanup -r: : exec: \"rpm-ostree\": executable file not found in $PATH", Node ip-10-0-147-189.us-east-2.compute.internal is reporting: "Failed to remove rollback: error running rpm-ostree cleanup -r: : exec: \"rpm-ostree\": executable file not found in $PATH"'
    reason: 3 nodes are reporting degraded status on sync
    status: "True"
    type: NodeDegraded


How reproducible:
CI Job:
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_installer/4321/pull-ci-openshift-installer-master-e2e-aws-workers-rhel7/1329424551041306624

CI Search:
https://search.ci.openshift.org/?search=error%3A+timed+out+waiting+for+the+condition+on+machineconfigpools%2Fworker&maxAge=48h&context=1&type=bug%2Bjunit&name=workers-rhel7&maxMatches=5&maxBytes=20971520&groupBy=job

Comment 2 Michael Nguyen 2020-11-24 14:28:57 UTC
Marking as verified.  The error did not show up in ci search for the last two days.

Comment 5 errata-xmlrpc 2021-02-24 15:35:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633