Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1760665

Summary: upgrade.yml playbook didn't update cri-o package during upgrade
Product: OpenShift Container Platform Reporter: Gaoyun Pei <gpei>
Component: InstallerAssignee: Russell Teague <rteague>
Installer sub component: openshift-ansible QA Contact: Gaoyun Pei <gpei>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: bleanhar, wzheng
Version: 4.2.0   
Target Milestone: ---   
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1760859 (view as bug list) Environment:
Last Closed: 2020-01-23 11:07:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1760859    

Description Gaoyun Pei 2019-10-11 05:15:07 UTC 
Description of problem:

cri-o package was not updated during upgrade.

https://github.com/openshift/openshift-ansible/blob/release-4.2/roles/openshift_node/defaults/main.yml#L13
https://github.com/openshift/openshift-ansible/blob/release-4.2/roles/openshift_node/tasks/install.yml#L50


TASK [openshift_node : Install openshift packages] *****************************
Thursday 10 October 2019  16:25:48 +0800 (0:00:00.084)       0:04:02.661 ****** 
changed: [rhel-1.qe-lxia-upg-share-1010.qe.devcluster.openshift.com] => {"ansible_job_id": "919582869957.1459", "attempts": 1, "changed": true, "changes": {"installed": ["openshift-clients-4.2.0", "openshift-hyperkube-4.2.0"]}, "finished": 1, "msg": "", "rc": 0, "results": ["cri-o-1.13.11-0.10.dev.rhaos4.1.gitbdeb2ca.el7.x86_64 providing cri-o is already installed", "Loaded plugins: product-id, search-disabled-repos, subscription-manager\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\nResolving Dependencies\n--> Running transaction check\n---> Package openshift-clients.x86_64 0:4.1.19-201910070609.git.0.6f9924b.el7 will be updated\n---> Package openshift-clients.x86_64 0:4.2.0-201910041700.git.1.c8c7aaa.el7 will be an update\n---> Package openshift-hyperkube.x86_64 0:4.1.19-201910070609.git.0.6f9924b.el7 will be updated\n---> Package openshift-hyperkube.x86_64 0:4.2.0-201910020731.git.0.463c73f.el7 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package              Arch    Version                                Repository\n                                                                           Size\n================================================================================\nUpdating:\n openshift-clients    x86_64  4.2.0-201910041700.git.1.c8c7aaa.el7   aos   17 M\n openshift-hyperkube  x86_64  4.2.0-201910020731.git.0.463c73f.el7   aos   34 M\n\nTransaction Summary\n================================================================================\nUpgrade  2 Packages\n\nTotal download size: 51 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal                                               12 MB/s |  51 MB  00:04     \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n  Updating   : openshift-hyperkube-4.2.0-201910020731.git.0.463c73f.el7.x   1/4 \n  Updating   : openshift-clients-4.2.0-201910041700.git.1.c8c7aaa.el7.x86   2/4 \n  Cleanup    : openshift-hyperkube-4.1.19-201910070609.git.0.6f9924b.el7.   3/4 \n  Cleanup    : openshift-clients-4.1.19-201910070609.git.0.6f9924b.el7.x8   4/4 \n  Verifying  : openshift-clients-4.2.0-201910041700.git.1.c8c7aaa.el7.x86   1/4 \n  Verifying  : openshift-hyperkube-4.2.0-201910020731.git.0.463c73f.el7.x   2/4 \n  Verifying  : openshift-clients-4.1.19-201910070609.git.0.6f9924b.el7.x8   3/4 \n  Verifying  : openshift-hyperkube-4.1.19-201910070609.git.0.6f9924b.el7.   4/4 \n\nUpdated:\n  openshift-clients.x86_64 0:4.2.0-201910041700.git.1.c8c7aaa.el7               \n  openshift-hyperkube.x86_64 0:4.2.0-201910020731.git.0.463c73f.el7             \n\nComplete!\n"]}



Version-Release number of the following components:
openshift-ansible-4.2.0-201909221318.git.193.0fd88d7.el7

How reproducible:

Steps to Reproduce:
1. Run upgrade.yml playbook to upgrade RHEL worker from 4.1 to 4.2
ansible-playbook -i jenkins_inventory -v /usr/share/ansible/openshift-ansible/playbooks/upgrade.yml

Actual results:
It's still cri-o-1.13.11-0.10.dev.rhaos4.1.gitbdeb2ca.el7.x86_64 on the RHEL node

Expected results:
Should be cri-o-1.14.11-0.17.dev.rhaos4.2.gitc41de67.el7.x86_64.rpm after upgrade to 4.2


Additional info:
Please attach logs from ansible-playbook with the -vvv flag
Comment 1 Brenton Leanhardt 2019-10-11 12:17:02 UTC 
Definitely seems like a bug and it should be a quick fix.  

Hi Gaoyun, so we understand the urgency, aside from the following bug fixes in cri-o, are there other obviously broken OpenShift features as a result of the cri-o upgrade failure?  If you don't know, that's a fair answer.

https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2019-10214 (med severity, token security)
https://bugzilla.redhat.com/show_bug.cgi?id=1726326 (jenkins)
https://bugzilla.redhat.com/show_bug.cgi?id=1731370 (disable fips)
Comment 3 Gaoyun Pei 2019-10-12 06:21:35 UTC 
(In reply to Brenton Leanhardt from comment #1)
> Definitely seems like a bug and it should be a quick fix.  
> 
> Hi Gaoyun, so we understand the urgency, aside from the following bug fixes
> in cri-o, are there other obviously broken OpenShift features as a result of
> the cri-o upgrade failure?  If you don't know, that's a fair answer.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2019-10214 (med severity,
> token security)
> https://bugzilla.redhat.com/show_bug.cgi?id=1726326 (jenkins)
> https://bugzilla.redhat.com/show_bug.cgi?id=1731370 (disable fips)

Hi Brenton, I didn't see any obvious error after the upgrade, actually the infra pods(like machine-config-daemon/dns) are running normally on the RHEL worker with cri-o-1.13.11, but I didn't make further testing on it.
Comment 4 Gaoyun Pei 2019-10-14 08:10:41 UTC 
Proposed PR has been merged into openshift-ansible-4.3.0-201910111316.git.190.ba71029.el7.noarch.rpm.

"state: latest" of `package` module could ensure packages are upgraded to the latest version when running the task, so move this bug to verified.
Comment 6 errata-xmlrpc 2020-01-23 11:07:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062