Bug 1809621

Summary: cri-o failed to be installed due to dependency issue when scaleup rhel7 node
Product: OpenShift Container Platform Reporter: Johnny Liu <jialiu>
Component: ReleaseAssignee: Luke Meyer <lmeyer>
Status: CLOSED ERRATA QA Contact: Johnny Liu <jialiu>
Severity: high Docs Contact:
Priority: urgent    
Version: 4.4CC: aos-bugs, gpei, jokerman, yanyang
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-13 20:05:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Johnny Liu 2020-03-03 14:37:24 UTC
Description of problem:
Use baseurl=http://download.eng.bos.redhat.com/rcm-guest/puddles/RHAOS/AtomicOpenShift/4.4/2020-03-03.2/x86_64/os to create yum repo on a RHEL7 node, then scaleup.

Failed at the following steps:
TASK [openshift_node : Install openshift packages] *****************************
Tuesday 03 March 2020  18:06:29 +0800 (0:00:00.096)       0:06:45.201 ********* 
FAILED - RETRYING: Install openshift packages (3 retries left).
FAILED - RETRYING: Install openshift packages (3 retries left).
FAILED - RETRYING: Install openshift packages (2 retries left).
FAILED - RETRYING: Install openshift packages (2 retries left).
FAILED - RETRYING: Install openshift packages (1 retries left).
FAILED - RETRYING: Install openshift packages (1 retries left).
fatal: [ip-10-0-58-205.us-east-2.compute.internal]: FAILED! => {"ansible_job_id": "602098970672.27401", "attempts": 3, "changed": true, "changes": {"installed": ["cri-o-1.17.*", "openshift-clients-4.4.*", "openshift-hyperkube-4.4.*"], "updated": []}, "finished": 1, "msg": "Error: Package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64 (aos-v4-devel-install)\n           Requires: skopeo-containers >= 1:0.1.40-1\n           Installed: 1:containers-common-0.1.37-3.el7.x86_64 (@oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.37-3.el7\n           Available: 1:containers-common-0.1.31-7.gitb0b750d.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.31-7.gitb0b750d.el7\n           Available: 1:containers-common-0.1.31-8.gitb0b750d.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.31-8.gitb0b750d.el7\n           Available: 1:containers-common-0.1.35-2.git404c5bd.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.35-2.git404c5bd.el7\n           Available: 1:containers-common-0.1.37-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.37-1.el7\n           Available: 1:skopeo-containers-0.1.17-0.7.git1f655f3.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.17-0.7.git1f655f3.el7\n           Available: 1:skopeo-containers-0.1.17-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.17-1.el7\n           Available: 1:skopeo-containers-0.1.18-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.18-1.el7\n           Available: 1:skopeo-containers-0.1.19-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.19-1.el7\n           Available: 1:skopeo-containers-0.1.20-2.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.20-2.el7\n           Available: 1:skopeo-containers-0.1.20-2.1.gite802625.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.20-2.1.gite802625.el7\n           Available: 1:skopeo-containers-0.1.23-1.git1bbd87f.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.23-1.git1bbd87f.el7\n           Available: 1:skopeo-containers-0.1.24-1.dev.git28d4e08.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.24-1.dev.git28d4e08.el7\n           Available: 1:skopeo-containers-0.1.26-2.dev.git2e8377a.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.26-2.dev.git2e8377a.el7\n           Available: 1:skopeo-containers-0.1.27-3.dev.git14245f2.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.27-3.dev.git14245f2.el7\n           Available: 1:skopeo-containers-0.1.28-1.git0270e56.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.28-1.git0270e56.el7\n           Available: 1:skopeo-containers-0.1.29-1.dev.gitb08350d.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.29-1.dev.gitb08350d.el7\n           Available: 1:skopeo-containers-0.1.29-3.dev.git7add6fc.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.29-3.dev.git7add6fc.el7\n           Available: 1:skopeo-containers-0.1.30-1.dev.gitca3bff6.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.30-1.dev.gitca3bff6.el7\n           Available: 1:skopeo-containers-0.1.31-1.dev.gitae64ff7.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.31-1.dev.gitae64ff7.el7\nError: Package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64 (aos-v4-devel-install)\n           Requires: conmon >= 2.0.2-2\n", "obsoletes": {"python-urllib3": {"dist": "noarch", "repo": "@anaconda/7.6", "version": "1.10.2-5.el7"}}, "rc": 1, "results": ["Loaded plugins: amazon-id, rhui-lb, search-disabled-repos\nResolving Dependencies\n--> Running transaction check\n---> Package cri-o.x86_64 0:1.17.0-4.dev.rhaos4.4.git186c230.el7 will be installed\n--> Processing Dependency: skopeo-containers >= 1:0.1.40-1 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n--> Processing Dependency: conmon >= 2.0.2-2 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n---> Package openshift-clients.x86_64 0:4.4.0-202002282323.git.0.d2acde5.el7 will be installed\n---> Package openshift-hyperkube.x86_64 0:4.4.0-202002282323.git.0.108c140.el7 will be installed\n--> Processing Dependency: socat for package: openshift-hyperkube-4.4.0-202002282323.git.0.108c140.el7.x86_64\n--> Running transaction check\n---> Package cri-o.x86_64 0:1.17.0-4.dev.rhaos4.4.git186c230.el7 will be installed\n--> Processing Dependency: skopeo-containers >= 1:0.1.40-1 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n--> Processing Dependency: conmon >= 2.0.2-2 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"]}
fatal: [ip-10-0-58-19.us-east-2.compute.internal]: FAILED! => {"ansible_job_id": "777886524480.27388", "attempts": 3, "changed": true, "changes": {"installed": ["cri-o-1.17.*", "openshift-clients-4.4.*", "openshift-hyperkube-4.4.*"], "updated": []}, "finished": 1, "msg": "Error: Package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64 (aos-v4-devel-install)\n           Requires: skopeo-containers >= 1:0.1.40-1\n           Installed: 1:containers-common-0.1.37-3.el7.x86_64 (@oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.37-3.el7\n           Available: 1:containers-common-0.1.31-7.gitb0b750d.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.31-7.gitb0b750d.el7\n           Available: 1:containers-common-0.1.31-8.gitb0b750d.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.31-8.gitb0b750d.el7\n           Available: 1:containers-common-0.1.35-2.git404c5bd.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.35-2.git404c5bd.el7\n           Available: 1:containers-common-0.1.37-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.37-1.el7\n           Available: 1:skopeo-containers-0.1.17-0.7.git1f655f3.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.17-0.7.git1f655f3.el7\n           Available: 1:skopeo-containers-0.1.17-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.17-1.el7\n           Available: 1:skopeo-containers-0.1.18-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.18-1.el7\n           Available: 1:skopeo-containers-0.1.19-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.19-1.el7\n           Available: 1:skopeo-containers-0.1.20-2.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.20-2.el7\n           Available: 1:skopeo-containers-0.1.20-2.1.gite802625.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.20-2.1.gite802625.el7\n           Available: 1:skopeo-containers-0.1.23-1.git1bbd87f.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.23-1.git1bbd87f.el7\n           Available: 1:skopeo-containers-0.1.24-1.dev.git28d4e08.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.24-1.dev.git28d4e08.el7\n           Available: 1:skopeo-containers-0.1.26-2.dev.git2e8377a.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.26-2.dev.git2e8377a.el7\n           Available: 1:skopeo-containers-0.1.27-3.dev.git14245f2.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.27-3.dev.git14245f2.el7\n           Available: 1:skopeo-containers-0.1.28-1.git0270e56.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.28-1.git0270e56.el7\n           Available: 1:skopeo-containers-0.1.29-1.dev.gitb08350d.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.29-1.dev.gitb08350d.el7\n           Available: 1:skopeo-containers-0.1.29-3.dev.git7add6fc.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.29-3.dev.git7add6fc.el7\n           Available: 1:skopeo-containers-0.1.30-1.dev.gitca3bff6.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.30-1.dev.gitca3bff6.el7\n           Available: 1:skopeo-containers-0.1.31-1.dev.gitae64ff7.el7.x86_64 (oso-rhui-rhel-server-extras)\n               skopeo-containers = 1:0.1.31-1.dev.gitae64ff7.el7\nError: Package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64 (aos-v4-devel-install)\n           Requires: conmon >= 2.0.2-2\n", "obsoletes": {"python-urllib3": {"dist": "noarch", "repo": "@anaconda/7.6", "version": "1.10.2-5.el7"}}, "rc": 1, "results": ["Loaded plugins: amazon-id, rhui-lb, search-disabled-repos\nResolving Dependencies\n--> Running transaction check\n---> Package cri-o.x86_64 0:1.17.0-4.dev.rhaos4.4.git186c230.el7 will be installed\n--> Processing Dependency: skopeo-containers >= 1:0.1.40-1 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n--> Processing Dependency: conmon >= 2.0.2-2 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n---> Package openshift-clients.x86_64 0:4.4.0-202002282323.git.0.d2acde5.el7 will be installed\n---> Package openshift-hyperkube.x86_64 0:4.4.0-202002282323.git.0.108c140.el7 will be installed\n--> Processing Dependency: socat for package: openshift-hyperkube-4.4.0-202002282323.git.0.108c140.el7.x86_64\n--> Running transaction check\n---> Package cri-o.x86_64 0:1.17.0-4.dev.rhaos4.4.git186c230.el7 will be installed\n--> Processing Dependency: skopeo-containers >= 1:0.1.40-1 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n--> Processing Dependency: conmon >= 2.0.2-2 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"]}


Seem like skopeo build is removed in https://errata.devel.redhat.com/advisory/47982#c723, which caused this failure.

4.4/2020-02-29.3 has no such issue.

Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Russell Teague 2020-03-03 16:23:35 UTC
Moving to the 'node' component as this is related to the availability of cri-o and dependent packages.

Comment 2 Luke Meyer 2020-03-04 16:47:42 UTC
I untagged builds of RPMs that we are not using in container images or RHCOS. It makes ART sad when we have to ship CVE rebuilds for things no one is actually using.

I was under the impression that install could use skopeo et al from RHEL directly. If that's not the case, or the ones shipped in RHEL 7 aren't recent enough builds, I'm happy to add these back and ship them. Requests for specific RPMs would be appreciated. Here it looks like skopeo and containers-common RPMs would be needed, anything else?

Since this is essentially a delivery/install problem, moving to Release component.

Comment 3 Johnny Liu 2020-03-05 02:19:59 UTC
So far, I only can see skopeo-0.1.37-3.el7 available from RHEL Extra repo. While cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7 want 0.1.40-2.el7, which is never shipped in RHEL Extras repo.

Comment 4 Johnny Liu 2020-03-05 12:55:25 UTC
In latest 2020-03-05.3 puddle, skopeo is back, but it still need fuse-overlayfs move back together as dependency, otherwise, will this such failure:
# yum install "cri-o-1.17*"
ansible29                                                                                                                                                              | 4.0 kB  00:00:00     
ose-v4                                                                                                                                                                 | 1.2 kB  00:00:00     
rhel7                                                                                                                                                                  | 3.5 kB  00:00:00     
rhel7-extra                                                                                                                                                            | 3.4 kB  00:00:00     
ose-v4/primary                                                                                                                                                         |  26 kB  00:00:00     
ose-v4                                                                                                                                                                                  86/86
Resolving Dependencies
--> Running transaction check
---> Package cri-o.x86_64 0:1.10.5-1.rhaos3.10.git4c81456.el7 will be updated
---> Package cri-o.x86_64 0:1.17.0-4.dev.rhaos4.4.git186c230.el7 will be an update
--> Processing Dependency: skopeo-containers >= 1:0.1.40-1 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64
--> Processing Dependency: containernetworking-plugins >= 0.8.2-3 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64
--> Processing Dependency: conmon >= 2.0.2-2 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64
--> Running transaction check
---> Package conmon.x86_64 0:2.0.9-1.rhaos4.4.el7 will be installed
---> Package containernetworking-plugins.x86_64 0:0.8.1-2.el7 will be updated
---> Package containernetworking-plugins.x86_64 0:0.8.2-3.el7 will be an update
---> Package containers-common.x86_64 1:0.1.37-3.el7 will be updated
--> Processing Dependency: containers-common = 1:0.1.37-3.el7 for package: 1:skopeo-0.1.37-3.el7.x86_64
---> Package containers-common.x86_64 1:0.1.40-1.rhaos4.3.el7 will be an update
--> Processing Dependency: slirp4netns for package: 1:containers-common-0.1.40-1.rhaos4.3.el7.x86_64
--> Processing Dependency: fuse-overlayfs for package: 1:containers-common-0.1.40-1.rhaos4.3.el7.x86_64
--> Running transaction check
---> Package containers-common.x86_64 1:0.1.40-1.rhaos4.3.el7 will be an update
--> Processing Dependency: fuse-overlayfs for package: 1:containers-common-0.1.40-1.rhaos4.3.el7.x86_64
---> Package skopeo.x86_64 1:0.1.37-3.el7 will be updated
---> Package skopeo.x86_64 1:0.1.40-1.rhaos4.3.el7 will be an update
---> Package slirp4netns.x86_64 0:0.3.0-1.el7 will be installed
--> Finished Dependency Resolution
Error: Package: 1:containers-common-0.1.40-1.rhaos4.3.el7.x86_64 (ose-v4)
           Requires: fuse-overlayfs
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

Comment 6 Johnny Liu 2020-03-09 06:16:39 UTC
Verified this bug with 4.4/2020-03-09.1, and PASS.


# yum install "cri-o-1.17*"
<--SNIP-->
Dependencies Resolved

==============================================================================================================================================================================================
 Package                                             Arch                           Version                                                         Repository                           Size
==============================================================================================================================================================================================
Updating:
 cri-o                                               x86_64                         1.17.0-8.dev.rhaos4.4.git36920a5.el7                            ose-v4                               27 M
Installing for dependencies:
 conmon                                              x86_64                         2.0.11-1.rhaos4.4.el7                                           ose-v4                               32 k
 fuse-overlayfs                                      x86_64                         0.7.2-1.el7                                                     ose-v4                               52 k
 fuse3-libs                                          x86_64                         3.6.1-4.el7                                                     ose-v4                               81 k
 slirp4netns                                         x86_64                         0.3.0-1.el7                                                     rhel7-extra                          77 k
Updating for dependencies:
 containernetworking-plugins                         x86_64                         0.8.2-3.el7                                                     ose-v4                               20 M
 containers-common                                   x86_64                         1:0.1.40-1.rhaos4.3.el7                                         ose-v4                               27 k
 skopeo                                              x86_64                         1:0.1.40-1.rhaos4.3.el7                                         ose-v4                              5.7 M

Transaction Summary
==============================================================================================================================================================================================
Install             ( 4 Dependent packages)
Upgrade  1 Package  (+3 Dependent packages)
<--SNIP-->

Also scale up successfully per https://openshift-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/ocp4-rhel-scaleup-runner/1158/consoleFull.

Comment 9 Luke Meyer 2020-05-13 20:05:26 UTC
This should have been closed with the 4.4 GA release.