Description of problem: Use baseurl=http://download.eng.bos.redhat.com/rcm-guest/puddles/RHAOS/AtomicOpenShift/4.4/2020-03-03.2/x86_64/os to create yum repo on a RHEL7 node, then scaleup. Failed at the following steps: TASK [openshift_node : Install openshift packages] ***************************** Tuesday 03 March 2020 18:06:29 +0800 (0:00:00.096) 0:06:45.201 ********* FAILED - RETRYING: Install openshift packages (3 retries left). FAILED - RETRYING: Install openshift packages (3 retries left). FAILED - RETRYING: Install openshift packages (2 retries left). FAILED - RETRYING: Install openshift packages (2 retries left). FAILED - RETRYING: Install openshift packages (1 retries left). FAILED - RETRYING: Install openshift packages (1 retries left). fatal: [ip-10-0-58-205.us-east-2.compute.internal]: FAILED! => {"ansible_job_id": "602098970672.27401", "attempts": 3, "changed": true, "changes": {"installed": ["cri-o-1.17.*", "openshift-clients-4.4.*", "openshift-hyperkube-4.4.*"], "updated": []}, "finished": 1, "msg": "Error: Package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64 (aos-v4-devel-install)\n Requires: skopeo-containers >= 1:0.1.40-1\n Installed: 1:containers-common-0.1.37-3.el7.x86_64 (@oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.37-3.el7\n Available: 1:containers-common-0.1.31-7.gitb0b750d.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.31-7.gitb0b750d.el7\n Available: 1:containers-common-0.1.31-8.gitb0b750d.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.31-8.gitb0b750d.el7\n Available: 1:containers-common-0.1.35-2.git404c5bd.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.35-2.git404c5bd.el7\n Available: 1:containers-common-0.1.37-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.37-1.el7\n Available: 1:skopeo-containers-0.1.17-0.7.git1f655f3.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.17-0.7.git1f655f3.el7\n Available: 1:skopeo-containers-0.1.17-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.17-1.el7\n Available: 1:skopeo-containers-0.1.18-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.18-1.el7\n Available: 1:skopeo-containers-0.1.19-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.19-1.el7\n Available: 1:skopeo-containers-0.1.20-2.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.20-2.el7\n Available: 1:skopeo-containers-0.1.20-2.1.gite802625.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.20-2.1.gite802625.el7\n Available: 1:skopeo-containers-0.1.23-1.git1bbd87f.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.23-1.git1bbd87f.el7\n Available: 1:skopeo-containers-0.1.24-1.dev.git28d4e08.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.24-1.dev.git28d4e08.el7\n Available: 1:skopeo-containers-0.1.26-2.dev.git2e8377a.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.26-2.dev.git2e8377a.el7\n Available: 1:skopeo-containers-0.1.27-3.dev.git14245f2.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.27-3.dev.git14245f2.el7\n Available: 1:skopeo-containers-0.1.28-1.git0270e56.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.28-1.git0270e56.el7\n Available: 1:skopeo-containers-0.1.29-1.dev.gitb08350d.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.29-1.dev.gitb08350d.el7\n Available: 1:skopeo-containers-0.1.29-3.dev.git7add6fc.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.29-3.dev.git7add6fc.el7\n Available: 1:skopeo-containers-0.1.30-1.dev.gitca3bff6.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.30-1.dev.gitca3bff6.el7\n Available: 1:skopeo-containers-0.1.31-1.dev.gitae64ff7.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.31-1.dev.gitae64ff7.el7\nError: Package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64 (aos-v4-devel-install)\n Requires: conmon >= 2.0.2-2\n", "obsoletes": {"python-urllib3": {"dist": "noarch", "repo": "@anaconda/7.6", "version": "1.10.2-5.el7"}}, "rc": 1, "results": ["Loaded plugins: amazon-id, rhui-lb, search-disabled-repos\nResolving Dependencies\n--> Running transaction check\n---> Package cri-o.x86_64 0:1.17.0-4.dev.rhaos4.4.git186c230.el7 will be installed\n--> Processing Dependency: skopeo-containers >= 1:0.1.40-1 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n--> Processing Dependency: conmon >= 2.0.2-2 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n---> Package openshift-clients.x86_64 0:4.4.0-202002282323.git.0.d2acde5.el7 will be installed\n---> Package openshift-hyperkube.x86_64 0:4.4.0-202002282323.git.0.108c140.el7 will be installed\n--> Processing Dependency: socat for package: openshift-hyperkube-4.4.0-202002282323.git.0.108c140.el7.x86_64\n--> Running transaction check\n---> Package cri-o.x86_64 0:1.17.0-4.dev.rhaos4.4.git186c230.el7 will be installed\n--> Processing Dependency: skopeo-containers >= 1:0.1.40-1 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n--> Processing Dependency: conmon >= 2.0.2-2 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"]} fatal: [ip-10-0-58-19.us-east-2.compute.internal]: FAILED! => {"ansible_job_id": "777886524480.27388", "attempts": 3, "changed": true, "changes": {"installed": ["cri-o-1.17.*", "openshift-clients-4.4.*", "openshift-hyperkube-4.4.*"], "updated": []}, "finished": 1, "msg": "Error: Package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64 (aos-v4-devel-install)\n Requires: skopeo-containers >= 1:0.1.40-1\n Installed: 1:containers-common-0.1.37-3.el7.x86_64 (@oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.37-3.el7\n Available: 1:containers-common-0.1.31-7.gitb0b750d.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.31-7.gitb0b750d.el7\n Available: 1:containers-common-0.1.31-8.gitb0b750d.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.31-8.gitb0b750d.el7\n Available: 1:containers-common-0.1.35-2.git404c5bd.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.35-2.git404c5bd.el7\n Available: 1:containers-common-0.1.37-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.37-1.el7\n Available: 1:skopeo-containers-0.1.17-0.7.git1f655f3.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.17-0.7.git1f655f3.el7\n Available: 1:skopeo-containers-0.1.17-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.17-1.el7\n Available: 1:skopeo-containers-0.1.18-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.18-1.el7\n Available: 1:skopeo-containers-0.1.19-1.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.19-1.el7\n Available: 1:skopeo-containers-0.1.20-2.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.20-2.el7\n Available: 1:skopeo-containers-0.1.20-2.1.gite802625.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.20-2.1.gite802625.el7\n Available: 1:skopeo-containers-0.1.23-1.git1bbd87f.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.23-1.git1bbd87f.el7\n Available: 1:skopeo-containers-0.1.24-1.dev.git28d4e08.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.24-1.dev.git28d4e08.el7\n Available: 1:skopeo-containers-0.1.26-2.dev.git2e8377a.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.26-2.dev.git2e8377a.el7\n Available: 1:skopeo-containers-0.1.27-3.dev.git14245f2.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.27-3.dev.git14245f2.el7\n Available: 1:skopeo-containers-0.1.28-1.git0270e56.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.28-1.git0270e56.el7\n Available: 1:skopeo-containers-0.1.29-1.dev.gitb08350d.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.29-1.dev.gitb08350d.el7\n Available: 1:skopeo-containers-0.1.29-3.dev.git7add6fc.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.29-3.dev.git7add6fc.el7\n Available: 1:skopeo-containers-0.1.30-1.dev.gitca3bff6.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.30-1.dev.gitca3bff6.el7\n Available: 1:skopeo-containers-0.1.31-1.dev.gitae64ff7.el7.x86_64 (oso-rhui-rhel-server-extras)\n skopeo-containers = 1:0.1.31-1.dev.gitae64ff7.el7\nError: Package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64 (aos-v4-devel-install)\n Requires: conmon >= 2.0.2-2\n", "obsoletes": {"python-urllib3": {"dist": "noarch", "repo": "@anaconda/7.6", "version": "1.10.2-5.el7"}}, "rc": 1, "results": ["Loaded plugins: amazon-id, rhui-lb, search-disabled-repos\nResolving Dependencies\n--> Running transaction check\n---> Package cri-o.x86_64 0:1.17.0-4.dev.rhaos4.4.git186c230.el7 will be installed\n--> Processing Dependency: skopeo-containers >= 1:0.1.40-1 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n--> Processing Dependency: conmon >= 2.0.2-2 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n---> Package openshift-clients.x86_64 0:4.4.0-202002282323.git.0.d2acde5.el7 will be installed\n---> Package openshift-hyperkube.x86_64 0:4.4.0-202002282323.git.0.108c140.el7 will be installed\n--> Processing Dependency: socat for package: openshift-hyperkube-4.4.0-202002282323.git.0.108c140.el7.x86_64\n--> Running transaction check\n---> Package cri-o.x86_64 0:1.17.0-4.dev.rhaos4.4.git186c230.el7 will be installed\n--> Processing Dependency: skopeo-containers >= 1:0.1.40-1 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n--> Processing Dependency: conmon >= 2.0.2-2 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64\n---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"]} Seem like skopeo build is removed in https://errata.devel.redhat.com/advisory/47982#c723, which caused this failure. 4.4/2020-02-29.3 has no such issue. Version-Release number of the following components: rpm -q openshift-ansible rpm -q ansible ansible --version How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated Expected results: Additional info: Please attach logs from ansible-playbook with the -vvv flag
Moving to the 'node' component as this is related to the availability of cri-o and dependent packages.
I untagged builds of RPMs that we are not using in container images or RHCOS. It makes ART sad when we have to ship CVE rebuilds for things no one is actually using. I was under the impression that install could use skopeo et al from RHEL directly. If that's not the case, or the ones shipped in RHEL 7 aren't recent enough builds, I'm happy to add these back and ship them. Requests for specific RPMs would be appreciated. Here it looks like skopeo and containers-common RPMs would be needed, anything else? Since this is essentially a delivery/install problem, moving to Release component.
So far, I only can see skopeo-0.1.37-3.el7 available from RHEL Extra repo. While cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7 want 0.1.40-2.el7, which is never shipped in RHEL Extras repo.
In latest 2020-03-05.3 puddle, skopeo is back, but it still need fuse-overlayfs move back together as dependency, otherwise, will this such failure: # yum install "cri-o-1.17*" ansible29 | 4.0 kB 00:00:00 ose-v4 | 1.2 kB 00:00:00 rhel7 | 3.5 kB 00:00:00 rhel7-extra | 3.4 kB 00:00:00 ose-v4/primary | 26 kB 00:00:00 ose-v4 86/86 Resolving Dependencies --> Running transaction check ---> Package cri-o.x86_64 0:1.10.5-1.rhaos3.10.git4c81456.el7 will be updated ---> Package cri-o.x86_64 0:1.17.0-4.dev.rhaos4.4.git186c230.el7 will be an update --> Processing Dependency: skopeo-containers >= 1:0.1.40-1 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64 --> Processing Dependency: containernetworking-plugins >= 0.8.2-3 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64 --> Processing Dependency: conmon >= 2.0.2-2 for package: cri-o-1.17.0-4.dev.rhaos4.4.git186c230.el7.x86_64 --> Running transaction check ---> Package conmon.x86_64 0:2.0.9-1.rhaos4.4.el7 will be installed ---> Package containernetworking-plugins.x86_64 0:0.8.1-2.el7 will be updated ---> Package containernetworking-plugins.x86_64 0:0.8.2-3.el7 will be an update ---> Package containers-common.x86_64 1:0.1.37-3.el7 will be updated --> Processing Dependency: containers-common = 1:0.1.37-3.el7 for package: 1:skopeo-0.1.37-3.el7.x86_64 ---> Package containers-common.x86_64 1:0.1.40-1.rhaos4.3.el7 will be an update --> Processing Dependency: slirp4netns for package: 1:containers-common-0.1.40-1.rhaos4.3.el7.x86_64 --> Processing Dependency: fuse-overlayfs for package: 1:containers-common-0.1.40-1.rhaos4.3.el7.x86_64 --> Running transaction check ---> Package containers-common.x86_64 1:0.1.40-1.rhaos4.3.el7 will be an update --> Processing Dependency: fuse-overlayfs for package: 1:containers-common-0.1.40-1.rhaos4.3.el7.x86_64 ---> Package skopeo.x86_64 1:0.1.37-3.el7 will be updated ---> Package skopeo.x86_64 1:0.1.40-1.rhaos4.3.el7 will be an update ---> Package slirp4netns.x86_64 0:0.3.0-1.el7 will be installed --> Finished Dependency Resolution Error: Package: 1:containers-common-0.1.40-1.rhaos4.3.el7.x86_64 (ose-v4) Requires: fuse-overlayfs You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest
Verified this bug with 4.4/2020-03-09.1, and PASS. # yum install "cri-o-1.17*" <--SNIP--> Dependencies Resolved ============================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================== Updating: cri-o x86_64 1.17.0-8.dev.rhaos4.4.git36920a5.el7 ose-v4 27 M Installing for dependencies: conmon x86_64 2.0.11-1.rhaos4.4.el7 ose-v4 32 k fuse-overlayfs x86_64 0.7.2-1.el7 ose-v4 52 k fuse3-libs x86_64 3.6.1-4.el7 ose-v4 81 k slirp4netns x86_64 0.3.0-1.el7 rhel7-extra 77 k Updating for dependencies: containernetworking-plugins x86_64 0.8.2-3.el7 ose-v4 20 M containers-common x86_64 1:0.1.40-1.rhaos4.3.el7 ose-v4 27 k skopeo x86_64 1:0.1.40-1.rhaos4.3.el7 ose-v4 5.7 M Transaction Summary ============================================================================================================================================================================================== Install ( 4 Dependent packages) Upgrade 1 Package (+3 Dependent packages) <--SNIP--> Also scale up successfully per https://openshift-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/ocp4-rhel-scaleup-runner/1158/consoleFull.
This should have been closed with the 4.4 GA release.