Hide Forgot
Description of problem: Containerized deployment fails with skopeo-containers conflicts during running ceph-ansible playbook. Issue is repated to latest container images, not reproducible for older images. Version-Release number of selected component (if applicable): RHCEPH-3.0-RHEL-7-20180427.ci.1 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph:ceph-3.0-rhel-7-docker-candidate-30343-20180427015126 How reproducible: Persistent Steps to Reproduce: 1. Prepare an environment to match hosts file 2. Edit ceph-ansible config 3. Run container playbook 4. Observe results hosts file: [mons] ceph-jenkins-build-run750-node2-osdmonmgr monitor_interface=eth0 ceph-jenkins-build-run750-node5-monmgr monitor_interface=eth0 ceph-jenkins-build-run750-node6-monmgr monitor_interface=eth0 [mgrs] ceph-jenkins-build-run750-node2-osdmonmgr monitor_interface=eth0 ceph-jenkins-build-run750-node5-monmgr monitor_interface=eth0 ceph-jenkins-build-run750-node6-monmgr monitor_interface=eth0 [osds] ceph-jenkins-build-run750-node9-osd monitor_interface=eth0 devices='["/dev/vdb", "/dev/vdc", "/dev/vdd", "/dev/vde"]' ceph-jenkins-build-run750-node3-osdrgw monitor_interface=eth0 devices='["/dev/vdb", "/dev/vdc", "/dev/vdd", "/dev/vde"]' ceph-jenkins-build-run750-node2-osdmonmgr monitor_interface=eth0 devices='["/dev/vdb", "/dev/vdc", "/dev/vdd", "/dev/vde"]' ceph-jenkins-build-run750-node4-osdmds monitor_interface=eth0 devices='["/dev/vdb", "/dev/vdc", "/dev/vdd", "/dev/vde"]' [mdss] ceph-jenkins-build-run750-node4-osdmds monitor_interface=eth0 ceph-jenkins-build-run750-node7-mds monitor_interface=eth0 [rgws] ceph-jenkins-build-run750-node8-rgw radosgw_interface=eth0 ceph-jenkins-build-run750-node3-osdrgw radosgw_interface=eth0 all.yml: ceph_conf_overrides: global: mon_max_pg_per_osd: 1024 osd_default_pool_size: 2 osd_pool_default_pg_num: 64 osd_pool_default_pgp_num: 64 mon: mon_allow_pool_delete: true ceph_docker_image: rhceph ceph_docker_image_tag: ceph-3.0-rhel-7-docker-candidate-30343-20180427015126 ceph_docker_registry: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 ceph_origin: distro ceph_repository: rhcs ceph_stable: true ceph_stable_release: luminous ceph_stable_rh_storage: true ceph_test: true containerized_deployment: true copy_admin_key: true fetch_directory: ~/fetch journal_size: 1024 osd_auto_discovery: false osd_scenario: collocated public_network: 172.16.0.0/12 Actual results: FAILED! => {"changed": false, "msg": "Error: skopeo-containers conflicts with 1:atomic-registries-1.22.1-1.gitd36c015.el7.x86_64\n" http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-1524851990969/containerized_ceph_ansible_0.log Expected results: http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-1525082779523/containerized_ceph_ansible_0.log The passing sample is taken under the same conditions, using the same ceph-ansible version and environment for brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph:ceph-3.0-rhel-7-docker-candidate-53533-20180320051359 (20-Mar-2018) Additional info:
Looks like a dependency issue during the installation of Docker, see: 2018-04-27 18:23:34,644 - ceph.ceph - INFO - fatal: [ceph-jenkins-build-run750-node2-osdmonmgr]: FAILED! => {"changed": false, "msg": "Error: skopeo-containers conflicts with 1:atomic-registries-1.22.1-1.gitd36c015.el7.x86_64\n", "rc": 1, "results": ["Loaded plugins: product-id, search-disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package docker.x86_64 2:1.13.1-58.git87f2fab.el7 will be installed\n--> Processing Dependency: docker-client = 2:1.13.1-58.git87f2fab.el7 for package: 2:docker-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: docker-common = 2:1.13.1-58.git87f2fab.el7 for package: 2:docker-1.13.1-58.git87f2fab.el7.x86_64\n--> Running transaction check\n---> Package docker-client.x86_64 2:1.13.1-58.git87f2fab.el7 will be installed\n---> Package docker-common.x86_64 2:1.13.1-58.git87f2fab.el7 will be installed\n--> Processing Dependency: docker-rhel-push-plugin = 2:1.13.1-58.git87f2fab.el7 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: contai 2018-04-27 18:23:34,644 - ceph.ceph - INFO - ner-selinux >= 2:2.51-1 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: container-storage-setup >= 0.9.0-1 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: lvm2 >= 2.02.112 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: oci-register-machine >= 1:0-5.13 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: oci-systemd-hook >= 1:0.1.4-9 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: oci-umount >= 2:2.3.3-3 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: skopeo-containers >= 1:0.1.26-2 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: atomic-registries for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: subscription-manager-plugin-container for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\ 2018-04-27 18:23:34,644 - ceph.ceph - INFO - n--> Running transaction check\n---> Package atomic-registries.x86_64 1:1.22.1-1.gitd36c015.el7 will be installed\n--> Processing Dependency: python-pytoml for package: 1:atomic-registries-1.22.1-1.gitd36c015.el7.x86_64\n---> Package container-selinux.noarch 2:2.55-1.el7 will be installed\n---> Package container-storage-setup.noarch 0:0.9.0-1.rhel75.gite0997c3.el7 will be installed\n---> Package docker-rhel-push-plugin.x86_64 2:1.13.1-58.git87f2fab.el7 will be installed\n---> Package lvm2.x86_64 7:2.02.177-4.el7 will be installed\n--> Processing Dependency: lvm2-libs = 7:2.02.177-4.el7 for package: 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: device-mapper-persistent-data >= 0.7.0-0.1.rc6 for package: 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: libdevmapper-event.so.1.02(Base)(64bit) for package: 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: liblvm2app.so.2.2(Base)(64bit) for package: 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: libdevmapper-event.so.1.02()(64bi 2018-04-27 18:23:34,644 - ceph.ceph - INFO - t) for package: 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: liblvm2app.so.2.2()(64bit) for package: 7:lvm2-2.02.177-4.el7.x86_64\n---> Package oci-register-machine.x86_64 1:0-6.git2b44233.el7 will be installed\n---> Package oci-systemd-hook.x86_64 1:0.1.15-2.gitc04483d.el7 will be installed\n--> Processing Dependency: libyajl.so.2()(64bit) for package: 1:oci-systemd-hook-0.1.15-2.gitc04483d.el7.x86_64\n---> Package oci-umount.x86_64 2:2.3.3-3.gite3c9055.el7 will be installed\n---> Package skopeo-containers.x86_64 1:0.1.29-3.dev.git7add6fc.el7 will be installed\n---> Package subscription-manager-plugin-container.x86_64 0:1.20.11-1.el7_5 will be installed\n--> Processing Dependency: subscription-manager = 1.20.11-1.el7_5 for package: subscription-manager-plugin-container-1.20.11-1.el7_5.x86_64\n--> Running transaction check\n---> Package device-mapper-event-libs.x86_64 7:1.02.146-4.el7 will be installed\n---> Package device-mapper-persistent-data.x86_64 0:0.7.3-3.el7 will be installed\n--> Processi 2018-04-27 18:23:34,645 - ceph.ceph - INFO - ng Dependency: libaio.so.1(LIBAIO_0.1)(64bit) for package: device-mapper-persistent-data-0.7.3-3.el7.x86_64\n--> Processing Dependency: libaio.so.1(LIBAIO_0.4)(64bit) for package: device-mapper-persistent-data-0.7.3-3.el7.x86_64\n--> Processing Dependency: libaio.so.1()(64bit) for package: device-mapper-persistent-data-0.7.3-3.el7.x86_64\n---> Package lvm2-libs.x86_64 7:2.02.177-4.el7 will be installed\n--> Processing Dependency: device-mapper-event = 7:1.02.146-4.el7 for package: 7:lvm2-libs-2.02.177-4.el7.x86_64\n---> Package python-pytoml.noarch 0:0.1.14-1.git7dea353.el7 will be installed\n---> Package subscription-manager.x86_64 0:1.20.10-1.el7 will be updated\n---> Package subscription-manager.x86_64 0:1.20.11-1.el7_5 will be an update\n--> Processing Dependency: subscription-manager-rhsm = 1.20.11 for package: subscription-manager-1.20.11-1.el7_5.x86_64\n---> Package yajl.x86_64 0:2.0.4-4.el7 will be installed\n--> Running transaction check\n---> Package device-mapper-event.x86_64 7:1.02.146-4.el7 will 2018-04-27 18:23:34,645 - ceph.ceph - INFO - be installed\n---> Package libaio.x86_64 0:0.3.109-13.el7 will be installed\n---> Package subscription-manager-rhsm.x86_64 0:1.20.10-1.el7 will be updated\n---> Package subscription-manager-rhsm.x86_64 0:1.20.11-1.el7_5 will be an update\n--> Processing Dependency: subscription-manager-rhsm-certificates = 1.20.11-1.el7_5 for package: subscription-manager-rhsm-1.20.11-1.el7_5.x86_64\n--> Running transaction check\n---> Package subscription-manager-rhsm-certificates.x86_64 0:1.20.10-1.el7 will be updated\n---> Package subscription-manager-rhsm-certificates.x86_64 0:1.20.11-1.el7_5 will be an update\n--> Processing Conflict: 1:skopeo-containers-0.1.29-3.dev.git7add6fc.el7.x86_64 conflicts atomic-registries <= 1:1.22.1-2\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"]} I don't think this has anything to do with ceph-ansible. We only try to install a package.
(In reply to leseb from comment #4) > Looks like a dependency issue during the installation of Docker, see: > > 2018-04-27 18:23:34,644 - ceph.ceph - INFO - fatal: > [ceph-jenkins-build-run750-node2-osdmonmgr]: FAILED! => {"changed": false, > "msg": "Error: skopeo-containers conflicts with > 1:atomic-registries-1.22.1-1.gitd36c015.el7.x86_64\n", "rc": 1, "results": > ["Loaded plugins: product-id, search-disabled-repos, > subscription-manager\nResolving Dependencies\n--> Running transaction > check\n---> Package docker.x86_64 2:1.13.1-58.git87f2fab.el7 will be > installed\n--> Processing Dependency: docker-client = > 2:1.13.1-58.git87f2fab.el7 for package: > 2:docker-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: > docker-common = 2:1.13.1-58.git87f2fab.el7 for package: > 2:docker-1.13.1-58.git87f2fab.el7.x86_64\n--> Running transaction > check\n---> Package docker-client.x86_64 2:1.13.1-58.git87f2fab.el7 will be > installed\n---> Package docker-common.x86_64 2:1.13.1-58.git87f2fab.el7 will > be installed\n--> Processing Dependency: docker-rhel-push-plugin = > 2:1.13.1-58.git87f2fab.el7 for package: > 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: > contai > 2018-04-27 18:23:34,644 - ceph.ceph - INFO - ner-selinux >= 2:2.51-1 for > package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing > Dependency: container-storage-setup >= 0.9.0-1 for package: > 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: > lvm2 >= 2.02.112 for package: > 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: > oci-register-machine >= 1:0-5.13 for package: > 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: > oci-systemd-hook >= 1:0.1.4-9 for package: > 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: > oci-umount >= 2:2.3.3-3 for package: > 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: > skopeo-containers >= 1:0.1.26-2 for package: > 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: > atomic-registries for package: > 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: > subscription-manager-plugin-container for package: > 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\ > 2018-04-27 18:23:34,644 - ceph.ceph - INFO - n--> Running transaction > check\n---> Package atomic-registries.x86_64 1:1.22.1-1.gitd36c015.el7 will > be installed\n--> Processing Dependency: python-pytoml for package: > 1:atomic-registries-1.22.1-1.gitd36c015.el7.x86_64\n---> Package > container-selinux.noarch 2:2.55-1.el7 will be installed\n---> Package > container-storage-setup.noarch 0:0.9.0-1.rhel75.gite0997c3.el7 will be > installed\n---> Package docker-rhel-push-plugin.x86_64 > 2:1.13.1-58.git87f2fab.el7 will be installed\n---> Package lvm2.x86_64 > 7:2.02.177-4.el7 will be installed\n--> Processing Dependency: lvm2-libs = > 7:2.02.177-4.el7 for package: 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing > Dependency: device-mapper-persistent-data >= 0.7.0-0.1.rc6 for package: > 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: > libdevmapper-event.so.1.02(Base)(64bit) for package: > 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: > liblvm2app.so.2.2(Base)(64bit) for package: > 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: > libdevmapper-event.so.1.02()(64bi > 2018-04-27 18:23:34,644 - ceph.ceph - INFO - t) for package: > 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: > liblvm2app.so.2.2()(64bit) for package: 7:lvm2-2.02.177-4.el7.x86_64\n---> > Package oci-register-machine.x86_64 1:0-6.git2b44233.el7 will be > installed\n---> Package oci-systemd-hook.x86_64 1:0.1.15-2.gitc04483d.el7 > will be installed\n--> Processing Dependency: libyajl.so.2()(64bit) for > package: 1:oci-systemd-hook-0.1.15-2.gitc04483d.el7.x86_64\n---> Package > oci-umount.x86_64 2:2.3.3-3.gite3c9055.el7 will be installed\n---> Package > skopeo-containers.x86_64 1:0.1.29-3.dev.git7add6fc.el7 will be > installed\n---> Package subscription-manager-plugin-container.x86_64 > 0:1.20.11-1.el7_5 will be installed\n--> Processing Dependency: > subscription-manager = 1.20.11-1.el7_5 for package: > subscription-manager-plugin-container-1.20.11-1.el7_5.x86_64\n--> Running > transaction check\n---> Package device-mapper-event-libs.x86_64 > 7:1.02.146-4.el7 will be installed\n---> Package > device-mapper-persistent-data.x86_64 0:0.7.3-3.el7 will be installed\n--> > Processi > 2018-04-27 18:23:34,645 - ceph.ceph - INFO - ng Dependency: > libaio.so.1(LIBAIO_0.1)(64bit) for package: > device-mapper-persistent-data-0.7.3-3.el7.x86_64\n--> Processing Dependency: > libaio.so.1(LIBAIO_0.4)(64bit) for package: > device-mapper-persistent-data-0.7.3-3.el7.x86_64\n--> Processing Dependency: > libaio.so.1()(64bit) for package: > device-mapper-persistent-data-0.7.3-3.el7.x86_64\n---> Package > lvm2-libs.x86_64 7:2.02.177-4.el7 will be installed\n--> Processing > Dependency: device-mapper-event = 7:1.02.146-4.el7 for package: > 7:lvm2-libs-2.02.177-4.el7.x86_64\n---> Package python-pytoml.noarch > 0:0.1.14-1.git7dea353.el7 will be installed\n---> Package > subscription-manager.x86_64 0:1.20.10-1.el7 will be updated\n---> Package > subscription-manager.x86_64 0:1.20.11-1.el7_5 will be an update\n--> > Processing Dependency: subscription-manager-rhsm = 1.20.11 for package: > subscription-manager-1.20.11-1.el7_5.x86_64\n---> Package yajl.x86_64 > 0:2.0.4-4.el7 will be installed\n--> Running transaction check\n---> Package > device-mapper-event.x86_64 7:1.02.146-4.el7 will > 2018-04-27 18:23:34,645 - ceph.ceph - INFO - be installed\n---> Package > libaio.x86_64 0:0.3.109-13.el7 will be installed\n---> Package > subscription-manager-rhsm.x86_64 0:1.20.10-1.el7 will be updated\n---> > Package subscription-manager-rhsm.x86_64 0:1.20.11-1.el7_5 will be an > update\n--> Processing Dependency: subscription-manager-rhsm-certificates = > 1.20.11-1.el7_5 for package: > subscription-manager-rhsm-1.20.11-1.el7_5.x86_64\n--> Running transaction > check\n---> Package subscription-manager-rhsm-certificates.x86_64 > 0:1.20.10-1.el7 will be updated\n---> Package > subscription-manager-rhsm-certificates.x86_64 0:1.20.11-1.el7_5 will be an > update\n--> Processing Conflict: > 1:skopeo-containers-0.1.29-3.dev.git7add6fc.el7.x86_64 conflicts > atomic-registries <= 1:1.22.1-2\n--> Finished Dependency Resolution\n You > could try using --skip-broken to work around the problem\n You could try > running: rpm -Va --nofiles --nodigest\n"]} > > > I don't think this has anything to do with ceph-ansible. We only try to > install a package. I wonder why would the previous docker image passes (I've restested this after the failure) and the current latest image fails under same conditions? We are using released RHEL7.4 and RHEL7.5 images in osp, the same subscription process. I do not see any change for subscription process requirements and it looks like regression issue from our side. In addition the latest 2.5 does not have this issue.
Unless I missed something, the issue is on the package install so I'm not sure why you're mentioning the docker image here.
Sebastein, I think Valeri meant to say this issue is seen in docker sanity runs but in prerequisites playbook at below line, so I am going to move this to build component https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-docker-common/tasks/pre_requisites/prerequisites.yml#L44-L51
The failure happens when installing docker-1.13.1-58.git87f2fab.el7, whereas the successful run installs docker-1.13.1-62.gitc6c9b51.el7 . My guess is that there's some problem going on with RHEL 7.5 Extras, but I need to know more details about the exact repo this node is using. Would you please provide the whole contents of /etc/yum.repos.d/* on the node that is failing to install the docker package? Specifically, which RHEL Extras compose is this using? (if it's using a "latest" symlink, let's figure out the time and exact RHEL Extras compose id)
(In reply to Ken Dreyer (Red Hat) from comment #10) > The failure happens when installing docker-1.13.1-58.git87f2fab.el7, whereas > the successful run installs docker-1.13.1-62.gitc6c9b51.el7 . My guess is > that there's some problem going on with RHEL 7.5 Extras, but I need to know > more details about the exact repo this node is using. > > Would you please provide the whole contents of /etc/yum.repos.d/* on the > node that is failing to install the docker package? Specifically, which RHEL > Extras compose is this using? (if it's using a "latest" symlink, let's > figure out the time and exact RHEL Extras compose id) I've tested this on the latest container available by yesterday (tag ceph-3.0-rhel-7-containers-candidate-61607-20180502085052) and see that is's passing. The conflict is specific to ceph-3.0-rhel-7-docker-candidate-30343-20180427015126 tag and not reproducible at the moment. http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-1525280184362/containerized_ceph_ansible_0.log The node is using generic repos from stage: sudo subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-optional-rpms --enable=rhel-7-server-extras-rpms The failure used to happen during the container deployment, docker was already installed I'm setting this to verified as not reproducible with latest container.
I think the bug was related to RHEL 7 Extras in stage, not related to the ceph container. Nevertheless thanks for reporting this, because if the prod CDN had the same problem as the stage CDN, customers would have been sad. If you find any package conflicts again, would you please mention that you're using the stage CDN in the bug description? We would need to escalate those to RCM and the product team that maintains the packages in RHEL.