Bug 1573279 - Containerized deployment fails with skopeo-containers conflicts
Summary: Containerized deployment fails with skopeo-containers conflicts
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: Build
Version: 3.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: z3
: 3.0
Assignee: ceph-eng-bugs
QA Contact: ceph-qe-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-30 17:54 UTC by Valerii Shevchenko
Modified: 2018-05-03 12:23 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-03 12:23:54 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Valerii Shevchenko 2018-04-30 17:54:38 UTC
Description of problem:
Containerized deployment fails with skopeo-containers conflicts during running ceph-ansible playbook. Issue is repated to latest container images, not reproducible for older images.

Version-Release number of selected component (if applicable):
RHCEPH-3.0-RHEL-7-20180427.ci.1
brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph:ceph-3.0-rhel-7-docker-candidate-30343-20180427015126

How reproducible:
Persistent

Steps to Reproduce:
1. Prepare an environment to match hosts file
2. Edit ceph-ansible config
3. Run container playbook
4. Observe results

hosts file: 
[mons]
ceph-jenkins-build-run750-node2-osdmonmgr monitor_interface=eth0
ceph-jenkins-build-run750-node5-monmgr monitor_interface=eth0
ceph-jenkins-build-run750-node6-monmgr monitor_interface=eth0
[mgrs]
ceph-jenkins-build-run750-node2-osdmonmgr monitor_interface=eth0
ceph-jenkins-build-run750-node5-monmgr monitor_interface=eth0
ceph-jenkins-build-run750-node6-monmgr monitor_interface=eth0
[osds]
ceph-jenkins-build-run750-node9-osd monitor_interface=eth0  devices='["/dev/vdb", "/dev/vdc", "/dev/vdd", "/dev/vde"]'
ceph-jenkins-build-run750-node3-osdrgw monitor_interface=eth0  devices='["/dev/vdb", "/dev/vdc", "/dev/vdd", "/dev/vde"]'
ceph-jenkins-build-run750-node2-osdmonmgr monitor_interface=eth0  devices='["/dev/vdb", "/dev/vdc", "/dev/vdd", "/dev/vde"]'
ceph-jenkins-build-run750-node4-osdmds monitor_interface=eth0  devices='["/dev/vdb", "/dev/vdc", "/dev/vdd", "/dev/vde"]'
[mdss]
ceph-jenkins-build-run750-node4-osdmds monitor_interface=eth0
ceph-jenkins-build-run750-node7-mds monitor_interface=eth0
[rgws]
ceph-jenkins-build-run750-node8-rgw radosgw_interface=eth0
ceph-jenkins-build-run750-node3-osdrgw radosgw_interface=eth0

all.yml:
ceph_conf_overrides:
  global:
    mon_max_pg_per_osd: 1024
    osd_default_pool_size: 2
    osd_pool_default_pg_num: 64
    osd_pool_default_pgp_num: 64
  mon:
    mon_allow_pool_delete: true
ceph_docker_image: rhceph
ceph_docker_image_tag: ceph-3.0-rhel-7-docker-candidate-30343-20180427015126
ceph_docker_registry: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888
ceph_origin: distro
ceph_repository: rhcs
ceph_stable: true
ceph_stable_release: luminous
ceph_stable_rh_storage: true
ceph_test: true
containerized_deployment: true
copy_admin_key: true
fetch_directory: ~/fetch
journal_size: 1024
osd_auto_discovery: false
osd_scenario: collocated
public_network: 172.16.0.0/12

Actual results:
FAILED! => {"changed": false, "msg": "Error: skopeo-containers conflicts with 1:atomic-registries-1.22.1-1.gitd36c015.el7.x86_64\n"

http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-1524851990969/containerized_ceph_ansible_0.log

Expected results:

http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-1525082779523/containerized_ceph_ansible_0.log

The passing sample is taken under the same conditions, using the same ceph-ansible version and environment for  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph:ceph-3.0-rhel-7-docker-candidate-53533-20180320051359 (20-Mar-2018)

Additional info:

Comment 4 Sébastien Han 2018-05-02 13:28:17 UTC
Looks like a dependency issue during the installation of Docker, see:

2018-04-27 18:23:34,644 - ceph.ceph - INFO - fatal: [ceph-jenkins-build-run750-node2-osdmonmgr]: FAILED! => {"changed": false, "msg": "Error: skopeo-containers conflicts with 1:atomic-registries-1.22.1-1.gitd36c015.el7.x86_64\n", "rc": 1, "results": ["Loaded plugins: product-id, search-disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package docker.x86_64 2:1.13.1-58.git87f2fab.el7 will be installed\n--> Processing Dependency: docker-client = 2:1.13.1-58.git87f2fab.el7 for package: 2:docker-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: docker-common = 2:1.13.1-58.git87f2fab.el7 for package: 2:docker-1.13.1-58.git87f2fab.el7.x86_64\n--> Running transaction check\n---> Package docker-client.x86_64 2:1.13.1-58.git87f2fab.el7 will be installed\n---> Package docker-common.x86_64 2:1.13.1-58.git87f2fab.el7 will be installed\n--> Processing Dependency: docker-rhel-push-plugin = 2:1.13.1-58.git87f2fab.el7 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: contai
2018-04-27 18:23:34,644 - ceph.ceph - INFO - ner-selinux >= 2:2.51-1 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: container-storage-setup >= 0.9.0-1 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: lvm2 >= 2.02.112 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: oci-register-machine >= 1:0-5.13 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: oci-systemd-hook >= 1:0.1.4-9 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: oci-umount >= 2:2.3.3-3 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: skopeo-containers >= 1:0.1.26-2 for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: atomic-registries for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency: subscription-manager-plugin-container for package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\
2018-04-27 18:23:34,644 - ceph.ceph - INFO - n--> Running transaction check\n---> Package atomic-registries.x86_64 1:1.22.1-1.gitd36c015.el7 will be installed\n--> Processing Dependency: python-pytoml for package: 1:atomic-registries-1.22.1-1.gitd36c015.el7.x86_64\n---> Package container-selinux.noarch 2:2.55-1.el7 will be installed\n---> Package container-storage-setup.noarch 0:0.9.0-1.rhel75.gite0997c3.el7 will be installed\n---> Package docker-rhel-push-plugin.x86_64 2:1.13.1-58.git87f2fab.el7 will be installed\n---> Package lvm2.x86_64 7:2.02.177-4.el7 will be installed\n--> Processing Dependency: lvm2-libs = 7:2.02.177-4.el7 for package: 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: device-mapper-persistent-data >= 0.7.0-0.1.rc6 for package: 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: libdevmapper-event.so.1.02(Base)(64bit) for package: 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: liblvm2app.so.2.2(Base)(64bit) for package: 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: libdevmapper-event.so.1.02()(64bi
2018-04-27 18:23:34,644 - ceph.ceph - INFO - t) for package: 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency: liblvm2app.so.2.2()(64bit) for package: 7:lvm2-2.02.177-4.el7.x86_64\n---> Package oci-register-machine.x86_64 1:0-6.git2b44233.el7 will be installed\n---> Package oci-systemd-hook.x86_64 1:0.1.15-2.gitc04483d.el7 will be installed\n--> Processing Dependency: libyajl.so.2()(64bit) for package: 1:oci-systemd-hook-0.1.15-2.gitc04483d.el7.x86_64\n---> Package oci-umount.x86_64 2:2.3.3-3.gite3c9055.el7 will be installed\n---> Package skopeo-containers.x86_64 1:0.1.29-3.dev.git7add6fc.el7 will be installed\n---> Package subscription-manager-plugin-container.x86_64 0:1.20.11-1.el7_5 will be installed\n--> Processing Dependency: subscription-manager = 1.20.11-1.el7_5 for package: subscription-manager-plugin-container-1.20.11-1.el7_5.x86_64\n--> Running transaction check\n---> Package device-mapper-event-libs.x86_64 7:1.02.146-4.el7 will be installed\n---> Package device-mapper-persistent-data.x86_64 0:0.7.3-3.el7 will be installed\n--> Processi
2018-04-27 18:23:34,645 - ceph.ceph - INFO - ng Dependency: libaio.so.1(LIBAIO_0.1)(64bit) for package: device-mapper-persistent-data-0.7.3-3.el7.x86_64\n--> Processing Dependency: libaio.so.1(LIBAIO_0.4)(64bit) for package: device-mapper-persistent-data-0.7.3-3.el7.x86_64\n--> Processing Dependency: libaio.so.1()(64bit) for package: device-mapper-persistent-data-0.7.3-3.el7.x86_64\n---> Package lvm2-libs.x86_64 7:2.02.177-4.el7 will be installed\n--> Processing Dependency: device-mapper-event = 7:1.02.146-4.el7 for package: 7:lvm2-libs-2.02.177-4.el7.x86_64\n---> Package python-pytoml.noarch 0:0.1.14-1.git7dea353.el7 will be installed\n---> Package subscription-manager.x86_64 0:1.20.10-1.el7 will be updated\n---> Package subscription-manager.x86_64 0:1.20.11-1.el7_5 will be an update\n--> Processing Dependency: subscription-manager-rhsm = 1.20.11 for package: subscription-manager-1.20.11-1.el7_5.x86_64\n---> Package yajl.x86_64 0:2.0.4-4.el7 will be installed\n--> Running transaction check\n---> Package device-mapper-event.x86_64 7:1.02.146-4.el7 will 
2018-04-27 18:23:34,645 - ceph.ceph - INFO - be installed\n---> Package libaio.x86_64 0:0.3.109-13.el7 will be installed\n---> Package subscription-manager-rhsm.x86_64 0:1.20.10-1.el7 will be updated\n---> Package subscription-manager-rhsm.x86_64 0:1.20.11-1.el7_5 will be an update\n--> Processing Dependency: subscription-manager-rhsm-certificates = 1.20.11-1.el7_5 for package: subscription-manager-rhsm-1.20.11-1.el7_5.x86_64\n--> Running transaction check\n---> Package subscription-manager-rhsm-certificates.x86_64 0:1.20.10-1.el7 will be updated\n---> Package subscription-manager-rhsm-certificates.x86_64 0:1.20.11-1.el7_5 will be an update\n--> Processing Conflict: 1:skopeo-containers-0.1.29-3.dev.git7add6fc.el7.x86_64 conflicts atomic-registries <= 1:1.22.1-2\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"]}


I don't think this has anything to do with ceph-ansible. We only try to install a package.

Comment 5 Valerii Shevchenko 2018-05-02 15:50:51 UTC
(In reply to leseb from comment #4)
> Looks like a dependency issue during the installation of Docker, see:
> 
> 2018-04-27 18:23:34,644 - ceph.ceph - INFO - fatal:
> [ceph-jenkins-build-run750-node2-osdmonmgr]: FAILED! => {"changed": false,
> "msg": "Error: skopeo-containers conflicts with
> 1:atomic-registries-1.22.1-1.gitd36c015.el7.x86_64\n", "rc": 1, "results":
> ["Loaded plugins: product-id, search-disabled-repos,
> subscription-manager\nResolving Dependencies\n--> Running transaction
> check\n---> Package docker.x86_64 2:1.13.1-58.git87f2fab.el7 will be
> installed\n--> Processing Dependency: docker-client =
> 2:1.13.1-58.git87f2fab.el7 for package:
> 2:docker-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency:
> docker-common = 2:1.13.1-58.git87f2fab.el7 for package:
> 2:docker-1.13.1-58.git87f2fab.el7.x86_64\n--> Running transaction
> check\n---> Package docker-client.x86_64 2:1.13.1-58.git87f2fab.el7 will be
> installed\n---> Package docker-common.x86_64 2:1.13.1-58.git87f2fab.el7 will
> be installed\n--> Processing Dependency: docker-rhel-push-plugin =
> 2:1.13.1-58.git87f2fab.el7 for package:
> 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency:
> contai
> 2018-04-27 18:23:34,644 - ceph.ceph - INFO - ner-selinux >= 2:2.51-1 for
> package: 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing
> Dependency: container-storage-setup >= 0.9.0-1 for package:
> 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency:
> lvm2 >= 2.02.112 for package:
> 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency:
> oci-register-machine >= 1:0-5.13 for package:
> 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency:
> oci-systemd-hook >= 1:0.1.4-9 for package:
> 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency:
> oci-umount >= 2:2.3.3-3 for package:
> 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency:
> skopeo-containers >= 1:0.1.26-2 for package:
> 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency:
> atomic-registries for package:
> 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\n--> Processing Dependency:
> subscription-manager-plugin-container for package:
> 2:docker-common-1.13.1-58.git87f2fab.el7.x86_64\
> 2018-04-27 18:23:34,644 - ceph.ceph - INFO - n--> Running transaction
> check\n---> Package atomic-registries.x86_64 1:1.22.1-1.gitd36c015.el7 will
> be installed\n--> Processing Dependency: python-pytoml for package:
> 1:atomic-registries-1.22.1-1.gitd36c015.el7.x86_64\n---> Package
> container-selinux.noarch 2:2.55-1.el7 will be installed\n---> Package
> container-storage-setup.noarch 0:0.9.0-1.rhel75.gite0997c3.el7 will be
> installed\n---> Package docker-rhel-push-plugin.x86_64
> 2:1.13.1-58.git87f2fab.el7 will be installed\n---> Package lvm2.x86_64
> 7:2.02.177-4.el7 will be installed\n--> Processing Dependency: lvm2-libs =
> 7:2.02.177-4.el7 for package: 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing
> Dependency: device-mapper-persistent-data >= 0.7.0-0.1.rc6 for package:
> 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency:
> libdevmapper-event.so.1.02(Base)(64bit) for package:
> 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency:
> liblvm2app.so.2.2(Base)(64bit) for package:
> 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency:
> libdevmapper-event.so.1.02()(64bi
> 2018-04-27 18:23:34,644 - ceph.ceph - INFO - t) for package:
> 7:lvm2-2.02.177-4.el7.x86_64\n--> Processing Dependency:
> liblvm2app.so.2.2()(64bit) for package: 7:lvm2-2.02.177-4.el7.x86_64\n--->
> Package oci-register-machine.x86_64 1:0-6.git2b44233.el7 will be
> installed\n---> Package oci-systemd-hook.x86_64 1:0.1.15-2.gitc04483d.el7
> will be installed\n--> Processing Dependency: libyajl.so.2()(64bit) for
> package: 1:oci-systemd-hook-0.1.15-2.gitc04483d.el7.x86_64\n---> Package
> oci-umount.x86_64 2:2.3.3-3.gite3c9055.el7 will be installed\n---> Package
> skopeo-containers.x86_64 1:0.1.29-3.dev.git7add6fc.el7 will be
> installed\n---> Package subscription-manager-plugin-container.x86_64
> 0:1.20.11-1.el7_5 will be installed\n--> Processing Dependency:
> subscription-manager = 1.20.11-1.el7_5 for package:
> subscription-manager-plugin-container-1.20.11-1.el7_5.x86_64\n--> Running
> transaction check\n---> Package device-mapper-event-libs.x86_64
> 7:1.02.146-4.el7 will be installed\n---> Package
> device-mapper-persistent-data.x86_64 0:0.7.3-3.el7 will be installed\n-->
> Processi
> 2018-04-27 18:23:34,645 - ceph.ceph - INFO - ng Dependency:
> libaio.so.1(LIBAIO_0.1)(64bit) for package:
> device-mapper-persistent-data-0.7.3-3.el7.x86_64\n--> Processing Dependency:
> libaio.so.1(LIBAIO_0.4)(64bit) for package:
> device-mapper-persistent-data-0.7.3-3.el7.x86_64\n--> Processing Dependency:
> libaio.so.1()(64bit) for package:
> device-mapper-persistent-data-0.7.3-3.el7.x86_64\n---> Package
> lvm2-libs.x86_64 7:2.02.177-4.el7 will be installed\n--> Processing
> Dependency: device-mapper-event = 7:1.02.146-4.el7 for package:
> 7:lvm2-libs-2.02.177-4.el7.x86_64\n---> Package python-pytoml.noarch
> 0:0.1.14-1.git7dea353.el7 will be installed\n---> Package
> subscription-manager.x86_64 0:1.20.10-1.el7 will be updated\n---> Package
> subscription-manager.x86_64 0:1.20.11-1.el7_5 will be an update\n-->
> Processing Dependency: subscription-manager-rhsm = 1.20.11 for package:
> subscription-manager-1.20.11-1.el7_5.x86_64\n---> Package yajl.x86_64
> 0:2.0.4-4.el7 will be installed\n--> Running transaction check\n---> Package
> device-mapper-event.x86_64 7:1.02.146-4.el7 will 
> 2018-04-27 18:23:34,645 - ceph.ceph - INFO - be installed\n---> Package
> libaio.x86_64 0:0.3.109-13.el7 will be installed\n---> Package
> subscription-manager-rhsm.x86_64 0:1.20.10-1.el7 will be updated\n--->
> Package subscription-manager-rhsm.x86_64 0:1.20.11-1.el7_5 will be an
> update\n--> Processing Dependency: subscription-manager-rhsm-certificates =
> 1.20.11-1.el7_5 for package:
> subscription-manager-rhsm-1.20.11-1.el7_5.x86_64\n--> Running transaction
> check\n---> Package subscription-manager-rhsm-certificates.x86_64
> 0:1.20.10-1.el7 will be updated\n---> Package
> subscription-manager-rhsm-certificates.x86_64 0:1.20.11-1.el7_5 will be an
> update\n--> Processing Conflict:
> 1:skopeo-containers-0.1.29-3.dev.git7add6fc.el7.x86_64 conflicts
> atomic-registries <= 1:1.22.1-2\n--> Finished Dependency Resolution\n You
> could try using --skip-broken to work around the problem\n You could try
> running: rpm -Va --nofiles --nodigest\n"]}
> 
> 
> I don't think this has anything to do with ceph-ansible. We only try to
> install a package.

I wonder why would the previous docker image passes (I've restested this after the failure) and the current latest image fails under same conditions? We are using released RHEL7.4 and RHEL7.5 images in osp, the same subscription process. I do not see any change for subscription process requirements and it looks like regression issue from our side. In addition the latest 2.5 does not have this issue.

Comment 6 Sébastien Han 2018-05-02 16:31:03 UTC
Unless I missed something, the issue is on the package install so I'm not sure why you're mentioning the docker image here.

Comment 9 Vasu Kulkarni 2018-05-03 02:58:12 UTC
Sebastein,

I think Valeri meant to say this issue is seen in docker sanity runs but in prerequisites playbook at below line, so I am going to move this to build component 

https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-docker-common/tasks/pre_requisites/prerequisites.yml#L44-L51

Comment 10 Ken Dreyer (Red Hat) 2018-05-03 03:25:11 UTC
The failure happens when installing docker-1.13.1-58.git87f2fab.el7, whereas the successful run installs docker-1.13.1-62.gitc6c9b51.el7 . My guess is that there's some problem going on with RHEL 7.5 Extras, but I need to know more details about the exact repo this node is using.

Would you please provide the whole contents of /etc/yum.repos.d/* on the node that is failing to install the docker package? Specifically, which RHEL Extras compose is this using? (if it's using a "latest" symlink, let's figure out the time and exact RHEL Extras compose id)

Comment 11 Valerii Shevchenko 2018-05-03 12:01:55 UTC
(In reply to Ken Dreyer (Red Hat) from comment #10)
> The failure happens when installing docker-1.13.1-58.git87f2fab.el7, whereas
> the successful run installs docker-1.13.1-62.gitc6c9b51.el7 . My guess is
> that there's some problem going on with RHEL 7.5 Extras, but I need to know
> more details about the exact repo this node is using.
> 
> Would you please provide the whole contents of /etc/yum.repos.d/* on the
> node that is failing to install the docker package? Specifically, which RHEL
> Extras compose is this using? (if it's using a "latest" symlink, let's
> figure out the time and exact RHEL Extras compose id)

I've tested this on the latest container available by yesterday (tag ceph-3.0-rhel-7-containers-candidate-61607-20180502085052) and see that is's passing. The conflict is specific to ceph-3.0-rhel-7-docker-candidate-30343-20180427015126 tag and not reproducible at the moment.
http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-1525280184362/containerized_ceph_ansible_0.log

The node is using generic repos from stage:
sudo subscription-manager repos --enable=rhel-7-server-rpms  --enable=rhel-7-server-optional-rpms --enable=rhel-7-server-extras-rpms

The failure used to happen during the container deployment, docker was already installed

I'm setting this to verified as not reproducible with latest container.

Comment 12 Ken Dreyer (Red Hat) 2018-05-03 12:23:54 UTC
I think the bug was related to RHEL 7 Extras in stage, not related to the ceph container. Nevertheless thanks for reporting this, because if the prod CDN had the same problem as the stage CDN, customers would have been sad.

If you find any package conflicts again, would you please mention that you're using the stage CDN in the bug description? We would need to escalate those to RCM and the product team that maintains the packages in RHEL.


Note You need to log in before you can comment on or make changes to this bug.