Created attachment 1244202 [details] ansible playbook log Description of problem: I have a setup where mds is colocated on the MON node. during rolling_update where MON update is tried, the ceph-base is not allowed to upgrade due to dependency on ceph-base. So this forms a cyclic dependency. TASK [ceph.ceph-common : install distro or red hat storage ceph mon] *********** fatal: [magna028]: FAILED! => {"changed": true, "failed": true, "msg": "Error: Package: 1:ceph-mds-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-tools-rpms)\n Requires: ceph-base = 1:10.2.3-17.el7cp\n Removing: 1:ceph-base-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-mon-rpms)\n ceph-base = 1:10.2.3-17.el7cp\n Updated By: 1:ceph-base-10.2.5-12.el7cp.x86_64 (download-node-02.eng.bos.redhat.com_rcm-guest_ceph-drops_auto_ceph-2-rhel-7-compose_latest-RHCEPH-2-RHEL-7_compose_MON_x86_64_os_)\n ceph-base = 1:10.2.5-12.el7cp\n Available: 1:ceph-base-10.2.2-38.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)\n ceph-base = 1:10.2.2-38.el7cp\n Available: 1:ceph-base-10.2.2-41.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)\n ceph-base = 1:10.2.2-41.el7cp\n Available: 1:ceph-base-10.2.3-13.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)\n ceph-base = 1:10.2.3-13.el7cp\n", "rc": 1, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\nResolving Dependencies\n--> Running transaction check\n---> Package ceph-mon.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package ceph-mon.x86_64 1:10.2.5-12.el7cp will be an update\n--> Processing Dependency: ceph-base = 1:10.2.5-12.el7cp for package: 1:ceph-mon-10.2.5-12.el7cp.x86_64\n--> Running transaction check\n---> Package ceph-base.x86_64 1:10.2.3-17.el7cp will be updated\n--> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-selinux-10.2.3-17.el7cp.x86_64\n--> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-mds-10.2.3-17.el7cp.x86_64\n---> Package ceph-base.x86_64 1:10.2.5-12.el7cp will be an update\n--> Processing Dependency: ceph-common = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64\n--> Processing Dependency: libcephfs1 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64\n--> Processing Dependency: librados2 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64\n--> Processing Dependency: librbd1 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64\n--> Processing Dependency: librgw2 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64\n--> Running transaction check\n---> Package ceph-base.x86_64 1:10.2.3-17.el7cp will be updated\n--> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-mds-10.2.3-17.el7cp.x86_64\n---> Package ceph-common.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package ceph-common.x86_64 1:10.2.5-12.el7cp will be an update\n--> Processing Dependency: python-cephfs = 1:10.2.5-12.el7cp for package: 1:ceph-common-10.2.5-12.el7cp.x86_64\n--> Processing Dependency: python-rados = 1:10.2.5-12.el7cp for package: 1:ceph-common-10.2.5-12.el7cp.x86_64\n--> Processing Dependency: python-rbd = 1:10.2.5-12.el7cp for package: 1:ceph-common-10.2.5-12.el7cp.x86_64\n---> Package ceph-selinux.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package ceph-selinux.x86_64 1:10.2.5-12.el7cp will be an update\n---> Package libcephfs1.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package libcephfs1.x86_64 1:10.2.5-12.el7cp will be an update\n---> Package librados2.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package librados2.x86_64 1:10.2.5-12.el7cp will be an update\n---> Package librbd1.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package librbd1.x86_64 1:10.2.5-12.el7cp will be an update\n---> Package librgw2.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package librgw2.x86_64 1:10.2.5-12.el7cp will be an update\n--> Running transaction check\n---> Package ceph-base.x86_64 1:10.2.3-17.el7cp will be updated\n--> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-mds-10.2.3-17.el7cp.x86_64\n---> Package python-cephfs.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package python-cephfs.x86_64 1:10.2.5-12.el7cp will be an update\n---> Package python-rados.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package python-rados.x86_64 1:10.2.5-12.el7cp will be an update\n---> Package python-rbd.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package python-rbd.x86_64 1:10.2.5-12.el7cp will be an update\n--> Finished Dependency Resolution\nError: Package: 1:ceph-mds-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-tools-rpms)\n Requires: ceph-base = 1:10.2.3-17.el7cp\n Removing: 1:ceph-base-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-mon-rpms)\n ceph-base = 1:10.2.3-17.el7cp\n Updated By: 1:ceph-base-10.2.5-12.el7cp.x86_64 (download-node-02.eng.bos.redhat.com_rcm-guest_ceph-drops_auto_ceph-2-rhel-7-compose_latest-RHCEPH-2-RHEL-7_compose_MON_x86_64_os_)\n ceph-base = 1:10.2.5-12.el7cp\n Available: 1:ceph-base-10.2.2-38.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)\n ceph-base = 1:10.2.2-38.el7cp\n Available: 1:ceph-base-10.2.2-41.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)\n ceph-base = 1:10.2.2-41.el7cp\n Available: 1:ceph-base-10.2.3-13.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)\n ceph-base = 1:10.2.3-13.el7cp\n**********************************************************************\nyum can be configured to try to resolve such errors by temporarily enabling\ndisabled repos and searching for missing dependencies.\nTo enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf\n**********************************************************************\n\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"]} Version-Release number of selected component (if applicable): ceph-ansible-2.1.3-1.el7scon.noarch ansible-2.2.1.0-1.el7.noarch How reproducible: Always Steps to Reproduce: 1. on a cluster where MOn and mds are colocated, during MON update rolling_update fails Additional info: root@magna028 ~]# yum update ceph-mon Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-disabled-repos, subscription-manager Loading mirror speeds from cached hostfile Resolving Dependencies --> Running transaction check ---> Package ceph-mon.x86_64 1:10.2.3-17.el7cp will be updated ---> Package ceph-mon.x86_64 1:10.2.5-12.el7cp will be an update --> Processing Dependency: ceph-base = 1:10.2.5-12.el7cp for package: 1:ceph-mon-10.2.5-12.el7cp.x86_64 --> Running transaction check ---> Package ceph-base.x86_64 1:10.2.3-17.el7cp will be updated --> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-selinux-10.2.3-17.el7cp.x86_64 --> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-mds-10.2.3-17.el7cp.x86_64 ---> Package ceph-base.x86_64 1:10.2.5-12.el7cp will be an update --> Processing Dependency: ceph-common = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64 --> Processing Dependency: libcephfs1 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64 --> Processing Dependency: librados2 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64 --> Processing Dependency: librbd1 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64 --> Processing Dependency: librgw2 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64 --> Running transaction check ---> Package ceph-base.x86_64 1:10.2.3-17.el7cp will be updated --> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-mds-10.2.3-17.el7cp.x86_64 ---> Package ceph-common.x86_64 1:10.2.3-17.el7cp will be updated ---> Package ceph-common.x86_64 1:10.2.5-12.el7cp will be an update --> Processing Dependency: python-cephfs = 1:10.2.5-12.el7cp for package: 1:ceph-common-10.2.5-12.el7cp.x86_64 --> Processing Dependency: python-rados = 1:10.2.5-12.el7cp for package: 1:ceph-common-10.2.5-12.el7cp.x86_64 --> Processing Dependency: python-rbd = 1:10.2.5-12.el7cp for package: 1:ceph-common-10.2.5-12.el7cp.x86_64 ---> Package ceph-selinux.x86_64 1:10.2.3-17.el7cp will be updated ---> Package ceph-selinux.x86_64 1:10.2.5-12.el7cp will be an update ---> Package libcephfs1.x86_64 1:10.2.3-17.el7cp will be updated ---> Package libcephfs1.x86_64 1:10.2.5-12.el7cp will be an update ---> Package librados2.x86_64 1:10.2.3-17.el7cp will be updated ---> Package librados2.x86_64 1:10.2.5-12.el7cp will be an update ---> Package librbd1.x86_64 1:10.2.3-17.el7cp will be updated ---> Package librbd1.x86_64 1:10.2.5-12.el7cp will be an update ---> Package librgw2.x86_64 1:10.2.3-17.el7cp will be updated ---> Package librgw2.x86_64 1:10.2.5-12.el7cp will be an update --> Running transaction check ---> Package ceph-base.x86_64 1:10.2.3-17.el7cp will be updated --> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-mds-10.2.3-17.el7cp.x86_64 ---> Package python-cephfs.x86_64 1:10.2.3-17.el7cp will be updated ---> Package python-cephfs.x86_64 1:10.2.5-12.el7cp will be an update ---> Package python-rados.x86_64 1:10.2.3-17.el7cp will be updated ---> Package python-rados.x86_64 1:10.2.5-12.el7cp will be an update ---> Package python-rbd.x86_64 1:10.2.3-17.el7cp will be updated ---> Package python-rbd.x86_64 1:10.2.5-12.el7cp will be an update --> Finished Dependency Resolution Error: Package: 1:ceph-mds-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-tools-rpms) Requires: ceph-base = 1:10.2.3-17.el7cp Removing: 1:ceph-base-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-mon-rpms) ceph-base = 1:10.2.3-17.el7cp Updated By: 1:ceph-base-10.2.5-12.el7cp.x86_64 (download-node-02.eng.bos.redhat.com_rcm-guest_ceph-drops_auto_ceph-2-rhel-7-compose_latest-RHCEPH-2-RHEL-7_compose_MON_x86_64_os_) ceph-base = 1:10.2.5-12.el7cp Available: 1:ceph-base-10.2.2-38.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms) ceph-base = 1:10.2.2-38.el7cp Available: 1:ceph-base-10.2.2-41.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms) ceph-base = 1:10.2.2-41.el7cp Available: 1:ceph-base-10.2.3-13.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms) ceph-base = 1:10.2.3-13.el7cp ********************************************************************** yum can be configured to try to resolve such errors by temporarily enabling disabled repos and searching for missing dependencies. To enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf ********************************************************************** Error: Package: 1:ceph-mds-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-tools-rpms) Requires: ceph-base = 1:10.2.3-17.el7cp Removing: 1:ceph-base-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-mon-rpms) ceph-base = 1:10.2.3-17.el7cp Updated By: 1:ceph-base-10.2.5-12.el7cp.x86_64 (download-node-02.eng.bos.redhat.com_rcm-guest_ceph-drops_auto_ceph-2-rhel-7-compose_latest-RHCEPH-2-RHEL-7_compose_MON_x86_64_os_) ceph-base = 1:10.2.5-12.el7cp Available: 1:ceph-base-10.2.2-38.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms) ceph-base = 1:10.2.2-38.el7cp Available: 1:ceph-base-10.2.2-41.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms) ceph-base = 1:10.2.2-41.el7cp Available: 1:ceph-base-10.2.3-13.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms) ceph-base = 1:10.2.3-13.el7cp You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest [root@magna028 ~]# Attaching the playbook log.
I checked magna028 today. I think this is because you've enabled the "rhel-7-server-rhceph-2-tools-rpms" repository on this node, but you didn't enable the "Tools" repo from the latest compose (only the "MON" repo is enabled in /etc/yum.repos.d). Would you please enable the Tools repository from the compose and re-try?
Ken, Thanks for the info. It is because the Tools repo is not enabled. This is not a bug.