Bug 1416330

Summary: [ceph-ansible]: MON update fails with rolling_update due to dependency on ceph-mds
Product: [Red Hat Storage] Red Hat Storage Console Reporter: Tejas <tchandra>
Component: ceph-ansibleAssignee: Ken Dreyer (Red Hat) <kdreyer>
Status: CLOSED NOTABUG QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 2CC: adeza, aschoen, ceph-eng-bugs, gmeno, kdreyer, nthomas, sankarshan, tchandra
Target Milestone: ---   
Target Release: 2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-31 06:08:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ansible playbook log none

Description Tejas 2017-01-25 10:01:23 UTC
Created attachment 1244202 [details]
ansible playbook  log

Description of problem:
      I have a setup where mds is colocated on the MON node. during rolling_update where MON update is tried, the ceph-base is not allowed to upgrade due to dependency on ceph-base. So this forms a cyclic dependency.

TASK [ceph.ceph-common : install distro or red hat storage ceph mon] ***********
fatal: [magna028]: FAILED! => {"changed": true, "failed": true, "msg": "Error: Package: 1:ceph-mds-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-tools-rpms)\n           Requires: ceph-base = 1:10.2.3-17.el7cp\n           Removing: 1:ceph-base-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-mon-rpms)\n               ceph-base = 1:10.2.3-17.el7cp\n           Updated By: 1:ceph-base-10.2.5-12.el7cp.x86_64 (download-node-02.eng.bos.redhat.com_rcm-guest_ceph-drops_auto_ceph-2-rhel-7-compose_latest-RHCEPH-2-RHEL-7_compose_MON_x86_64_os_)\n               ceph-base = 1:10.2.5-12.el7cp\n           Available: 1:ceph-base-10.2.2-38.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)\n               ceph-base = 1:10.2.2-38.el7cp\n           Available: 1:ceph-base-10.2.2-41.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)\n               ceph-base = 1:10.2.2-41.el7cp\n           Available: 1:ceph-base-10.2.3-13.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)\n               ceph-base = 1:10.2.3-13.el7cp\n", "rc": 1, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n              : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\nResolving Dependencies\n--> Running transaction check\n---> Package ceph-mon.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package ceph-mon.x86_64 1:10.2.5-12.el7cp will be an update\n--> Processing Dependency: ceph-base = 1:10.2.5-12.el7cp for package: 1:ceph-mon-10.2.5-12.el7cp.x86_64\n--> Running transaction check\n---> Package ceph-base.x86_64 1:10.2.3-17.el7cp will be updated\n--> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-selinux-10.2.3-17.el7cp.x86_64\n--> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-mds-10.2.3-17.el7cp.x86_64\n---> Package ceph-base.x86_64 1:10.2.5-12.el7cp will be an update\n--> Processing Dependency: ceph-common = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64\n--> Processing Dependency: libcephfs1 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64\n--> Processing Dependency: librados2 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64\n--> Processing Dependency: librbd1 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64\n--> Processing Dependency: librgw2 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64\n--> Running transaction check\n---> Package ceph-base.x86_64 1:10.2.3-17.el7cp will be updated\n--> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-mds-10.2.3-17.el7cp.x86_64\n---> Package ceph-common.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package ceph-common.x86_64 1:10.2.5-12.el7cp will be an update\n--> Processing Dependency: python-cephfs = 1:10.2.5-12.el7cp for package: 1:ceph-common-10.2.5-12.el7cp.x86_64\n--> Processing Dependency: python-rados = 1:10.2.5-12.el7cp for package: 1:ceph-common-10.2.5-12.el7cp.x86_64\n--> Processing Dependency: python-rbd = 1:10.2.5-12.el7cp for package: 1:ceph-common-10.2.5-12.el7cp.x86_64\n---> Package ceph-selinux.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package ceph-selinux.x86_64 1:10.2.5-12.el7cp will be an update\n---> Package libcephfs1.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package libcephfs1.x86_64 1:10.2.5-12.el7cp will be an update\n---> Package librados2.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package librados2.x86_64 1:10.2.5-12.el7cp will be an update\n---> Package librbd1.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package librbd1.x86_64 1:10.2.5-12.el7cp will be an update\n---> Package librgw2.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package librgw2.x86_64 1:10.2.5-12.el7cp will be an update\n--> Running transaction check\n---> Package ceph-base.x86_64 1:10.2.3-17.el7cp will be updated\n--> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-mds-10.2.3-17.el7cp.x86_64\n---> Package python-cephfs.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package python-cephfs.x86_64 1:10.2.5-12.el7cp will be an update\n---> Package python-rados.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package python-rados.x86_64 1:10.2.5-12.el7cp will be an update\n---> Package python-rbd.x86_64 1:10.2.3-17.el7cp will be updated\n---> Package python-rbd.x86_64 1:10.2.5-12.el7cp will be an update\n--> Finished Dependency Resolution\nError: Package: 1:ceph-mds-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-tools-rpms)\n           Requires: ceph-base = 1:10.2.3-17.el7cp\n           Removing: 1:ceph-base-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-mon-rpms)\n               ceph-base = 1:10.2.3-17.el7cp\n           Updated By: 1:ceph-base-10.2.5-12.el7cp.x86_64 (download-node-02.eng.bos.redhat.com_rcm-guest_ceph-drops_auto_ceph-2-rhel-7-compose_latest-RHCEPH-2-RHEL-7_compose_MON_x86_64_os_)\n               ceph-base = 1:10.2.5-12.el7cp\n           Available: 1:ceph-base-10.2.2-38.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)\n               ceph-base = 1:10.2.2-38.el7cp\n           Available: 1:ceph-base-10.2.2-41.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)\n               ceph-base = 1:10.2.2-41.el7cp\n           Available: 1:ceph-base-10.2.3-13.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)\n               ceph-base = 1:10.2.3-13.el7cp\n**********************************************************************\nyum can be configured to try to resolve such errors by temporarily enabling\ndisabled repos and searching for missing dependencies.\nTo enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf\n**********************************************************************\n\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"]}


Version-Release number of selected component (if applicable):
ceph-ansible-2.1.3-1.el7scon.noarch
ansible-2.2.1.0-1.el7.noarch


How reproducible:
Always

Steps to Reproduce:
1. on a cluster where MOn and mds are colocated, during MON update rolling_update fails


Additional info:


root@magna028 ~]# yum update ceph-mon
Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-disabled-repos, subscription-manager
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package ceph-mon.x86_64 1:10.2.3-17.el7cp will be updated
---> Package ceph-mon.x86_64 1:10.2.5-12.el7cp will be an update
--> Processing Dependency: ceph-base = 1:10.2.5-12.el7cp for package: 1:ceph-mon-10.2.5-12.el7cp.x86_64
--> Running transaction check
---> Package ceph-base.x86_64 1:10.2.3-17.el7cp will be updated
--> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-selinux-10.2.3-17.el7cp.x86_64
--> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-mds-10.2.3-17.el7cp.x86_64
---> Package ceph-base.x86_64 1:10.2.5-12.el7cp will be an update
--> Processing Dependency: ceph-common = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64
--> Processing Dependency: libcephfs1 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64
--> Processing Dependency: librados2 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64
--> Processing Dependency: librbd1 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64
--> Processing Dependency: librgw2 = 1:10.2.5-12.el7cp for package: 1:ceph-base-10.2.5-12.el7cp.x86_64
--> Running transaction check
---> Package ceph-base.x86_64 1:10.2.3-17.el7cp will be updated
--> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-mds-10.2.3-17.el7cp.x86_64
---> Package ceph-common.x86_64 1:10.2.3-17.el7cp will be updated
---> Package ceph-common.x86_64 1:10.2.5-12.el7cp will be an update
--> Processing Dependency: python-cephfs = 1:10.2.5-12.el7cp for package: 1:ceph-common-10.2.5-12.el7cp.x86_64
--> Processing Dependency: python-rados = 1:10.2.5-12.el7cp for package: 1:ceph-common-10.2.5-12.el7cp.x86_64
--> Processing Dependency: python-rbd = 1:10.2.5-12.el7cp for package: 1:ceph-common-10.2.5-12.el7cp.x86_64
---> Package ceph-selinux.x86_64 1:10.2.3-17.el7cp will be updated
---> Package ceph-selinux.x86_64 1:10.2.5-12.el7cp will be an update
---> Package libcephfs1.x86_64 1:10.2.3-17.el7cp will be updated
---> Package libcephfs1.x86_64 1:10.2.5-12.el7cp will be an update
---> Package librados2.x86_64 1:10.2.3-17.el7cp will be updated
---> Package librados2.x86_64 1:10.2.5-12.el7cp will be an update
---> Package librbd1.x86_64 1:10.2.3-17.el7cp will be updated
---> Package librbd1.x86_64 1:10.2.5-12.el7cp will be an update
---> Package librgw2.x86_64 1:10.2.3-17.el7cp will be updated
---> Package librgw2.x86_64 1:10.2.5-12.el7cp will be an update
--> Running transaction check
---> Package ceph-base.x86_64 1:10.2.3-17.el7cp will be updated
--> Processing Dependency: ceph-base = 1:10.2.3-17.el7cp for package: 1:ceph-mds-10.2.3-17.el7cp.x86_64
---> Package python-cephfs.x86_64 1:10.2.3-17.el7cp will be updated
---> Package python-cephfs.x86_64 1:10.2.5-12.el7cp will be an update
---> Package python-rados.x86_64 1:10.2.3-17.el7cp will be updated
---> Package python-rados.x86_64 1:10.2.5-12.el7cp will be an update
---> Package python-rbd.x86_64 1:10.2.3-17.el7cp will be updated
---> Package python-rbd.x86_64 1:10.2.5-12.el7cp will be an update
--> Finished Dependency Resolution
Error: Package: 1:ceph-mds-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-tools-rpms)
           Requires: ceph-base = 1:10.2.3-17.el7cp
           Removing: 1:ceph-base-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-mon-rpms)
               ceph-base = 1:10.2.3-17.el7cp
           Updated By: 1:ceph-base-10.2.5-12.el7cp.x86_64 (download-node-02.eng.bos.redhat.com_rcm-guest_ceph-drops_auto_ceph-2-rhel-7-compose_latest-RHCEPH-2-RHEL-7_compose_MON_x86_64_os_)
               ceph-base = 1:10.2.5-12.el7cp
           Available: 1:ceph-base-10.2.2-38.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)
               ceph-base = 1:10.2.2-38.el7cp
           Available: 1:ceph-base-10.2.2-41.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)
               ceph-base = 1:10.2.2-41.el7cp
           Available: 1:ceph-base-10.2.3-13.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)
               ceph-base = 1:10.2.3-13.el7cp
**********************************************************************
yum can be configured to try to resolve such errors by temporarily enabling
disabled repos and searching for missing dependencies.
To enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf
**********************************************************************

Error: Package: 1:ceph-mds-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-tools-rpms)
           Requires: ceph-base = 1:10.2.3-17.el7cp
           Removing: 1:ceph-base-10.2.3-17.el7cp.x86_64 (@rhel-7-server-rhceph-2-mon-rpms)
               ceph-base = 1:10.2.3-17.el7cp
           Updated By: 1:ceph-base-10.2.5-12.el7cp.x86_64 (download-node-02.eng.bos.redhat.com_rcm-guest_ceph-drops_auto_ceph-2-rhel-7-compose_latest-RHCEPH-2-RHEL-7_compose_MON_x86_64_os_)
               ceph-base = 1:10.2.5-12.el7cp
           Available: 1:ceph-base-10.2.2-38.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)
               ceph-base = 1:10.2.2-38.el7cp
           Available: 1:ceph-base-10.2.2-41.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)
               ceph-base = 1:10.2.2-41.el7cp
           Available: 1:ceph-base-10.2.3-13.el7cp.x86_64 (rhel-7-server-rhceph-2-mon-rpms)
               ceph-base = 1:10.2.3-13.el7cp
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
[root@magna028 ~]# 


Attaching the playbook log.

Comment 2 Ken Dreyer (Red Hat) 2017-01-25 16:45:05 UTC
I checked magna028 today. I think this is because you've enabled the "rhel-7-server-rhceph-2-tools-rpms" repository on this node, but you didn't enable the "Tools" repo from the latest compose (only the "MON" repo is enabled in /etc/yum.repos.d).

Would you please enable the Tools repository from the compose and re-try?

Comment 3 Tejas 2017-01-31 06:08:12 UTC
Ken,
 
   Thanks for the info. It is because the Tools repo is not enabled.
This is not a bug.