Bug 1609966 - FFU: controllers upgrade fail with Error: Package: 2:ceph-base-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon) Requires: libcephfs.so.1()(64bit) dependency
Summary: FFU: controllers upgrade fail with Error: Package: 2:ceph-base-10.2.10-28.el7...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z2
: 13.0 (Queens)
Assignee: Giulio Fidente
QA Contact: Yogev Rabl
URL:
Whiteboard:
: 1609459 (view as bug list)
Depends On:
Blocks: 1488066 1558787 1599764
TreeView+ depends on / blocked
 
Reported: 2018-07-30 22:49 UTC by Marius Cornea
Modified: 2018-12-24 11:40 UTC (History)
17 users (show)

Fixed In Version: openstack-tripleo-heat-templates-8.0.4-18.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-08-29 16:39:18 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1785279 None None None 2018-08-03 16:20:36 UTC
OpenStack gerrit 590746 None MERGED Skip Ceph packages when running tripleo-packages 2020-10-25 14:44:13 UTC
Red Hat Product Errata RHBA-2018:2574 None None None 2018-08-29 16:40:19 UTC

Description Marius Cornea 2018-07-30 22:49:50 UTC
Description of problem:

FFU: controllers upgrade fail with ceph rpm packages dependencies:

u'TASK [Update all packages] *****************************************************',
 u'Monday 30 July 2018  17:51:55 -0400 (0:00:02.955)       0:15:51.535 *********** ',
 u'fatal: [192.168.24.15]: FAILED! => {"changed": true, "msg": "Error: Package: 2:ceph-base-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n           Requires: libcephfs.so.1()(64bit)\\n           Removing: 2:libcephfs1-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n               libcephfs.so.1()(64bit)\\n           Obsoleted By: 2:libcephfs2-12.2.4-30.el7.x86_64 (rhelosp-13.0-puddle)\\n              ~libcephfs.so.2()(64bit)\\nError: Package: 2:ceph-base-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n           Requires: librgw2 = 2:10.2.10-28.el7cp\\n           Removing: 2:librgw2-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n               librgw2 = 2:10.2.10-28.el7cp\\n           Updated By: 2:librgw2-12.2.4-30.el7.x86_64 (rhelosp-13.0-puddle)\\n               librgw2 = 2:12.2.4-30.el7\\nError: Package: 2:ceph-radosgw-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-tools)\\n           Requires: librgw2 = 2:10.2.10-28.el7cp\\n           Removing: 2:librgw2-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n               librgw2 = 2:10.2.10-28.el7cp\\n           Updated By: 2:librgw2-12.2.4-30.el7.x86_64 (rhelosp-13.0-puddle)\\n               librgw2 = 2:12.2.4-30.el7\\nError: Package: 2:ceph-radosgw-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-tools)\\n           Requires: ceph-common = 2:10.2.10-28.el7cp\\n           Removing: 2:ceph-common-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n               ceph-common = 2:10.2.10-28.el7cp\\n           Updated By: 2:ceph-common-12.2.4-30.el7.x86_64 (rhelosp-13.0-puddle)\\n               ceph-common = 2:12.2.4-30.el7\\n           Available: 1:ceph-common-0.80.7-2.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               ceph-common = 1:0.80.7-2.el7\\n           Available: 1:ceph-common-0.80.7-3.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               ceph-common = 1:0.80.7-3.el7\\n           Available: 1:ceph-common-0.94.5-1.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               ceph-common = 1:0.94.5-1.el7\\n           Available: 1:ceph-common-0.94.5-2.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               ceph-common = 1:0.94.5-2.el7\\nError: Package: 2:ceph-base-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n           Requires: ceph-common = 2:10.2.10-28.el7cp\\n           Removing: 2:ceph-common-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n               ceph-common = 2:10.2.10-28.el7cp\\n           Updated By: 2:ceph-common-12.2.4-30.el7.x86_64 (rhelosp-13.0-puddle)\\n               ceph-common = 2:12.2.4-30.el7\\n           Available: 1:ceph-common-0.80.7-2.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               ceph-common = 1:0.80.7-2.el7\\n           Available: 1:ceph-common-0.80.7-3.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               ceph-common = 1:0.80.7-3.el7\\n           Available: 1:ceph-common-0.94.5-1.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               ceph-common = 1:0.94.5-1.el7\\n           Available: 1:ceph-common-0.94.5-2.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               ceph-common = 1:0.94.5-2.el7\\nError: Package: 2:ceph-base-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n           Requires: librados2 = 2:10.2.10-28.el7cp\\n           Removing: 2:librados2-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n               librados2 = 2:10.2.10-28.el7cp\\n           Updated By: 2:librados2-12.2.4-30.el7.x86_64 (rhelosp-13.0-puddle)\\n               librados2 = 2:12.2.4-30.el7\\n           Available: 1:librados2-0.80.7-2.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               librados2 = 1:0.80.7-2.el7\\n           Available: 1:librados2-0.80.7-3.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               librados2 = 1:0.80.7-3.el7\\n           Available: 1:librados2-0.94.5-1.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               librados2 = 1:0.94.5-1.el7\\n           Available: 1:librados2-0.94.5-2.el7.i686 (rhelosp-rhel-7.5-server)\\n               librados2 = 1:0.94.5-2.el7\\nError: Package: 2:ceph-base-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n           Requires: librbd1 = 2:10.2.10-28.el7cp\\n           Removing: 2:librbd1-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n               librbd1 = 2:10.2.10-28.el7cp\\n           Updated By: 2:librbd1-12.2.4-30.el7.x86_64 (rhelosp-13.0-puddle)\\n               librbd1 = 2:12.2.4-30.el7\\n           Available: 1:librbd1-0.80.7-2.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               librbd1 = 1:0.80.7-2.el7\\n           Available: 1:librbd1-0.80.7-3.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               librbd1 = 1:0.80.7-3.el7\\n           Available: 1:librbd1-0.94.5-1.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               librbd1 = 1:0.94.5-1.el7\\n           Available: 1:librbd1-0.94.5-2.el7.i686 (rhelosp-rhel-7.5-server)\\n               librbd1 = 1:0.94.5-2.el7\\nError: Package: 2:ceph-base-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n           Requires: libcephfs1 = 2:10.2.10-28.el7cp\\n           Removing: 2:libcephfs1-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n               libcephfs1 = 2:10.2.10-28.el7cp\\n           Obsoleted By: 2:libcephfs2-12.2.4-30.el7.x86_64 (rhelosp-13.0-puddle)\\n              ~libcephfs2 = 2:12.2.4-30.el7\\nError: Package: 2:ceph-radosgw-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-tools)\\n           Requires: librados2 = 2:10.2.10-28.el7cp\\n           Removing: 2:librados2-10.2.10-28.el7cp.x86_64 (@rhelosp-ceph-2.0-mon)\\n               librados2 = 2:10.2.10-28.el7cp\\n           Updated By: 2:librados2-12.2.4-30.el7.x86_64 (rhelosp-13.0-puddle)\\n               librados2 = 2:12.2.4-30.el7\\n           Available: 1:librados2-0.80.7-2.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               librados2 = 1:0.80.7-2.el7\\n           Available: 1:librados2-0.80.7-3.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               librados2 = 1:0.80.7-3.el7\\n           Available: 1:librados2-0.94.5-1.el7.x86_64 (rhelosp-rhel-7.5-server)\\n               librados2 = 1:0.94.5-1.el7\\n           Available: 1:librados2-0.94.5-2.el7.i686 (rhelosp-rhel-7.5-server)\\n               librados2 = 1:0.94.5-2.el7\\n", "rc": 1, "results": ["Loaded plugins: product-id, search-disabled-repos, subscription-manager\\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\\nResolving Dependencies\\n--> Running transaction check\\n---> Package Cython.x86_64 0:0.21.1-1.el7ost will be obsoleted\\n---> Package bootswatch-common.noarch 0:3.3.5.3-2.1.el7ost will be updated\\n---> Package bootswatch-common.noarch 0:3.3.7.0-1.el7ost will be an update\\n---> Package bootswatch-fonts.noarch 0:3.3.5.3-2.1.el7ost will be updated\\n---> Package bootswatch-fonts.noarch 0:3.3.7.0-1.el7ost will be an update\\n---> Package ceph-common.x86_64 2:10.2.10-28.el7cp will be updated\\n--> Processing Dependency: ceph-common = 2:10.2.10-28.el7cp for package: 2:ceph-radosgw-10.2.10-28.el7cp.x86_64\\n--> Processing Dependency: ceph-common = 2:10.2.10-28.el7cp for package: 2:ceph-base-10.2.10-28.el7cp.x86_64\\n---> Package ceph-common.x86_64 2:12.2.4-30.el7 will be an update\\n--> Processing Dependency: python-rgw = 2:12.2.4-30.el7 for package: 2:ceph-common-12.2.4-30.el7.x86_64\\n--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-common-12.2.4-30.el7.x86_64\\n---> 


Version-Release number of selected component (if applicable):
2018-07-30.2 puddle

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP10 z8
2. Minor update to OSP10 lates
3. Run through the FFU procedure to OSP13

Actual results:
During the controller upgrade stage the upgrade process fails due to rpm packages dependencies.

Expected results:
Clean rpm packages update.

Additional info:

Comment 2 Giulio Fidente 2018-07-31 14:18:36 UTC
Looks like we have Ceph 12. packages distributed via rhelosp-13.0-puddle repo which try to bring forward the rest of the Ceph 10. packages installed via @rhelosp-ceph-2.0-mon repo ... but in the ceph-2.0-mon there are not 12.x versions.

I believe the only workaround is to bump up the Ceph repos on the nodes to point to ceph-3.0 repos, see BZ #1558787.

It's not clear to me though why this is only happening starting from the latest 10z8 update; not sure if the version of any of the ceph packages shipping in z8 changed, compared to z7, and is now triggering the cascading dependencies issue.

Comment 16 John Fulton 2018-08-08 13:24:24 UTC
*** Bug 1609459 has been marked as a duplicate of this bug. ***

Comment 32 Joanne O'Flynn 2018-08-15 07:51:35 UTC
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

* Select the documentation type from the "Doc Type" drop down field.

* A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.

Comment 36 Tzach Shefi 2018-08-20 12:27:36 UTC
Yogev/Ronnie 

I just sucsessuly upgraded an OPS10 to OSP13 with internal ceph. 
openstack-tripleo-heat-templates-8.0.4-20.el7ost.noarch. 

Upgrade completed if you wish to verify based on my system.

Comment 38 Tzach Shefi 2018-08-20 12:48:08 UTC
One odd issue I noticed post upgrade

openstack endpoint list
Failed to discover available identity versions when contacting http://10.0.0.108:5000//v3. Attempting to parse version from URL.
Could not determine a suitable URL for the plugin


cat overcloudrc
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="}  /^OS_/ {print $1}' ); do unset $key ; done
export OS_NO_CACHE=True
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,10.0.0.108,192.168.24.12
export OS_USER_DOMAIN_NAME=Default
export OS_VOLUME_API_VERSION=3
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=http://10.0.0.108:5000//v3  -> why double "//"

Don't recall adding this my self may have been a fluke or caused by upgrade?
After removing the second "/" and sourcing overcloudrc again->

. overcloudrc
(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list
Failed to discover available identity versions when contacting http://10.0.0.108:5000/v3. Attempting to parse version from URL.

I'll try a fresh FFU again on another host and report on findings.

Comment 39 Tzach Shefi 2018-08-21 03:51:08 UTC
I suspect OK to verify this time, after running on a new fresh deployment (Titan30) upgrade completed without issues.
-------------------

Stack overcloud UPDATE_COMPLETE 

Started Mistral Workflow tripleo.deployment.v1.get_horizon_url. Execution ID: 53cf715d-43fd-464c-bb35-ef1690f0bf9a
Overcloud Endpoint: http://10.0.0.101:5000/
Overcloud Horizon Dashboard URL: http://10.0.0.101:80/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed
FFWD Upgrade Converge on stack overcloud complete.

--------------

Sourced overcloudrc.v3, basic cinder create instance boot work as expected. 
Dockers are being used.  
I'll start a tempest waiting for output before I verify.

Comment 40 Yogev Rabl 2018-08-21 18:56:02 UTC
Verified on openstack-tripleo-heat-templates-8.0.4-20.el7ost.noarch

Comment 42 errata-xmlrpc 2018-08-29 16:39:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2574


Note You need to log in before you can comment on or make changes to this bug.