Bug 1466345 - Cannot upgrade OSP11 to OSP12: Overcloud upgrade failed due to dependency issue for openstack-dashboard-theme
Cannot upgrade OSP11 to OSP12: Overcloud upgrade failed due to dependency iss...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-django-horizon (Show other bugs)
12.0 (Pike)
x86_64 Linux
urgent Severity high
: beta
: 12.0 (Pike)
Assigned To: Radomir Dopieralski
Ido Ovadia
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-29 09:13 EDT by Artem Hrechanychenko
Modified: 2017-12-13 16:35 EST (History)
14 users (show)

See Also:
Fixed In Version: python-django-horizon-12.0.0-0.20170821142112.el7ost.1 openstack-dashboard-theme-12.0.0-2.el7ost.1
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-13 16:35:30 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
tvignaud: needinfo+


Attachments (Terms of Use)

  None (edit)
Description Artem Hrechanychenko 2017-06-29 09:13:44 EDT
Description of problem:
On controller node:
--> Processing Dependency: openstack-dashboard = 1:11.0.1-5.el7ost for package: 1:openstack-dashboard-theme-11.0.1-5.el7ost.noarch
---> Package python-html5lib.noarch 1:0.999-6.el7ost will be installed
--> Finished Dependency Resolution
Error: Package: 1:openstack-dashboard-theme-11.0.1-5.el7ost.noarch (@rhos-11.0-signed)
           Requires: openstack-dashboard = 1:11.0.1-5.el7ost
           Removing: 1:openstack-dashboard-11.0.1-5.el7ost.noarch (@rhos-11.0-signed)
               openstack-dashboard = 1:11.0.1-5.el7ost
           Updated By: 1:openstack-dashboard-12.0.0-0.20170624124155.e8d1dd9.el7ost.noarch (rhelosp-12.0-puddle)
               openstack-dashboard = 1:12.0.0-0.20170624124155.e8d1dd9.el7ost
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest


Version-Release number of selected component (if applicable):
OSP12

openstack-nova-api-16.0.0-0.20170624031428.3863eca.el7ost.noarch
openstack-keystone-12.0.0-0.20170623233743.9070172.el7ost.noarch
openstack-ironic-inspector-5.1.1-0.20170622234613.05896b5.el7ost.noarch
puppet-openstack_extras-11.2.0-0.20170613085321.0b0ea62.el7ost.noarch
openstack-tripleo-common-containers-7.1.1-0.20170623115707.4ba7d56.el7ost.noarch
openstack-neutron-common-11.0.0-0.20170624003801.11acb1d.el7ost.noarch
openstack-neutron-11.0.0-0.20170624003801.11acb1d.el7ost.noarch
openstack-nova-compute-16.0.0-0.20170624031428.3863eca.el7ost.noarch
openstack-mistral-executor-5.0.0-0.20170623025911.6a1ca0f.el7ost.noarch
openstack-tempest-16.0.1-0.20170623203531.4386df8.el7ost.noarch
openstack-neutron-openvswitch-11.0.0-0.20170624003801.11acb1d.el7ost.noarch
openstack-nova-placement-api-16.0.0-0.20170624031428.3863eca.el7ost.noarch
openstack-heat-api-cfn-9.0.0-0.20170623110018.173f03a.el7ost.noarch
openstack-mistral-api-5.0.0-0.20170623025911.6a1ca0f.el7ost.noarch
openstack-ironic-api-8.0.1-0.20170621044433.f0e6a07.el7ost.noarch
python-openstack-mistral-5.0.0-0.20170623025911.6a1ca0f.el7ost.noarch
openstack-heat-common-9.0.0-0.20170623110018.173f03a.el7ost.noarch
openstack-tripleo-image-elements-7.0.0-0.20170607161959.401d861.el7ost.noarch
openstack-tripleo-puppet-elements-7.0.0-0.20170614005502.9285877.el7ost.noarch
openstack-swift-object-2.14.1-0.20170622024006.2d18ecd.el7ost.noarch
openstack-neutron-ml2-11.0.0-0.20170624003801.11acb1d.el7ost.noarch
openstack-glance-15.0.0-0.20170623215940.8188eca.el7ost.noarch
openstack-nova-scheduler-16.0.0-0.20170624031428.3863eca.el7ost.noarch
openstack-heat-engine-9.0.0-0.20170623110018.173f03a.el7ost.noarch
openstack-mistral-engine-5.0.0-0.20170623025911.6a1ca0f.el7ost.noarch
openstack-ironic-conductor-8.0.1-0.20170621044433.f0e6a07.el7ost.noarch
openstack-tripleo-common-7.1.1-0.20170623115707.4ba7d56.el7ost.noarch
python-openstackclient-3.11.0-0.20170613232431.c69304e.el7ost.noarch
openstack-ironic-common-8.0.1-0.20170621044433.f0e6a07.el7ost.noarch
openstack-swift-account-2.14.1-0.20170622024006.2d18ecd.el7ost.noarch
openstack-selinux-0.8.8-0.20170622195307.74ddc0e.el7ost.noarch
openstack-swift-container-2.14.1-0.20170622024006.2d18ecd.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170624031428.3863eca.el7ost.noarch
openstack-tripleo-validations-7.1.1-0.20170621111847.fb7346f.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170624014919.el7ost.noarch
puppet-openstacklib-11.2.0-0.20170613150439.93b8e7d.el7ost.noarch
python-openstacksdk-0.9.17-0.20170621195806.7946243.el7ost.noarch
openstack-swift-proxy-2.14.1-0.20170622024006.2d18ecd.el7ost.noarch
openstack-heat-api-9.0.0-0.20170623110018.173f03a.el7ost.noarch
openstack-zaqar-5.0.0-0.20170623003642.68eac5a.el7ost.noarch
openstack-tripleo-ui-7.1.1-0.20170624213423.cb896fd.el7ost.noarch
openstack-puppet-modules-10.0.0-1.el7ost.noarch
openstack-mistral-common-5.0.0-0.20170623025911.6a1ca0f.el7ost.noarch
openstack-nova-common-16.0.0-0.20170624031428.3863eca.el7ost.noarch

How reproducible:


Steps to Reproduce:
Steps to Reproduce:
1.Install OSP11
infrared virsh -v --host-address 10.9.76.22 --host-key ~/.ssh/id_rsa --cleanup yes && infrared virsh -v --host-address 10.9.76.22 --host-key ~/.ssh/id_rsa  --topology-nodes undercloud:1,controller:1,compute:1  -e  override.controller.cpu=8 -e override.controller.memory=16384 -e  override.undercloud.disks.disk1.size=100G && infrared tripleo-undercloud --version 11 --images-task=rpm && infrared tripleo-overcloud -v --introspect yes --tagging yes --post no --deployment-files virt --version 11 --deploy yes 

2.Upgrade undercloud and overcloud to osp11 latest+ rhel7.4
ir tripleo-undercloud -v --update-undercloud yes --mirror qeos --build 7.4-testing --osrelease 7.4 

ir tripleo-overcloud -v --updateto 7.4-testing --deployment-files virt --mirror qeos --osrelease 7.4
#Please note, during yum update stage on overcloud controller run "sudo pcs cluster start and sudo pcs cleanup resources" - w/a for https://bugzilla.redhat.com/show_bug.cgi?id=1464588

3. Upgrade undercloud and overcloud to osp12
http://etherpad.corp.redhat.com/osp12-upgrade
#please note, before updating overcloud apply w/a for https://bugzilla.redhat.com/show_bug.cgi?id=1460421
https://bugzilla.redhat.com/show_bug.cgi?id=1463227
https://bugzilla.redhat.com/show_bug.cgi?id=1466219
https://bugzilla.redhat.com/show_bug.cgi?id=1466308

Actual results:
overcloud update failed


Expected results:
overcloud update succeed

Additional info:
http://pastebin.test.redhat.com/498883
Comment 1 Red Hat Bugzilla Rules Engine 2017-06-29 09:13:49 EDT
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
Comment 2 Jiri Stransky 2017-06-29 09:46:06 EDT
There seems to be a difference between available packages on undercloud vs. overcloud. Undercloud:

(undercloud) [stack@undercloud-0 ~]$ yum list available | grep openstack-dashboard
openstack-dashboard.noarch        1:12.0.0-0.20170624124155.e8d1dd9.el7ost
openstack-dashboard-theme.noarch  12.0.0-2.el7ost        rhelosp-12.0-puddle    


Overcloud:

[root@controller-0 ~]# yum list available | grep openstack-dashboard
openstack-dashboard.noarch        1:12.0.0-0.20170624124155.e8d1dd9.el7ost

It could be an issue with how we setup repositories on OC during upgrade initialization.
Comment 3 Artem Hrechanychenko 2017-06-29 11:38:31 EDT
yum clean all;rhos-release -x; rhos-release 12-director ; reboot on overcloud nodes didn't helped
Comment 4 Jiri Stransky 2017-06-30 04:46:34 EDT
The problem is that the openstack-dashboard and openstack-dashboard-theme versions in the latest puddle repo don't match, but it looks like they're supposed to be version-locked together, so i guess yum ignores the newer openstack-dashboard-theme. (Perhaps the reason why yum listed it on undercloud as available is that it's not yet installed on undercloud, and maybe some of the version lock checks weren't triggered then... Just guessing...)

[   ] openstack-dashboard-12.0.0-0.20170624124155.e8d1dd9.el7ost.noarch.rpm                       26-Jun-2017 10:39  3.4M  
[   ] openstack-dashboard-theme-12.0.0-2.el7ost.noarch.rpm                                        23-Jun-2017 04:38  703K  

E.g. in OSP 11 latest puddle repos, they are exactly the same version:


[   ] openstack-dashboard-11.0.1-5.el7ost.noarch.rpm                               04-Apr-2017 13:42   13M  
[   ] openstack-dashboard-theme-11.0.1-5.el7ost.noarch.rpm                         04-Apr-2017 13:42  6.2K  

Let's try adding

yum -y remove openstack-dasboard-theme

into UpgradeInitCommand as a workaround for now.
Comment 5 Jiri Stransky 2017-06-30 04:47:43 EDT
Typo fix:

yum -y remove openstack-dashboard-theme
Comment 7 Mike Burns 2017-06-30 08:22:30 EDT
It appears the issue here is the epoch.

in python-django-horizon specfile, openstack-dashboard has:

Requires:   openstack-dashboard-theme => 1:12.0

The version of openstack-dashboard-theme is 12.0.0 with no epoch.  

Note, the => may cause a problem too, though I'm not certain.  We should probably clean it up to be >=.
Comment 8 Artem Hrechanychenko 2017-06-30 08:25:26 EDT
w/a - 
create local mirror with this rpms
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=13578817
on all oc nodes and re-start oc upgrade
Comment 9 Thierry Vignaud 2017-06-30 09:07:37 EDT
Good catch.
Comment 10 Thierry Vignaud 2017-06-30 09:09:56 EDT
Note that "=>" is automatically converted to ">=" but let's clean it for clarity/readability
Comment 14 Marius Cornea 2017-08-29 09:07:47 EDT
This issue reported initially is still showing up in the latest puddle:

[root@controller-0 heat-admin]# yum info python-django-horizon
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Installed Packages
Name        : python-django-horizon
Arch        : noarch
Epoch       : 1
Version     : 11.0.3
Release     : 3.el7ost
Size        : 2.9 M
Repo        : installed
From repo   : rhos-11.0-signed
Summary     : Django application for talking to Openstack
URL         : http://horizon.openstack.org/
License     : ASL 2.0 and BSD
Description : Horizon is a Django application for providing Openstack UI components.
            : It allows performing site administrator (viewing account resource usage,
            : configuring users, accounts, quotas, flavors, etc.) and end user
            : operations (start/stop/delete instances, create/restore snapshots, view
            : instance VNC console, etc.)

Available Packages
Name        : python-django-horizon
Arch        : noarch
Epoch       : 1
Version     : 12.0.0
Release     : 0.20170821142112.el7ost
Size        : 697 k
Repo        : rhelosp-12.0-puddle/x86_64
Summary     : Django application for talking to Openstack
URL         : http://horizon.openstack.org/
License     : ASL 2.0 and BSD
Description : Horizon is a Django application for providing Openstack UI components.
            : It allows performing site administrator (viewing account resource usage,
            : configuring users, accounts, quotas, flavors, etc.) and end user
            : operations (start/stop/delete instances, create/restore snapshots, view
            : instance VNC console, etc.)



--> Finished Dependency Resolution
Error: Package: 1:openstack-dashboard-theme-11.0.3-3.el7ost.noarch (@rhos-11.0-signed)
           Requires: openstack-dashboard = 1:11.0.3-3.el7ost
           Removing: 1:openstack-dashboard-11.0.3-3.el7ost.noarch (@rhos-11.0-signed)
               openstack-dashboard = 1:11.0.3-3.el7ost
           Updated By: 1:openstack-dashboard-12.0.0-0.20170821142112.el7ost.noarch (rhelosp-12.0-puddle)
               openstack-dashboard = 1:12.0.0-0.20170821142112.el7ost
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
Comment 16 Thierry Vignaud 2017-08-30 09:44:10 EDT
Could you attach the full yum logs?
Comment 17 Thierry Vignaud 2017-08-30 10:51:18 EDT
I suspect a repo issue
Comment 18 Thierry Vignaud 2017-08-30 11:09:39 EDT
From some manual tests with dnf (bad I know) & rpm, I think the issue is that EVR of the new openstack-dashboard-theme is lower than the one in osp11 (VR is newer but epoch isn't)
I think we might need to add back the Epoch in the splitted openstack-dashboard-theme.
I'll test that theory
Comment 19 Thierry Vignaud 2017-08-30 11:27:55 EDT
This seems to do the trick (openstack-dashboard-theme-12.0.0-2.el7ost.1)
(I should have bump the release to 3 instead of appending ".1" btw)
We now need to adjust the requires in python-django-horizon so that it includes the epoch again
Comment 24 errata-xmlrpc 2017-12-13 16:35:30 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.