Bug 1444643 - OSP9 -> OSP10 -> OSP11 upgrade: cinder api returns ERROR: Internal Server Error (HTTP 500) after upgrade
Summary: OSP9 -> OSP10 -> OSP11 upgrade: cinder api returns ERROR: Internal Server Err...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z4
: 10.0 (Newton)
Assignee: Sofer Athlan-Guyot
QA Contact: Marius Cornea
URL:
Whiteboard:
: 1420860 (view as bug list)
Depends On:
Blocks: 1444866 1493961
TreeView+ depends on / blocked
 
Reported: 2017-04-23 14:47 UTC by Marius Cornea
Modified: 2023-02-22 23:02 UTC (History)
10 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.3.0-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1444866 (view as bug list)
Environment:
Last Closed: 2017-09-06 17:09:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1685750 0 None None None 2017-04-24 09:12:27 UTC
OpenStack gerrit 459271 0 None None None 2017-04-24 15:10:22 UTC
Red Hat Product Errata RHBA-2017:2654 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 director Bug Fix Advisory 2017-09-06 20:55:36 UTC

Description Marius Cornea 2017-04-23 14:47:55 UTC
Description of problem:
After OSP9 -> OSP10 -> OSP11 upgrade cinder api returns ERROR: Internal Server Error (HTTP 500)


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-6.0.0-5.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP9
2. Upgrade to OSP10
3. Upgrade to OSP11

Actual results:
Cinder API is broken.

Expected results:
Cinder API works as expected.

Additional info:
[root@controller-0 heat-admin]# tail -1 /var/log/httpd/cinder_wsgi_error.log 
[Sun Apr 23 14:43:30.733678 2017] [:error] [pid 10112] [remote 172.17.1.12:360] ImportError: No module named sizelimit

In /etc/cinder/api-paste.ini:
[filter:sizelimit]
paste.filter_factory = cinder.api.middleware.sizelimit:RequestBodySizeLimiter.factory

Comment 2 Marius Cornea 2017-04-23 15:05:41 UTC
Workaround:
sed -i s/cinder.api.middleware.sizelimit/oslo_middleware.sizelimit/ /etc/cinder/api-paste.ini

I'm not sure if this file gets overwritten by puppet on further stack updates.

Comment 3 Marios Andreou 2017-04-24 08:07:49 UTC
(In reply to Marius Cornea from comment #2)
> Workaround:
> sed -i s/cinder.api.middleware.sizelimit/oslo_middleware.sizelimit/
> /etc/cinder/api-paste.ini
> 
> I'm not sure if this file gets overwritten by puppet on further stack
> updates.

just sanity checked and afaics we are not explicitly setting this for stable/newton tripleo-heat-templates or puppet-tripleo (grep -rni sizelimit ./* )

sofer via irc suspects this is set in packaging I will go look at puppet-cinder defaults too. So no, we don't do this explicitly in puppet config mcornea.

Comment 4 Sofer Athlan-Guyot 2017-04-24 08:35:03 UTC
Hi,

so in stable/mitaka (osp9) when you enabled ssl you had that code[1] which modifies the /etc/cinder/api-paste.ini file.

From there on the packaging replacement wouldn't take place anymore and we have instead /etc/cinder/api-paste.ini.rpmnew file created.

In stable/newton (osp10) this mechanism was removed (or handled another way) so upgrading from osp10 to osp11 let the packaging system replace the api-paste.ini.

In no-ssl deployment this wouldn't occur as well as the packaging system would kick in and replace that file.

[1] https://github.com/openstack/puppet-tripleo/blob/stable/mitaka/manifests/ssl/cinder_config.pp#L21..L29

This is tricky to solve as it would mean that we need to completely manage the api-paste.ini file from puppet-tripleo to match what each release (and packaging) does.

The consensus here seemed to have been "let the api-paste.ini file alone so that packaging does the jobs and replace it with the right value for each release".  

Checking with other what could be the best course of action.

Comment 5 Marios Andreou 2017-04-24 08:48:39 UTC
thanks chem... so i came across this fwiw/debug/info the config in question here (api_paste.ini) was added with https://review.openstack.org/#/c/268647/  https://review.openstack.org/#/c/322078/ which is where the config you point at in comment #5 should have been removed but wasn't.

Comment 7 Jaromir Coufal 2017-04-24 13:58:00 UTC
Not OSP11 blocker.

Fix goes to OSP10.z, for OSP11 GA we cloned for doc_text until we have OSP10 async out.

Comment 8 Sofer Athlan-Guyot 2017-07-04 11:46:29 UTC
Still not merged downstream.

Comment 9 Sofer Athlan-Guyot 2017-07-10 11:09:16 UTC
*** Bug 1420860 has been marked as a duplicate of this bug. ***

Comment 13 errata-xmlrpc 2017-09-06 17:09:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2654


Note You need to log in before you can comment on or make changes to this bug.