Bug 1444643

Summary: OSP9 -> OSP10 -> OSP11 upgrade: cinder api returns ERROR: Internal Server Error (HTTP 500) after upgrade
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-heat-templatesAssignee: Sofer Athlan-Guyot <sathlang>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 11.0 (Ocata)CC: dbecker, djuran, jcoufal, jjoyce, mandreou, mburns, morazi, mschuppe, rhel-osp-director-maint, sathlang
Target Milestone: z4Keywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-5.3.0-2.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1444866 (view as bug list) Environment:
Last Closed: 2017-09-06 17:09:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1444866, 1493961    

Description Marius Cornea 2017-04-23 14:47:55 UTC
Description of problem:
After OSP9 -> OSP10 -> OSP11 upgrade cinder api returns ERROR: Internal Server Error (HTTP 500)


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-6.0.0-5.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP9
2. Upgrade to OSP10
3. Upgrade to OSP11

Actual results:
Cinder API is broken.

Expected results:
Cinder API works as expected.

Additional info:
[root@controller-0 heat-admin]# tail -1 /var/log/httpd/cinder_wsgi_error.log 
[Sun Apr 23 14:43:30.733678 2017] [:error] [pid 10112] [remote 172.17.1.12:360] ImportError: No module named sizelimit

In /etc/cinder/api-paste.ini:
[filter:sizelimit]
paste.filter_factory = cinder.api.middleware.sizelimit:RequestBodySizeLimiter.factory

Comment 2 Marius Cornea 2017-04-23 15:05:41 UTC
Workaround:
sed -i s/cinder.api.middleware.sizelimit/oslo_middleware.sizelimit/ /etc/cinder/api-paste.ini

I'm not sure if this file gets overwritten by puppet on further stack updates.

Comment 3 Marios Andreou 2017-04-24 08:07:49 UTC
(In reply to Marius Cornea from comment #2)
> Workaround:
> sed -i s/cinder.api.middleware.sizelimit/oslo_middleware.sizelimit/
> /etc/cinder/api-paste.ini
> 
> I'm not sure if this file gets overwritten by puppet on further stack
> updates.

just sanity checked and afaics we are not explicitly setting this for stable/newton tripleo-heat-templates or puppet-tripleo (grep -rni sizelimit ./* )

sofer via irc suspects this is set in packaging I will go look at puppet-cinder defaults too. So no, we don't do this explicitly in puppet config mcornea.

Comment 4 Sofer Athlan-Guyot 2017-04-24 08:35:03 UTC
Hi,

so in stable/mitaka (osp9) when you enabled ssl you had that code[1] which modifies the /etc/cinder/api-paste.ini file.

From there on the packaging replacement wouldn't take place anymore and we have instead /etc/cinder/api-paste.ini.rpmnew file created.

In stable/newton (osp10) this mechanism was removed (or handled another way) so upgrading from osp10 to osp11 let the packaging system replace the api-paste.ini.

In no-ssl deployment this wouldn't occur as well as the packaging system would kick in and replace that file.

[1] https://github.com/openstack/puppet-tripleo/blob/stable/mitaka/manifests/ssl/cinder_config.pp#L21..L29

This is tricky to solve as it would mean that we need to completely manage the api-paste.ini file from puppet-tripleo to match what each release (and packaging) does.

The consensus here seemed to have been "let the api-paste.ini file alone so that packaging does the jobs and replace it with the right value for each release".  

Checking with other what could be the best course of action.

Comment 5 Marios Andreou 2017-04-24 08:48:39 UTC
thanks chem... so i came across this fwiw/debug/info the config in question here (api_paste.ini) was added with https://review.openstack.org/#/c/268647/  https://review.openstack.org/#/c/322078/ which is where the config you point at in comment #5 should have been removed but wasn't.

Comment 7 Jaromir Coufal 2017-04-24 13:58:00 UTC
Not OSP11 blocker.

Fix goes to OSP10.z, for OSP11 GA we cloned for doc_text until we have OSP10 async out.

Comment 8 Sofer Athlan-Guyot 2017-07-04 11:46:29 UTC
Still not merged downstream.

Comment 9 Sofer Athlan-Guyot 2017-07-10 11:09:16 UTC
*** Bug 1420860 has been marked as a duplicate of this bug. ***

Comment 13 errata-xmlrpc 2017-09-06 17:09:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2654