Bug 1420860

Summary: OSP9 -> OSP10 upgrade do not replace cinder api-paste.ini and API v3 connection fail
Product: Red Hat OpenStack Reporter: Martin Schuppert <mschuppe>
Component: openstack-tripleo-heat-templatesAssignee: Sofer Athlan-Guyot <sathlang>
Status: CLOSED DUPLICATE QA Contact: Amit Ugol <augol>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: augol, djuran, jcoufal, mburns, mschuppe, rhel-osp-director-maint, sathlang
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-10 11:09:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Martin Schuppert 2017-02-09 16:21:31 UTC
Description of problem:

After a successful OSP9->OSP10 upgrade cinder api V3 connections fail due to missing replacement of old api-paste.ini in /etc/cinder. E.g. by default Heat does use Cinder API v3

[stack@bb1-osd01 ~]$ cinder --os-volume-api-version 3 create 10
ERROR: 'unicode' object has no attribute 'get'

On the API side we see 
2017-02-09 13:57:40.725 657640 INFO eventlet.wsgi.server [-] 172.17.0.18 "POST /v3/6354e06f244d43b995f477c8178305cf/volumes HTTP/1.1" status: 404  len: 243 time: 0.1442511

Looking at the api-paste, the upgrade did not replace the old api-paste.ini
This results that the API v3 was not available.

Version-Release number of selected component (if applicable):
OSP10

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
cinder v3 API requests fail after upgrade

Expected results:
cinder v3 works without manual modifications

Additional info:

Fixed like this
$ mv /etc/cinder/api-paste.ini.rpmnew /etc/cinder/api-paste.ini
$ systemctl restart openstack-cinder-api

Comment 1 Sofer Athlan-Guyot 2017-02-14 09:08:37 UTC
Hi,

On a fresh osp9->osp10 upgrade, the required v3 definition are in the api-paste.ini.

On a osp8->osp9 upgrade as well.  On both platform the 

    cinder --os-volume-api-version 3 create 10

worked.

Looking at the puppet code, it does look like there is no special handling of the v3 configuration for cinder/api-paste.ini.  So it looks like this file has the required (in the package) bits since at least some version of osp8.


So, do you happen to have moved this platform from osp7 all the way to osp10 ?

Did you keep the original api-paste.ini so that I can have a look at it ?

Comment 2 David Juran 2017-02-14 14:13:46 UTC
Actually, on a freshly installed OSP8, 

cinder --os-volume-api-version 3 create 10

tells me that version must be one of 1 or 2

And indeed, the api.paste.ini only mentions v1 and v2

Comment 3 Sofer Athlan-Guyot 2017-02-16 16:27:56 UTC
Hi David,

yes, you're right, on a new osp8 installation only v2 is presents.  But at the controller-major upgrade step during osp8->osp9 upgrade, the api-paste.ini get updated to include v3 definition.  This is package installation only.  My guess is that there was some local modification done to the file and the package management system didn't replace with the new version.

So I think this should be a documentation warning, as currently, no provision is done to manage this file at all (only package installation and upgrade).  Something along those lines:

"check setting for new configuration file with local modification and review the differences and adjust accordingly:

  find /etc -print | egrep "rpmnew$|rpmsave$"
"

The whole rpmnew/rpmsave is well described there[1]

[1]: https://www.redhat.com/archives/rhl-list/2003-December/msg04713.html

Would it be a valuable solution.

Asking pm as well.

Comment 4 Jaromir Coufal 2017-02-28 19:49:20 UTC
If there was local modification, we are unable to detect such changes and support successful upgrade. In this case manual steps would be needed. Is it the case David?

Comment 5 David Juran 2017-03-06 15:20:05 UTC
Sorry for the delay, I was merely providing an additional data point.

But looking at "my" deployment, the /etc/cinder/api-paste.ini seem to have updated well during the upgrade to OSP9, it now has sections on cinder v3. However something else is is still fishy, 

cinder --os-volume-api-version 3 create 10

still tell me that the version must be one of 1 or 2. And looking at 

openstack catalog list

there is an endpoint for cinder and cinderv2 but none for cinderv3.

But I guess that's a topic for a different Bz....

In regards to #3, in my opinion, we should automate the process rather then asking the user to review rpmnew files. If files have been modified _manually_, i.e. outside the control of the OSP-d all bets are off anyway. But if a user has used a parameter like 

controllerExtraConfig:
  cinder::config::api_paste_ini_config

to modify the api_paste, we shouldn't force the user to go through a manual verification. Wouldn't it make more sense to instead put the api-paste under puppet-control rather then relying on the RPM?

Comment 6 Amit Ugol 2017-03-30 04:51:05 UTC
With regards to comment #4 and the first half of comment #5, could you please ask how cinder ini file got changed? If it is manual, then it is well documented that those changes are the changer's responsibility to maintain. If the change came in the form as in the 2nd part of comment #5, then it is indeed a director bug, though I did not reproduce it.

Comment 7 Martin Schuppert 2017-03-30 06:49:14 UTC
I'll check with the user, but what I know is:
* the templates do not contain the cinder::config::api_paste_ini_config 
* the environment got updated from OSP8 before, I just wanted to mention it. As said in comment #3 the file should have been updated in the upgrade process.

Comment 8 Martin Schuppert 2017-04-28 14:03:31 UTC
(In reply to Martin Schuppert from comment #7)
> I'll check with the user, but what I know is:
> * the templates do not contain the cinder::config::api_paste_ini_config 
> * the environment got updated from OSP8 before, I just wanted to mention it.
> As said in comment #3 the file should have been updated in the upgrade
> process.

Amit, while the env was on OSP7, there was work done to integrate a 3rd party self service portal, but it is not sure if the /etc/cinder/api-paste.ini was modified during that time. What we can definitely say is that the file was not managed at any time via OSP-d.

Comment 9 Sofer Athlan-Guyot 2017-05-17 09:17:55 UTC
Hi,

We discovered that enabling tls was enough to have this file modified.

so this is one use case where the costumer has nothing to do to have this file updated which then disable the normal package upgrade handling of this file. 

I proposed a patch upstream to remove the tls done is this file when we upgrade to 10.  TLS is handled in another way now but this leftover will cause problem.