Bug 1255359 - Flavor cannot be deleted
Flavor cannot be deleted
Status: CLOSED NEXTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-novaclient (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 8.0 (Liberty)
Assigned To: Diana Clarke
nlevinki
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-20 07:20 EDT by Marko Myllynen
Modified: 2015-10-29 01:52 EDT (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-28 22:38:12 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
output.txt (12.71 KB, text/plain)
2015-08-20 07:22 EDT, Marko Myllynen
no flags Details
test.py (3.05 KB, text/plain)
2015-10-14 08:41 EDT, Marko Myllynen
no flags Details
logs.tar.bz2 (27.72 KB, application/x-bzip)
2015-10-28 12:35 EDT, Marko Myllynen
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1446850 None None None Never

  None (edit)
Description Marko Myllynen 2015-08-20 07:20:18 EDT
Description of problem:
When doing some tests with private flavors we noticed that in some cases private flavor are undeleteable. Please see the attached script and output from two test runs.

It's unclear whether the issue is with API usage or with the nova command but either way, we should not end up in a situation where some of the flavors can't be deleted.

Version-Release number of selected component (if applicable):
openstack-nova-api-2015.1.0-17.el7ost.noarch
openstack-nova-cert-2015.1.0-17.el7ost.noarch
openstack-nova-common-2015.1.0-17.el7ost.noarch
openstack-nova-conductor-2015.1.0-17.el7ost.noarch
openstack-nova-console-2015.1.0-17.el7ost.noarch
openstack-nova-novncproxy-2015.1.0-17.el7ost.noarch
openstack-nova-scheduler-2015.1.0-17.el7ost.noarch
python-nova-2015.1.0-17.el7ost.noarch
python-novaclient-2.23.0-1.el7ost.noarch
Comment 3 Marko Myllynen 2015-08-20 07:22:06 EDT
Created attachment 1065212 [details]
test.py
Comment 4 Marko Myllynen 2015-08-20 07:22:32 EDT
Created attachment 1065213 [details]
output.txt
Comment 5 Diana Clarke 2015-10-02 16:48:28 EDT
Hi Marko:

I just tried running your script locally, and I'm not having any difficulty deleting public and private flavors using both the API and the command line interface. Note that I tested against master rather than the same version as you though. Here's my output:

    http://paste.openstack.org/show/475214/

Are you still able to reproduce this bug? If so, can you please send me the following additional information.

1. I noticed that you are using both a 'demo' and an 'admin' user. Can you describe how you created those users? That is, what roles did you associate with the user/tenant combos etc?

2. Did you modify the 'os_compute_api:os-flavor-manage' entry in the policy.json file to allow the 'demo' user to create and delete flavors, or otherwise change the permissions?

3. When you only use the 'admin' user, do you still have difficulty deleting private flavors?

4. Can you paste the 'test_1' and 'Jack' rows from the database into bugzilla? For example:

    $ mysql nova
    [nova]> select * from instance_types where name = 'Jack'\G
    [nova]> select * from instance_types where name = 'test_1'\G

5. Please also attach the nova logs that span both the API requests and nova commands you are issuing.

Thanks!
Comment 6 Diana Clarke 2015-10-05 16:58:55 EDT
In addition to being able to successfully delete public and private flavors using devstack master (approximately liberty/stable), I have also tested deleting public & private flavors with devstack kilo/stable (works, no errors). In both cases, I was testing with the admin user. I also checked upstream for similar error reports, but I haven't found any.
Comment 8 Marko Myllynen 2015-10-14 08:41:07 EDT
Hi Diana,

thanks for looking into this.

I was using the following script to create the tenant after a PackStack installation (which used defaults pretty much for everything expect for IPs):

#!/bin/sh

set -e

# Become as admin
. /root/keystonerc_admin || exit 1

# Create minimal test flavor
nova flavor-create --is-public true m1.nano 10 128 1 1

# Create a test tenant and its init file
keystone tenant-create --name demo
keystone user-create --tenant demo --name demo --pass demo \
  --email root@localhost
cp -p /root/keystonerc_admin /root/keystonerc_demo
sed -i -e 's,admin,demo,g' /root/keystonerc_demo

# Become as tenant
. /root/keystonerc_demo || exit 2

And the script finished I also adjusted policy.json to have:

     "compute_extension:flavormanage": "rule:admin_or_owner",

I'm still able to reproduce this on latest RHEL OSP 7 + updates with the updated test.py script - but only if the test flavor is named something like "Jack", using "test_1" does not bring up this issue (apologies for having attached a wrong version of the script initially).

Thanks.
Comment 9 Marko Myllynen 2015-10-14 08:41 EDT
Created attachment 1082813 [details]
test.py
Comment 10 Diana Clarke 2015-10-16 05:18:52 EDT
Thanks a ton for the additional information, Marko!

I'll take a look at this again next week.

Have a great weekend!

Cheers,

--diana
Comment 11 Diana Clarke 2015-10-23 16:53:35 EDT
Hi Marko:

In comment #8, you mentioned that you changed the following in your policy.json file.

    "compute_extension:flavormanage": "rule:admin_or_owner",

Did you perhaps mean "os_compute_api:os-flavor-manage": "rule:admin_or_owner"? As far as I can tell, changing "compute_extension:flavormanage" will not allow the demo user to manage flavors, but changing "os_compute_api:os-flavor-manage" will.

    $ diff policy.json policy.json.bak 
    <     "os_compute_api:os-flavor-manage": "rule:admin_or_owner",
    ---
    >     "os_compute_api:os-flavor-manage": "rule:admin_api",

Be default, only admin users can create and delete flavors. Changing "os_compute_api:os-flavor-manage" to "rule:admin_or_owner" will allow the tenant owner (demo) to create and delete flavors too.

The exception to that rule is private flavors. Only admin users can list and delete private flavors. Oddly enough, the tenant owner (demo) can create private flavors, but they cannot delete or list that private flavor.

If there is a bug here, I suspect it's the fact that tenant owners like the demo user can create private flavors that they can't list or later delete. When a tenant owner tries to delete a private flavor they get a 404 error.

This may be by design though. I can create a bug report upstream and see what people say. Thoughts? Do you think a tenant owner should be able to create private flavors? Do you think tenant owners should be able to list and delete their private flavors?

PS. I have reviewed and tested this code at length, and I don't think it has anything to do with what you name the flavor ("Jack" vs "test_1").

Have a great weekend!
Comment 12 Marko Myllynen 2015-10-26 05:16:30 EDT
(In reply to Diana Clarke from comment #11)
> 
> In comment #8, you mentioned that you changed the following in your
> policy.json file.
> 
>     "compute_extension:flavormanage": "rule:admin_or_owner",
> 
> Did you perhaps mean "os_compute_api:os-flavor-manage":
> "rule:admin_or_owner"?

Nope, I only changed compute_extension:flavormanage.

> The exception to that rule is private flavors. Only admin users can list and
> delete private flavors. Oddly enough, the tenant owner (demo) can create
> private flavors, but they cannot delete or list that private flavor.
> 
> If there is a bug here, I suspect it's the fact that tenant owners like the
> demo user can create private flavors that they can't list or later delete.
> When a tenant owner tries to delete a private flavor they get a 404 error.
> 
> This may be by design though. I can create a bug report upstream and see
> what people say. Thoughts? Do you think a tenant owner should be able to
> create private flavors? Do you think tenant owners should be able to list
> and delete their private flavors?

As you say, it's all a bit confusing but I think what a tenant can create, they should be also able to list/delete. Also, could be that the script is not perfect or a different policy change would be better here. But at least there's one clear bug identified already, as we end up in a situation with an undeleteable flavor.

> PS. I have reviewed and tested this code at length, and I don't think it has
> anything to do with what you name the flavor ("Jack" vs "test_1").

I seem to be doing rather terrible job here providing all the details. So far I've only seen this with flavor names starting upper case. Here's a "screenshot" illustrating the issue after running the test.py script twice (using both test_1 and Jack) and differentiating between 400 and 404 errors:

[root@rhel-7-osp-1 ~(keystone_demo)]# nova flavor-delete Jack
ERROR (BadRequest): The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400) (Request-ID: req-5b1974f3-82fa-462d-9c7f-a55873760235)
[root@rhel-7-osp-1 ~(keystone_demo)]# nova flavor-delete test_1
ERROR (NotFound): Flavor test_1 could not be found. (HTTP 404) (Request-ID: req-ba027ab7-a707-404b-809c-2a6d7e7f30e6)
[root@rhel-7-osp-1 ~(keystone_demo)]# . /root/keystonerc_admin
[root@rhel-7-osp-1 ~(keystone_admin)]# nova flavor-delete Jack
ERROR (BadRequest): The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400) (Request-ID: req-40640ee3-0968-418f-812f-fc1123db6b74)
[root@rhel-7-osp-1 ~(keystone_admin)]# nova flavor-delete test_1
+--------+--------+-----------+------+-----------+------+-------+-------------+-----------+
| ID     | Name   | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+--------+--------+-----------+------+-----------+------+-------+-------------+-----------+
| test_1 | test_1 | 1024      | 10   | 0         |      | 1     | 1.0         | False     |
+--------+--------+-----------+------+-----------+------+-------+-------------+-----------+
[root@rhel-7-osp-1 ~(keystone_admin)]# 

Thanks.
Comment 13 Diana Clarke 2015-10-27 10:08:38 EDT
Hi Marko:

Can you please add the following information to this bug:

    1) Your nova policy.json file.

    2) The compute logs generated around the time you saw the 400 errors.

        - I'm especially interested in any logs with (or near) these ids:

            req-5b1974f3-82fa-462d-9c7f-a55873760235
            req-40640ee3-0968-418f-812f-fc1123db6b74

    3) The output from the following commands:

        $ openstack user list

        $ openstack role list

        $ openstack role list --user demo --project demo

        $ openstack role list --user admin --project demo

Thanks!
Comment 14 Marko Myllynen 2015-10-28 12:35 EDT
Created attachment 1087279 [details]
logs.tar.bz2

Hi Diana,

I'm now attaching a tarball containing the additional info as well as the PackStack answers.cfg file I used in case you want to try reproduce locally.

Thanks.
Comment 15 Diana Clarke 2015-10-28 13:55:15 EDT
From Marko:

[root@rhel-7-osp-1 logs(keystone_admin)]# nova flavor-delete Jack
ERROR (BadRequest): The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400) (Request-ID: req-bea9bd41-6ee3-46da-a0b2-07df044d8f7c)

---------------------------------------------

2015-10-28 18:08:13.733 2530 ERROR nova.api.openstack.wsgi [req-bea9bd41-6ee3-46da-a0b2-07df044d8f7c 58d5f1e8d6bd494c85cc7edffc6d7781 1bb3b821c9174e179707302c860adfa7 - - -] Exception handling resource: 'NoneType' object has no attribute '__getitem__'
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi Traceback (most recent call last):
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 710, in post_process_extensions
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi     **action_args)
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/flavor_access.py", line 86, in show
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi     self._extend_flavor(resp_obj.obj['flavor'], db_flavor)
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/flavor_access.py", line 78, in _extend_flavor
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi     flavor_rval[key] = flavor_ref['is_public']
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi TypeError: 'NoneType' object has no attribute '__getitem__'
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi 
2015-10-28 18:08:13.735 2530 INFO nova.osapi_compute.wsgi.server [req-bea9bd41-6ee3-46da-a0b2-07df044d8f7c 58d5f1e8d6bd494c85cc7edffc6d7781 1bb3b821c9174e179707302c860adfa7 - - -] 192.168.122.101 "GET /v2/1bb3b821c9174e179707302c860adfa7/flavors/jack HTTP/1.1" status: 400 len: 362 time: 0.0708370

---------------------------------------------


[root@rhel-7-osp-1 logs(keystone_admin)]# openstack user list
+----------------------------------+---------+
| ID                               | Name    |
+----------------------------------+---------+
| 047798b793f042429e41092b2697b935 | cinder  |
| 0dc84a75b98a49df80ed71af9860d9df | nova    |
| 257bb204f554415a9f6f46ed2747c64c | neutron |
| 58d5f1e8d6bd494c85cc7edffc6d7781 | admin   |
| 69451d65e7e549d7a7117d4fb45f3205 | glance  |
| c8d0a6b11bb84de78fdbff2edd9cb945 | demo    |
+----------------------------------+---------+
[root@rhel-7-osp-1 logs(keystone_admin)]# openstack role list
+----------------------------------+----------+
| ID                               | Name     |
+----------------------------------+----------+
| 886ff45ad8a74d00a1b65dec0bceba01 | admin    |
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |
+----------------------------------+----------+
[root@rhel-7-osp-1 logs(keystone_admin)]# openstack role list --user demo --project demo
+----------------------------------+----------+---------+------+
| ID                               | Name     | Project | User |
+----------------------------------+----------+---------+------+
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | demo    | demo |
+----------------------------------+----------+---------+------+
[root@rhel-7-osp-1 logs(keystone_admin)]# openstack role list --user admin --project demo

[root@rhel-7-osp-1 logs(keystone_admin)]# 

---------------------------------------------

[diana@localhost ~]$ diff marko_policy.json /etc/nova/policy.json | grep flavor
<     "compute_extension:flavormanage": "rule:admin_or_owner",
>     "compute_extension:flavormanage": "rule:admin_api",
Comment 16 Diana Clarke 2015-10-28 14:10:13 EDT
Hmmm... notice that the case changed (with the same request ID):

nova flavor-delete Jack

vs.

GET /v2/1bb3b821c9174e179707302c860adfa7/flavors/jack

I'll start this pass of digging there.

Also note that this error happened with the admin user this time, so the permissions are less interesting now.
Comment 17 Diana Clarke 2015-10-28 22:38:12 EDT
I have finally tracked down the root of the problem :)

You were right, Marko! It did have something to do with capital letters in the flavor ID.

The following upstream change introduced the bug in Kilo:

    https://bugs.launchpad.net/python-novaclient/+bug/1423885
    https://review.openstack.org/#/c/158270/10/novaclient/v2/shell.py

And the following upstream change fixed the bug in Liberty:

    https://bugs.launchpad.net/python-novaclient/+bug/1446850
    https://review.openstack.org/#/c/176106/1/novaclient/v2/shell.py

Workaround: Use lower case or numeric flavor IDs only, or upgrade to Liberty.

I suspect it's probably not worth back-porting a fix to Kilo (upstream or downstream) unless we have a customer that really needs the back-port. I'll close this bug as WONTFIX in the mean time.

Marko: Thanks a ton for your patience working with me on this bug (answering my questions, getting back to me with additional data, etc). I'm new to both Red Hat & OpenStack, and I learned a ton in the process of debugging this issue (how to create a devstack for an old release, how permissions work, everything I now know about flavors, etc).

Cheers,

--diana
Comment 18 Marko Myllynen 2015-10-29 01:52:22 EDT
(In reply to Diana Clarke from comment #17)
> I have finally tracked down the root of the problem :)
> 
> You were right, Marko! It did have something to do with capital letters in
> the flavor ID.
> 
> The following upstream change introduced the bug in Kilo:
> 
>     https://bugs.launchpad.net/python-novaclient/+bug/1423885
>     https://review.openstack.org/#/c/158270/10/novaclient/v2/shell.py
> 
> And the following upstream change fixed the bug in Liberty:
> 
>     https://bugs.launchpad.net/python-novaclient/+bug/1446850
>     https://review.openstack.org/#/c/176106/1/novaclient/v2/shell.py

Good find, thanks!

> I suspect it's probably not worth back-porting a fix to Kilo (upstream or
> downstream) unless we have a customer that really needs the back-port.

The workaround is certainly easy enough, absolutely no need for backporting.

> I'll close this bug as WONTFIX in the mean time.
> 
> Marko: Thanks a ton for your patience working with me on this bug (answering
> my questions, getting back to me with additional data, etc).

No problem, thanks for your prompt replies. I'll leave it up to you determine whether it's worth continuing in upstream wrt tenant owner being able to list/delete flavor their created.

> I'm new to both
> Red Hat & OpenStack, and I learned a ton in the process of debugging this
> issue (how to create a devstack for an old release, how permissions work,
> everything I now know about flavors, etc).

There's actually still one more learning experience left here :) When the issue is real and it has been fixed in upstream already, WONTFIX is a bit misleading, NEXTRELEASE (or perhaps UPSTREAM) is a better choice.

Cheers.

Note You need to log in before you can comment on or make changes to this bug.