Bug 1255359

Created attachment 1065212 [details]
test.py

Created attachment 1065213 [details]
output.txt

Hi Marko:

I just tried running your script locally, and I'm not having any difficulty deleting public and private flavors using both the API and the command line interface. Note that I tested against master rather than the same version as you though. Here's my output:

    http://paste.openstack.org/show/475214/

Are you still able to reproduce this bug? If so, can you please send me the following additional information.

1. I noticed that you are using both a 'demo' and an 'admin' user. Can you describe how you created those users? That is, what roles did you associate with the user/tenant combos etc?

2. Did you modify the 'os_compute_api:os-flavor-manage' entry in the policy.json file to allow the 'demo' user to create and delete flavors, or otherwise change the permissions?

3. When you only use the 'admin' user, do you still have difficulty deleting private flavors?

4. Can you paste the 'test_1' and 'Jack' rows from the database into bugzilla? For example:

    $ mysql nova
    [nova]> select * from instance_types where name = 'Jack'\G
    [nova]> select * from instance_types where name = 'test_1'\G

5. Please also attach the nova logs that span both the API requests and nova commands you are issuing.

Thanks!

In addition to being able to successfully delete public and private flavors using devstack master (approximately liberty/stable), I have also tested deleting public & private flavors with devstack kilo/stable (works, no errors). In both cases, I was testing with the admin user. I also checked upstream for similar error reports, but I haven't found any.

Hi Diana,

thanks for looking into this.

I was using the following script to create the tenant after a PackStack installation (which used defaults pretty much for everything expect for IPs):

#!/bin/sh

set -e

# Become as admin
. /root/keystonerc_admin || exit 1

# Create minimal test flavor
nova flavor-create --is-public true m1.nano 10 128 1 1

# Create a test tenant and its init file
keystone tenant-create --name demo
keystone user-create --tenant demo --name demo --pass demo \
  --email root@localhost
cp -p /root/keystonerc_admin /root/keystonerc_demo
sed -i -e 's,admin,demo,g' /root/keystonerc_demo

# Become as tenant
. /root/keystonerc_demo || exit 2

And the script finished I also adjusted policy.json to have:

     "compute_extension:flavormanage": "rule:admin_or_owner",

I'm still able to reproduce this on latest RHEL OSP 7 + updates with the updated test.py script - but only if the test flavor is named something like "Jack", using "test_1" does not bring up this issue (apologies for having attached a wrong version of the script initially).

Thanks.

Created attachment 1082813 [details]
test.py

Thanks a ton for the additional information, Marko!

I'll take a look at this again next week.

Have a great weekend!

Cheers,

--diana

Hi Marko:

In comment #8, you mentioned that you changed the following in your policy.json file.

    "compute_extension:flavormanage": "rule:admin_or_owner",

Did you perhaps mean "os_compute_api:os-flavor-manage": "rule:admin_or_owner"? As far as I can tell, changing "compute_extension:flavormanage" will not allow the demo user to manage flavors, but changing "os_compute_api:os-flavor-manage" will.

    $ diff policy.json policy.json.bak 
    <     "os_compute_api:os-flavor-manage": "rule:admin_or_owner",
    ---
    >     "os_compute_api:os-flavor-manage": "rule:admin_api",

Be default, only admin users can create and delete flavors. Changing "os_compute_api:os-flavor-manage" to "rule:admin_or_owner" will allow the tenant owner (demo) to create and delete flavors too.

The exception to that rule is private flavors. Only admin users can list and delete private flavors. Oddly enough, the tenant owner (demo) can create private flavors, but they cannot delete or list that private flavor.

If there is a bug here, I suspect it's the fact that tenant owners like the demo user can create private flavors that they can't list or later delete. When a tenant owner tries to delete a private flavor they get a 404 error.

This may be by design though. I can create a bug report upstream and see what people say. Thoughts? Do you think a tenant owner should be able to create private flavors? Do you think tenant owners should be able to list and delete their private flavors?

PS. I have reviewed and tested this code at length, and I don't think it has anything to do with what you name the flavor ("Jack" vs "test_1").

Have a great weekend!

(In reply to Diana Clarke from comment #11)
> 
> In comment #8, you mentioned that you changed the following in your
> policy.json file.
> 
>     "compute_extension:flavormanage": "rule:admin_or_owner",
> 
> Did you perhaps mean "os_compute_api:os-flavor-manage":
> "rule:admin_or_owner"?

Nope, I only changed compute_extension:flavormanage.

> The exception to that rule is private flavors. Only admin users can list and
> delete private flavors. Oddly enough, the tenant owner (demo) can create
> private flavors, but they cannot delete or list that private flavor.
> 
> If there is a bug here, I suspect it's the fact that tenant owners like the
> demo user can create private flavors that they can't list or later delete.
> When a tenant owner tries to delete a private flavor they get a 404 error.
> 
> This may be by design though. I can create a bug report upstream and see
> what people say. Thoughts? Do you think a tenant owner should be able to
> create private flavors? Do you think tenant owners should be able to list
> and delete their private flavors?

As you say, it's all a bit confusing but I think what a tenant can create, they should be also able to list/delete. Also, could be that the script is not perfect or a different policy change would be better here. But at least there's one clear bug identified already, as we end up in a situation with an undeleteable flavor.

> PS. I have reviewed and tested this code at length, and I don't think it has
> anything to do with what you name the flavor ("Jack" vs "test_1").

I seem to be doing rather terrible job here providing all the details. So far I've only seen this with flavor names starting upper case. Here's a "screenshot" illustrating the issue after running the test.py script twice (using both test_1 and Jack) and differentiating between 400 and 404 errors:

[root@rhel-7-osp-1 ~(keystone_demo)]# nova flavor-delete Jack
ERROR (BadRequest): The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400) (Request-ID: req-5b1974f3-82fa-462d-9c7f-a55873760235)
[root@rhel-7-osp-1 ~(keystone_demo)]# nova flavor-delete test_1
ERROR (NotFound): Flavor test_1 could not be found. (HTTP 404) (Request-ID: req-ba027ab7-a707-404b-809c-2a6d7e7f30e6)
[root@rhel-7-osp-1 ~(keystone_demo)]# . /root/keystonerc_admin
[root@rhel-7-osp-1 ~(keystone_admin)]# nova flavor-delete Jack
ERROR (BadRequest): The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400) (Request-ID: req-40640ee3-0968-418f-812f-fc1123db6b74)
[root@rhel-7-osp-1 ~(keystone_admin)]# nova flavor-delete test_1
+--------+--------+-----------+------+-----------+------+-------+-------------+-----------+
| ID     | Name   | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+--------+--------+-----------+------+-----------+------+-------+-------------+-----------+
| test_1 | test_1 | 1024      | 10   | 0         |      | 1     | 1.0         | False     |
+--------+--------+-----------+------+-----------+------+-------+-------------+-----------+
[root@rhel-7-osp-1 ~(keystone_admin)]# 

Thanks.

Hi Marko:

Can you please add the following information to this bug:

    1) Your nova policy.json file.

    2) The compute logs generated around the time you saw the 400 errors.

        - I'm especially interested in any logs with (or near) these ids:

            req-5b1974f3-82fa-462d-9c7f-a55873760235
            req-40640ee3-0968-418f-812f-fc1123db6b74

    3) The output from the following commands:

        $ openstack user list

        $ openstack role list

        $ openstack role list --user demo --project demo

        $ openstack role list --user admin --project demo

Thanks!

Created attachment 1087279 [details]
logs.tar.bz2

Hi Diana,

I'm now attaching a tarball containing the additional info as well as the PackStack answers.cfg file I used in case you want to try reproduce locally.

Thanks.

From Marko:

[root@rhel-7-osp-1 logs(keystone_admin)]# nova flavor-delete Jack
ERROR (BadRequest): The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400) (Request-ID: req-bea9bd41-6ee3-46da-a0b2-07df044d8f7c)

---------------------------------------------

2015-10-28 18:08:13.733 2530 ERROR nova.api.openstack.wsgi [req-bea9bd41-6ee3-46da-a0b2-07df044d8f7c 58d5f1e8d6bd494c85cc7edffc6d7781 1bb3b821c9174e179707302c860adfa7 - - -] Exception handling resource: 'NoneType' object has no attribute '__getitem__'
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi Traceback (most recent call last):
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 710, in post_process_extensions
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi     **action_args)
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/flavor_access.py", line 86, in show
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi     self._extend_flavor(resp_obj.obj['flavor'], db_flavor)
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/flavor_access.py", line 78, in _extend_flavor
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi     flavor_rval[key] = flavor_ref['is_public']
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi TypeError: 'NoneType' object has no attribute '__getitem__'
2015-10-28 18:08:13.733 2530 TRACE nova.api.openstack.wsgi 
2015-10-28 18:08:13.735 2530 INFO nova.osapi_compute.wsgi.server [req-bea9bd41-6ee3-46da-a0b2-07df044d8f7c 58d5f1e8d6bd494c85cc7edffc6d7781 1bb3b821c9174e179707302c860adfa7 - - -] 192.168.122.101 "GET /v2/1bb3b821c9174e179707302c860adfa7/flavors/jack HTTP/1.1" status: 400 len: 362 time: 0.0708370

---------------------------------------------


[root@rhel-7-osp-1 logs(keystone_admin)]# openstack user list
+----------------------------------+---------+
| ID                               | Name    |
+----------------------------------+---------+
| 047798b793f042429e41092b2697b935 | cinder  |
| 0dc84a75b98a49df80ed71af9860d9df | nova    |
| 257bb204f554415a9f6f46ed2747c64c | neutron |
| 58d5f1e8d6bd494c85cc7edffc6d7781 | admin   |
| 69451d65e7e549d7a7117d4fb45f3205 | glance  |
| c8d0a6b11bb84de78fdbff2edd9cb945 | demo    |
+----------------------------------+---------+
[root@rhel-7-osp-1 logs(keystone_admin)]# openstack role list
+----------------------------------+----------+
| ID                               | Name     |
+----------------------------------+----------+
| 886ff45ad8a74d00a1b65dec0bceba01 | admin    |
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |
+----------------------------------+----------+
[root@rhel-7-osp-1 logs(keystone_admin)]# openstack role list --user demo --project demo
+----------------------------------+----------+---------+------+
| ID                               | Name     | Project | User |
+----------------------------------+----------+---------+------+
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | demo    | demo |
+----------------------------------+----------+---------+------+
[root@rhel-7-osp-1 logs(keystone_admin)]# openstack role list --user admin --project demo

[root@rhel-7-osp-1 logs(keystone_admin)]# 

---------------------------------------------

[diana@localhost ~]$ diff marko_policy.json /etc/nova/policy.json | grep flavor
<     "compute_extension:flavormanage": "rule:admin_or_owner",
>     "compute_extension:flavormanage": "rule:admin_api",

Hmmm... notice that the case changed (with the same request ID):

nova flavor-delete Jack

vs.

GET /v2/1bb3b821c9174e179707302c860adfa7/flavors/jack

I'll start this pass of digging there.

Also note that this error happened with the admin user this time, so the permissions are less interesting now.

I have finally tracked down the root of the problem :)

You were right, Marko! It did have something to do with capital letters in the flavor ID.

The following upstream change introduced the bug in Kilo:

    https://bugs.launchpad.net/python-novaclient/+bug/1423885
    https://review.openstack.org/#/c/158270/10/novaclient/v2/shell.py

And the following upstream change fixed the bug in Liberty:

    https://bugs.launchpad.net/python-novaclient/+bug/1446850
    https://review.openstack.org/#/c/176106/1/novaclient/v2/shell.py

Workaround: Use lower case or numeric flavor IDs only, or upgrade to Liberty.

I suspect it's probably not worth back-porting a fix to Kilo (upstream or downstream) unless we have a customer that really needs the back-port. I'll close this bug as WONTFIX in the mean time.

Marko: Thanks a ton for your patience working with me on this bug (answering my questions, getting back to me with additional data, etc). I'm new to both Red Hat & OpenStack, and I learned a ton in the process of debugging this issue (how to create a devstack for an old release, how permissions work, everything I now know about flavors, etc).

Cheers,

--diana

(In reply to Diana Clarke from comment #17)
> I have finally tracked down the root of the problem :)
> 
> You were right, Marko! It did have something to do with capital letters in
> the flavor ID.
> 
> The following upstream change introduced the bug in Kilo:
> 
>     https://bugs.launchpad.net/python-novaclient/+bug/1423885
>     https://review.openstack.org/#/c/158270/10/novaclient/v2/shell.py
> 
> And the following upstream change fixed the bug in Liberty:
> 
>     https://bugs.launchpad.net/python-novaclient/+bug/1446850
>     https://review.openstack.org/#/c/176106/1/novaclient/v2/shell.py

Good find, thanks!

> I suspect it's probably not worth back-porting a fix to Kilo (upstream or
> downstream) unless we have a customer that really needs the back-port.

The workaround is certainly easy enough, absolutely no need for backporting.

> I'll close this bug as WONTFIX in the mean time.
> 
> Marko: Thanks a ton for your patience working with me on this bug (answering
> my questions, getting back to me with additional data, etc).

No problem, thanks for your prompt replies. I'll leave it up to you determine whether it's worth continuing in upstream wrt tenant owner being able to list/delete flavor their created.

> I'm new to both
> Red Hat & OpenStack, and I learned a ton in the process of debugging this
> issue (how to create a devstack for an old release, how permissions work,
> everything I now know about flavors, etc).

There's actually still one more learning experience left here :) When the issue is real and it has been fixed in upstream already, WONTFIX is a bit misleading, NEXTRELEASE (or perhaps UPSTREAM) is a better choice.

Cheers.