Bug 1938966

Summary:	Quota API returns invalid JSON when 'reserved' > 0
Product:	Red Hat OpenStack	Reporter:	Matthew Booth <mbooth>
Component:	openstack-neutron	Assignee:	Rodolfo Alonso <ralonsoh>
Status:	CLOSED ERRATA	QA Contact:	Candido Campos <ccamposr>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	16.2 (Train)	CC:	chrisw, ralonsoh, scohen
Target Milestone:	ga	Keywords:	Triaged
Target Release:	16.2 (Train on RHEL 8.4)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-neutron-15.3.5-2.20210520174810.e00f444.el8ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-09-15 06:39:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Matthew Booth 2021-03-15 11:23:48 UTC

When a resource (e.g. port) is requested, QuotaDetail returns invalid JSON until all resources of that type are created (e.g. all ports requested are now created and there is no ongoing request to create any other port). The invalid JSON is only for the "reserved" key, which returns a string instead of an integer.

This is incompatible with what the API is supposed to return:
https://docs.openstack.org/api-ref/network/v2/index.html?expanded=show-quota-details-for-a-tenant-detail

"The value for each resource type is itself an object (the quota set) containing the quota’s used, limit and reserved integer values."

It is problematic in the context of gophercloud, which expect an integer to be returned, and not a string:
https://github.com/gophercloud/gophercloud/blob/cd9c207e93f4f76af2c0a06c6d449ab342bfbe56/openstack/networking/v2/extensions/quotas/results.go#L125

# Pre-conditions

* Neutron QuotaDB driver is used.
* Resources have to be under creation (e.g. ports)

# Step-by-step reproduction steps

* Get a token from Keystone:
$ openstack token issue

And export its ID as $token.

* Run this loop:
$ while true; do curl -H "X-Auth-Token: $token" <your-neutron-endpoint>:9696/v2.0/quotas/<your-project-id>/details.json | jq '.'|& tee -a logs; done

* In another terminal, run this command:
$ tail -f logs|grep 'reserved": "'

* Now, create ports (e.g. 3 ports)
$ openstack port create (...)

* Expected output

{
  "quota": {
    "port": {
      "limit": 500,
      "used": 0,
      "reserved": 3
    },
...
}

Then:

{
  "quota": {
    "port": {
      "limit": 500,
      "used": 1,
      "reserved": 2
    },
...
}

Then:

{
  "quota": {
    "port": {
      "limit": 500,
      "used": 2,
      "reserved": 1
    },
...
}

And then:

{
  "quota": {
    "port": {
      "limit": 500,
      "used": 3,
      "reserved": 0
    },
...
}

* Actual output

{
  "quota": {
    "port": {
      "limit": 500,
      "used": 0,
      "reserved": "3"
    },
...
}

Then:

{
  "quota": {
    "port": {
      "limit": 500,
      "used": 1,
      "reserved": "2"
    },
...
}

Then:

{
  "quota": {
    "port": {
      "limit": 500,
      "used": 2,
      "reserved": "1"
    },
...
}

And then:

{
  "quota": {
    "port": {
      "limit": 500,
      "used": 3,
      "reserved": 0 // an integer!
    },
...
}

# Version:

Master and stable branches.
RHEL 8.3
TripleO based deployment (OSP 17)

# Environment:
OSP17 deployed in standalone, OVN backend.

# Perceived severity:
Workaround is possible but currently blocking gophercloud to be working for QuotaDetails.

Comment 1 Matthew Booth 2021-03-15 11:41:01 UTC

Setting severity to medium because realistically we need to work round this bug in Gophercloud anyway, and we already have a patch to do that.

I've already posted a patch for it, just discussing the required testing upstream. It sounds like @ralonsoh is going to be ok with the existing SQLite-only unit test coverage, which would obviously make life easy :) However, if it looks like we need a proper regression test then I'm going to have to bow out because I can't justify that with my team. Incidentally, I have implemented tests using oslo.db's MySQLOpportunisticFixture in a past life when I worked on Nova. You may find that much of the infrastructure for this is already in place, depending on how the existing db tests do their setup. You'd still need to wire it up in CI, though.

Please take my filing this bug as explicit permission to modify and/or abandon my proposed upstream patch :) If it stays on its current course (no new DB testing framework) then I'll push it over the line myself. If not I'll have to leave it to you to prioritise. Either way, please don't wait for me if you want to move on with it.

Comment 2 Rodolfo Alonso 2021-03-16 18:41:31 UTC

Hello Matthew:

I'll take care of this BZ.

I've pushed [1] in order to test this change using two different DB backends: MySQL and PostgreeSQL. You are right on the reply to my comment in [2]: Neutron UTs use SQLite and we cannot see the regression you are fixing. [2] is the translation of this proposal to FT, that uses both DB backends.

Regards.


[1]https://review.opendev.org/c/openstack/neutron/+/780946
[2]https://review.opendev.org/c/openstack/neutron/+/779878

Comment 3 Matthew Booth 2021-03-16 20:15:24 UTC

(In reply to Rodolfo Alonso from comment #2)
> Hello Matthew:
> 
> I'll take care of this BZ.
> 
> I've pushed [1] in order to test this change using two different DB
> backends: MySQL and PostgreeSQL. You are right on the reply to my comment in
> [2]: Neutron UTs use SQLite and we cannot see the regression you are fixing.
> [2] is the translation of this proposal to FT, that uses both DB backends.
> 
> Regards.
> 
> 
> [1]https://review.opendev.org/c/openstack/neutron/+/780946
> [2]https://review.opendev.org/c/openstack/neutron/+/779878

Thanks for doing this! I briefly started looking at doing this for all existing DB tests, but I wasn't going to have time to see it through. Looks like the targetted test both ran and passed in functional.

I'm currently babysitting my patch through a flaky CI as it's already approved, but if you want to add the functional test to it first feel free.

Comment 11 errata-xmlrpc 2021-09-15 06:39:15 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenStack Platform 16.2 (openstack-neutron) security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3488