Bug 1706222

I was able to reproduce this on my own setup.  I believe that there are some changes with how Accept header handling is managed, in a Neutron dependency library called WebOb [1].  Neutron uses Pecan which uses WebOb, and Pecan already had to adapt to the changes in Accept header handling [2].  But even using a version of Neutron that has the fix, I still see the issue you describe.

I will note that the error seen in the Neutron server.log when this happens tells me that it is not handling parsing the string correctly.  

% curl -s -H "Accept: application/json;charset=utf-8" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/networks" 
{
    "NeutronError": {
        "detail": "",
        "message": "The server could not comply with the request since it is either malformed or otherwise incorrect.",
        "type": "HTTPNotAcceptable"
    }
}

The logs: 

2019-05-07 14:47:47.341 77 DEBUG neutron.wsgi [-] (77) accepted ('172.17.1.31', 36530) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2019-05-07 14:47:47.345 77 ERROR pecan.core [req-58872c3a-e917-4e3e-82cf-d3dbd931b6b2 21ec7039924a46e9aec6a9fa490da066 fb53e9370fb0427fadb680238d90987a - default default] Controller 'index' defined does not support content_type 'None'. Supported type(s): ['application/json']
2019-05-07 14:47:47.346 77 INFO neutron.pecan_wsgi.hooks.translation [req-58872c3a-e917-4e3e-82cf-d3dbd931b6b2 21ec7039924a46e9aec6a9fa490da066 fb53e9370fb0427fadb680238d90987a - default default] GET failed (client error): The server could not comply with the request since it is either malformed or otherwise incorrect.
2019-05-07 14:47:47.347 77 INFO neutron.wsgi [req-58872c3a-e917-4e3e-82cf-d3dbd931b6b2 21ec7039924a46e9aec6a9fa490da066 fb53e9370fb0427fadb680238d90987a - default default] 172.17.1.31 "GET /v2.0/networks HTTP/1.1" status: 406  len: 360 time: 0.0057609

It gets weirder.  This is only a problem for certain methods.

/v2.0/address-scopes is fine:

[stack@undercloud-0 ~]$ curl -s -H "Accept: application/json;charset=utf-8" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/address-scopes" | python -mjson.tool
{
    "address_scopes": []
}

/v2.0/ports is no good:

[stack@undercloud-0 ~]$ curl -s -H "Accept: application/json;charset=utf-8" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/ports" | python -mjson.tool
{
    "NeutronError": {
        "detail": "",
        "message": "The server could not comply with the request since it is either malformed or otherwise incorrect.",
        "type": "HTTPNotAcceptable"
    }
}

/v2.0/flavors is fine:

[stack@undercloud-0 ~]$ curl -s -H "Accept: application/json;charset=utf-8" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/flavors" | python -mjson.tool
{
    "flavors": []
}

/v2.0/routers is fine:

[stack@undercloud-0 ~]$ curl -s -H "Accept: application/json;charset=utf-8" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/routers" | python -mjson.tool
{
    "routers": []
}

/v2.0/subnets is a problem again:

[stack@undercloud-0 ~]$ curl -s -H "Accept: application/json;charset=utf-8" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/subnets" | python -mjson.tool
{
    "NeutronError": {
        "detail": "",
        "message": "The server could not comply with the request since it is either malformed or otherwise incorrect.",
        "type": "HTTPNotAcceptable"
    }
}

And it's definitely a combination of "charset=UTF-8" and those methods.  A semicolon followed by "hamburger" is fine.

[stack@undercloud-0 ~]$ curl -s -H "Accept: application/json;hamburger" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/subnets" | python -mjson.tool | head -5
{
    "subnets": [
        {
            "allocation_pools": [
                {

Using "charset" on it's own is fine:

[stack@undercloud-0 ~]$ curl -s -H "Accept: application/json;charset" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/subnets" | python -mjson.tool | head -5
{
    "subnets": [
        {
            "allocation_pools": [
                {

Using "hamburger=food" is an error:

[stack@undercloud-0 ~]$ curl -s -H "Accept: application/json;hamburger=food" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/subnets" | python -mjson.tool | head -5
{
    "NeutronError": {
        "detail": "",
        "message": "The server could not comply with the request since it is either malformed or otherwise incorrect.",
        "type": "HTTPNotAcceptable"

And of course, the original request "charset=UTF-8" is an error:

[stack@undercloud-0 ~]$ curl -s -H "Accept: application/json;charset=UTF-8" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/subnets" | python -mjson.tool | head -5
{
    "NeutronError": {
        "detail": "",
        "message": "The server could not comply with the request since it is either malformed or otherwise incorrect.",
        "type": "HTTPNotAcceptable"

Interestingly enough, reversing the order and saying "charset=UTF-8;application/json" works just fine:

[stack@undercloud-0 ~]$ curl -s -H "Accept: charset=UTF-8;application/json" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/subnets" | python -mjson.tool | head -5
{
    "subnets": [
        {
            "allocation_pools": [
                {

Based on this I think that we can say that:

1. The issue only occurs for certain methods (/v2.0/subnets, /v2.0/networks) but not others.
2. The issue only occurs when there is a semicolon with an equals sign after it.

I also started to look into this issue, comparing working and non-working requests, the requests going through neutron.api.v2.resource (flavor, router) work, while the failing ones (nets/subnets) do not use that.

OK so a few additional comments.

This is only happening for "base" resources not handled by neutron.api.v2 and wsgi.request: the list seems to be limited to ports,networks and subnets
This was broken by WebOb >= 1.8.0, which changed a lot in Accept handling [0]. 13 uses 1.7.x, and as a reproducer here, it will break if updating WebOb to 1.8
Other resources are not affected as there is additional processing done on neutron side for them (including the content type)

This can be reproduced on devstack master, I filled a launchpad bug [1]

Short-term, the workaround is (if possible in tool) to use a separate header for charset:
$ curl -s -H "Accept: application/json" -H "Accept-Charset: UTF-8" -H "X-Auth-Token: $OS_TOKEN" "http://localhost:9696/v2.0/ports" | python -mjson.tool
{
    "ports": []
}


[0] https://github.com/Pylons/webob/blob/master/docs/whatsnew-1.8.txt#L24
[1] https://bugs.launchpad.net/neutron/+bug/1829042

 #26 (Customer) Make PrivatePrivate Helps Resolution? 0
Created By: Brent Becker  (5/15/2019 4:26 PM)

We do know about this working around, but unfortunately the request is being sent by a Vendor supplied application so we are unable to make this change.  Is there any sort of timeline for a fix?

Hi,

I'm just curious if this is still being worked upon?

Thanks,

Andreas

Yes, this is still on the docket, although it is below a number of lower priority items.

Since the upstream bug covers this issue in detail, I am going to mark this CLOSED UPSTREAM and we can track progress there.  I hope to get some upstream experts to look at this, people that have different kinds of expertise than is available in-house.

Hi,

RHOSP 16 is still affected:

(overcloud) [stack@undercloud-0 ~]$ cat /etc/rhosp-release 
Red Hat OpenStack Platform release 16.0.2 (Train)
(overcloud) [stack@undercloud-0 ~]$ TOKEN=`openstack token issue -f value -c id`; time curl -s -H "Accept: application/json; charset=utf-8" -H "X-Auth-Token: $TOKEN" "http://10.0.0.147:9696/v2.0/networks" | python3 -mjson.tool
{
    "NeutronError": {
        "type": "HTTPNotAcceptable",
        "message": "The server could not comply with the request since it is either malformed or otherwise incorrect.",
        "detail": ""
    }
}

real    0m0.653s
user    0m0.058s
sys     0m0.012s

I'd prefer to keep the downstream bug open as the customer case refers to the downstream bug; and the downstream bug refers to the upstream bug in turn.

Hi Andreas, this bug was for OSP14, the 16 one is bug 1854443 (visible in the clones list) which is under active investigation

Great thanks. Sorry that I missed this.