Bug 2283075 - HAProxy URL length limit
Summary: HAProxy URL length limit
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-placement
Version: 16.2 (Train)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: OSP DFG:Compute
QA Contact: OSP DFG:Compute
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-05-24 01:37 UTC by Robin Cernin
Modified: 2025-01-18 02:18 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2025-01-18 02:18:22 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 2068893 0 None None None 2024-06-12 17:01:51 UTC
OpenStack gerrit 921665 0 None NEW scheduler: fix _get_sharing_providers to support unlimited aggr 2024-06-12 17:02:26 UTC
Red Hat Issue Tracker OSP-32158 0 None None None 2024-05-24 01:38:09 UTC

Description Robin Cernin 2024-05-24 01:37:10 UTC
Description of problem:

The instance delete command fails due to a HAProxy limit reached on the placement URL query from Nova API.

Nova API makes a placement GET request of length 22k characters. When this requests goes thru HAProxy we get immediately 400.

~~~
    def _get_sharing_providers(self, context, agg_uuids):
        """Queries the placement API for a list of the resource providers
        associated with any of the specified aggregates and possessing the
        MISC_SHARES_VIA_AGGREGATE trait.

        :param context: The security context
        :param agg_uuids: Iterable of string UUIDs of aggregates to filter on.
        :return: A list of dicts of resource provider information, which may be
                 empty if no provider exists with the specified UUID.
        :raise: ResourceProviderRetrievalFailed on error.
        """
        if not agg_uuids:
            return []

        aggs = ','.join(agg_uuids)
        url = "/resource_providers?member_of=in:%s&required=%s" % (
            aggs, os_traits.MISC_SHARES_VIA_AGGREGATE)
        resp = self.get(url, version='1.18',
                        global_request_id=context.global_id)
        if resp.status_code == 200:
            return resp.json()['resource_providers']

        msg = _("[%(placement_req_id)s] Failed to retrieve sharing resource "
                "providers associated with the following aggregates from "
                "placement API: %(aggs)s. Got %(status_code)d: %(err_text)s.")
        args = {
            'aggs': aggs,
            'status_code': resp.status_code,
            'err_text': resp.text,
            'placement_req_id': get_placement_request_id(resp),
        }
        LOG.error(msg, args)
        raise exception.ResourceProviderRetrievalFailed(message=msg % args)
~~~

With large number of aggregate UUIDs the default length of 16k characters is exceeded and HAProxy ends up rejecting the request.


~~~
curl -s -H "accept: application/json" -H "User-Agent: nova-scheduler keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" -H "OpenStack-API-Version: placement 1.17" -H "X-Auth-Token: $OC_TOKEN" "$PLACEMENT_ENDPOINT/resource_providers?member_of=in:TH1RXv8cd1ulc2Obtzu82mYfIf74vDXHGVwV9vCX9YyzlKwxbuPN7CNLxOI8A2KybBKutkq7AYrtay0BO7tba5XtECf7XjIKqNytqUFGHqWIha1tBga5Vdd2pNCVGq9Zfw1YJ1TM3hTaATsU9Jv3JyMiD2CDlcR7hOgroGaWwYIEezZwiL5uWRmcE873hS5VWBz2rTkaLhsbHcRvLIgBOwJ8sAuQP5jiasik2qtv44kAkNHaAAn5cbZOPeEdjLrQL9k3xWUU2nlfPdXiuiN2flnAPNv38HaGuFkEx3DHolphMRstcLUUQ9LC9hiRnexAmLWuE6uXUfPrfMqPDubPgxyne4LYEZyBjUXAS8EylZIusNVOmz0o4GGbFLNh4f23Qiu373kjsqxfucYQZfo4gMGIDXSmrx36p9hNblN3hu7Ce1Gmkh59wW2ibbefX2ga708XMSdDCVeB9E9xU3VYPdraoC54dSLV7ZgVwE4vWfuTo69w3RwwZyAU4I3J2XkKaJJhh2DqsIEq9luFxg1A5bgYufpX2gcAuLcl8woWvQGTssGtlGG40ajsddcHZy8LNLW0lnJJIKfINPStZusneVU54oD7mv2A3OKw8MYvDk32yB59Iv5gSaQ3NrD41yvTukhaIwymWaeknQitrDDHe1BhwQOUAjiaCkoRLXqAOnd8byMSdxnhTNFEIgDgQRSLqGAt97hQOuQTxaySN9sRbOqNnWmmpsYjJiM5DWB9VhDcJcQN44Bb6DabWuS6wEVPKMvV69bvBgYPtbLen9sGf2cTDc1ZGptiM3ydrpAOwGCXj7z3aaOOEyiTN59F07Ymy38kyWhlXe0AmBPshaaA0hY92bnxgcxdEonxRfS7D94URnMf9lXAaubheZPlQVeN7RGE8fh5zswvbip5sQwAirfDBloiwqRnhdjBq5Uria6oEJmXqZVRnYIMAjXp0MIz0eRgsnvNljHtQDW6IIeZZXPS0Xn6lmQDkOCSQdRYqNPClJsTvlpNb2dFWhbPl3ViPpSVosfNwkEAKVugjaDvt8BlPMbMoYBoL81IWBPeayN2lWy6JC1of1UziVn9NwmFiOdGo4tbu3gHOzKWlxoyKNDJ4vHqkFDE9HePGUjYqeXZVRO6aFvCXq7EQKewAnQd4LeoUSda6xuJK9rzegQDfqogMOmPiE68EYWIzTBiP2N2FVVl4lhrh9XUovrPBZYGIjcNj0izCg0ZYag5Lopj2SgidY9q5b8CQ4QMf96K85tQ0FiL1kxun7b3AkqNIybSnuDOqFNbt406VeG9RPDncL0JwlZ8K7PljaLF4tOCHkgVysGbdf5iS5A1DCdjoWS8B406FjZAj6JGDDjJ4IGVnWUalychwVdFa8Bfh1vkYmzUn234MthMy9fsyzAdFbc7PJ5Wwfzzp0ojd7YZiDFXmvNjgmzPfIngBTc9ZkLnVZ1DYVFwX7RBq44spwms7jX5nkN4y6c153hpnHSoVBx0oJLrxd1Dy3EXUw0EGOOxAoifmHN3avivpP9C6GDq1SEJAdTrTiURehRQ0W9HC1o6T1dDquDd5YtlAVWtJhCVH9sS26yKGmGoD3RUhDVAFtSptUIk93bMUQ3lSEmQjl5fs7Z73PE5cls3AOr9JLGOWJL7whp44vnZmQBpM3HQAHISI293SvLL0PC1hp9PGlWgkNTzkMq4CAV2NMfYnNtcNJK32wox98OUDMuyq1YjjFDL9D6PvmNTBq9RppiB9HGflQi7FXgIvNdJBhzCuhj0j4xeLCJm4kCLgfeHAJfbpIsCrGCKkXF2CaWxJoXPDfaCNH9QIpZIp2Ba2fbmAk0eQoQGYLfxn3tYkY5Q933MJz98RwkBbrNyKEEvPlx5RbC5h7hZOPU9hIPsq0NK9CfAWiM9hNz7rX4sSxH2WwhhlSSlsciJ91rZWqiBQ6tiuls7D9hk3X8enKVBBQ0cwOHYPMFPe4SiVQQyURsYd2eGF1XAdeKdXa72IPbdZVrXhV3z7CGPPASbVXOoJsF3LdctYX6OEZmlvfmLl5RCuF5kjAe6KFP5SIMQyFg1k4Tv2GgWspyUhDb1fUxhQSAWhWX3JVRJKKfCDcYdmvPxRROvsBV3b8dthOGWxS2Ba3jd96O13X8rSUYQxt4HW01KCTp6Ro75WtETiqPgAjzkYs3iPpLQXagGdb9LMBjhC48G1vRoDiZA7XrGDhilh2wQZLB0mdvlSS0IjLas3GM3S9Th2G6oPyi0ClIZOXc9OraJ3NNna8tquC0QYUr5k4hPRkPiMx8j76yXm4cDYKG67cTUaG5zuTkIFDC6jXm8rUZyYygvuYF4rYfj4J8CUou9cYzPgoo9zAhuByUKIX77bgAvfvNRQejIitDz1NsnGyjATf1PptEjlmjTpP33euEOiuiSPBE3sZ9WTbNS9RwyTXmqregmq9J7KoJ4KXrEwnuyy0mL35O5FcfqXht21n5HBc6XkZN4sWgVoDZWG6UOWLBjHhn8Lo7RSzChqrbLGVKE8513pg2ihWHXIyAcsD23o0ai1JRsHZjJuAp1bI4zKat0n4Nwjz4vdlHF0NeLSV1NLdEZFt7ngNNzGIK1pMPBxreon08OmWrRVyCecNNcTK3RTARLvwqmnFmJVGpNEzor1sZP2a5JLgtkYExZNOmtYkSHw0LCX14LwzjN87RS4GgeIu3t9P7HhXSt35ZsLxSI2zeb7of4WHcD5hgtuzK1yjnvw28oYkL6giTVvmfyUlli6x05MQGtobGi1jPZxzHgTgaFxPEaiPrhGFXVgWbAOxMYmWWT3Zj0jrDGcNqPiHH2yhR6SMnboq7alm4hvCL33OAUTrM6vYsbf7kekRarr5LD9D7Znfv6Exvq3RX4dfDXVOIe2yFszvEskfNb8KUmRp6rpujFTxl5RzLz82PNzJDVILBpQraCCT7IlO6CQkE8JtRdMO6cXc3ORJEMkc9EgEyKReuFJp4IVgt0nlbmvqJOG7M27ZXRUW7aaPIc4EQnXFxKqK7rzK3SyT00FuOuxkGthB5kONWCAYxQ4g7dzveaOAtWsU3gsGP9H0hTvgtm7oy7BoEJsdlxxZReAt5vD4hXo8pxQKqY2gh6u032kdyHbXaEVtYChj0sENDv0qJvghIrx86Qk5NyMItRu0l50rpnc9FPdyCOXaD0QutVWJhI73sjx4eXUgfi566gIL5EWdKjUY31fzy6a4fZe9LG6WeOPndB1cvWoDsWJFPyiEbWAkaDzDNpEgFbIAkQPdj3OdSm6Rk0al6TPYw7JlbuvBEKtQ1Qcx0rammbfR7Fa5zYxA9L3ybh7bjuuWI66awTc1nT9ZMzCilMKrUfMJ55DJWw42H4iF52Bylsc0CeODcoMv5TDVsh2uUVSnTPgkTldcg8OAc7LiA9g7DxA67Ig06m3cv4XRrXa1fIxyNrVFxBPyhJSf4d6iDgHtjvfUUn6oQFRQ5HpUsAplzJ7sTRq6AI7JwOBo3TSy5EVlqehzQvhAhwd8w0KCZ6p3yZLRp76Bqqoo11d9tmNOtrTT6BpuQHkjZUkY62Sqa4w1FzGZH2M7KX8xof9Vxate4OgHLXByPUeditDRd3M2NF8qKAytsFcPHzX6Axvv5QyA7DRu7uV4C2dQwwUQpLYmqenkdjYARASy2qa611J2GiOFGKqourR9Gl8mW6SNIcc1gvsJDcNGq18UEIJAjHJ9RgT079BopikIYgXWYP7ct0h38LAWDjpNsSKLYB8uqI8Lyppw5wrelh7hICiZYaxkUENoLvxUNUWPsxZscivYnwANsilCrQR5inwdZXDFOOFpLEG9KuQmCFx4E844HGlvdY0iAY5I2tm1gzkUBCJjzGYIEPVVUzt6RaQCKM5fItJNGOOQKMYdlDNayYTMSkKy3MnVSjdQHZeJ1HBFZpTx7pm7uI448gOkXOJdVCgaMzFEOlEL5x4Lwo9g43O59PjkQZBnOjLTjJFqxpviWAYE0uYdl9XoHQGDGoYUSroSXV7Mo2je8K5f5M7sUzAyZ4FfeH3rAi9FlY6iqgCCkoMR7Jwmm7ibTwHgX7PTUVZ8DcCz3FoH1pu2ri6C8VwGWmXmiHdaqvtC7j04nfFqrbptsXzIVOmEvPFCbeapae34v6XAnTTJbohdCwNEHcymbddhMhsS1FrH6Ytngx6xX4K9RPnXE7idaORhOhrpwwGtJbJRRBBKqZhaVHuesxNKdxTucq8MWRUJfvQMGTLuuIhBQFUeVk6dV7AWGSvUYrBqZlEBTLWLgH3AFxXGRp610WjnBnymp1NUpgp3NhFfjLTfSDkEV2aKvSqpAEhEtDN4xGcym1mDqOwWdxh3OUU0OPjFoKtC9JCRf10JscRLd8wklE4adlcLuo5tj78L4VX2LPMFLjseSmQREH6G5WwiMCsC2orPJYylrbqgIdRljNlXGq21KLgCEWobmDxBuXaCTyfxcRXGLBM3Fx5Jg4LSctm2OESXeIHqlVVAyP2SwesY74WDANfVBdwUfx6bDOMVMfumYwSOrmEokqpiB4Kg3FithgCN1iahNrgKYpR6yuYmubOIKo4diyxXc8QNOAgy7ARHu0aQBeb7Ycck7MG57YWt36OFA2y1ygz0qS2ynhXBnPCqlI3UmGu0UZpTWc7FFCFfYdlKjbGnDDr9e45CUzWyRDzQs7bkG26bIERZ0sXMkqF4F1VHsZMXtemShBSegWNIZbfiypLWasQvl0Z6Qs7iDCXSd2MIynes1FRLKNYOzg14dGwDmiI8NIopHz23U7D7Yu194SdIIWWPlzwCHMkZMZUAWlNDXMDxtL732fx0eieEWPaBom1YYccFWQk8s2MEQWiQy0S9bccHg2VwuXiysDjcoJXROu4PQ8gqRpA30oYJPcykXScoFafko8G6srFl4TfkcdQjQ1RaasfL5c2xGbgNx49n8JVaBsk2dhXtU05UDhbcJYCv4pc0VKhQwdRO6wsWzcyYusuTPRkQUkYAiGrFHUycPDNScRAQcISm3FLKy9SX4M3ByqQWVVx59zl17FK4fjjdSCeqEK7rMja8cGZ1NENNiL9IeSUkHp4Df00aNxvYdWWxSWCwUY01lNDxwvpP9ZIGZ038yafQR5WECVzmSZJA34lpzhPFGTyUigsZmrUdxBNyxKE7CRnBTc1NgbNQ7jLM0ZyrmS9rsJHJeLiE5Kw06ZW4G7LmI8kl2N6YZ2v4WgPd4lHoAT4QNsrFWbOJjCFnOmEP7Rcplg7gpFPo2kjSFGEj0OttzFnQTtQnopheu0r1AKg0vQYugA0bazVzD3eJEkbYDkL8rqdPnbE5Y1b67jHm8B4PquXiHFNhGs7kbCjTQZZ1VHTbnDW3kCby0tCrZptjLIbDhImN2NVRYbdz2yWOEvZINQJ1ZUA80YvfsOmGlGiTn2sVsWVkEHqQYVlFwYyzug83suuDb7Dg4SZO8vezszOZBFNwLfTSIMRBLEz7TekNEHxipusBWPSEOrVgtS7mCr2ZEyyLVCMOxmQRp16U5G7htjS2ZJZoSUGFHs7CxQMkvl5LPycgsTJYWQpw5Z9lLQJvI4iLkaduboFJnbw0dC82RT0JPAJzIxGV6CJXlLzZL30Q2HMyhQSQ1kdh0jGZPQLGtRgpDXnDHvdsZSi0m1fTPSzpMvOTYr6SMoR8BcK.......w78FSFIL7i3G8ghz14surP7OOCLfiMGJORn8aPtR2wCoiK1IYSfRd4jS8VuDawSj6nycmWovJ5QmdKexzj" 
<html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>
~~~

Version-Release number of selected component (if applicable):

OpenStack 16.2.4

How reproducible:

Follow https://access.redhat.com/solutions/3537351 and perform a query with large number of characters. (reproducing the large number of aggregate uuids)

Actual results:

HAProxy rejects the long query with:

~~~
<html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>
~~~

Expected results:

The query should pass thru the HAProxy:

~~~
curl -s -H "accept: application/json" -H "User-Agent: nova-scheduler keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" -H "OpenStack-API-Version: placement 1.17" -H "X-Auth-Token: $OC_TOKEN" "$PLACEMENT_ENDPOINT/resource_providers?member_of=in:1e48a974..." {"errors": [{"status": 400, "title": "Bad Request", "detail": "The server could not comply with the request since it is either malformed or otherwise incorrect.\n\n Invalid query string parameters: Expected 'member_of' parameter to contain valid UUID(s)....
~~~

Additional info:

Workaround using:

~~~
tune.bufsize 65536
~~~

Inside the the global section of /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

Then restart the HAProxy bundle

~~~
pcs resource restart haproxy-bundle
~~~

Comment 6 Artom Lifshitz 2025-01-18 02:18:22 UTC
At this point in time we've run out of runway for backporting this to 17.1, which is now limited to critical/urgent customer issues, and CVEs. Closing as UPSTREAM since it was done in a subsequent upstream release.


Note You need to log in before you can comment on or make changes to this bug.