Red Hat Bugzilla – Bug 962557
Issue with 65k message limit in qpid
Last modified: 2016-04-27 00:48:30 EDT
Qpid has a limitation where it cannot serialize a Python dict
containing a string longer than 65535 characters. This can result
in problems when making a conductor call that returns a large
structure - for example, instance_get_all_by_host on one of my
systems returns 38 instances, which when serialized as JSON is
too long for Qpid to handle.
Sounds like an issue (a) only seen at scale and (b) specific to nova-conductor and, therefore, Grizzly
This isn't as bad as it looks at first. It doesn't really affect grizzly, unless it's receiving a message sent by havana. The fix for havana will likely require a grizzly change to make sure grizzly can still understand havana messages, though.
Fixed upstream in havana:
Author: Andrew Laski <firstname.lastname@example.org>
Date: Wed Jun 5 10:02:07 2013 -0400
Update rpc/impl_qpid.py from oslo
The current qpid driver cannot serialize objects containing strings
longer than 65535 characters. This just became a breaking issue when
the message to scheduler_run_instance went over that limit. The fix has
been commited to oslo, so this just syncs it over to Nova.
I was able to boot 120+ instance on the same hypervisor one by one, without an ERROR, and without any suspicious log message.
several related command:
nova quota-update --cores -1 $TENANT
nova quota-update --ram -1 $TENANT
nova quota-update --instances -1 $TENANT
a=1; while nova boot server-$a --image cirros-0.3.1-x86_64-uec --flavor 42 --poll ;do a=$((a+1)); done
The new oslo rpc code is in the python-nova package.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.