See: https://bugs.launchpad.net/nova/+bug/1175808 Qpid has a limitation where it cannot serialize a Python dict containing a string longer than 65535 characters. This can result in problems when making a conductor call that returns a large structure - for example, instance_get_all_by_host on one of my systems returns 38 instances, which when serialized as JSON is too long for Qpid to handle. Sounds like an issue (a) only seen at scale and (b) specific to nova-conductor and, therefore, Grizzly
This isn't as bad as it looks at first. It doesn't really affect grizzly, unless it's receiving a message sent by havana. The fix for havana will likely require a grizzly change to make sure grizzly can still understand havana messages, though. https://review.openstack.org/#/c/28711/ https://code.launchpad.net/bugs/1175808
Fixed upstream in havana: commit 781a8f908cd3e5e69ff8b88d998fa93c48532e15 Author: Andrew Laski <andrew.laski> Date: Wed Jun 5 10:02:07 2013 -0400 Update rpc/impl_qpid.py from oslo The current qpid driver cannot serialize objects containing strings longer than 65535 characters. This just became a breaking issue when the message to scheduler_run_instance went over that limit. The fix has been commited to oslo, so this just syncs it over to Nova. Bug 1175808 Bug 1187595 Change-Id: If95c11a7e03c81d89133f6cad0dcbb6d8acb8148
I was able to boot 120+ instance on the same hypervisor one by one, without an ERROR, and without any suspicious log message. several related command: nova quota-update --cores -1 $TENANT nova quota-update --ram -1 $TENANT nova quota-update --instances -1 $TENANT a=1; while nova boot server-$a --image cirros-0.3.1-x86_64-uec --flavor 42 --poll ;do a=$((a+1)); done packages: openstack-nova-conductor-2013.2-5.el6ost.noarch openstack-nova-scheduler-2013.2-5.el6ost.noarch The new oslo rpc code is in the python-nova package.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2013-1859.html