This is a problem observed in a lab deployment of nova.


qpidd was regularly disconnecting the nova-compute service due to a
timeout (as seen in the log).  A review of a packet capture with
wireshark showed that the cause was nova-compute failing to send a
heartbeat message on time.  By default, we set the heartbeat interval to
5 seconds.

The timeouts were happening at a regular interval.  When compared with
the nova-compute log, it was happening when nova-compute ran its
periodic tasks (which is done 60 seconds after the last run of periodic
tasks completes by default).

Digging further, the update_available_resource() periodic task was
taking roughly 15 seconds.  Here's the relevant code:


https://github.com/openstack/nova/blob/stable/essex/nova/compute/manager.py#L2425


https://github.com/openstack/nova/blob/stable/essex/nova/virt/libvirt/connection.py#L1945

In the libvirt driver we have this code gathering a bunch of stats:

        # Updating host information
        dic = {'vcpus': self.get_vcpu_total(),
               'memory_mb': self.get_memory_mb_total(),
               'local_gb': self.get_local_gb_total(),
               'vcpus_used': self.get_vcpu_used(),
               'memory_mb_used': self.get_memory_mb_used(),
               'local_gb_used': self.get_local_gb_used(),
               'hypervisor_type': self.get_hypervisor_type(),
               'hypervisor_version': self.get_hypervisor_version(),
               'cpu_info': self.get_cpu_info(),
               'service_id': service_ref['id'],
               'disk_available_least': self.get_disk_available_least()}

self.get_vcpu_total() was taking about 5 seconds and
self.get_disk_available_least() was taking about 10 seconds.  (You can
take a look at the driver to see what libvirt calls are being used in
each of these.)

When we block in native code (such as libvirt), other greenthreads can
not be scheduled.  This is because eventlet relies on hooking into
various parts of the Python stdlib:

    http://eventlet.net/doc/patching.html

It appears that there was enough blocking done in this periodic task
that the greenthread handling the qpid heartbeat didn't get to run in
time and caused qpidd to drop the connection.

There is a bit of support for making libvirt calls non-blocking, but it
appears to only apply when connecting to libvirt, not when making other
calls.


The fix should be to include a couple of changes:

1) Include the default for qpid_heartbeat to a more reasonable value:

    https://review.openstack.org/#/c/12981/

This one needs to go into openstack-common, then nova master, and then backported to stable/essex.

2) Yield to other threads during time consuming periodic tasks:

    https://review.openstack.org/#/c/12985

This one is proposed for nova stable/essex.