Description of problem:

If libvirt is blocked because of inaccessible storage, (e.g. in 
virConnectGetAllDomainStats), calling virDomain.getMemoryStats or
virDomain.diskErrors may block the entire process for 30 seconds.

The issue is not releasing the GIL when calling virDomainMemoryStats() and
virDomainGetDiskErrors().

Version-Release number of selected component (if applicable):
- virDomainMemoryStats issue exists since it was introduced in 2009
  commit d8d01cc94ce4a2955ef272319429dfd5a3e9df01
- virDomainGetDiskErrors issue exists since it was introduced in 2012
  commit 340ef52421ef1ac903a97b64ab0572da05175970

How reproducible:
Always

Steps to Reproduce:
1. Setup oVirt 4.2
2. Start vm connected to a cd on iso domain
3. Block access to iso domain (e.g. using iptables)
4. In vdsm log we can see that the process freeze for about 30 seconds every
   minute or so.

Actual results:
Blocking cause loss of heartbeats with oVirt engine, host becomes non-responsive,
system never recover.

Expected results:
Only the vm connected to faulty storage is affected, other vms or subsystems
on this host are not affected by the faulty storage.


More info:

Vdsm is calling virConnect.getAllDomainStats every 15 seconds. This call is
blocked for 30 seconds when an iso domain is not accessible, and a vm has a cd
on that iso domain.

Vdsm is calling virDomain.getMemoryStats on another thread every 15 seconds. We
see in vdsm log that vdsm is freezed for 30 seconds after this call.

Example vdsm log show 30 seconds without any log (impossible in debug level)

2017-09-26 17:05:42,713+0300 DEBUG (periodic/2) [storage.TaskManager.Task] (Task='9280fb0f-eda3-4838-bb06-c0b7ca151ccb') ref 0 aborting False (task:999)
2017-09-26 17:06:12,726+0300 DEBUG (monitor/410c3f4) [storage.oop] Removing idle ioprocess Global (outOfProcess:76)


Attaching to vdsm show two libvirt calls waiting:

Thread 17 (Thread 0x7f94a77f6700 (LWP 6775)):
#4 Waiting for a lock (e.g. GIL)
#8 Frame 0x7f94b8017320, for file /usr/lib64/python2.7/site-packages/libvirt.py, line 5239, in getAllDomainStats (self=<virConnect(restore=<function at remote 0x3987848>, nodeDeviceEventRegisterAny=<function at remote 0x39855f0>, nodeDeviceLookupSCSIHostByWWN=<function at remote 0x39857d0>, listAllSecrets=<function at remote 0x3982b90>, saveImageGetXMLDesc=<function at remote 0x3987b18>, virConnGetLastError=<function at remote 0x3989e60>, listDomainsID=<function at remote 0x39842a8>, storagePoolLookupByUUIDString=<function at remote 0x39899b0>, domainEventCallbackID={0: 0, 1: 1, 2: 2, 3: 6, 4: 5, 5: 8, 6: 3, 7: 21, 8: 15}, getCPUMap=<function at remote 0x3789e60>, setKeepAlive=<function at remote 0x3989230>, pingLibvirt=<instancemethod at remote 0x366d820>, changeBegin=<function at remote 0x3788c08>, _o=<PyCapsule at remote 0x2c896f0>, saveImageDefineXML=<function at remote 0x3987a28>, listDefinedDomains=<function at remote 0x3982d70>, getCapabilities=<function at remote 0x39811b8>, getFreeMemory=<function at re...(truncated)
    ret = libvirtmod.virConnectGetAllDomainStats(self._o, stats, flags)

Thread 3 (Thread 0x7f94bd7fa700 (LWP 7037)):
#10 Frame 0x7f94cc0efc90, for file /usr/lib64/python2.7/site-packages/libvirt.py, line 1449, in memoryStats (self=<virDomain(addIOThread=<function at remote 0x2c1e8c0>, pinIOThread=<function at remote 0x2c316e0>, getTime=<function at remote 0x2c33938>, maxMemory=<function at remote 0x2c08b90>, setBlockThreshold=<function at remote 0x2c29ed8>, securityLabelList=<function at remote 0x2c29a28>, _conn=<virConnect(restore=<function at remote 0x3987848>, nodeDeviceEventRegisterAny=<function at remote 0x39855f0>, nodeDeviceLookupSCSIHostByWWN=<function at remote 0x39857d0>, listAllSecrets=<function at remote 0x3982b90>, saveImageGetXMLDesc=<function at remote 0x3987b18>, virConnGetLastError=<function at remote 0x3989e60>, listDomainsID=<function at remote 0x39842a8>, storagePoolLookupByUUIDString=<function at remote 0x39899b0>, domainEventCallbackID={0: 0, 1: 1, 2: 2, 3: 6, 4: 5, 5: 8, 6: 3, 7: 21, 8: 15}, getCPUMap=<function at remote 0x3789e60>, setKeepAlive=<function at remote 0x3989230>, pingLibvirt=<instancemethod ...(truncated)
    ret = libvirtmod.virDomainMemoryStats(self._o)

We are trying to reproduce this now on oVirt 4.1 (stable version).

I sent a patch for fixing virDomain.getMemoryStats:
https://www.redhat.com/archives/libvir-list/2017-September/msg00939.html