Bug 1376806

Summary: After live Migration of Windows Guests, VM is not reachable via RDP and console
Product: Red Hat Enterprise Linux 7 Reporter: VIKRANT <vaggarwa>
Component: qemu-kvm-rhevAssignee: Dr. David Alan Gilbert <dgilbert>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: ailan, berrange, chayang, dasmith, dgilbert, eglynn, huding, juzhang, kchamart, knoel, ochalups, pablo.iranzo, pmannidi, pmorey, qizhu, qzhang, rbalakri, rbryant, sbauza, sferdjao, sgordon, srevivo, vaggarwa, virt-bugs, virt-maint, vromanso, xianwang, zhengtli
Target Milestone: pre-dev-freezeKeywords: ZStream
Target Release: 7.3   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-09 12:42:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1401400    

Description VIKRANT 2016-09-16 12:37:13 UTC 
Description of problem:

After live Migration of Windows Guests, VM is not reachable via RDP and console.
Sometimes this issue appeared after first migration, sometimes after completing the first migration successfully it's happening for successive migrations.

Migration at the KVM level is working well (after a while, the VM is running on the destination host). live migration of Linux guests is working well without any issue. 

IMP : - telnet on port 3389, it works even after the migration (so the TCP/IP stack is still OK)

In Cu test : vnc console is showing blank.

In my test : Not able to take any action on console using mouse after the migration. Console is accepting "ctrl+alt+del" but mouse is not working. 

Version-Release number of selected component (if applicable):
RHEL OSP 8

How reproducible:
Everytime. 

Steps to Reproduce:
1. Download an image from [a]
2. Create cinder volume using qcow2 image.
3. Spawn windows instance using cinder volume. 
4. Live-migrate the windows instance couple of times. 

[a] https://cloudbase.it/windows-cloud-images/#download

Actual results:
Windows instance becoming inaccessible after live-migration.

Expected results:
Windows instance should remain accessible after live-migration.

Additional info:

No error reported in live-migration. This issue is easily reproducible.
Comment 2 Sahid Ferdjaoui 2016-09-26 15:36:14 UTC 
After discussing with team, that issue looks to be related to the guest image but since at this step we don"t have so much information, I will try to reproduce the issue.

During the time I'm building that env and doing tests, it's possible to ask customer to return sosreport ?

Thanks,
s.
Comment 7 Sahid Ferdjaoui 2016-10-10 13:42:28 UTC 
So if you were able to confirm the same issue on OSP7 I also have a setup ready.

Thanks,
s.
Comment 8 VIKRANT 2016-10-11 03:55:30 UTC 
Yes, I was able to reproduce it on OSP 7 setup.
Comment 13 Sahid Ferdjaoui 2016-10-19 11:11:49 UTC 
So it seems that the issue is not related to Nova for two reasons:

 - I tried to connect to a live-migrated guest stuck by using virt-manager (directly connected to the host through SSH) same problem: I'm able to see the screen but not able to interact with it.

 - I compared the libvirt XML of a live-migrated guest stuck and a sane guest running and there are no difference, the domains have the same devices defined after a live migration.

I would say the problem is in the image or at least we should to transfer that issue to the team responsible to support Windows OS on KVM, they probably have more information about why the guest get stuck after live-migration.
Comment 16 Sahid Ferdjaoui 2016-10-19 13:41:23 UTC 
I discussed with vladik and we think that you can probably move that issue to the KVM team, they have experts who can probably handle it better than us.
Comment 28 Sahid Ferdjaoui 2016-10-21 12:34:21 UTC 
Hello,

David is trying to do some tests but the setup seems to have too many VMs running, can you try to clean it and ping David to let him know he can use it.

(his nick on IRC is dgilbert)

Thanks,
s.
Comment 29 VIKRANT 2016-10-24 04:48:32 UTC 
Sahid,

Test setup is having some issue. 

I am able to create the cinder volume from newly uploaded windows image but while trying to spin up the instance it's getting failed with below error. It's happening with all images.

~~~
2016-10-24 09:48:34.246 2773417 ERROR nova.virt.libvirt.driver [req-f3d2fd61-5df0-4f29-a826-6714a1e3e61b 67dea3e02e344a978e17addd264e4eb
5 06f93042e8e64a9fbe0517d08c12e8af - - -] Error launching a defined domain with XML: <domain type='kvm'>
  <name>instance-00000081</name>
  <uuid>e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="2015.1.4-13.el7ost"/>
      <nova:name>vikrant</nova:name>
      <nova:creationTime>2016-10-24 04:18:31</nova:creationTime>
      <nova:flavor name="m1.medium">
        <nova:memory>4096</nova:memory>
        <nova:disk>40</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>2</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="67dea3e02e344a978e17addd264e4eb5">admin</nova:user>
        <nova:project uuid="06f93042e8e64a9fbe0517d08c12e8af">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="d1e998c1-5129-4eba-9626-0e471f91e95b"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <shares>2048</shares>
  </cputune>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Red Hat</entry>
      <entry name='product'>OpenStack Compute</entry>
      <entry name='version'>2015.1.4-13.el7ost</entry>
      <entry name='serial'>f37fc981-aa7c-4ef0-afa9-afdba8cb1469</entry>
      <entry name='uuid'>e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='2' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='e243141d-88a0-448d-9dec-3289b347d34c'/>
      </auth>
      <source protocol='rbd' name='vms/e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1_disk'>
        <host name='10.74.128.30' port='6789'/>
        <host name='10.74.128.31' port='6789'/>
        <host name='10.74.128.32' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='bridge'>
      <mac address='fa:16:3e:ee:94:11'/>
      <source bridge='qbra9796782-fe'/>
      <target dev='tapa9796782-fe'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1/console.log'/>
      <target port='0'/>
    </serial>
    <serial type='pty'>
      <target port='1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1/console.log'/>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

2016-10-24 09:48:34.247 2773417 ERROR nova.compute.manager [req-f3d2fd61-5df0-4f29-a826-6714a1e3e61b 67dea3e02e344a978e17addd264e4eb5 06f93042e8e64a9fbe0517d08c12e8af - - -] [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] Instance failed to spawn
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] Traceback (most recent call last):
last):
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2510, in _build_resources
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     yield resources
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2382, in _build_and_run_instance
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     block_device_info=block_device_info)
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2592, in spawn
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     block_device_info=block_device_info)
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4674, in _create_domain_and_network
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     power_on=power_on)
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4605, in _create_domain
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     LOG.error(err)
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     six.reraise(self.type_, self.value, self.tb)
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4595, in _create_domain
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     domain.createWithFlags(launch_flags)
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     result = proxy_call(self
._autowrap, f, *args, **kwargs)
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     rv = execute(f, *args, **kwargs)
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     six.reraise(c, e, tb)
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     rv = meth(*args, **kwargs)
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] libvirtError: internal error: process exited while connecting to monitor: 
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] (process:2839667): GLib-WARNING **: gmem.c:482: custom memory allocation vtable not supported
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] 2016-10-24T04:18:34.105911Z qemu-kvm: CPU feature arat not found
2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] 
2016-10-24 09:48:34.423 2773417 INFO nova.compute.manager [req-f3d2fd61-5df0-4f29-a826-6714a1e3e61b 67dea3e02e344a978e17addd264e4eb5 06f93042e8e64a9fbe0517d08c12e8af - - -] [instance: e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] Terminating instance
~~~

any Idea what is leading to this issue ?
Comment 32 Dr. David Alan Gilbert 2016-10-24 10:17:21 UTC 
(In reply to VIKRANT from comment #29)
> Sahid,
> 
> Test setup is having some issue. 
> 
> I am able to create the cinder volume from newly uploaded windows image but
> while trying to spin up the instance it's getting failed with below error.
> It's happening with all images.
> 
> ~~~
> 2016-10-24 09:48:34.246 2773417 ERROR nova.virt.libvirt.driver
> [req-f3d2fd61-5df0-4f29-a826-6714a1e3e61b 67dea3e02e344a978e17addd264e4eb
> 5 06f93042e8e64a9fbe0517d08c12e8af - - -] Error launching a defined domain
> with XML: <domain type='kvm'>
>   <name>instance-00000081</name>
>   <uuid>e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1</uuid>
>   <metadata>
>     <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
>       <nova:package version="2015.1.4-13.el7ost"/>
>       <nova:name>vikrant</nova:name>
>       <nova:creationTime>2016-10-24 04:18:31</nova:creationTime>
>       <nova:flavor name="m1.medium">
>         <nova:memory>4096</nova:memory>
>         <nova:disk>40</nova:disk>
>         <nova:swap>0</nova:swap>
>         <nova:ephemeral>0</nova:ephemeral>
>         <nova:vcpus>2</nova:vcpus>
>       </nova:flavor>
>       <nova:owner>
>         <nova:user uuid="67dea3e02e344a978e17addd264e4eb5">admin</nova:user>
>         <nova:project
> uuid="06f93042e8e64a9fbe0517d08c12e8af">admin</nova:project>
>       </nova:owner>
>       <nova:root type="image" uuid="d1e998c1-5129-4eba-9626-0e471f91e95b"/>
>     </nova:instance>
>   </metadata>
>   <memory unit='KiB'>4194304</memory>
>   <currentMemory unit='KiB'>4194304</currentMemory>
>   <vcpu placement='static'>2</vcpu>
>   <cputune>
>     <shares>2048</shares>
>   </cputune>
>   <sysinfo type='smbios'>
>     <system>
>       <entry name='manufacturer'>Red Hat</entry>
>       <entry name='product'>OpenStack Compute</entry>
>       <entry name='version'>2015.1.4-13.el7ost</entry>
>       <entry name='serial'>f37fc981-aa7c-4ef0-afa9-afdba8cb1469</entry>
>       <entry name='uuid'>e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1</entry>
>     </system>
>   </sysinfo>
>   <os>
>     <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
>     <boot dev='hd'/>
>     <smbios mode='sysinfo'/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>   </features>
>   <cpu mode='host-model'>
>     <model fallback='allow'/>
>     <topology sockets='2' cores='1' threads='1'/>
>   </cpu>
>   <clock offset='utc'>
>     <timer name='pit' tickpolicy='delay'/>
>     <timer name='rtc' tickpolicy='catchup'/>
>     <timer name='hpet' present='no'/>
>   </clock>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>destroy</on_crash>
>   <devices>
>     <emulator>/usr/libexec/qemu-kvm</emulator>
>     <disk type='network' device='disk'>
>       <driver name='qemu' type='raw' cache='writeback'/>
>       <auth username='cinder'>
>         <secret type='ceph' uuid='e243141d-88a0-448d-9dec-3289b347d34c'/>
>       </auth>
>       <source protocol='rbd'
> name='vms/e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1_disk'>
>         <host name='10.74.128.30' port='6789'/>
>         <host name='10.74.128.31' port='6789'/>
>         <host name='10.74.128.32' port='6789'/>
>       </source>
>       <target dev='vda' bus='virtio'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
>     </disk>
>     <controller type='usb' index='0'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x2'/>
>     </controller>
>     <controller type='pci' index='0' model='pci-root'/>
>     <interface type='bridge'>
>       <mac address='fa:16:3e:ee:94:11'/>
>       <source bridge='qbra9796782-fe'/>
>       <target dev='tapa9796782-fe'/>
>       <model type='virtio'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </interface>
>     <serial type='file'>
>       <source
> path='/var/lib/nova/instances/e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1/console.
> log'/>
>       <target port='0'/>
>     </serial>
>     <serial type='pty'>
>       <target port='1'/>
>     </serial>
>     <console type='file'>
>       <source
> path='/var/lib/nova/instances/e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1/console.
> log'/>
>       <target type='serial' port='0'/>
>     </console>
>     <input type='tablet' bus='usb'>
>       <address type='usb' bus='0' port='1'/>
>     </input>
>     <input type='mouse' bus='ps2'/>
>     <input type='keyboard' bus='ps2'/>
>     <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'
> keymap='en-us'>
>       <listen type='address' address='0.0.0.0'/>
>     </graphics>
>     <video>
>       <model type='cirrus' vram='16384' heads='1' primary='yes'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
>     </video>
>     <memballoon model='virtio'>
>       <stats period='10'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0'/>
>     </memballoon>
>   </devices>
> </domain>
> 
> 2016-10-24 09:48:34.247 2773417 ERROR nova.compute.manager
> [req-f3d2fd61-5df0-4f29-a826-6714a1e3e61b 67dea3e02e344a978e17addd264e4eb5
> 06f93042e8e64a9fbe0517d08c12e8af - - -] [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] Instance failed to spawn
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] Traceback (most recent call last):
> last):
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2510, in
> _build_resources
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     yield resources
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File
> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2382, in
> _build_and_run_instance
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]    
> block_device_info=block_device_info)
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2592,
> in spawn
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]    
> block_device_info=block_device_info)
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4674,
> in _create_domain_and_network
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     power_on=power_on)
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4605,
> in _create_domain
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     LOG.error(err)
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in
> __exit__
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     six.reraise(self.type_,
> self.value, self.tb)
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4595,
> in _create_domain
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]    
> domain.createWithFlags(launch_flags)
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File
> "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     result = proxy_call(self
> ._autowrap, f, *args, **kwargs)
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File
> "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     rv = execute(f, *args, **kwargs)
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File
> "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     six.reraise(c, e, tb)
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File
> "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     rv = meth(*args, **kwargs)
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]   File
> "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in
> createWithFlags
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1]     if ret == -1: raise libvirtError
> ('virDomainCreateWithFlags() failed', dom=self)
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] libvirtError: internal error: process
> exited while connecting to monitor: 
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] (process:2839667): GLib-WARNING **:
> gmem.c:482: custom memory allocation vtable not supported
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] 2016-10-24T04:18:34.105911Z qemu-kvm:
> CPU feature arat not found
> 2016-10-24 09:48:34.247 2773417 TRACE nova.compute.manager [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] 
> 2016-10-24 09:48:34.423 2773417 INFO nova.compute.manager
> [req-f3d2fd61-5df0-4f29-a826-6714a1e3e61b 67dea3e02e344a978e17addd264e4eb5
> 06f93042e8e64a9fbe0517d08c12e8af - - -] [instance:
> e8eac58c-19b4-4d87-8ab8-0c8a214a6ee1] Terminating instance
> ~~~
> 
> any Idea what is leading to this issue ?

Vikrant:
  I think that problem is bz 1371617 (I've added some detail to that bz);  if you configure the CPU model in the nova.conf I think it should go away.
Comment 59 Dr. David Alan Gilbert 2017-01-09 12:42:02 UTC 
see comment 40/41 - this is fixed in 7.3