Bug 1562747

Summary: Nova reports overcloud instance in error state after failed instance-ha evacuation. This does NOT happen while doing the same procedure (see steps to reproduce) with 'evacuable' tagged images (using: openstack server --tag evacuable)
Product: Red Hat OpenStack Reporter: pkomarov
Component: openstack-tripleo-heat-templatesAssignee: Andrew Beekhof <abeekhof>
Status: CLOSED DUPLICATE QA Contact: Gurenko Alex <agurenko>
Severity: high Docs Contact:
Priority: high    
Version: 12.0 (Pike)CC: abeekhof, berrange, chjones, dasmith, eglynn, jhakimra, kchamart, lyarwood, mburns, pkomarov, sbauza, sferdjao, sgordon, srevivo, vromanso
Target Milestone: asyncKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1567606 (view as bug list) Environment:
Last Closed: 2018-05-08 08:36:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1264181, 1567606, 1570429, 1582827, 1649253    
Attachments:
Description Flags
sos reports from compute-0
none
sos reports from compute-1 none

Description pkomarov 2018-04-02 11:09:22 UTC 
Created attachment 1416224 [details]
sos reports from compute-0

Description of problem:
Nova reports overcloud instance in error state after failed instance-ha evacuation
This does NOT happen while doing the same procedure (see steps to reproduce) with 'evacuable' tagged images (using: openstack server --tag evacuable)

Environment :
OSP12 on virt setup, with instance-ha enabled,no shared storage



How reproducible:
Always

Steps to Reproduce:
1) deploy OSP12 with HA env (3 controllers + 2 computes)
2) deploy instance-ha with non_shared_storage option using the following procedure :
- https://github.com/redhat-openstack/tripleo-quickstart-utils
- https://github.com/redhat-openstack/tripleo-quickstart-utils/tree/master/roles/instance-ha
3) boot 4 instances with fip's (no volumes attached as we are using non_shared_storage instance-ha setup)
4) failover the compute hypervisors (echo b >/proc/sysrq-trigger), each two times, with each failover ,wait for evacuation of the instances.
5) after two failovers instances go into ERROR state.

Expected result : 
after two failovers instances go into AcTIVE state on a different hypevisor.

Attachments: the sos-reports from the computes.


Additional info:


The logs below are collected from NOVA right after the ERROR, the lines are sorted chronologically:  


(overcloud) [stack@undercloud ~]$ ./check_servers f604d3d9-7670-46aa-926c-c9f78f50e79a|sort -k2

2018-04-02 08:00:54.860 1 INFO nova.compute.claims [req-50752304-45cd-4b09-bc50-8ced3698f69a 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Attempting claim on node overcloud-compute-1.localdomain: memory 64 MB, disk 1 GB, vcpus 1 CPU 
2018-04-02 08:00:54.861 1 INFO nova.compute.claims [req-50752304-45cd-4b09-bc50-8ced3698f69a 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Total memory: 12287 MB, used: 4096.00 MB 
2018-04-02 08:00:54.862 1 INFO nova.compute.claims [req-50752304-45cd-4b09-bc50-8ced3698f69a 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] memory limit not specified, defaulting to unlimited 
2018-04-02 08:00:54.862 1 INFO nova.compute.claims [req-50752304-45cd-4b09-bc50-8ced3698f69a 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Total disk: 49 GB, used: 0.00 GB 
2018-04-02 08:00:54.863 1 INFO nova.compute.claims [req-50752304-45cd-4b09-bc50-8ced3698f69a 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] disk limit not specified, defaulting to unlimited 
2018-04-02 08:00:54.864 1 INFO nova.compute.claims [req-50752304-45cd-4b09-bc50-8ced3698f69a 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Total vcpu: 8 VCPU, used: 0.00 VCPU 
2018-04-02 08:00:54.864 1 INFO nova.compute.claims [req-50752304-45cd-4b09-bc50-8ced3698f69a 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] vcpu limit not specified, defaulting to unlimited 
2018-04-02 08:00:54.865 1 INFO nova.compute.claims [req-50752304-45cd-4b09-bc50-8ced3698f69a 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Claim successful on node overcloud-compute-1.localdomain 
2018-04-02 08:00:56.711 1 INFO nova.virt.libvirt.driver [req-50752304-45cd-4b09-bc50-8ced3698f69a 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Creating image 
2018-04-02 08:01:14.227 1 INFO nova.compute.manager [-] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] VM Started (Lifecycle Event) 
2018-04-02 08:01:14.295 1 INFO nova.compute.manager [req-f7927ea3-560b-4209-b288-2c39288bba7d - - - - -] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] VM Paused (Lifecycle Event) 
2018-04-02 08:01:14.447 1 INFO nova.compute.manager [req-f7927ea3-560b-4209-b288-2c39288bba7d - - - - -] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] During sync_power_state the instance has a pending task (spawning). Skip. 
2018-04-02 08:01:16.637 1 INFO nova.compute.manager [req-f7927ea3-560b-4209-b288-2c39288bba7d - - - - -] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] VM Resumed (Lifecycle Event) 
2018-04-02 08:01:16.643 1 INFO nova.compute.manager [req-50752304-45cd-4b09-bc50-8ced3698f69a 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Took 19.93 seconds to spawn the instance on the hypervisor. 
2018-04-02 08:01:16.643 1 INFO nova.virt.libvirt.driver [-] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Instance spawned successfully. 
2018-04-02 08:01:16.826 1 INFO nova.compute.manager [req-f7927ea3-560b-4209-b288-2c39288bba7d - - - - -] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] VM Resumed (Lifecycle Event) 
2018-04-02 08:01:16.899 1 INFO nova.compute.manager [req-50752304-45cd-4b09-bc50-8ced3698f69a 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Took 22.10 seconds to build instance. 
2018-04-02 08:17:50.375 1 INFO nova.compute.manager [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Rebuilding instance 
2018-04-02 08:17:51.584 1 INFO nova.compute.claims [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Attempting claim on node overcloud-compute-0.localdomain: memory 64 MB, disk 1 GB, vcpus 1 CPU 
2018-04-02 08:17:51.585 1 INFO nova.compute.claims [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Total memory: 12287 MB, used: 4224.00 MB 
2018-04-02 08:17:51.586 1 INFO nova.compute.claims [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] disk limit not specified, defaulting to unlimited 
2018-04-02 08:17:51.586 1 INFO nova.compute.claims [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] memory limit not specified, defaulting to unlimited 
2018-04-02 08:17:51.586 1 INFO nova.compute.claims [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Total disk: 49 GB, used: 2.00 GB 
2018-04-02 08:17:51.587 1 INFO nova.compute.claims [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Total vcpu: 8 VCPU, used: 2.00 VCPU 
2018-04-02 08:17:51.587 1 INFO nova.compute.claims [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] vcpu limit not specified, defaulting to unlimited 
2018-04-02 08:17:51.588 1 INFO nova.compute.claims [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Claim successful on node overcloud-compute-0.localdomain 
2018-04-02 08:17:51.811 1 INFO nova.compute.resource_tracker [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] Updating from migration f604d3d9-7670-46aa-926c-c9f78f50e79a 
2018-04-02 08:17:57.155 1 WARNING nova.compute.manager [req-dd538a92-b426-44f2-a7b5-96e63ce5c1ca b3b5b0100e684ed1a41375b66e0b4c38 d70ba71eb42748b49bd1df205d37d4e4 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Received unexpected event network-vif-unplugged-341d9cc2-21d6-4419-8630-eb44875cc6c4 for instance 
2018-04-02 08:17:58.013 1 INFO nova.network.neutronv2.api [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Updating port 341d9cc2-21d6-4419-8630-eb44875cc6c4 with attributes {'binding:profile': {}, 'binding:host_id': 'overcloud-compute-0.localdomain'} 
2018-04-02 08:18:00.123 1 WARNING nova.compute.manager [req-79a813d7-0cb9-4208-ae0b-e167deaad7a2 b3b5b0100e684ed1a41375b66e0b4c38 d70ba71eb42748b49bd1df205d37d4e4 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Received unexpected event network-vif-plugged-341d9cc2-21d6-4419-8630-eb44875cc6c4 for instance 
2018-04-02 08:18:01.742 1 WARNING nova.compute.manager [req-4b4b548c-2903-4c5f-8340-30ddf2870c4a b3b5b0100e684ed1a41375b66e0b4c38 d70ba71eb42748b49bd1df205d37d4e4 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Received unexpected event network-vif-unplugged-341d9cc2-21d6-4419-8630-eb44875cc6c4 for instance 
2018-04-02 08:18:04.596 1 INFO nova.virt.libvirt.driver [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Creating image 
2018-04-02 08:18:19.746 1 WARNING nova.compute.resource_tracker [req-8651f0f6-1c64-4eaf-b8cb-634d7e4bb060 - - - - -] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Instance not resizing, skipping migration. 
2018-04-02 08:18:27.926 1 WARNING nova.compute.resource_tracker [req-aec14588-6508-4542-adcb-167b9c344c1b - - - - -] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Instance not resizing, skipping migration. 
2018-04-02 08:18:28.457 1 WARNING nova.compute.manager [req-dd538a92-b426-44f2-a7b5-96e63ce5c1ca b3b5b0100e684ed1a41375b66e0b4c38 d70ba71eb42748b49bd1df205d37d4e4 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Received unexpected event network-vif-unplugged-341d9cc2-21d6-4419-8630-eb44875cc6c4 for instance 
2018-04-02 08:18:28.485 1 WARNING nova.compute.manager [req-79a813d7-0cb9-4208-ae0b-e167deaad7a2 b3b5b0100e684ed1a41375b66e0b4c38 d70ba71eb42748b49bd1df205d37d4e4 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Received unexpected event network-vif-plugged-341d9cc2-21d6-4419-8630-eb44875cc6c4 for instance 
2018-04-02 08:18:28.517 1 WARNING nova.compute.manager [req-4b4b548c-2903-4c5f-8340-30ddf2870c4a b3b5b0100e684ed1a41375b66e0b4c38 d70ba71eb42748b49bd1df205d37d4e4 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Received unexpected event network-vif-unplugged-341d9cc2-21d6-4419-8630-eb44875cc6c4 for instance 
2018-04-02 08:18:34.099 1 INFO nova.compute.manager [req-d0cf908d-e3ed-4ef9-980f-3bfe948b2ddf - - - - -] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] VM Resumed (Lifecycle Event) 
2018-04-02 08:18:34.154 1 INFO nova.virt.libvirt.driver [-] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Instance spawned successfully. 
2018-04-02 08:18:34.338 1 INFO nova.compute.manager [req-d0cf908d-e3ed-4ef9-980f-3bfe948b2ddf - - - - -] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] During the sync_power process the instance has moved from host overcloud-compute-1.localdomain to host overcloud-compute-0.localdomain 
2018-04-02 08:18:34.338 1 INFO nova.compute.manager [req-d0cf908d-e3ed-4ef9-980f-3bfe948b2ddf - - - - -] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] VM Started (Lifecycle Event) 
2018-04-02 08:18:34.577 1 INFO nova.compute.manager [req-d0cf908d-e3ed-4ef9-980f-3bfe948b2ddf - - - - -] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] During the sync_power process the instance has moved from host overcloud-compute-1.localdomain to host overcloud-compute-0.localdomain 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     bdms, recreate, on_shared_storage, preserve_ephemeral) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     columns_to_join=_expected_cols(expected_attrs)) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     context, instance_uuid, values, expected, original=instance_ref)) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     ctxt, self, fn.__name__, args, kwargs) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     ectxt.value = e.inner_exc 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     expected=expected) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     instance.save(expected_task_state=[task_states.REBUILD_SPAWNING]) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     objmethod=objmethod, args=args, kwargs=kwargs) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     raise exc(**exc_props) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     raise result 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     retry=retry) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     retry=self.retry) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     return f(*args, **kwargs) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     return f(*args, **kwargs) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     return f(context, *args, **kwargs) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     return fn(self, *args, **kwargs) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     return getattr(target, method)(*args, **kwargs) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     self._do_rebuild_instance(*args, **kwargs) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     self.force_reraise() 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     six.reraise(self.type_, self.value, self.tb) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     timeout=timeout, retry=retry) 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     yield 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2823, in rebuild_instance 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2871, in _do_rebuild_instance_with_claim 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2995, in _do_rebuild_instance 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6863, in _error_out_instance_on_exception 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 124, in _object_dispatch 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 245, in object_action 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 859, in instance_update_and_get_original 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 250, in wrapped 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2666, in instance_update_and_get_original 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2802, in _instance_update 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 785, in save 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 210, in wrapper 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Traceback (most recent call last): 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Traceback (most recent call last): 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] UnexpectedTaskStateError: Conflict updating instance f604d3d9-7670-46aa-926c-c9f78f50e79a. Expected: {'task_state': [u'rebuild_spawning']}. Actual: {'task_state': None} 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] UnexpectedTaskStateError_Remote: Conflict updating instance f604d3d9-7670-46aa-926c-c9f78f50e79a. Expected: {'task_state': [u'rebuild_spawning']}. Actual: {'task_state': None} 
2018-04-02 08:18:35.086 1 ERROR nova.compute.manager [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Setting instance vm_state to ERROR: UnexpectedTaskStateError_Remote: Conflict updating instance f604d3d9-7670-46aa-926c-c9f78f50e79a. Expected: {'task_state': [u'rebuild_spawning']}. Actual: {'task_state': None} 
2018-04-02 08:18:35.321 1 WARNING nova.compute.manager [req-f57bbfd6-152d-4486-9403-7d4b9cbe2227 b3b5b0100e684ed1a41375b66e0b4c38 d70ba71eb42748b49bd1df205d37d4e4 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Received unexpected event network-vif-plugged-341d9cc2-21d6-4419-8630-eb44875cc6c4 for instance 
2018-04-02 08:18:35.402 1 WARNING nova.compute.manager [req-f57bbfd6-152d-4486-9403-7d4b9cbe2227 b3b5b0100e684ed1a41375b66e0b4c38 d70ba71eb42748b49bd1df205d37d4e4 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Received unexpected event network-vif-plugged-341d9cc2-21d6-4419-8630-eb44875cc6c4 for instance 
2018-04-02 08:18:35.732 1 INFO nova.compute.manager [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] Task possibly preempted: Conflict updating instance f604d3d9-7670-46aa-926c-c9f78f50e79a. Expected: {'task_state': [u'rebuild_spawning']}. Actual: {'task_state': None} 
2018-04-02 08:18:35.791 1 ERROR oslo_messaging.rpc.server [req-28359149-ae5c-4072-a5de-cdf6b881cfb2 e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] Exception during message handling: UnexpectedTaskStateError_Remote: Conflict updating instance f604d3d9-7670-46aa-926c-c9f78f50e79a. Expected: {'task_state': [u'rebuild_spawning']}. Actual: {'task_state': None} 
2018-04-02 08:18:35.791 1 ERROR oslo_messaging.rpc.server UnexpectedTaskStateError: Conflict updating instance f604d3d9-7670-46aa-926c-c9f78f50e79a. Expected: {'task_state': [u'rebuild_spawning']}. Actual: {'task_state': None} 
2018-04-02 08:18:35.791 1 ERROR oslo_messaging.rpc.server UnexpectedTaskStateError_Remote: Conflict updating instance f604d3d9-7670-46aa-926c-c9f78f50e79a. Expected: {'task_state': [u'rebuild_spawning']}. Actual: {'task_state': None} 
2018-04-02 08:19:11.566 1 INFO nova.compute.manager [-] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. 
2018-04-02 08:41:06.663 1 INFO nova.compute.manager [req-996df209-5745-40cf-9e1b-bac395f28bac e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Rebuilding instance 
2018-04-02 08:41:09.256 1 INFO nova.compute.claims [req-996df209-5745-40cf-9e1b-bac395f28bac e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Attempting claim on node overcloud-compute-0.localdomain: memory 64 MB, disk 1 GB, vcpus 1 CPU 
2018-04-02 08:41:09.257 1 INFO nova.compute.claims [req-996df209-5745-40cf-9e1b-bac395f28bac e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] memory limit not specified, defaulting to unlimited 
2018-04-02 08:41:09.257 1 INFO nova.compute.claims [req-996df209-5745-40cf-9e1b-bac395f28bac e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Total memory: 12287 MB, used: 4160.00 MB 
2018-04-02 08:41:09.258 1 INFO nova.compute.claims [req-996df209-5745-40cf-9e1b-bac395f28bac e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Total disk: 49 GB, used: 1.00 GB 
2018-04-02 08:41:09.259 1 INFO nova.compute.claims [req-996df209-5745-40cf-9e1b-bac395f28bac e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] disk limit not specified, defaulting to unlimited 
2018-04-02 08:41:09.259 1 INFO nova.compute.claims [req-996df209-5745-40cf-9e1b-bac395f28bac e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Total vcpu: 8 VCPU, used: 1.00 VCPU 
2018-04-02 08:41:09.260 1 INFO nova.compute.claims [req-996df209-5745-40cf-9e1b-bac395f28bac e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] vcpu limit not specified, defaulting to unlimited 
2018-04-02 08:41:09.261 1 INFO nova.compute.claims [req-996df209-5745-40cf-9e1b-bac395f28bac e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Claim successful on node overcloud-compute-0.localdomain 
2018-04-02 08:41:09.682 1 INFO nova.compute.resource_tracker [req-996df209-5745-40cf-9e1b-bac395f28bac e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] Updating from migration f604d3d9-7670-46aa-926c-c9f78f50e79a 
2018-04-02 08:41:11.468 1 WARNING nova.compute.resource_tracker [req-481f29ac-b8fb-4535-8388-beb406e4ae30 - - - - -] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Instance not resizing, skipping migration. 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]  
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     bdms, recreate, on_shared_storage, preserve_ephemeral) 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     raise exception.InstanceExists(name=instance.name) 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     self._check_instance_exists(context, instance) 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     self._do_rebuild_instance(*args, **kwargs) 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]     yield 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1386, in _check_instance_exists 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2823, in rebuild_instance 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2871, in _do_rebuild_instance_with_claim 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2890, in _do_rebuild_instance 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6863, in _error_out_instance_on_exception 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] InstanceExists: Instance instance-00000567 already exists. 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Traceback (most recent call last): 
2018-04-02 08:41:11.914 1 ERROR nova.compute.manager [req-996df209-5745-40cf-9e1b-bac395f28bac e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Setting instance vm_state to ERROR: InstanceExists: Instance instance-00000567 already exists. 
2018-04-02 08:41:12.549 1 INFO nova.compute.manager [req-996df209-5745-40cf-9e1b-bac395f28bac e926c3dcb82440f49d78cd1e79ed710b b1dea719142a49f4bc91350870388851 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Successfully reverted task state from rebuilding on failure for instance. 
2018-04-02 08:51:31.571 1 INFO nova.compute.manager [req-42eb1a98-4e63-4f38-9fda-ebf0cc89b1ce 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Terminating instance 
2018-04-02 08:51:31.600 1 INFO nova.virt.libvirt.driver [-] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Instance destroyed successfully. 
2018-04-02 08:51:33.187 1 INFO nova.virt.libvirt.driver [req-42eb1a98-4e63-4f38-9fda-ebf0cc89b1ce 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Deleting instance files /var/lib/nova/instances/f604d3d9-7670-46aa-926c-c9f78f50e79a_del 
2018-04-02 08:51:33.189 1 INFO nova.virt.libvirt.driver [req-42eb1a98-4e63-4f38-9fda-ebf0cc89b1ce 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Deletion of /var/lib/nova/instances/f604d3d9-7670-46aa-926c-c9f78f50e79a_del complete 
2018-04-02 08:51:33.359 1 INFO nova.compute.manager [req-42eb1a98-4e63-4f38-9fda-ebf0cc89b1ce 2c7115d0b3524251a0574cf7128d27bd 03e5a1679c7b4864898667e84d55dba8 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Took 1.77 seconds to destroy the instance on the hypervisor. 
2018-04-02 08:51:36.072 1 INFO nova.compute.manager [req-241ed97c-a7d7-4056-9c23-f8d8d05bbd69 b3b5b0100e684ed1a41375b66e0b4c38 d70ba71eb42748b49bd1df205d37d4e4 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Neutron deleted interface 341d9cc2-21d6-4419-8630-eb44875cc6c4; detaching it from the instance and deleting it from the info cache 
2018-04-02 08:51:36.107 1 INFO nova.compute.manager [req-241ed97c-a7d7-4056-9c23-f8d8d05bbd69 b3b5b0100e684ed1a41375b66e0b4c38 d70ba71eb42748b49bd1df205d37d4e4 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Neutron deleted interface 341d9cc2-21d6-4419-8630-eb44875cc6c4; detaching it from the instance and deleting it from the info cache 
2018-04-02 08:51:36.151 1 WARNING nova.compute.manager [req-241ed97c-a7d7-4056-9c23-f8d8d05bbd69 b3b5b0100e684ed1a41375b66e0b4c38 d70ba71eb42748b49bd1df205d37d4e4 - default default] [instance: f604d3d9-7670-46aa-926c-c9f78f50e79a] Detach interface failed, port_id=341d9cc2-21d6-4419-8630-eb44875cc6c4, reason: Instance f604d3d9-7670-46aa-926c-c9f78f50e79a could not be found.: InstanceNotFound: Instance f604d3d9-7670-46aa-926c-c9f78f50e79a could not be found.
Comment 1 pkomarov 2018-04-02 11:11:09 UTC 
Created attachment 1416225 [details]
sos reports from compute-1
Comment 6 Andrew Beekhof 2018-04-09 11:28:20 UTC 
Can we get sosreports for the controller nodes too please?
I need to verify that fencing occurred.
Comment 8 Andrew Beekhof 2018-04-15 11:02:21 UTC 
Still need those controller node sos reports
Comment 9 Lee Yarwood 2018-05-08 08:36:40 UTC 

*** This bug has been marked as a duplicate of bug 1567606 ***