Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1441368

Summary:

Openstack HA Cluster- Sequential Evacuation of compute nodes leaves Virtual machines on nodes in Error state

Product:

Red Hat OpenStack

Reporter:

Kumar Abhinav <kumar.abhinav>

Component:

openstack-nova

Assignee:

Sylvain Bauza <sbauza>

Status:

CLOSED DUPLICATE

QA Contact:

Prasanth Anbalagan <panbalag>

Severity:

high

Docs Contact:

Priority:

high

Version:

10.0 (Newton)

CC:

arkady_kanevsky, berrange, cdevine, christopher_dearborn, dasmith, dcain, eglynn, John_walsh, kchamart, kumar.abhinav, kurt_hey, mbooth, mburns, morazi, rajini.karthik, randy_perryman, sbauza, sferdjao, sgordon, smerrow, sreichar, srevivo, vromanso

Target Milestone:

async

Keywords:

Triaged, ZStream

Target Release:

10.0 (Newton)

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2017-10-25 15:58:21 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1335596, 1356451

Attachments:

Description	Flags
SOS report compute -1	none
SOS report compute -2	none
SOS report compute-0	none
compute-0 SOS report	none
compute-1 SOS report	none
compute-2 SOS report	none
controller--0 Part 1	none
controller-0 Part 2	none
Controller-1 Part 1	none
Controller-1 Part 2	none
controller--2 Part 1	none
controller--2 Part 2	none

Description Kumar Abhinav 2017-04-11 19:33:31 UTC

Description of problem:





How reproducible:
pre req
Openstack cluster with HA enabled.
atleast 4 vms running on 2 different compute nodes on the cluster. 
watch sudo virsh list on each compute nodes should be running in 3 seperate windows. 
nova list --fields name,host,instance_name should show the vms and the host they are running on.

Steps to Reproduce:

1. Evacuate VMS from compute 1 using the following command (or as root)
sudo echo c > /proc/sysrq-trigger
2. run nova list --fields name,host,instance_name to ensure the vms are evacuated to compute-0,compute-2. virsh list should also  show the vms have moved.
3.Now log on to compute-2 or compute-0 and run the following command
sudo echo c > /proc/sysrq-trigger (or as root).
4. nova list --fields name,host,instance_name will still show the old output. virsh list on compute-1 and compute-0 will not show all vms' have moved over. virsh list on compute-2 will show no vms running either. This is the case where I evacuated vms from compute-2

Actual results:
nova list output is incorrect. VM's are lost during migration.

Expected results:
All vms should have been evacuated from compute-2 to either compute-0 or compute-1. nova list command should show correct output after migration is complete


Additional info: SOS report from  nodes is attached

Comment 1 Kumar Abhinav 2017-04-11 19:34:42 UTC

Created attachment 1270930 [details]
SOS report compute -1

Comment 2 Kumar Abhinav 2017-04-11 19:35:21 UTC

Created attachment 1270931 [details]
SOS report compute -2

Comment 3 Kumar Abhinav 2017-04-11 19:35:56 UTC

Created attachment 1270933 [details]
SOS report compute-0

Comment 4 Sylvain Bauza 2017-04-21 09:54:53 UTC

I don't have a lot of context with the attachments since they're not including the controller logs so I can't match the request IDs and see which instances are been migrated and when.

That said, trying to look over the possible log problems I could see, I noticed a stacktrace that the root cause is known by me :

2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [req-d6b1e183-4b54-416e-b2c6-2cb5f367eb13 1827d1c9f1204ec58854b6c37696914d db18d0b6324e4b0492b453ae907fea55 - - -] [instance: f69e7961-b98b-4028-a66c-7f99e5113640] Setting instance vm_state to ERROR
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640] Traceback (most recent call last):
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6571, in _error_out_instance_on_exception
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]     yield
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2690, in rebuild_instance
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]     bdms, recreate, on_shared_storage, preserve_ephemeral)
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2734, in _do_rebuild_instance_with_claim
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]     self._do_rebuild_instance(*args, **kwargs)
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2849, in _do_rebuild_instance
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]     self._rebuild_default_impl(**kwargs)
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2612, in _rebuild_default_impl
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]     block_device_info=new_block_device_info)
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2607, in spawn
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]     self._ensure_console_log_for_instance(instance)
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2869, in _ensure_console_log_for_instance
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]     libvirt_utils.file_open(console_file, 'a').close()
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 323, in file_open
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640]     return open(*args, **kwargs)
2017-04-10 23:05:02.866 5725 ERROR nova.compute.manager [instance: f69e7961-b98b-4028-a66c-7f99e5113640] IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/f69e7961-b98b-4028-a66c-7f99e5113640/console.log'

Could you please confirm that when you're trying to evacuate instances back to hosts that were previously having instances, you could see the instances in ERROR state ?

What I'd ideally like is you trying to reproduce the issue that way :
1/ On a cloud with 3 compute nodes (and 3 controllers), do a single nova boot. Then, notice the host where the instance is running, and kill the host so the instance evacuates. 
2/ Do a nova show on that instance before the evacuation and after
3/ Kill the host that hosted the evacuated instance and do a nova show of the instance before and after the second evacuation.
4/ Provide the controllers and compute logs for that scenario and let me know the instance UUID and where the instance was located during that scenario.

FWIW, we have a current known issue with OSP10 where permission issues prevent us to evacuate an instance back to its orginal source host but the fix recently merged upstream in the stable/newton stable branch and will be released soon in a nova stable release, so OSP10 could benefit from it in the next OSP10 rebase.
The change in question is https://review.openstack.org/#/c/454593/

Comment 5 Kumar Abhinav 2017-05-09 16:13:30 UTC

Could you please confirm that when you're trying to evacuate instances back to hosts that were previously having instances, you could see the instances in ERROR state ?

yes, that was the case I was testing.

Comment 6 Kumar Abhinav 2017-06-30 19:00:51 UTC

Attaching logs for controller 0-2, compute 0-2

Here is what I did to repro this problem

1. Create VM on compute-2
2. Force evacuation from compute-2
3. VM evacuate's to compute-1
4. Force evacuation from compute-1
5. VM from compute-2 goes into error state

VM id used in this experiment- ee8f834c-3bad-4c89-84d4-717c3682e83a

This bug is consistently reproducible.

Comment 7 Kumar Abhinav 2017-06-30 19:02:28 UTC

Created attachment 1293322 [details]
compute-0 SOS report

Comment 8 Kumar Abhinav 2017-06-30 19:03:31 UTC

Created attachment 1293323 [details]
compute-1 SOS report

Comment 9 Kumar Abhinav 2017-06-30 19:04:12 UTC

Created attachment 1293324 [details]
compute-2 SOS report

Comment 10 Kumar Abhinav 2017-06-30 19:05:11 UTC

Created attachment 1293325 [details]
controller--0 Part 1

Comment 11 Kumar Abhinav 2017-06-30 19:08:40 UTC

Created attachment 1293326 [details]
controller-0 Part 2

Comment 12 Kumar Abhinav 2017-06-30 19:10:37 UTC

Created attachment 1293327 [details]
Controller-1 Part 1

Comment 13 Kumar Abhinav 2017-06-30 19:11:20 UTC

Created attachment 1293328 [details]
Controller-1 Part 2

Comment 14 Kumar Abhinav 2017-06-30 19:12:06 UTC

Created attachment 1293329 [details]
controller--2 Part 1

Comment 15 Kumar Abhinav 2017-06-30 19:13:07 UTC

Created attachment 1293330 [details]
controller--2 Part 2

Comment 16 Sean Merrow 2017-07-06 15:57:00 UTC

Hi Sylvain, let us know if the data uploaded is helpful or if there is anything else needed as part of the investigation.

Thanks,
Sean

Comment 17 Sylvain Bauza 2017-07-07 08:29:02 UTC

Sorry, seems like I didn't get the replies because I wasn't the assigned folk. Will respond to your questions really soon.

Comment 18 Sylvain Bauza 2017-07-07 11:04:47 UTC

First, thanks for providing me all the sosreports and reproducing the problem given my comment, it really helped me.

When looking at compute-2 log, you can see a exception about permission issues like I said in my comment :

2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [req-4e49e1e0-c4af-4620-8dbe-77ec1e6a7a27 16b58768a0f74d2383442d0748c4c46d 444691671cf64048b55556038f9ccb87 - - -] [instance: ee8f834c-3bad-4c89-84d4-717c3
682e83a] Setting instance vm_state to ERROR
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a] Traceback (most recent call last):
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6598, in _error_out_instance_on_ex
ception
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]     yield
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2695, in rebuild_instance
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]     bdms, recreate, on_shared_storage, preserve_ephemeral)
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2739, in _do_rebuild_instance_with
_claim
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]     self._do_rebuild_instance(*args, **kwargs)
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2854, in _do_rebuild_instance
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]     self._rebuild_default_impl(**kwargs)
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2617, in _rebuild_default_impl
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]     block_device_info=new_block_device_info)
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2609, in spawn
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]     self._ensure_console_log_for_instance(instance)
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2873, in _ensure_console_log_f
or_instance
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]     libvirt_utils.file_open(console_file, 'a').close()
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 323, in file_open
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a]     return open(*args, **kwargs)
2017-06-30 15:10:11.372 4986 ERROR nova.compute.manager [instance: ee8f834c-3bad-4c89-84d4-717c3682e83a] IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/ee8f834c-3bad-4c89-84d4-717c3682e83a/conso
le.log'


That problem was fixed in master but also backported to stable/newton in https://review.openstack.org/#/c/454593/1 so nova-14.0.6 was supporting it.

Red Hat rebased our OSP10 services by this Nova point release with the package openstack-nova-14.0.7-6.el7ost

Could you please try to use the above nova package for your compute services and see if it fixes your problem ? Thanks.

Comment 19 Kumar Abhinav 2017-07-10 16:04:59 UTC

Sylvain
As we dont install individual packages but rather install the full OSP solution, in order to verify this bug I will have to wait till this package is released in z- stream. I can then verify the bug fix.

Comment 24 Sylvain Bauza 2017-10-25 15:58:21 UTC

This is correct, that one is a duplicate.

*** This bug has been marked as a duplicate of bug 1386420 ***