1279451 – Instance resize fails due to missing disk on destination compute

Bug 1279451 - Instance resize fails due to missing disk on destination compute

Summary: Instance resize fails due to missing disk on destination compute

Keywords:
Status:	CLOSED INSUFFICIENT_DATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	6.0 (Juno)
Hardware:	Unspecified
OS:	All
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	8.0 (Liberty)
Assignee:	Lee Yarwood
QA Contact:	nlevinki
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2015-11-09 13:35 UTC by Jaison Raju
Modified:	2019-09-12 09:15 UTC (History)
CC List:	13 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2015-11-26 11:27:33 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
Destination (2.59 MB, text/plain) 2015-11-09 13:39 UTC, Jaison Raju	no flags	Details
Host (3.23 MB, text/plain) 2015-11-09 13:40 UTC, Jaison Raju	no flags	Details
View All

Description Jaison Raju 2015-11-09 13:35:15 UTC

Description of problem:
Instance resize fails due to missing disk on destination compute .

grep image_service ../../*/*/etc/nova/nova.conf 
../../150-sosreport-compute1-aragaa.01530875-20151102123242.tar.xz/etc/nova/nova.conf:image_service=nova.image.glance.GlanceImageService
../../160-sosreport-compute2-aragaa.01530875-20151102123312.tar.xz/etc/nova/nova.conf:image_service=nova.image.glance.GlanceImageService


[root@os-compute-1 ~]# kill -SIGUSR1 $(pgrep nova-compute) ; sleep 5 ; journalctl -u openstack-nova-compute | grep my_ip
Nov 08 11:47:48 os-compute-1.sahaba.com nova-compute[8441]: my_ip = 127.0.0.1

[root@os-compute-2 ~]# kill -SIGUSR1 $(pgrep nova-compute) ; sleep 5 ; journalctl -u openstack-nova-compute | grep my_ip
Nov 08 11:46:30 os-compute-2.sahaba.com nova-compute[145832]: my_ip = 127.0.0.1

Version-Release number of selected component (if applicable):
SpinalStack deployment
OSP6

How reproducible:
Always at customer end

Steps to Reproduce:
1.
2.
3.

Actual results:
Re-size fails due to missing disk on destination node .

Expected results:
Re-size is successful

Additional info:
resize on same node works using 'allow_resize_to_same_host=true' 

compute1 (dest)
81823: DEBUG nova.openstack.common.lockutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Releasing semaphore "refresh_cache-a35c068d-b4d6-4bc2-bd3b-f535ae711b43" lock /usr/lib/pyt
hon2.7/site-packages/nova/openstack/common/lockutils.py:238
81823: DEBUG nova.virt.libvirt.driver [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43] Starting finish_migration finish_migration /usr/
lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6127
81823: DEBUG nova.virt.disk.api [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Checking if we can resize image /var/lib/nova/instances/a35c068d-b4d6-4bc2-bd3b-f535ae711b43/disk. size
=85899345920 can_resize_image /usr/lib/python2.7/site-packages/nova/virt/disk/api.py:192
81823: ERROR nova.compute.manager [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43] Setting instance vm_state to ERROR
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43] Traceback (most recent call last):
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3903, in finish_resize
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]     disk_info, image)
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3871, in _finish_resize
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]     old_instance_type, sys_meta)
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]     six.reraise(self.type_, self.value, self.tb)
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3866, in _finish_resize
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]     block_device_info, power_on)
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6134, in finish_migration
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]     self._disk_resize(info, size)
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6110, in _disk_resize
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]     disk.can_resize_image(pth, size) and
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]   File "/usr/lib/python2.7/site-packages/nova/virt/disk/api.py", line 195, in can_resize_image
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]     virt_size = get_disk_size(image)
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]   File "/usr/lib/python2.7/site-packages/nova/virt/disk/api.py", line 151, in get_disk_size
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]     return images.qemu_img_info(path).virtual_size
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]   File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 53, in qemu_img_info
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43]     raise exception.InvalidDiskInfo(reason=msg)
81823: TRACE nova.compute.manager [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43] InvalidDiskInfo: Disk info file is invalid: Path does not exist /var/lib/nova/instances/a35c068d-b4d6-4bc2-bd3b-f535ae711b43/dis
k

On compute2 (source)

140853: INFO nova.virt.libvirt.driver [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43] Instance failed to shutdown in 60 seconds.
140853: DEBUG nova.virt.driver [-] Emitting event <LifecycleEvent: 1446640102.07, a35c068d-b4d6-4bc2-bd3b-f535ae711b43 => Stopped> emit_event /usr/lib/python2.7/site-packages/nova/virt/driver.py:1298
140853: INFO nova.compute.manager [-] [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43] VM Stopped (Lifecycle Event)                    <<<<<<<<<<<<<<<<<<<<------ instance stopped !!!
140853: INFO nova.virt.libvirt.driver [-] [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43] Instance destroyed successfully.
140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Running cmd (subprocess): mv /var/lib/nova/instances/a35c068d-b4d6-4bc2-bd3b-f535ae711
b43 /var/lib/nova/instances/a35c068d-b4d6-4bc2-bd3b-f535ae711b43_resize execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161  <<<<<<--- Resize
140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Result was 0 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processuti
ls.py:195
140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/a35c068d-b4d6-4bc2-bd3b-f53
5ae711b43 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161   <<<<<<<=====mkdir ??

140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Running cmd (subprocess): qemu-img convert -f qcow2 -O qcow2 /var/lib/nova/instances/a35c068d-b4d6-4bc2-bd3b-f535ae711b43_resize/disk /var/lib/nova/instances/a35c068d-b4d6-4bc2-bd3b-f535ae711b43_resize/disk_rbase execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161
140853: DEBUG nova.compute.manager [-] [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43] Synchronizing instance power state after lifecycle event "Stopped"; current vm_state: active, current task_state: resize_migrating, current DB power_state: 1, VM power_state: 4 handle_lifecycle_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:1105
140853: INFO nova.compute.manager [-] [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43] During sync_power_state the instance has a pending task (resize_migrating). Skip.  <<<<<<======
140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Result was 0 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:195
140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Running cmd (subprocess): mv /var/lib/nova/instances/a35c068d-b4d6-4bc2-bd3b-f535ae711b43_resize/disk_rbase /var/lib/nova/instances/a35c068d-b4d6-4bc2-bd3b-f535ae711b43/disk execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161
140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Result was 0 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:195
140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf cat /etc/iscsi/initiatorname.iscsi execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161
140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Result was 0 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:195
140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf systool -c fc_host -v execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161       <<<<<<===== FAILED
140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Result was 1 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:195
140853: DEBUG nova.virt.libvirt.driver [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43] Could not determine fibre channel world wide node names get_volume_connector /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:1280
140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf systool -c fc_host -v execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:161       <<<<<<===== FAILED
140853: DEBUG nova.openstack.common.processutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Result was 1 execute /usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py:195
140853: DEBUG nova.virt.libvirt.driver [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] [instance: a35c068d-b4d6-4bc2-bd3b-f535ae711b43] Could not determine fibre channel world wide port names get_volume_connector /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:1287
140853: DEBUG nova.openstack.common.lockutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Created new semaphore "a35c068d-b4d6-4bc2-bd3b-f535ae711b43-events" internal_lock /usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py:206
140853: DEBUG nova.openstack.common.lockutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Acquired semaphore "a35c068d-b4d6-4bc2-bd3b-f535ae711b43-events" lock /usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py:229
140853: DEBUG nova.openstack.common.lockutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Got semaphore / lock "_clear_events" inner /usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py:271
140853: DEBUG nova.openstack.common.lockutils [req-a538a459-6ac2-4d30-8b02-734ce362d3ec 5bc15739d13343719261fca7c34307a2] Releasing semaphore "a35c068d-b4d6-4bc2-bd3b-f535ae711b43-events" lock /usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py:238

Comment 2 Jaison Raju 2015-11-09 13:39:15 UTC

Created attachment 1091750 [details]
Destination

Comment 3 Jaison Raju 2015-11-09 13:40:12 UTC

Created attachment 1091751 [details]
Host

Comment 5 Lee Yarwood 2015-11-10 09:24:56 UTC

(In reply to Jaison Raju from comment #0)
> Description of problem:
>
> [..]
>
> [root@os-compute-1 ~]# kill -SIGUSR1 $(pgrep nova-compute) ; sleep 5 ;
> journalctl -u openstack-nova-compute | grep my_ip
> Nov 08 11:47:48 os-compute-1.sahaba.com nova-compute[8441]: my_ip = 127.0.0.1
> 
> [root@os-compute-2 ~]# kill -SIGUSR1 $(pgrep nova-compute) ; sleep 5 ;
> journalctl -u openstack-nova-compute | grep my_ip
> Nov 08 11:46:30 os-compute-2.sahaba.com nova-compute[145832]: my_ip =
> 127.0.0.1

Jaison, as discussed the issue here is that my_ip is being set to 127.0.0.1 on both compute hosts thus causing the resize logic to assume that storage is shared between these hosts. Restarting nova-compute on both hosts or manually setting my_ip should workaround this for now. Once confirmed I'll look into how we could end up with 127.0.0.1 in the first place, I assume this is a race between networking coming up fully and nova-compute starting.

Setting a needinfo against you while we await feedback from the customer.

Comment 6 Lee Yarwood 2015-11-26 11:27:33 UTC

Closing with INSUFFICIENT_DATA, Jaison please reopen if/when the customer comes back.

Note You need to log in before you can comment on or make changes to this bug.