Bug 1478399 - after successful resize Nova recreates instance directory
after successful resize Nova recreates instance directory
Status: ASSIGNED
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
9.0 (Mitaka)
Unspecified Unspecified
medium Severity medium
: ---
: 9.0 (Mitaka)
Assigned To: Lee Yarwood
Joe H. Rahme
: Triaged, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-04 09:42 EDT by Martin Schuppert
Modified: 2017-10-24 11:57 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1666831 None None None 2017-08-04 09:42 EDT

  None (edit)
Description Martin Schuppert 2017-08-04 09:42:07 EDT
Description of problem:

After successful resize action on SRC host instance directory get created with 
disk.info file. As a consequence migrations back to this compute will fail with:

2017-07-29 13:37:30.899 15150 ERROR oslo_messaging.rpc.dispatcher DestinationDiskExists: The supplied disk path (/var/lib/nova/instances/9b975758-1379-42b0-8329-87100acbed8f) already exists, it is expected not to exist.


Version-Release number of selected component (if applicable):
python-nova-13.1.4-5.el7ost.noarch


How reproducible:
always

Steps to Reproduce:

[root@compute-1 ~]# ll /var/lib/nova/instances/
total 4
drwxr-xr-x. 2 nova nova  82 Aug  3 14:05 _base
-rw-r--r--. 1 nova nova  44 Aug  4 10:33 compute_nodes
drwxr-xr-x. 2 nova nova 126 Aug  3 14:05 locks

[root@compute-0 ~]# ll /var/lib/nova/instances/
total 4
drwxr-xr-x. 2 nova nova  82 Aug  3 16:12 _base
-rw-r--r--. 1 nova nova  44 Aug  4 10:38 compute_nodes
drwxr-xr-x. 2 nova nova 126 Aug  3 16:12 locks

[stack@undercloud-0 ~]$ openstack server create --flavor m1.tiny --image cirros --nic net-id=8008d543-d9e3-464c-99c0-7b25a24c330d test1

[stack@undercloud-0 ~]$ openstack server show test1
+--------------------------------------+----------------------------------------------------------+
| Field                                | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                   |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | compute-1.localdomain                                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute-1.localdomain                                    |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000026                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | None                                                     |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2017-08-04T10:58:42.000000                               |
| OS-SRV-USG:terminated_at             | None                                                     |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| addresses                            | public=10.0.0.221                                        |
| config_drive                         |                                                          |
| created                              | 2017-08-04T10:58:31Z                                     |
| flavor                               | m1.tiny (1)                                              |
| hostId                               | c8723dfdbc0143505a13afb820b05bc41fbe4022759d76826d2cb4a3 |
| id                                   | f5febdb5-6726-4cfa-8077-348b2a01703e                     |
| image                                | cirros (e0c662b1-0e51-4753-b38d-27f1a43a32b8)            |
| key_name                             | None                                                     |
| name                                 | test1                                                    |
| os-extended-volumes:volumes_attached | []                                                       |
| progress                             | 0                                                        |
| project_id                           | a126a6b887d6452bb4c8ae99774a07ef                         |
| properties                           |                                                          |
| security_groups                      | [{u'name': u'default'}]                                  |
| status                               | ACTIVE                                                   |
| updated                              | 2017-08-04T10:58:42Z                                     |
| user_id                              | fb4114459eec4a039caaace6ad1c2583                         |
+--------------------------------------+----------------------------------------------------------+

[stack@undercloud-0 ~]$ openstack server resize --flavor m1.small test1
[stack@undercloud-0 ~]$ openstack server resize --confirm test1

[stack@undercloud-0 ~]$ openstack server show test1
+--------------------------------------+----------------------------------------------------------+
| Field                                | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                   |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | compute-0.localdomain                                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute-0.localdomain                                    |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000026                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | None                                                     |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2017-08-04T11:02:00.000000                               |
| OS-SRV-USG:terminated_at             | None                                                     |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| addresses                            | public=10.0.0.221                                        |
| config_drive                         |                                                          |
| created                              | 2017-08-04T10:58:31Z                                     |
| flavor                               | m1.small (2)                                             |
| hostId                               | a536a72e96016e5359aeefa4a60939a4bc888a548965f204984c579f |
| id                                   | f5febdb5-6726-4cfa-8077-348b2a01703e                     |
| image                                | cirros (e0c662b1-0e51-4753-b38d-27f1a43a32b8)            |
| key_name                             | None                                                     |
| name                                 | test1                                                    |
| os-extended-volumes:volumes_attached | []                                                       |
| progress                             | 0                                                        |
| project_id                           | a126a6b887d6452bb4c8ae99774a07ef                         |
| properties                           |                                                          |
| security_groups                      | [{u'name': u'default'}]                                  |
| status                               | ACTIVE                                                   |
| updated                              | 2017-08-04T11:02:16Z                                     |
| user_id                              | fb4114459eec4a039caaace6ad1c2583                         |
+--------------------------------------+----------------------------------------------------------+

[root@compute-1 ~]# ll /var/lib/nova/instances/
total 4
drwxr-xr-x. 2 nova nova 130 Aug  4 10:58 _base
-rw-r--r--. 1 nova nova  44 Aug  4 10:33 compute_nodes
drwxr-xr-x. 2 nova nova  23 Aug  4 11:02 f5febdb5-6726-4cfa-8077-348b2a01703e
drwxr-xr-x. 2 nova nova 179 Aug  4 10:58 locks


[root@compute-0 ~]# ll /var/lib/nova/instances/
total 4
drwxr-xr-x. 2 nova nova 130 Aug  4 11:01 _base
-rw-r--r--. 1 nova nova  44 Aug  4 10:38 compute_nodes
drwxr-xr-x. 2 nova root  54 Aug  4 11:01 f5febdb5-6726-4cfa-8077-348b2a01703e
drwxr-xr-x. 2 nova nova 179 Aug  4 11:01 locks

on the src compute-1.localdomain we see:

- instance folder moved to _resize
2017-08-04 11:01:51.996 16307 DEBUG oslo_concurrency.processutils [req-eddb8f6f-0b53-42a9-8530-2ed70881ac13 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Running cmd (subprocess): mv /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e_resize execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:344
2017-08-04 11:01:52.012 16307 DEBUG oslo_concurrency.processutils [req-eddb8f6f-0b53-42a9-8530-2ed70881ac13 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] CMD "mv /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e_resize" returned: 0 in 0.016s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:374

- copy files to the other compute
2017-08-04 11:01:52.014 16307 DEBUG nova.virt.libvirt.volume.remotefs [req-eddb8f6f-0b53-42a9-8530-2ed70881ac13 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Copying file /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e_resize/disk to 192.168.24.8:/var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e/disk copy_file /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py:121
2017-08-04 11:01:52.014 16307 DEBUG oslo_concurrency.processutils [req-eddb8f6f-0b53-42a9-8530-2ed70881ac13 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Running cmd (subprocess): scp /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e_resize/disk 192.168.24.8:/var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e/disk execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:344
2017-08-04 11:01:52.594 16307 DEBUG oslo_concurrency.processutils [req-eddb8f6f-0b53-42a9-8530-2ed70881ac13 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] CMD "scp /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e_resize/disk 192.168.24.8:/var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e/disk" returned: 0 in 0.579s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:374
2017-08-04 11:01:52.595 16307 DEBUG nova.virt.libvirt.volume.remotefs [req-eddb8f6f-0b53-42a9-8530-2ed70881ac13 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Copying file /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e_resize/disk.info to 192.168.24.8:/var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e/disk.info copy_file /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py:121
2017-08-04 11:01:52.596 16307 DEBUG oslo_concurrency.processutils [req-eddb8f6f-0b53-42a9-8530-2ed70881ac13 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Running cmd (subprocess): scp /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e_resize/disk.info 192.168.24.8:/var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e/disk.info execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:344
2017-08-04 11:01:53.134 16307 DEBUG oslo_concurrency.processutils [req-eddb8f6f-0b53-42a9-8530-2ed70881ac13 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] CMD "scp /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e_resize/disk.info 192.168.24.8:/var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e/disk.info" returned: 0 in 0.539s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:374

...

- cleanup of the resize
2017-08-04 11:02:15.155 16307 DEBUG oslo_concurrency.processutils [req-ec235f97-e8ee-4f3f-8a30-5de0b7901c58 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Running cmd (subprocess): rm -rf /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e_resize execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:344
2017-08-04 11:02:15.172 16307 DEBUG oslo_concurrency.processutils [req-ec235f97-e8ee-4f3f-8a30-5de0b7901c58 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] CMD "rm -rf /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e_resize" returned: 0 in 0.017s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:374

- here we write a disk.info:
2017-08-04 11:02:15.175 16307 DEBUG oslo_concurrency.lockutils [req-ec235f97-e8ee-4f3f-8a30-5de0b7901c58 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Lock "/var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e/disk.info" acquired by "nova.virt.libvirt.imagebackend.write_to_disk_info_file" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-08-04 11:02:15.176 16307 DEBUG oslo_concurrency.lockutils [req-ec235f97-e8ee-4f3f-8a30-5de0b7901c58 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Lock "/var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e/disk.info" released by "nova.virt.libvirt.imagebackend.write_to_disk_info_file" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282

And this is the remaining information in the instance dir on the src compute:
[root@compute-1 ~]# ll /var/lib/nova/instances/f5febdb5-6726-4cfa-8077-348b2a01703e/
total 4
-rw-r--r--. 1 nova nova 79 Aug  4 11:02 disk.info



Additional info:

https://bugs.launchpad.net/nova/+bug/1666831

Note You need to log in before you can comment on or make changes to this bug.