Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1919083

Summary: openstack server resize fails on shared storage, libvirt cannot access storage file /var/lib/nova/instances/UUID/disk.swap
Product: Red Hat OpenStack Reporter: Masayuki Igawa <migawa>
Component: documentationAssignee: Irina <igallagh>
Status: CLOSED CURRENTRELEASE QA Contact: RHOS Documentation Team <rhos-docs>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.1 (Train)CC: alifshit, astupnik, bdobreli, dasmith, eglynn, igallagh, jbeaudoi, jhakimra, kchamart, lyarwood, rribaud, sbauza, sgordon, smooney, vromanso
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-07-28 10:16:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Masayuki Igawa 2021-01-22 04:45:40 UTC 
Description of problem:

`openstack server resize` an instance with swap fails on NFS backend nova

Version-Release number of selected component (if applicable):


How reproducible:
almost 100%

Steps to Reproduce:
1. Create an instance with a flavor that has swap
2. Resize the instance with a flavor that has swap
3. An error occurs

If the "2." step doesn't fail, try to resize a couple of times.


Actual results:
Resize fails.

Expected results:
Resize succeeds.

Additional info:
This cannot be reproducible on non NFS backend nova.

I can see the error log in nova-compute.log.
~~~
...
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [req-c64928e8-bb8a-4950-a101-8f1e6868afac 276f72c19c1948b59e2f93e71f3b8e7d b022c1df41ba46fea657f5500a947c83 - default default] [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437] Setting instance vm_state to ERROR: libvirt.libvirtError: Cannot access storage file '/var/lib/nova/instances/8cdfbaf8-2f16-4342-857b-b5c9ebd5d437/disk.swap' (as uid:107, gid:107): No such file or directory
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437] Traceback (most recent call last):
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9014, in _error_out_instance_on_exception
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     yield
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 5181, in _finish_resize_helper
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     request_spec)
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 5117, in _finish_resize
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     self._set_instance_info(instance, old_flavor)
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     self.force_reraise()
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     six.reraise(self.type_, self.value, self.tb)
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     raise value
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 5105, in _finish_resize
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     block_device_info, power_on)
...
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     self.force_reraise()
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     six.reraise(self.type_, self.value, self.tb)
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     raise value
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 138, in launch
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     return self._domain.createWithFlags(flags)
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     rv = execute(f, *args, **kwargs)
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     six.reraise(c, e, tb)
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     raise value
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     rv = meth(*args, **kwargs)
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]   File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1265, in createWithFlags
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437] libvirt.libvirtError: Cannot access storage file '/var/lib/nova/instances/8cdfbaf8-2f16-4342-857b-b5c9ebd5d437/disk.swap' (as uid:107, gid:107): No such file or directory
2021-01-22 04:05:33.677 7 ERROR nova.compute.manager [instance: 8cdfbaf8-2f16-4342-857b-b5c9ebd5d437] 
2021-01-22 04:05:33.999 7 ERROR oslo_messaging.rpc.server [req-c64928e8-bb8a-4950-a101-8f1e6868afac 276f72c19c1948b59e2f93e71f3b8e7d b022c1df41ba46fea657f5500a947c83 - default default] Exception during message handling: libvirt.libvirtError: Cannot access storage file '/var/lib/nova/instances/8cdfbaf8-2f16-4342-857b-b5c9ebd5d437/disk.swap' (as uid:107, gid:107): No such file or directory
2021-01-22 04:05:33.999 7 ERROR oslo_messaging.rpc.server libvirt.libvirtError: Cannot access storage file '/var/lib/nova/instances/8cdfbaf8-2f16-4342-857b-b5c9ebd5d437/disk.swap' (as uid:107, gid:107): No such file or directory
~~~
Comment 1 Masayuki Igawa 2021-01-24 23:26:02 UTC 
Hi team,

Additional information, it seems that "console.log" and "disk.swap" aren't in the directory but only in the "_resize" directory.
~~~
[root@compute-0 heat-admin]# ls -l /var/lib/nova/instances/8cdfbaf8-2f16-4342-857b-b5c9ebd5d437*
/var/lib/nova/instances/8cdfbaf8-2f16-4342-857b-b5c9ebd5d437:
total 4
-rw-r--r--. 1 42436 42436 84 Jan 22 04:05 disk.info

/var/lib/nova/instances/8cdfbaf8-2f16-4342-857b-b5c9ebd5d437_resize:
total 228
-rw-------. 1 root  root   26233 Jan 22 04:05 console.log
-rw-r--r--. 1 42436 42436     84 Jan 22 04:04 disk.info
-rw-r--r--. 1 root  root  196616 Jan 22 04:04 disk.swap
~~~

Please let me know if you need further information such as sosreport. I can provide it.

Best Regards.
Comment 28 René Ribaud 2022-06-29 16:26:27 UTC 
Documenting it sounds like a reasonable option.
Alex, would it be possible for you to document this in KMS ?
Comment 35 Irina 2022-07-28 10:16:56 UTC 
Updated content available on the Customer Portal:

16.2:
- https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/advanced_overcloud_customization/assembly_storage-configuration#con_nfs-supported-configurations-and-limitations_storage-configuration

- https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/advanced_overcloud_customization/assembly_storage-configuration#con_nfs-supported-configurations-and-limitations_storage-configuration

16.1:

- https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/advanced_overcloud_customization/assembly_storage-configuration#con_nfs-supported-configurations-and-limitations_storage-configuration

- https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/configuring_the_compute_service_for_instance_creation/assembly_creating-flavors-for-launching-instances_instance-flavors#ref_flavor-arguments_instance-flavors