1412454 – [RHOSP10] When trying to create an instance with creating volume or attaching existing cinder volume to an instance, these operations fails under cinder NFS backend.

Bug 1412454 - [RHOSP10] When trying to create an instance with creating volume or attaching existing cinder volume to an instance, these operations fails under cinder NFS backend.

Summary: [RHOSP10] When trying to create an instance with creating volume or attachin...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	8.0 (Liberty)
Hardware:	x86_64
OS:	Linux
Priority:	urgent
Severity:	urgent
Target Milestone:	z2
Target Release:	10.0 (Newton)
Assignee:	Lee Yarwood
QA Contact:	Gabriel Szasz
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1419045 1419047
TreeView+	depends on / blocked

Reported:	2017-01-12 04:27 UTC by Keigo Noha
Modified:	2020-04-15 15:05 UTC (History)
CC List:	15 users (show)
Fixed In Version:	openstack-nova-14.0.3-8.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1419045 (view as bug list)
Environment:
Last Closed:	2017-02-23 21:16:04 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
OpenStack gerrit	420146	None	MERGED	libvirt: workaround findmnt behaviour change	2020-07-14 18:05:42 UTC
Red Hat Knowledge Base (Solution)	2858491	None	None	None	2017-01-18 06:56:50 UTC
Red Hat Product Errata	RHBA-2017:0319	normal	SHIPPED_LIVE	openstack-nova bug fix advisory	2017-02-24 02:12:47 UTC

Description Keigo Noha 2017-01-12 04:27:45 UTC

Description of problem:
When trying to create an instance with creating volume or attaching existing cinder volume to an instance, nova generates 'libvirtError: Cannot access storage file 'XXX': No such file or directory.' then these operations fails under cinder NFS backend.

Version-Release number of selected component (if applicable):
RHOSP8 and RHOSP10 is confirmed. RHOSP9 is still being checked.

How reproducible:
100%

Steps to Reproduce:
1. Install RHOSP with 'packstack --allinone' described in https://access.redhat.com/articles/1127153
2. Configure cinder NFS backend described in https://access.redhat.com/articles/1323213
3. Remove pre-existing cinder type, iscsi.
4. One of following operations causes this issue.
4-a Launch a cirros instance with creating volume at instance creation
or 
4-b Create a cinder volume in NFS and attach it to the existing instance.

Actual results:
4-a causes following error and an instance can not be booted from image.
~~~
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [req-5c6aed5d-8491-4a6f-9638-0f6db3375e15 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] [instance: 5a819dd6-dfba-4239-a161-905943b24f53] Instance failed to spawn
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53] Traceback (most recent call last):
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2078, in _build_resources
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     yield resources
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1920, in _build_and_run_instance
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     block_device_info=block_device_info)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2597, in spawn
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     post_xml_callback=gen_confdrive)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4845, in _create_domain_and_network
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     post_xml_callback=post_xml_callback)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4774, in _create_domain
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     guest.launch(pause=pause)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     self._encoded_xml, errors='ignore')
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     self.force_reraise()
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     six.reraise(self.type_, self.value, self.tb)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     return self._domain.createWithFlags(flags)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     rv = execute(f, *args, **kwargs)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     six.reraise(c, e, tb)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     rv = meth(*args, **kwargs)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53] libvirtError: Cannot access storage file '/var/lib/nova/mnt/09fb4552f711a03c79ab026fd565aead/volume-b1687cc4-6fc4-40db-8498-1899d481e613' (as uid:107, gid:107): No such file or directory
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53] 
2017-01-12 12:59:09.210 2238 INFO nova.compute.manager [req-5c6aed5d-8491-4a6f-9638-0f6db3375e15 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] [instance: 5a819dd6-dfba-4239-a161-905943b24f53] Terminating instance
2017-01-12 12:59:09.210 2238 WARNING nova.image.glance [req-5c6aed5d-8491-4a6f-9638-0f6db3375e15 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] No protocol specified in for api_server 'http://192.168.122.138:9292', please update [glance] api_servers with fully qualified url including scheme (http / https)
2017-01-12 12:59:09.217 2238 INFO nova.virt.libvirt.driver [-] [instance: 5a819dd6-dfba-4239-a161-905943b24f53] Instance destroyed successfully.
2017-01-12 12:59:09.278 2238 INFO os_vif [req-5c6aed5d-8491-4a6f-9638-0f6db3375e15 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:0c:7f:73,bridge_name='qbr874d54be-cf',has_traffic_filtering=True,id=874d54be-cf9f-46bc-820f-b13d6fa4a5e4,network=Network(5b3e313b-77a9-4ead-b748-ee44374745b7),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap874d54be-cf')
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs [req-5c6aed5d-8491-4a6f-9638-0f6db3375e15 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] Couldn't unmount the NFS share 192.168.122.167:/cinder1
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs Traceback (most recent call last):
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/nfs.py", line 59, in disconnect_volume
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs     utils.execute('umount', mount_path, run_as_root=True)
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 295, in execute
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs     return RootwrapProcessHelper().execute(*cmd, **kwargs)
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 178, in execute
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs     return processutils.execute(*cmd, **kwargs)
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 389, in execute
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs     cmd=sanitized_cmd)
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs ProcessExecutionError: Unexpected error while running command.
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs Command: sudo nova-rootwrap /etc/nova/rootwrap.conf umount /var/lib/nova/mnt/09fb4552f711a03c79ab026fd565aead
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs Exit code: 32
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs Stdout: u''
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs Stderr: u'umount: /var/lib/nova/mnt/09fb4552f711a03c79ab026fd565aead: mountpoint not found\n'
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs 
~~~

4-b causes following error and the operation failed
~~~
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [req-22296231-5766-47a8-8dd4-2604e55324f0 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5] Failed to attach d12f2eba-4fdc-4d65-afd4-6b0c2e3011c4 at /dev/vdb
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5] Traceback (most recent call last):
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4685, in _attach_volume
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     do_check_attach=False, do_driver_attach=True)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 49, in wrapped
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     ret_val = method(obj, context, *args, **kwargs)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 283, in attach
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     connector)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     self.force_reraise()
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 274, in attach
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     device_type=self['device_type'], encryption=encryption)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1166, in attach_volume
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     self._disconnect_volume(connection_info, disk_dev)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     self.force_reraise()
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1155, in attach_volume
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     guest.attach_device(conf, persistent=True, live=live)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 294, in attach_device
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     rv = execute(f, *args, **kwargs)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     six.reraise(c, e, tb)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     rv = meth(*args, **kwargs)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5] libvirtError: Cannot access storage file '/var/lib/nova/mnt/09fb4552f711a03c79ab026fd565aead/volume-d12f2eba-4fdc-4d65-afd4-6b0c2e3011c4' (as uid:107, gid:107): No such file or directory
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5] 
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server [req-22296231-5766-47a8-8dd4-2604e55324f0 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] Exception during message handling
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4675, in attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     do_attach_volume(context, instance, driver_bdm)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4673, in do_attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     bdm.destroy()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4670, in do_attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     return self._attach_volume(context, instance, driver_bdm)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4693, in _attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.volume_api.unreserve_volume(context, bdm.volume_id)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4685, in _attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     do_check_attach=False, do_driver_attach=True)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 49, in wrapped
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     ret_val = method(obj, context, *args, **kwargs)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 283, in attach
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     connector)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 274, in attach
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     device_type=self['device_type'], encryption=encryption)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1166, in attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self._disconnect_volume(connection_info, disk_dev)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1155, in attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     guest.attach_device(conf, persistent=True, live=live)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 294, in attach_device
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server libvirtError: Cannot access storage file '/var/lib/nova/mnt/09fb4552f711a03c79ab026fd565aead/volume-d12f2eba-4fdc-4d65-afd4-6b0c2e3011c4' (as uid:107, gid:107): No such file or directory
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server 
2017-01-12 13:24:04.189 2238 INFO nova.compute.resource_tracker [req-ab7a08af-be1e-44dd-9cfb-f6bbd3d0b437 - - - - -] Auditing locally available compute resources for node localhost
2017-01-12 13:24:04.196 2238 WARNING nova.virt.libvirt.driver [req-ab7a08af-be1e-44dd-9cfb-f6bbd3d0b437 - - - - -] couldn't obtain the vcpu count from domain id: c286be11-76f5-4067-b77a-ecc7fe0504a5, exception: Requested operation is not valid: cpu affinity is not supported
~~~

Expected results:
Those operations are done successfully.

Additional info:
The issue seeems to be related to RHBZ#1331286 and the upstream launchpad, https://bugs.launchpad.net/nova/+bug/1421550.
But the issue occurs under there is no suspended instances.

Comment 9 Lee Yarwood 2017-02-03 14:19:11 UTC

My change to remove the use of findmnt has landed upstream ahead of Ocata RC1 so I'm reusing this bug to track the fix into stable/newton and OSP 10. I'll clone this now for 9 and 8.

Comment 16 errata-xmlrpc 2017-02-23 21:16:04 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0319.html

Note You need to log in before you can comment on or make changes to this bug.