Bug 1412454

Summary: [RHOSP10] When trying to create an instance with creating volume or attaching existing cinder volume to an instance, these operations fails under cinder NFS backend.
Product: Red Hat OpenStack Reporter: Keigo Noha <knoha>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: Gabriel Szasz <gszasz>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.0 (Liberty)CC: agallego, awaugama, berrange, dasmith, ddomingo, eglynn, gpaterno, jschluet, kchamart, lyarwood, sbauza, sferdjao, sgordon, srevivo, vromanso
Target Milestone: z2Keywords: ZStream
Target Release: 10.0 (Newton)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-14.0.3-8.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1419045 (view as bug list) Environment:
Last Closed: 2017-02-23 21:16:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1419045, 1419047    

Description Keigo Noha 2017-01-12 04:27:45 UTC 
Description of problem:
When trying to create an instance with creating volume or attaching existing cinder volume to an instance, nova generates 'libvirtError: Cannot access storage file 'XXX': No such file or directory.' then these operations fails under cinder NFS backend.

Version-Release number of selected component (if applicable):
RHOSP8 and RHOSP10 is confirmed. RHOSP9 is still being checked.

How reproducible:
100%

Steps to Reproduce:
1. Install RHOSP with 'packstack --allinone' described in https://access.redhat.com/articles/1127153
2. Configure cinder NFS backend described in https://access.redhat.com/articles/1323213
3. Remove pre-existing cinder type, iscsi.
4. One of following operations causes this issue.
4-a Launch a cirros instance with creating volume at instance creation
or 
4-b Create a cinder volume in NFS and attach it to the existing instance.

Actual results:
4-a causes following error and an instance can not be booted from image.
~~~
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [req-5c6aed5d-8491-4a6f-9638-0f6db3375e15 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] [instance: 5a819dd6-dfba-4239-a161-905943b24f53] Instance failed to spawn
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53] Traceback (most recent call last):
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2078, in _build_resources
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     yield resources
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1920, in _build_and_run_instance
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     block_device_info=block_device_info)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2597, in spawn
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     post_xml_callback=gen_confdrive)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4845, in _create_domain_and_network
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     post_xml_callback=post_xml_callback)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4774, in _create_domain
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     guest.launch(pause=pause)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     self._encoded_xml, errors='ignore')
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     self.force_reraise()
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     six.reraise(self.type_, self.value, self.tb)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     return self._domain.createWithFlags(flags)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     rv = execute(f, *args, **kwargs)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     six.reraise(c, e, tb)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     rv = meth(*args, **kwargs)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53] libvirtError: Cannot access storage file '/var/lib/nova/mnt/09fb4552f711a03c79ab026fd565aead/volume-b1687cc4-6fc4-40db-8498-1899d481e613' (as uid:107, gid:107): No such file or directory
2017-01-12 12:59:09.206 2238 ERROR nova.compute.manager [instance: 5a819dd6-dfba-4239-a161-905943b24f53] 
2017-01-12 12:59:09.210 2238 INFO nova.compute.manager [req-5c6aed5d-8491-4a6f-9638-0f6db3375e15 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] [instance: 5a819dd6-dfba-4239-a161-905943b24f53] Terminating instance
2017-01-12 12:59:09.210 2238 WARNING nova.image.glance [req-5c6aed5d-8491-4a6f-9638-0f6db3375e15 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] No protocol specified in for api_server 'http://192.168.122.138:9292', please update [glance] api_servers with fully qualified url including scheme (http / https)
2017-01-12 12:59:09.217 2238 INFO nova.virt.libvirt.driver [-] [instance: 5a819dd6-dfba-4239-a161-905943b24f53] Instance destroyed successfully.
2017-01-12 12:59:09.278 2238 INFO os_vif [req-5c6aed5d-8491-4a6f-9638-0f6db3375e15 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:0c:7f:73,bridge_name='qbr874d54be-cf',has_traffic_filtering=True,id=874d54be-cf9f-46bc-820f-b13d6fa4a5e4,network=Network(5b3e313b-77a9-4ead-b748-ee44374745b7),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap874d54be-cf')
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs [req-5c6aed5d-8491-4a6f-9638-0f6db3375e15 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] Couldn't unmount the NFS share 192.168.122.167:/cinder1
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs Traceback (most recent call last):
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/nfs.py", line 59, in disconnect_volume
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs     utils.execute('umount', mount_path, run_as_root=True)
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 295, in execute
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs     return RootwrapProcessHelper().execute(*cmd, **kwargs)
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 178, in execute
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs     return processutils.execute(*cmd, **kwargs)
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 389, in execute
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs     cmd=sanitized_cmd)
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs ProcessExecutionError: Unexpected error while running command.
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs Command: sudo nova-rootwrap /etc/nova/rootwrap.conf umount /var/lib/nova/mnt/09fb4552f711a03c79ab026fd565aead
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs Exit code: 32
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs Stdout: u''
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs Stderr: u'umount: /var/lib/nova/mnt/09fb4552f711a03c79ab026fd565aead: mountpoint not found\n'
2017-01-12 12:59:09.380 2238 ERROR nova.virt.libvirt.volume.nfs 
~~~

4-b causes following error and the operation failed
~~~
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [req-22296231-5766-47a8-8dd4-2604e55324f0 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5] Failed to attach d12f2eba-4fdc-4d65-afd4-6b0c2e3011c4 at /dev/vdb
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5] Traceback (most recent call last):
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4685, in _attach_volume
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     do_check_attach=False, do_driver_attach=True)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 49, in wrapped
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     ret_val = method(obj, context, *args, **kwargs)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 283, in attach
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     connector)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     self.force_reraise()
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 274, in attach
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     device_type=self['device_type'], encryption=encryption)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1166, in attach_volume
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     self._disconnect_volume(connection_info, disk_dev)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     self.force_reraise()
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1155, in attach_volume
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     guest.attach_device(conf, persistent=True, live=live)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 294, in attach_device
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     rv = execute(f, *args, **kwargs)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     six.reraise(c, e, tb)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     rv = meth(*args, **kwargs)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5] libvirtError: Cannot access storage file '/var/lib/nova/mnt/09fb4552f711a03c79ab026fd565aead/volume-d12f2eba-4fdc-4d65-afd4-6b0c2e3011c4' (as uid:107, gid:107): No such file or directory
2017-01-12 13:23:03.173 2238 ERROR nova.compute.manager [instance: c286be11-76f5-4067-b77a-ecc7fe0504a5] 
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server [req-22296231-5766-47a8-8dd4-2604e55324f0 2c75536f961545a29b0a470cdd714b6c bb942319a85946d8b543ff8cdb62d66b - - -] Exception during message handling
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4675, in attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     do_attach_volume(context, instance, driver_bdm)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4673, in do_attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     bdm.destroy()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4670, in do_attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     return self._attach_volume(context, instance, driver_bdm)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4693, in _attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.volume_api.unreserve_volume(context, bdm.volume_id)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4685, in _attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     do_check_attach=False, do_driver_attach=True)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 49, in wrapped
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     ret_val = method(obj, context, *args, **kwargs)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 283, in attach
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     connector)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 274, in attach
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     device_type=self['device_type'], encryption=encryption)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1166, in attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self._disconnect_volume(connection_info, disk_dev)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1155, in attach_volume
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     guest.attach_device(conf, persistent=True, live=live)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 294, in attach_device
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     self._domain.attachDeviceFlags(device_xml, flags=flags)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server libvirtError: Cannot access storage file '/var/lib/nova/mnt/09fb4552f711a03c79ab026fd565aead/volume-d12f2eba-4fdc-4d65-afd4-6b0c2e3011c4' (as uid:107, gid:107): No such file or directory
2017-01-12 13:23:03.292 2238 ERROR oslo_messaging.rpc.server 
2017-01-12 13:24:04.189 2238 INFO nova.compute.resource_tracker [req-ab7a08af-be1e-44dd-9cfb-f6bbd3d0b437 - - - - -] Auditing locally available compute resources for node localhost
2017-01-12 13:24:04.196 2238 WARNING nova.virt.libvirt.driver [req-ab7a08af-be1e-44dd-9cfb-f6bbd3d0b437 - - - - -] couldn't obtain the vcpu count from domain id: c286be11-76f5-4067-b77a-ecc7fe0504a5, exception: Requested operation is not valid: cpu affinity is not supported
~~~

Expected results:
Those operations are done successfully.

Additional info:
The issue seeems to be related to RHBZ#1331286 and the upstream launchpad, https://bugs.launchpad.net/nova/+bug/1421550.
But the issue occurs under there is no suspended instances.
Comment 9 Lee Yarwood 2017-02-03 14:19:11 UTC 
My change to remove the use of findmnt has landed upstream ahead of Ocata RC1 so I'm reusing this bug to track the fix into stable/newton and OSP 10. I'll clone this now for 9 and 8.
Comment 16 errata-xmlrpc 2017-02-23 21:16:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0319.html