Overcloud deployed with NFS backend: unable to create cinder volume. Unable to mount the NFS share from controllers: mount.nfs: /var/lib/cinder/mnt/884306f0cb952b10bf6c43194652b6da is busy or already mounted Environment: python-cinderclient-3.1.0-1.el7ost.noarch puppet-cinder-11.3.1-0.20171010073336.d6dcc60.el7ost.noarch openstack-cinder-11.0.0-2.el7ost.noarch python-cinder-11.0.0-2.el7ost.noarch openstack-tripleo-heat-templates-7.0.3-3.el7ost.noarch openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch instack-undercloud-7.4.3-2.el7ost.noarch Steps to reproduce: 1) Deploy overcloud with NFS backend: openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /home/stack/templates/nodes_data.yaml \ -e /home/stack/nfs.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/rhos12.yaml (undercloud) [stack@undercloud-0 ~]$ cat /home/stack/nfs.yaml parameter_defaults: CinderEnableIscsiBackend: false CinderEnableRbdBackend: false CinderEnableNfsBackend: true CinderNfsMountOptions: 'retry=1' CinderNfsServers: 192.168.24.1:/export/cinder GlanceBackend: 'file' GlanceNfsEnabled: true GlanceNfsShare: '192.168.24.1:/export/glance' 2) Try to create cinder volume. Result: The cinder volume is not created. Checking the setup - I see that no mount can be made to NFS share from controllers in that setup. Computes (same subnet) are able to mount the share with no issue. Looking in cinder's volume.log: 2017-11-15 02:08:51.519 129662 INFO cinder.volume.manager [req-bc5f6a68-985e-4c40-b85c-b90eb919ac40 - - - - -] Starting volume driver NfsDriver (1.4.0) 2017-11-15 02:08:56.160 129662 ERROR cinder.volume.drivers.nfs [req-bc5f6a68-985e-4c40-b85c-b90eb919ac40 - - - - -] Mount failure for 192.168.24.1:/export/cinder after 3 attempts.: BrickExce ption: NFS mount failed for share 192.168.24.1:/export/cinder. Error - {'pnfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nf s -o retry=1,vers=4,minorversion=1 192.168.24.1:/export/cinder /var/lib/cinder/mnt/884306f0cb952b10bf6c43194652b6da\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: /var/lib/cinder/mnt/8843 06f0cb952b10bf6c43194652b6da is busy or already mounted\\n'", 'nfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o retry= 1 192.168.24.1:/export/cinder /var/lib/cinder/mnt/884306f0cb952b10bf6c43194652b6da\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: /var/lib/cinder/mnt/884306f0cb952b10bf6c43194652b6da is b usy or already mounted\\n'"} 2017-11-15 02:08:56.161 129662 ERROR cinder.volume.drivers.remotefs [req-bc5f6a68-985e-4c40-b85c-b90eb919ac40 - - - - -] Exception during mounting NFS mount failed for share 192.168.24.1:/ex port/cinder. Error - {'pnfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o retry=1,vers=4,minorversion=1 192.168.24.1:/e xport/cinder /var/lib/cinder/mnt/884306f0cb952b10bf6c43194652b6da\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: /var/lib/cinder/mnt/884306f0cb952b10bf6c43194652b6da is busy or already mo unted\\n'", 'nfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o retry=1 192.168.24.1:/export/cinder /var/lib/cinder/mnt/ 884306f0cb952b10bf6c43194652b6da\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: /var/lib/cinder/mnt/884306f0cb952b10bf6c43194652b6da is busy or already mounted\\n'"}: NfsException: NFS mo unt failed for share 192.168.24.1:/export/cinder. Error - {'pnfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o retry=1, vers=4,minorversion=1 192.168.24.1:/export/cinder /var/lib/cinder/mnt/884306f0cb952b10bf6c43194652b6da\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: /var/lib/cinder/mnt/884306f0cb952b10b f6c43194652b6da is busy or already mounted\\n'", 'nfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o retry=1 192.168.24. 1:/export/cinder /var/lib/cinder/mnt/884306f0cb952b10bf6c43194652b6da\nExit code: 32\nStdout: u''\nStderr: u'mount.nfs: /var/lib/cinder/mnt/884306f0cb952b10bf6c43194652b6da is busy or alread y mounted\\n'"} 2017-11-15 02:08:56.162 129662 ERROR cinder.volume.manager [req-bc5f6a68-985e-4c40-b85c-b90eb919ac40 - - - - -] Failed to initialize driver.: NfsNoSharesMounted: No mounted NFS shares found 2017-11-15 02:08:56.162 129662 ERROR cinder.volume.manager Traceback (most recent call last): 2017-11-15 02:08:56.162 129662 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 429, in init_host 2017-11-15 02:08:56.162 129662 ERROR cinder.volume.manager self.driver.do_setup(ctxt) 2017-11-15 02:08:56.162 129662 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 197, in do_setup 2017-11-15 02:08:56.162 129662 ERROR cinder.volume.manager self.set_nas_security_options(self._is_voldb_empty_at_startup) 2017-11-15 02:08:56.162 129662 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 395, in set_nas_security_options 2017-11-15 02:08:56.162 129662 ERROR cinder.volume.manager raise exception.NfsNoSharesMounted() 2017-11-15 02:08:56.162 129662 ERROR cinder.volume.manager NfsNoSharesMounted: No mounted NFS shares found 2017-11-15 02:08:56.162 129662 ERROR cinder.volume.manager 2017-11-15 02:08:56.182 129662 WARNING oslo_config.cfg [req-bc5f6a68-985e-4c40-b85c-b90eb919ac40 - - - - -] Option "rabbit_port" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url). Its value may be silently ignored in the future. 2017-11-15 02:08:56.182 129662 WARNING oslo_config.cfg [req-bc5f6a68-985e-4c40-b85c-b90eb919ac40 - - - - -] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url). Its value may be silently ignored in the future. 2017-11-15 02:08:56.183 129662 WARNING oslo_config.cfg [req-bc5f6a68-985e-4c40-b85c-b90eb919ac40 - - - - -] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url). Its value may be silently ignored in the future. 2017-11-15 02:08:56.400 129662 INFO cinder.volume.manager [req-bc5f6a68-985e-4c40-b85c-b90eb919ac40 - - - - -] Initializing RPC dependent components of volume driver NfsDriver (1.4.0) 2017-11-15 02:08:56.401 129662 ERROR cinder.utils [req-bc5f6a68-985e-4c40-b85c-b90eb919ac40 - - - - -] Volume driver NfsDriver not initialized 2017-11-15 02:08:56.401 129662 ERROR cinder.volume.manager [req-bc5f6a68-985e-4c40-b85c-b90eb919ac40 - - - - -] Cannot complete RPC initialization because driver isn't initialized properly.: DriverNotInitialized: Volume driver not ready. 2017-11-15 02:09:06.402 129662 ERROR cinder.service [-] Manager for service cinder-volume hostgroup@tripleo_nfs is reporting problems, not sending heartbeat. Service will appear "down". 2017-11-15 02:09:16.404 129662 ERROR cinder.service [-] Manager for service cinder-volume hostgroup@tripleo_nfs is reporting problems, not sending heartbeat. Service will appear "down"
Created attachment 1352376 [details] cinder logs from controller.
Could this be a dup of bug #1491597? I notice both Cinder and Glance are using NFS. The symptom is "SELinux: mount invalid. Same superblock, different security settings" will appear in /var/log/messages.
Might be related: https://bugzilla.redhat.com/show_bug.cgi?id=1486995 which got fixed with this upstream patch: https://review.openstack.org/#/c/502403
Can we get a copy of /var/log/messages from the node where the mount operation fails? That would help confirm if this is the same issue seen in bug 1491597, which seems likely.
While this seems like the same issue, bug #1491597 is for OSP11, so we should track both separately. The w/a mentioned in bug #1491597 works: Need to include: CinderNfsMountOptions: 'nosharecache' GlanceNfsMountOptions: 'nosharecache' Do we switch this to be a doc bug? Note: it used to work in previous releases without this setting.
Bug #1491597 is still under investigation, and the'nosharecache' option is not a suitable workaround. There's a rather alarming note in the Linux manpage for NFS that states: As of kernel 2.6.18, the behavior specified by nosharecache is legacy caching behavior. This is considered a data risk since multiple cached copies of the same file on the same client can become out of sync following a local update of one of the copies.
This definitely seems like an existing issue and not a regression. That said, Eric raised a good point, do we know if anything has changed that makes this more likely to happen in OSP12 - when we deploy NFS be default to we get this configuration where they point to same NFS server for glance and cinder. It also sounds like separate NFS servers is the work around.
Eric Correct me if I'm wrong, suspect this bz's will remain with us going forward. Shouldn't we clone for OSP13-14, so as to no lose track of this issue. Owing that OPS12 is EOL soon, this bz might fade away.
Updated target release to 13, a long term release.
*** Bug 1647654 has been marked as a duplicate of this bug. ***
Shall we close this bug, after it was not handled for several years? If not, what's the next step?
(In reply to Yaniv Kaul from comment #34) > Shall we close this bug, after it was not handled for several years? If not, > what's the next step? The issue is still there, but as usual there are other priorities. We will discuss it in the not far future.