Bug 1126981 - cinder NFS volume not mounting in HA staypuft deployment
Summary: cinder NFS volume not mounting in HA staypuft deployment
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 5.0 (RHEL 7)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 5.0 (RHEL 7)
Assignee: Eric Harney
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-05 18:44 UTC by Kambiz Aghaiepour
Modified: 2016-04-26 14:11 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-08-26 18:26:05 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Kambiz Aghaiepour 2014-08-05 18:44:10 UTC
Description of problem:

Installing osp5 using staypuft, and selecting HA deployment, glance and cinder NFS exports are required in order to create the deployment.  However, only the glance NFS volume appears on the controller hosts.  Cinder NFS seems to fail with the following error message in /var/log/messages:

Aug  3 03:19:18 macb8ca3a60bde8 cinder-volume: 2014-08-03 03:19:18.870 29362 WARNING cinder.volume.drivers.nfs [-] Exception during mounting NFS mount failed for share host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_cinder.Error - {'pnfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o vers=4,minorversion=1 host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_cinder /var/lib/cinder/mnt/4683cefc3a2d299755e2fee6d9945b4b\nExit code: 32\nStdout: ''\nStderr: 'mount.nfs: Protocol not supported\\n'", 'nfs': u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o  host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_cinder /var/lib/cinder/mnt/4683cefc3a2d299755e2fee6d9945b4b\nExit code: 32\nStdout: ''\nStderr: 'mount.nfs: /var/lib/cinder/mnt/4683cefc3a2d299755e2fee6d9945b4b is busy or already mounted\\n'"}

The foreman host has:

 openstack-foreman-installer-2.0.16-1.el6ost.noarch
 openstack-puppet-modules-2014.1-19.8.el6ost.noarch

and other associated packages from 07.29.3 poodle.

Version-Release number of selected component (if applicable):


How reproducible:

Deploy HA deployment using rhel-osp-installer. Deploy.

Comment 1 Eric Harney 2014-08-05 18:48:54 UTC
What is the state of /var/lib/cinder/mnt/4683... on the host?

It looks like it failed to mount with pnfs (4.1) which isn't surprising.  But then mount.nfs says it's already mounted?  What's mounted there?

Comment 3 Kambiz Aghaiepour 2014-08-05 19:35:02 UTC
Nothing is mounted there.  Here is the df output:

Filesystem                                                               1K-blocks     Used  Available Use% Mounted on
/dev/mapper/rhel_macb8ca3a60bde8-root                                     52403200  6268428   46134772  12% /
devtmpfs                                                                  65902744        0   65902744   0% /dev
tmpfs                                                                     65909272    39216   65870056   1% /dev/shm
tmpfs                                                                     65909272  1410036   64499236   3% /run
tmpfs                                                                     65909272        0   65909272   0% /sys/fs/cgroup
/dev/mapper/rhel_macb8ca3a60bde8-home                                   1894379776    32928 1894346848   1% /home
/dev/sda1                                                                   508588   118184     390404  24% /boot
host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_glance  960777216 94028800  817943552  11% /var/lib/glance/images

The expected NFS server path:

   host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_cinder

is exported to the host.  Here is what I find.  The glance options used (however the NFS mount is done), based on the output of "mount" are:

host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_glance on /var/lib/glance/images type nfs4 (rw,relatime,context=system_u:object_r:glance_var_lib_t:s0,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.1.252.6,local_lock=none,addr=10.1.8.30)

I can mount the cinder export manually using:

[root@macb8ca3a60bde8 ~]# mount -o rw,relatime,context=system_u:object_r:glance_var_lib_t:s0,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.1.252.6,local_lock=none,addr=10.1.8.30 host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_cinder /mnt

and the NFS volume is mounted fine as per:

[root@macb8ca3a60bde8 ~]# df
Filesystem                                                               1K-blocks     Used  Available Use% Mounted on
/dev/mapper/rhel_macb8ca3a60bde8-root                                     52403200  6269760   46133440  12% /
devtmpfs                                                                  65902744        0   65902744   0% /dev
tmpfs                                                                     65909272    39216   65870056   1% /dev/shm
tmpfs                                                                     65909272  1418228   64491044   3% /run
tmpfs                                                                     65909272        0   65909272   0% /sys/fs/cgroup
/dev/mapper/rhel_macb8ca3a60bde8-home                                   1894379776    32928 1894346848   1% /home
/dev/sda1                                                                   508588   118184     390404  24% /boot
host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_glance  960777216 94028800  817943552  11% /var/lib/glance/images
host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_cinder  960777216 94028800  817943552  11% /mnt

and the mount output:

[root@macb8ca3a60bde8 ~]# mount | grep cinder
host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_cinder on /mnt type nfs4 (rw,relatime,context=system_u:object_r:glance_var_lib_t:s0,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.1.252.6,local_lock=none,addr=10.1.8.30)

However, if I attempt to mount the cinder NFS volume using the two forms that appear to be getting logged in /var/log/messages, I get the same errors as per:

[root@macb8ca3a60bde8 ~]# cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o vers=4,minorversion=1 host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_cinder /mnt
mount.nfs: Protocol not supported
[root@macb8ca3a60bde8 ~]# cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o  "" host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_cinder /mnt
mount.nfs: /mnt is busy or already mounted

What does seem to work is :

[root@macb8ca3a60bde8 ~]# cinder-rootwrap /etc/cinder/rootwrap.conf mount -t nfs -o  "rw,relatime,context=system_u:object_r:glance_var_lib_t:s0,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.1.252.6,local_lock=none,addr=10.1.8.30" host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_cinder /mnt

So it appears that some NFS options need to be passed to the cinder-rootwrap, though there are none being used other than "vers=4,minorversion=1" and the second attempt with "" as the options.  The other oddity is why the target directory is "/var/lib/cinder/mnt/4683cefc3a2d299755e2fee6d9945b4b"?   That directory does exist though, so I suspect that is "normal"?

Comment 4 Jiri Stransky 2014-08-06 08:53:22 UTC
The system was installed with Staypuft/Quickstack. The relevant parameters for Cinder puppet module are:

    nfs_mount_options: ''
    nfs_shares:
    - host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_cinder

Which corresponds to what's in cinder.conf:

nfs_mount_options=

and in shares-nfs.conf file:

host01-rack02.scale.openstack.engineering.redhat.com:/srv/nfs/c1_cinder


Eric, do you know if there's something in particular we should be always setting for nfs_mount_options on RHEL 7?


As i read to logs and conversation, it seems the problem might be NFS version 4.0 vs 4.1? Kambiz, maybe you could try to set the nfs_mount_options parameter to "vers=4.0" as a workaround? I'm not sure if Staypuft devs put this parameter into the wizard so maybe you'll need to edit it as an advanced parameter on the host group.

Comment 5 Kambiz Aghaiepour 2014-08-18 13:39:51 UTC
After reinstallation from the latest poodle last week, the cinder volume appears to mount correctly on the controller nodes.   I have not done any testing with cinder in my installation as of yet.  Should I see the NFS mount on the compute nodes as well?

Comment 6 Eric Harney 2014-08-26 18:26:05 UTC
(In reply to Kambiz Aghaiepour from comment #5)
> Should I see the NFS
> mount on the compute nodes as well?

You will see one on the compute nodes when performing an attach.


Closing since it sounds like this is not happening on a setup currently, let me know if you run into this again.


Note You need to log in before you can comment on or make changes to this bug.