Bug 984460 - Gluster mount fails when provided in CONFIG_CINDER_GLUSTER_MOUNTS during packstack installation.
Summary: Gluster mount fails when provided in CONFIG_CINDER_GLUSTER_MOUNTS during pack...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: openstack-packstack
Version: 19
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Martin Magr
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 984465 986174
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-15 09:57 UTC by Gowrishankar Rajaiyan
Modified: 2015-02-17 16:09 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-02-17 16:09:53 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
/var/log/glusterfs/var-lib-cinder-mnt-23c1c46a20484743ca082b9d8c835e8c.log (23.11 KB, text/plain)
2013-07-15 09:57 UTC, Gowrishankar Rajaiyan
no flags Details
sosreport (893.39 KB, application/x-zip)
2013-07-15 11:20 UTC, Gowrishankar Rajaiyan
no flags Details

Description Gowrishankar Rajaiyan 2013-07-15 09:57:10 UTC
Created attachment 773682 [details]
/var/log/glusterfs/var-lib-cinder-mnt-23c1c46a20484743ca082b9d8c835e8c.log

Description of problem:


Version-Release number of selected component (if applicable):
1. GlusterFS: glusterfs-3.4.0-0.8.beta4.fc19.x86_64, glusterfs-fuse-3.4.0-0.8.beta4.fc19.x86_64
2. Openstack: openstack-cinder-2013.2.a342.g5610092-0.1.1432.5610092_6959372.noarch
3. Packstack: last commit "commit b73f51d3b419b5f3bc092a826097573085dd3126"

How reproducible: Always


Steps to Reproduce:
1. Install Fedora 19
2. git clone --recursive git://github.com/stackforge/packstack.git
3. cd packstack
4. ./bin/packstack --gen-answer-file=ans.txt
5. update ans.txt:
 5.1 CONFIG_CINDER_BACKEND=gluster
 5.2 CONFIG_CINDER_GLUSTER_MOUNTS=10.65.201.186:/cinder-vol
6. ./bin/packstack --answer-file=ans.txt


Actual results: Gluster volume fails to mount during packstack installation.


Expected results: Gluster volume provided in ans.tx mounts successfully. 


Additional info: /var/log/cinder/volume.log
2013-07-16 01:58:11.370 10465 WARNING cinder.volume.drivers.glusterfs [req-38c28b36-3f61-4268-9458-146ce4495058 None None] Exception during mounting Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.65.201.186:/cinder-vol /var/lib/cinder/mnt/23c1c46a20484743ca082b9d8c835e8c
Exit code: 1
Stdout: 'Mount failed. Please check the log file for more details.\n'
Stderr: ''
2013-07-16 01:58:11.415 10465 ERROR cinder.openstack.common.rpc.impl_qpid [req-38c28b36-3f61-4268-9458-146ce4495058 None None] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 1 seconds
2013-07-16 01:58:12.416 10465 ERROR cinder.openstack.common.rpc.impl_qpid [req-38c28b36-3f61-4268-9458-146ce4495058 None None] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 2 seconds
2013-07-16 01:58:14.421 10465 ERROR cinder.openstack.common.rpc.impl_qpid [req-38c28b36-3f61-4268-9458-146ce4495058 None None] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 4 seconds


/var/log/glusterfs/var-lib-cinder-mnt-23c1c46a20484743ca082b9d8c835e8c.log
[2013-07-15 20:28:11.348532] I [glusterfsd.c:1878:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0beta4 (/usr/sbin/glusterfs --volfile-id=/cinder-vol --volfile-server=10.65.201.186 /var/lib/cinder/mnt/23c1c46a20484743ca082b9d8c835e8c)
[2013-07-15 20:28:11.349157] E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse: cannot open /dev/fuse (Permission denied)
[2013-07-15 20:28:11.349180] E [xlator.c:390:xlator_init] 0-fuse: Initialization of volume 'fuse' failed, review your volfile again
...
[2013-07-15 20:34:31.583518] W [glusterfsd.c:970:cleanup_and_exit] (-->/usr/lib64/libc.so.6(clone+0x6d) [0x7f2335371ecd] (-->/usr/lib64/libpthread.so.0(+0x7c53) [0x7f2335a08c53] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7f23366f6e35]))) 0-: received signum (15), shutting down
[2013-07-15 20:34:31.583526] I [fuse-bridge.c:5212:fini] 0-fuse: Unmounting '/var/lib/cinder/mnt/23c1c46a20484743ca082b9d8c835e8c'.
(END)

Comment 1 Gowrishankar Rajaiyan 2013-07-15 09:58:23 UTC
Mount works fine when initiated manually on the terminal:

[root@dhcp201-146 ~]# cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.65.201.186:/cinder-vol /var/lib/cinder/mnt/23c1c46a20484743ca082b9d8c835e8c/

[root@dhcp201-146 ~]# df -h
Filesystem                            Size  Used Avail Use% Mounted on
/dev/mapper/fedora_dhcp201--146-root   50G  2.5G   45G   6% /
devtmpfs                              7.8G     0  7.8G   0% /dev
tmpfs                                 7.8G     0  7.8G   0% /dev/shm
tmpfs                                 7.8G  604K  7.8G   1% /run
tmpfs                                 7.8G     0  7.8G   0% /sys/fs/cgroup
tmpfs                                 7.8G  116K  7.8G   1% /tmp
/dev/sda1                             477M   79M  373M  18% /boot
/dev/mapper/fedora_dhcp201--146-home  860G   72M  816G   1% /home
tmpfs                                 7.8G  604K  7.8G   1% /run/netns
10.65.201.186:/cinder-vol             297G  434M  296G   1% /var/lib/cinder/mnt/23c1c46a20484743ca082b9d8c835e8c

[root@dhcp201-146 ~]# umount /var/lib/cinder/mnt/23c1c46a20484743ca082b9d8c835e8c

Comment 2 Gowrishankar Rajaiyan 2013-07-15 11:20:15 UTC
Created attachment 773699 [details]
sosreport

Comment 3 Martin Magr 2013-07-17 16:05:36 UTC
I have a similar problem:

2013-07-17 17:25:21  WARNING [cinder.volume.drivers.glusterfs] Exception during mounting Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 192.168.122.60:/testvol /var/lib/cinder/mnt/123839f0aa037c18af0b383288016134
Exit code: 1
Stdout: 'Mount failed. Please check the log file for more details.\n'
Stderr: '/sbin/mount.glusterfs: line 128: /usr/sbin/glusterfs: Permission denied\n'

Maybe Eric will have some hints how to fix this.

Comment 4 Eric Harney 2013-07-17 17:35:16 UTC
Martin: This generally means that the Gluster server/volume couldn't be reached for some reason.  There should be a log file in /var/log/gluster corresponding to that mount point that will give a more precise reason for the failure.

Comment 5 Martin Magr 2013-07-18 09:00:15 UTC
In my case I see following errors in /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on my GlusterFS VM:

[2013-07-17 16:28:18.932468] E [rpc-transport.c:677:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.2.7/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2013-07-17 16:28:18.932499] E [rpc-transport.c:681:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2013-07-17 16:28:18.932507] W [rpcsvc.c:1288:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2013-07-17 16:28:18.932540] E [glusterd-store.c:1150:glusterd_store_handle_retrieve] 0-glusterd: Unable to retrieve store handle for /var/lib/glusterd/glusterd.info, error: No such file or directory
[2013-07-17 16:28:18.932549] E [glusterd-store.c:1251:glusterd_retrieve_uuid] 0-: Unable to get store handle!

And in my /var/log/glusterfs/var-lib-cinder-mnt-123839f0aa037c18af0b383288016134.log on my cotroller VM it seems everything is OK:
[2013-07-17 17:27:23.865963] I [glusterfsd.c:1493:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.2.7
[2013-07-17 17:27:24.192061] W [write-behind.c:3023:init] 0-testvol-write-behind: disabling write-behind for first 0 bytes
[2013-07-17 17:27:24.193218] I [client.c:1935:notify] 0-testvol-client-0: parent translators are ready, attempting connect on transport
Given volfile:
+------------------------------------------------------------------------------+
  1: volume testvol-client-0
  2:     type protocol/client
  3:     option remote-host 192.168.122.60
  4:     option remote-subvolume /data
  5:     option transport-type tcp
  6: end-volume
  7: 
  8: volume testvol-write-behind
  9:     type performance/write-behind
 10:     subvolumes testvol-client-0
 11: end-volume
 12: 
 13: volume testvol-read-ahead
 14:     type performance/read-ahead
 15:     subvolumes testvol-write-behind
 16: end-volume
 17: 
 18: volume testvol-io-cache
 19:     type performance/io-cache
 20:     subvolumes testvol-read-ahead
 21: end-volume
 22: 
 23: volume testvol-quick-read
 24:     type performance/quick-read
 25:     subvolumes testvol-io-cache
 26: end-volume
 27: 
 28: volume testvol-stat-prefetch
 29:     type performance/stat-prefetch
 30:     subvolumes testvol-quick-read
 31: end-volume
 32: 
 33: volume testvol
 34:     type debug/io-stats
 35:     option latency-measurement off
 36:     option count-fop-hits off
 37:     subvolumes testvol-stat-prefetch
 38: end-volume

+------------------------------------------------------------------------------+
[2013-07-17 17:27:24.203108] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-testvol-client-0: changing port to 24009 (from 0)
[2013-07-17 17:27:28.150241] I [client-handshake.c:1090:select_server_supported_programs] 0-testvol-client-0: Using Program GlusterFS 3.2.7, Num (1298437), Version (310)
[2013-07-17 17:27:28.151618] I [client-handshake.c:913:client_setvolume_cbk] 0-testvol-client-0: Connected to 192.168.122.60:24009, attached to remote volume '/data'.
[2013-07-17 17:27:28.156788] I [fuse-bridge.c:3461:fuse_graph_setup] 0-fuse: switched to graph 0
[2013-07-17 17:27:28.157102] I [fuse-bridge.c:3049:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13

In Gowrishankar's case I see following errors in var-lib-cinder-mnt-23c1c46a20484743ca082b9d8c835e8c.log:
[2013-07-15 20:28:11.349157] E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse: cannot open /dev/fuse (Permission denied)
[2013-07-15 20:28:11.349180] E [xlator.c:390:xlator_init] 0-fuse: Initialization of volume 'fuse' failed, review your volfile again

Anyway for Gowrishankar and also for me manual mount (sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 192.168.122.60:/testvol /var/lib/cinder/mnt/123839f0aa037c18af0b383288016134) worked without problem, so maybe it's not because volume was not unreachable

Comment 6 Eric Harney 2013-07-18 11:36:12 UTC
(In reply to Martin Magr from comment #5)
> [2013-07-15 20:28:11.349157] E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse:
> cannot open /dev/fuse (Permission denied)
> [2013-07-15 20:28:11.349180] E [xlator.c:390:xlator_init] 0-fuse:
> Initialization of volume 'fuse' failed, review your volfile again
> 
> Anyway for Gowrishankar and also for me manual mount (sudo cinder-rootwrap
> /etc/cinder/rootwrap.conf mount -t glusterfs 192.168.122.60:/testvol
> /var/lib/cinder/mnt/123839f0aa037c18af0b383288016134) worked without
> problem, so maybe it's not because volume was not unreachable

Right, it doesn't look like it.  Can you see if there are SELinux avcs in /var/log/audit?

Comment 7 Martin Magr 2013-07-18 12:18:20 UTC
Indeed there is on controller node:
[para@virtual-rhel ~]$ sudo ausearch -m avc
----
time->Wed Jul 17 17:12:42 2013
type=SYSCALL msg=audit(1374073962.968:3909): arch=c000003e syscall=59 success=no exit=-13 a0=21a9be0 a1=21913d0 a2=21910f0 a3=30 items=0 ppid=8130 pid=8197 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=86 comm="mount.glusterfs" exe="/bin/bash" subj=unconfined_u:system_r:mount_t:s0 key=(null)
type=AVC msg=audit(1374073962.968:3909): avc:  denied  { execute } for  pid=8197 comm="mount.glusterfs" name="glusterfsd" dev=dm-0 ino=27972 scontext=unconfined_u:system_r:mount_t:s0 tcontext=system_u:object_r:glusterd_exec_t:s0 tclass=file
----

Comment 8 Gowrishankar Rajaiyan 2013-07-18 14:55:15 UTC
I had filed bug 984465 for the selinux issue.

- shanks

Comment 9 Gowrishankar Rajaiyan 2013-07-19 05:34:54 UTC
Update: Filed bug 986154 while installing with the latest packstack from upstream.

Comment 10 Gowrishankar Rajaiyan 2013-07-19 06:40:50 UTC
Another SELinux avc. Filed bug 986174.

Comment 11 David Bompart 2013-08-09 13:38:05 UTC
(In reply to Martin Magr from comment #3)
> I have a similar problem:
> 
> 2013-07-17 17:25:21  WARNING [cinder.volume.drivers.glusterfs] Exception
> during mounting Unexpected error while running command.
> Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs
> 192.168.122.60:/testvol /var/lib/cinder/mnt/123839f0aa037c18af0b383288016134
> Exit code: 1
> Stdout: 'Mount failed. Please check the log file for more details.\n'
> Stderr: '/sbin/mount.glusterfs: line 128: /usr/sbin/glusterfs: Permission
> denied\n'
> 
> Maybe Eric will have some hints how to fix this.

After creating the volume, did you assign the right permissions to the gluster volume?

if volname==cinder-vol being 165 cinder gid (#id cinder -> output 165)

volume set cinder-vol storage.owner-uid=165
volume set cinder-vol storage.owner-gid=165

Comment 12 David Bompart 2013-08-09 13:40:00 UTC
(In reply to David Bompart from comment #11)
> (In reply to Martin Magr from comment #3)
> > I have a similar problem:
> > 
> > 2013-07-17 17:25:21  WARNING [cinder.volume.drivers.glusterfs] Exception
> > during mounting Unexpected error while running command.
> > Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs
> > 192.168.122.60:/testvol /var/lib/cinder/mnt/123839f0aa037c18af0b383288016134
> > Exit code: 1
> > Stdout: 'Mount failed. Please check the log file for more details.\n'
> > Stderr: '/sbin/mount.glusterfs: line 128: /usr/sbin/glusterfs: Permission
> > denied\n'
> > 
> > Maybe Eric will have some hints how to fix this.
> 
> After creating the volume, did you assign the right permissions to the
> gluster volume?
> 
> if volname==cinder-vol being 165 cinder gid (#id cinder -> output 165)
> 
> volume set cinder-vol storage.owner-uid=165
> volume set cinder-vol storage.owner-gid=165

That was a: 


gluster volume set cinder-vol storage.owner-uid=165
gluster volume set cinder-vol storage.owner-gid=165

Sorry.

Comment 13 Fedora Admin XMLRPC Client 2013-10-14 09:51:06 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 14 Fedora Admin XMLRPC Client 2013-10-14 09:51:54 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 15 Fedora End Of Life 2015-01-09 18:55:39 UTC
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 16 Fedora End Of Life 2015-02-17 16:09:53 UTC
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.