Bug 984460
Summary: | Gluster mount fails when provided in CONFIG_CINDER_GLUSTER_MOUNTS during packstack installation. | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Gowrishankar Rajaiyan <grajaiya> | ||||||
Component: | openstack-packstack | Assignee: | Martin Magr <mmagr> | ||||||
Status: | CLOSED EOL | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
Severity: | unspecified | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 19 | CC: | david.bompart, derekh, eharney, grajaiya, Jan.van.Eldik, mmagr, p, sandro | ||||||
Target Milestone: | --- | ||||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2015-02-17 16:09:53 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | 984465, 986174 | ||||||||
Bug Blocks: | |||||||||
Attachments: |
|
Mount works fine when initiated manually on the terminal: [root@dhcp201-146 ~]# cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.65.201.186:/cinder-vol /var/lib/cinder/mnt/23c1c46a20484743ca082b9d8c835e8c/ [root@dhcp201-146 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/fedora_dhcp201--146-root 50G 2.5G 45G 6% / devtmpfs 7.8G 0 7.8G 0% /dev tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs 7.8G 604K 7.8G 1% /run tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup tmpfs 7.8G 116K 7.8G 1% /tmp /dev/sda1 477M 79M 373M 18% /boot /dev/mapper/fedora_dhcp201--146-home 860G 72M 816G 1% /home tmpfs 7.8G 604K 7.8G 1% /run/netns 10.65.201.186:/cinder-vol 297G 434M 296G 1% /var/lib/cinder/mnt/23c1c46a20484743ca082b9d8c835e8c [root@dhcp201-146 ~]# umount /var/lib/cinder/mnt/23c1c46a20484743ca082b9d8c835e8c Created attachment 773699 [details]
sosreport
I have a similar problem: 2013-07-17 17:25:21 WARNING [cinder.volume.drivers.glusterfs] Exception during mounting Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 192.168.122.60:/testvol /var/lib/cinder/mnt/123839f0aa037c18af0b383288016134 Exit code: 1 Stdout: 'Mount failed. Please check the log file for more details.\n' Stderr: '/sbin/mount.glusterfs: line 128: /usr/sbin/glusterfs: Permission denied\n' Maybe Eric will have some hints how to fix this. Martin: This generally means that the Gluster server/volume couldn't be reached for some reason. There should be a log file in /var/log/gluster corresponding to that mount point that will give a more precise reason for the failure. In my case I see following errors in /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on my GlusterFS VM: [2013-07-17 16:28:18.932468] E [rpc-transport.c:677:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.2.7/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2013-07-17 16:28:18.932499] E [rpc-transport.c:681:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine [2013-07-17 16:28:18.932507] W [rpcsvc.c:1288:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2013-07-17 16:28:18.932540] E [glusterd-store.c:1150:glusterd_store_handle_retrieve] 0-glusterd: Unable to retrieve store handle for /var/lib/glusterd/glusterd.info, error: No such file or directory [2013-07-17 16:28:18.932549] E [glusterd-store.c:1251:glusterd_retrieve_uuid] 0-: Unable to get store handle! And in my /var/log/glusterfs/var-lib-cinder-mnt-123839f0aa037c18af0b383288016134.log on my cotroller VM it seems everything is OK: [2013-07-17 17:27:23.865963] I [glusterfsd.c:1493:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.2.7 [2013-07-17 17:27:24.192061] W [write-behind.c:3023:init] 0-testvol-write-behind: disabling write-behind for first 0 bytes [2013-07-17 17:27:24.193218] I [client.c:1935:notify] 0-testvol-client-0: parent translators are ready, attempting connect on transport Given volfile: +------------------------------------------------------------------------------+ 1: volume testvol-client-0 2: type protocol/client 3: option remote-host 192.168.122.60 4: option remote-subvolume /data 5: option transport-type tcp 6: end-volume 7: 8: volume testvol-write-behind 9: type performance/write-behind 10: subvolumes testvol-client-0 11: end-volume 12: 13: volume testvol-read-ahead 14: type performance/read-ahead 15: subvolumes testvol-write-behind 16: end-volume 17: 18: volume testvol-io-cache 19: type performance/io-cache 20: subvolumes testvol-read-ahead 21: end-volume 22: 23: volume testvol-quick-read 24: type performance/quick-read 25: subvolumes testvol-io-cache 26: end-volume 27: 28: volume testvol-stat-prefetch 29: type performance/stat-prefetch 30: subvolumes testvol-quick-read 31: end-volume 32: 33: volume testvol 34: type debug/io-stats 35: option latency-measurement off 36: option count-fop-hits off 37: subvolumes testvol-stat-prefetch 38: end-volume +------------------------------------------------------------------------------+ [2013-07-17 17:27:24.203108] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-testvol-client-0: changing port to 24009 (from 0) [2013-07-17 17:27:28.150241] I [client-handshake.c:1090:select_server_supported_programs] 0-testvol-client-0: Using Program GlusterFS 3.2.7, Num (1298437), Version (310) [2013-07-17 17:27:28.151618] I [client-handshake.c:913:client_setvolume_cbk] 0-testvol-client-0: Connected to 192.168.122.60:24009, attached to remote volume '/data'. [2013-07-17 17:27:28.156788] I [fuse-bridge.c:3461:fuse_graph_setup] 0-fuse: switched to graph 0 [2013-07-17 17:27:28.157102] I [fuse-bridge.c:3049:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13 In Gowrishankar's case I see following errors in var-lib-cinder-mnt-23c1c46a20484743ca082b9d8c835e8c.log: [2013-07-15 20:28:11.349157] E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse: cannot open /dev/fuse (Permission denied) [2013-07-15 20:28:11.349180] E [xlator.c:390:xlator_init] 0-fuse: Initialization of volume 'fuse' failed, review your volfile again Anyway for Gowrishankar and also for me manual mount (sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 192.168.122.60:/testvol /var/lib/cinder/mnt/123839f0aa037c18af0b383288016134) worked without problem, so maybe it's not because volume was not unreachable (In reply to Martin Magr from comment #5) > [2013-07-15 20:28:11.349157] E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse: > cannot open /dev/fuse (Permission denied) > [2013-07-15 20:28:11.349180] E [xlator.c:390:xlator_init] 0-fuse: > Initialization of volume 'fuse' failed, review your volfile again > > Anyway for Gowrishankar and also for me manual mount (sudo cinder-rootwrap > /etc/cinder/rootwrap.conf mount -t glusterfs 192.168.122.60:/testvol > /var/lib/cinder/mnt/123839f0aa037c18af0b383288016134) worked without > problem, so maybe it's not because volume was not unreachable Right, it doesn't look like it. Can you see if there are SELinux avcs in /var/log/audit? Indeed there is on controller node: [para@virtual-rhel ~]$ sudo ausearch -m avc ---- time->Wed Jul 17 17:12:42 2013 type=SYSCALL msg=audit(1374073962.968:3909): arch=c000003e syscall=59 success=no exit=-13 a0=21a9be0 a1=21913d0 a2=21910f0 a3=30 items=0 ppid=8130 pid=8197 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=86 comm="mount.glusterfs" exe="/bin/bash" subj=unconfined_u:system_r:mount_t:s0 key=(null) type=AVC msg=audit(1374073962.968:3909): avc: denied { execute } for pid=8197 comm="mount.glusterfs" name="glusterfsd" dev=dm-0 ino=27972 scontext=unconfined_u:system_r:mount_t:s0 tcontext=system_u:object_r:glusterd_exec_t:s0 tclass=file ---- I had filed bug 984465 for the selinux issue. - shanks Update: Filed bug 986154 while installing with the latest packstack from upstream. Another SELinux avc. Filed bug 986174. (In reply to Martin Magr from comment #3) > I have a similar problem: > > 2013-07-17 17:25:21 WARNING [cinder.volume.drivers.glusterfs] Exception > during mounting Unexpected error while running command. > Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs > 192.168.122.60:/testvol /var/lib/cinder/mnt/123839f0aa037c18af0b383288016134 > Exit code: 1 > Stdout: 'Mount failed. Please check the log file for more details.\n' > Stderr: '/sbin/mount.glusterfs: line 128: /usr/sbin/glusterfs: Permission > denied\n' > > Maybe Eric will have some hints how to fix this. After creating the volume, did you assign the right permissions to the gluster volume? if volname==cinder-vol being 165 cinder gid (#id cinder -> output 165) volume set cinder-vol storage.owner-uid=165 volume set cinder-vol storage.owner-gid=165 (In reply to David Bompart from comment #11) > (In reply to Martin Magr from comment #3) > > I have a similar problem: > > > > 2013-07-17 17:25:21 WARNING [cinder.volume.drivers.glusterfs] Exception > > during mounting Unexpected error while running command. > > Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs > > 192.168.122.60:/testvol /var/lib/cinder/mnt/123839f0aa037c18af0b383288016134 > > Exit code: 1 > > Stdout: 'Mount failed. Please check the log file for more details.\n' > > Stderr: '/sbin/mount.glusterfs: line 128: /usr/sbin/glusterfs: Permission > > denied\n' > > > > Maybe Eric will have some hints how to fix this. > > After creating the volume, did you assign the right permissions to the > gluster volume? > > if volname==cinder-vol being 165 cinder gid (#id cinder -> output 165) > > volume set cinder-vol storage.owner-uid=165 > volume set cinder-vol storage.owner-gid=165 That was a: gluster volume set cinder-vol storage.owner-uid=165 gluster volume set cinder-vol storage.owner-gid=165 Sorry. This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component. This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component. This message is a notice that Fedora 19 is now at end of life. Fedora has stopped maintaining and issuing updates for Fedora 19. It is Fedora's policy to close all bug reports from releases that are no longer maintained. Approximately 4 (four) weeks from now this bug will be closed as EOL if it remains open with a Fedora 'version' of '19'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 19 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. |
Created attachment 773682 [details] /var/log/glusterfs/var-lib-cinder-mnt-23c1c46a20484743ca082b9d8c835e8c.log Description of problem: Version-Release number of selected component (if applicable): 1. GlusterFS: glusterfs-3.4.0-0.8.beta4.fc19.x86_64, glusterfs-fuse-3.4.0-0.8.beta4.fc19.x86_64 2. Openstack: openstack-cinder-2013.2.a342.g5610092-0.1.1432.5610092_6959372.noarch 3. Packstack: last commit "commit b73f51d3b419b5f3bc092a826097573085dd3126" How reproducible: Always Steps to Reproduce: 1. Install Fedora 19 2. git clone --recursive git://github.com/stackforge/packstack.git 3. cd packstack 4. ./bin/packstack --gen-answer-file=ans.txt 5. update ans.txt: 5.1 CONFIG_CINDER_BACKEND=gluster 5.2 CONFIG_CINDER_GLUSTER_MOUNTS=10.65.201.186:/cinder-vol 6. ./bin/packstack --answer-file=ans.txt Actual results: Gluster volume fails to mount during packstack installation. Expected results: Gluster volume provided in ans.tx mounts successfully. Additional info: /var/log/cinder/volume.log 2013-07-16 01:58:11.370 10465 WARNING cinder.volume.drivers.glusterfs [req-38c28b36-3f61-4268-9458-146ce4495058 None None] Exception during mounting Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.65.201.186:/cinder-vol /var/lib/cinder/mnt/23c1c46a20484743ca082b9d8c835e8c Exit code: 1 Stdout: 'Mount failed. Please check the log file for more details.\n' Stderr: '' 2013-07-16 01:58:11.415 10465 ERROR cinder.openstack.common.rpc.impl_qpid [req-38c28b36-3f61-4268-9458-146ce4495058 None None] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 1 seconds 2013-07-16 01:58:12.416 10465 ERROR cinder.openstack.common.rpc.impl_qpid [req-38c28b36-3f61-4268-9458-146ce4495058 None None] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 2 seconds 2013-07-16 01:58:14.421 10465 ERROR cinder.openstack.common.rpc.impl_qpid [req-38c28b36-3f61-4268-9458-146ce4495058 None None] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 4 seconds /var/log/glusterfs/var-lib-cinder-mnt-23c1c46a20484743ca082b9d8c835e8c.log [2013-07-15 20:28:11.348532] I [glusterfsd.c:1878:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0beta4 (/usr/sbin/glusterfs --volfile-id=/cinder-vol --volfile-server=10.65.201.186 /var/lib/cinder/mnt/23c1c46a20484743ca082b9d8c835e8c) [2013-07-15 20:28:11.349157] E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse: cannot open /dev/fuse (Permission denied) [2013-07-15 20:28:11.349180] E [xlator.c:390:xlator_init] 0-fuse: Initialization of volume 'fuse' failed, review your volfile again ... [2013-07-15 20:34:31.583518] W [glusterfsd.c:970:cleanup_and_exit] (-->/usr/lib64/libc.so.6(clone+0x6d) [0x7f2335371ecd] (-->/usr/lib64/libpthread.so.0(+0x7c53) [0x7f2335a08c53] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7f23366f6e35]))) 0-: received signum (15), shutting down [2013-07-15 20:34:31.583526] I [fuse-bridge.c:5212:fini] 0-fuse: Unmounting '/var/lib/cinder/mnt/23c1c46a20484743ca082b9d8c835e8c'. (END)