Bug 1259441

Summary: Can't create new Gluster storage domain - Permission denied
Product: [oVirt] vdsm Reporter: Aharon Canan <acanan>
Component: GeneralAssignee: Nobody <nobody>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Aharon Canan <acanan>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: ---CC: acanan, amureini, bazulay, bugs, ecohen, gklein, lsurette, sabose, ycui, yeylon, ylavi
Target Milestone: ovirt-3.5.6Keywords: Regression
Target Release: ---Flags: ylavi: ovirt-3.5.z?
ylavi: ovirt-3.6.0?
rule-engine: blocker?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: gluster
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-21 12:00:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Gluster RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Logs01 none

Description Aharon Canan 2015-09-02 15:31:04 UTC
Created attachment 1069479 [details]
Logs01

Description of problem:
-----------------------
Can't create new gluster storage domain
Permission denied for directory: /rhev/data-center/mnt/glusterSD/10.35.160.6:_acanan02 with permissions:7

Manual everything works fine
10.35.160.6:/acanan02 on /mnt type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)


Version-Release number of selected component (if applicable):
vt16.9

How reproducible:
100%

Steps to Reproduce:
1. Try to create new gluster storage domain
2.
3.

Actual results:
fails with "Permission denied for directory: /rhev/data-center/mnt/glusterSD/10.35.160.6:_acanan02 with permissions:7"

Expected results:
should work

From vdsm log:
--------------
Thread-945::DEBUG::2015-09-02 15:19:38,804::__init__::481::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.connectStorageServer' in bridge with {u'connectionParams': [{u'port': u'', u'connection': u'10.35.160.6:/acanan02', u'
iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'id': u'00000000-0000-0000-0000-000000000000'}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 7}
Thread-945::DEBUG::2015-09-02 15:19:38,805::task::595::Storage.TaskManager.Task::(_updateState) Task=`15ba8664-a87e-43e4-a934-558db50fbb1f`::moving from state init -> state preparing
Thread-945::INFO::2015-09-02 15:19:38,805::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'port': u'', u'connection': u'10.35.160.6:/acanan0
2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '******', u'id': u'00000000-0000-0000-0000-000000000000'}], options=None)
Thread-945::DEBUG::2015-09-02 15:19:38,807::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/glusterSD/10.35.160.6:_acanan02
Thread-945::DEBUG::2015-09-02 15:19:38,808::mount::227::Storage.Misc.excCmd::(_runcmd) /usr/bin/sudo -n /usr/bin/mount -t glusterfs 10.35.160.6:/acanan02 /rhev/data-center/mnt/glusterSD/10.35.160.6:_acanan02 (cwd None)
Thread-945::DEBUG::2015-09-02 15:19:38,918::__init__::298::IOProcessClient::(_run) Starting IOProcess...
Thread-945::WARNING::2015-09-02 15:19:38,927::outOfProcess::206::Storage.oop::(validateAccess) Permission denied for directory: /rhev/data-center/mnt/glusterSD/10.35.160.6:_acanan02 with permissions:7
Thread-945::DEBUG::2015-09-02 15:19:38,927::mount::227::Storage.Misc.excCmd::(_runcmd) /usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/glusterSD/10.35.160.6:_acanan02 (cwd None)
Thread-945::ERROR::2015-09-02 15:19:38,941::hsm::2449::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2446, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 231, in connect
    raise e
StorageServerAccessPermissionError: Permission settings on the specified path do not allow access to the storage. Verify permission settings on the specified storage path.: 'path = /rhev/data-center/mnt/glusterSD/10.35.160.6:_acanan02'


RPMs - 
-------
[root@camel-vdsc ~]# rpm -qa |grep vdsm
vdsm-cli-4.16.26-1.el7ev.noarch
vdsm-4.16.26-1.el7ev.x86_64
vdsm-python-zombiereaper-4.16.26-1.el7ev.noarch
vdsm-yajsonrpc-4.16.26-1.el7ev.noarch
vdsm-xmlrpc-4.16.26-1.el7ev.noarch
vdsm-jsonrpc-4.16.26-1.el7ev.noarch
vdsm-python-4.16.26-1.el7ev.noarch
[root@camel-vdsc ~]# rpm -qa |grep gluster
glusterfs-client-xlators-3.7.1-11.el7.x86_64
glusterfs-rdma-3.7.1-11.el7.x86_64
glusterfs-3.7.1-11.el7.x86_64
glusterfs-api-3.7.1-11.el7.x86_64
glusterfs-api-devel-3.7.1-11.el7.x86_64
glusterfs-fuse-3.7.1-11.el7.x86_64
glusterfs-libs-3.7.1-11.el7.x86_64
glusterfs-devel-3.7.1-11.el7.x86_64
glusterfs-cli-3.7.1-11.el7.x86_64


Gluster server- 
----------------
[root@gluster-storage-03 ~]# rpm -qa |grep gluster
glusterfs-fuse-3.7.1-11.el7rhgs.x86_64
glusterfs-libs-3.7.1-11.el7rhgs.x86_64
glusterfs-3.7.1-11.el7rhgs.x86_64
glusterfs-api-3.7.1-11.el7rhgs.x86_64
glusterfs-server-3.7.1-11.el7rhgs.x86_64
glusterfs-client-xlators-3.7.1-11.el7rhgs.x86_64
glusterfs-cli-3.7.1-11.el7rhgs.x86_64


[root@gluster-storage-03 ~]# gluster volume info acanan02
Volume Name: acanan02
Type: Distribute
Volume ID: 8bb98061-eb50-44f7-bd03-cc011c31ac2c
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 10.35.160.6:/gluster_volumes/acanan02
Options Reconfigured:
storage.owner-gid: 36
storage.owner-uid: 36
performance.readdir-ahead: on
server.allow-insecure: on
client.bind-insecure: off

Comment 1 Aharon Canan 2015-09-02 15:33:26 UTC
hypervisor is rhel 7.2 (BTW)

Comment 2 Sahina Bose 2015-09-22 13:04:22 UTC
Is SELinux in enforcing mode? Can you check if there are any AVC denial errors?

Comment 3 Aharon Canan 2015-10-08 11:34:12 UTC
Tried to reproduce
Using 3.5.5 (vt17.1) on RHEL 7.1 - Works.
Using 3.6.0-15 on RHEL 7.2 - Works

In case you can't find the cause from the logs we can close it.
If it will happen again I will reopen.

Comment 4 Yaniv Lavi 2015-10-14 14:45:15 UTC
should we close?

Comment 5 Allon Mureinik 2015-10-15 08:08:38 UTC
My instinct is yes, but since this BZ has whiteboard=gluster, I'll leave it to Sahina to decide.

Comment 6 Sahina Bose 2015-10-15 12:53:21 UTC
Yes, please go ahead and close.

Comment 7 Red Hat Bugzilla Rules Engine 2015-10-21 11:49:31 UTC
This bug is not marked for z-stream, yet the milestone is for a z-stream version, therefore the milestone has been reset.
Please set the correct milestone or add the z-stream flag.

Comment 8 Red Hat Bugzilla Rules Engine 2015-10-21 11:55:37 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 9 Yaniv Lavi 2015-10-21 12:00:56 UTC
Closing based on comment #6