Bug 1250660

Summary: Local storage can't be added
Product: [Retired] oVirt Reporter: jniederm
Component: vdsmAssignee: Greg Padgett <gpadgett>
Status: CLOSED WORKSFORME QA Contact: Kevin Alon Goldblatt <kgoldbla>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.6CC: acanan, amureini, bazulay, bugs, ecohen, gklein, gpadgett, jniederm, lsurette, mgoldboi, rbalakri, ycui, yeylon
Target Milestone: m1   
Target Release: 3.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-12 14:06:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
webadmin_error
none
engine log
none
vdsm.log none

Description jniederm 2015-08-05 17:27:32 UTC 
Created attachment 1059575 [details]
webadmin_error

Description of problem:
Local storage domain can't be added to local DC. It seems that scritp /usr/share/vdsm/storage/storageServer.py is trying to add path 'path = <path entered by user>' instead of just '<path entered by user>'.

Version-Release number of selected component (if applicable):
engine 3.6, commit e8c46f8
vdsm.noarch 4.17.0-1240.git487a19e.fc22 from http://resources.ovirt.org/pub/ovirt-3.6-pre/rpm/fc$releasever/

How reproducible:
100%

Steps to Reproduce:
1. (In webadmin, clean engine installation:) Set DC Default to local.
2. (Set architecture in cluster)
3. Add host
4. Make a directory for domain at the host
4. Add local storage, path like '/home/user/localdomain'

Actual results:
Error popup in webadmin (see screenshot): Error while executing action Add Storage Connection: Permission settings on the specified path do not allow access to the storage.
Verify permission settings on the specified storage path. 

Expected results:
Local domain is added

Additional info:
Interesting part from vdsm.log:
Thread-199::DEBUG::2015-08-05 12:07:56,107::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.connectStorageServer' in bridge with {u'connectionParams': [{u'id': u'00000000-0000-0000-0000-000000000000', u'connection': u'/home/jakub/localdomain', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 4}
Thread-199::DEBUG::2015-08-05 12:07:56,108::task::595::Storage.TaskManager.Task::(_updateState) Task=`c4268b63-d48c-41a2-a7d3-ec16c414a13b`::moving from state init -> state preparing
Thread-199::INFO::2015-08-05 12:07:56,108::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=4, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'00000000-0000-0000-0000-000000000000', u'connection': u'/home/jakub/localdomain', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '********', u'port': u''}], options=None)
Thread-199::ERROR::2015-08-05 12:07:56,108::hsm::2455::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2452, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 591, in connect
    self.checkTarget()
  File "/usr/share/vdsm/storage/storageServer.py", line 576, in checkTarget
    raise se.StorageServerLocalNotDirError(self._path)
StorageServerLocalNotDirError: The specified local path is not a directory.: u'path = /home/jakub/localdomain'
Thread-199::DEBUG::2015-08-05 12:07:56,108::hsm::2474::Storage.HSM::(connectStorageServer) knownSDs: {8af55ace-5b14-4ed6-a640-553f84b53124: storage.nfsSD.findDomain}
Thread-199::INFO::2015-08-05 12:07:56,108::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 469, 'id': u'00000000-0000-0000-0000-000000000000'}]}
Comment 1 jniederm 2015-08-05 17:28:16 UTC 
Created attachment 1059576 [details]
engine log
Comment 2 jniederm 2015-08-05 17:28:40 UTC 
Created attachment 1059577 [details]
vdsm.log
Comment 3 Allon Mureinik 2015-08-06 20:05:43 UTC 
Greg, you recently handled some issues in this area.
Can you take a look please?
Comment 4 Greg Padgett 2015-08-07 00:50:49 UTC 
(In reply to jniederm from comment #0)
> Steps to Reproduce:
> 1. (In webadmin, clean engine installation:) Set DC Default to local.
> 2. (Set architecture in cluster)
> 3. Add host
> 4. Make a directory for domain at the host
> 4. Add local storage, path like '/home/user/localdomain'
> 
> Actual results:
> Error popup in webadmin (see screenshot): Error while executing action Add
> Storage Connection: Permission settings on the specified path do not allow
> access to the storage.
> Verify permission settings on the specified storage path. 

Hi Jakub,

Can you verify the permissions for your directory?  The local storage should be owned by vdsm:kvm (36:36) and have permissions 0750 or higher, and each component of the path should also be navigable by that user/group.  The safest test would be something like /localstorage, just to make sure it's not really a permissions issue.

The 'path = <somepath>' error looks a little odd, but the 'path =' prefix was added by the exception code and so that shouldn't be an issue.

Let me know what you find.  Thanks.
Comment 5 jniederm 2015-08-10 10:16:52 UTC 
$ ll -d ~/localdomain/
drwxrwxrwx. 2 vdsm kvm 6 Aug  5 12:02 /home/jakub/localdomain/
... so I guess permissions should be ok.

IMHO: I'd be really suspicious about the code:

574     def checkTarget(self):
575         if not os.path.isdir(self._path):
576             raise se.StorageServerLocalNotDirError(self._path)

It looks like the string thrown in exception is the same as the string that is tested for being a directory i.e. the string with 'path = ' prefix.
Comment 6 Greg Padgett 2015-08-10 17:27:37 UTC 
(In reply to jniederm from comment #5)
> $ ll -d ~/localdomain/
> drwxrwxrwx. 2 vdsm kvm 6 Aug  5 12:02 /home/jakub/localdomain/
> ... so I guess permissions should be ok.

Can you check /home/jakub also, and ensure it's at least rx for the group?  By default it would be 0700 owned by your user, which would prevent vdsm from accessing that path.

> IMHO: I'd be really suspicious about the code:
> 
> 574     def checkTarget(self):
> 575         if not os.path.isdir(self._path):
> 576             raise se.StorageServerLocalNotDirError(self._path)
> 
> It looks like the string thrown in exception is the same as the string that
> is tested for being a directory i.e. the string with 'path = ' prefix.

Agreed, it looks suspicious--however the exception itself is doing the magic :)

class StorageServerLocalNotDirError(StorageException):
    code = 469
    message = "The specified local path is not a directory."

    def __init__(self, targetPath):
        self.value = "path = %s" % targetPath
Comment 7 jniederm 2015-08-11 16:32:27 UTC 
$ ll -d ~
drwx--x---+ 47 jakub jakub 4096 Aug 11 18:25 /home/jakub

Missing 'x' permission on my home directory was really the cause of the problem.
Thank you for explanation.
Feel free to close.
Comment 8 Greg Padgett 2015-08-12 14:06:45 UTC 
(In reply to jniederm from comment #7)
> $ ll -d ~
> drwx--x---+ 47 jakub jakub 4096 Aug 11 18:25 /home/jakub
> 
> Missing 'x' permission on my home directory was really the cause of the
> problem.
> Thank you for explanation.
> Feel free to close.

Happy it now works!