Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 890842

Summary: 3.1.z vdsm: connectStorageServer fails for posix domain
Product: Red Hat Enterprise Linux 6 Reporter: Dafna Ron <dron>
Component: vdsmAssignee: Federico Simoncelli <fsimonce>
Status: CLOSED NOTABUG QA Contact: Haim <hateya>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 6.4CC: abaron, bazulay, cpelland, iheim, lpeer, yeylon, ykaul
Target Milestone: rcKeywords: Regression, TestBlocker
Target Release: 6.4   
Hardware: x86_64   
OS: Linux   
Whiteboard: storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-21 15:00:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 882907    
Attachments:
Description Flags
logs none

Description Dafna Ron 2012-12-30 15:48:12 UTC 
Created attachment 670433 [details]
logs

Description of problem:

I failed creating a posix domain (nfs type) while I am able to create an nfs domain to the exact target. 

looking at the logs connectStorageServer fails. 

Version-Release number of selected component (if applicable):

sf2
vdsm-4.10.2-2.0.el6.x86_64

How reproducible:

100%

Steps to Reproduce:
1. try to create posix domain (type nfs)
2.
3.
  
Actual results:

we fail to create the domain 

Expected results:

we should succeed to create a posix domain

Additional info: logs

in the logs I used the same target for posix and than for nfs. 

we fail on posix with type nfs and succeed on regular nfs

I echoed into the logs before each one (search >>>>NFS and >>>>POSIX) 


Thread-3254::ERROR::2012-12-30 17:33:55,992::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2206, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 208, in connect
    fileSD.validateDirAccess(self.getMountObj().getRecord().fs_file)
  File "/usr/share/vdsm/storage/mount.py", line 236, in getRecord
    (self.fs_spec, self.fs_file))
OSError: [Errno 2] Mount of `filer01.qa.lab.tlv.redhat.com:/daffi_test` at `/rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test` does not exist


mount succeeded on nfs: 

root@gold-vdsc ~]# mount
/dev/mapper/vg0-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vg0-lv_home on /home type ext4 (rw,nobarrier)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
filer01.qa.lab.tlv.redhat.com:/daffi_test on /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test type nfs (rw,soft,nosharecache,timeo=600,retrans=6,nfsvers=3,addr=10.35.64.21)
Comment 2 Federico Simoncelli 2013-01-03 09:05:34 UTC 
The nfs mount that succeeds uses nfs version 3:

Thread-3362::DEBUG::2012-12-30 17:34:52,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 filer01.qa.lab.tlv.redhat.com:/daffi_test /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)

The one that fails uses version 4 (default):

Thread-3254::DEBUG::2012-12-30 17:31:50,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs filer01.qa.lab.tlv.redhat.com:/daffi_test /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)

If the server is gluster this is expected since it only supports version 3.

Can you try again specifying the vers=3 option.
Comment 4 Haim 2013-01-07 12:44:52 UTC 
(In reply to comment #2)
> The nfs mount that succeeds uses nfs version 3:
> 
> Thread-3362::DEBUG::2012-12-30
> 17:34:52,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n
> /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3
> filer01.qa.lab.tlv.redhat.com:/daffi_test
> /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)
> 
> The one that fails uses version 4 (default):
> 
> Thread-3254::DEBUG::2012-12-30
> 17:31:50,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n
> /bin/mount -t nfs filer01.qa.lab.tlv.redhat.com:/daffi_test
> /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)
> 
> If the server is gluster this is expected since it only supports version 3.
> 
> Can you try again specifying the vers=3 option.

Fede, we tried to execute the failed mount (which vdsm used), so it doesn't make sense.
the only difference that affects success or failure is if i'm using vdsClient connectStorageServer or mount command (which vdsm wraps it up).
Comment 5 Federico Simoncelli 2013-01-21 13:31:37 UTC 
I extracted from the attached logs all the mount commands. For what I see all the failed commands are related to missing glusterfs shares (requires double checking on the storage) or tries to access the same storage through nfs version 4 (which are correctly failing). There are some successful connection when nfs version 3 is used.

Thread-772::DEBUG::2012-12-30 16:23:13,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t glusterfs filer01.qa.lab.tlv.redhat.com:/Daffi /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_Daffi' (cwd None)
Thread-772::ERROR::2012-12-30 16:23:13,334::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-788::DEBUG::2012-12-30 16:23:34,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t glusterfs -o vers=3 filer01.qa.lab.tlv.redhat.com:/Daffi /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_Daffi' (cwd None)
Thread-788::ERROR::2012-12-30 16:23:34,640::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-859::DEBUG::2012-12-30 16:25:31,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t glusterfs filer01.qa.lab.tlv.redhat.com:/Dafna /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_Dafna' (cwd None)
Thread-859::ERROR::2012-12-30 16:25:31,925::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-879::DEBUG::2012-12-30 16:26:01,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t glusterfs filer01.qa.lab.tlv.redhat.com:/Dafna /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_Dafna' (cwd None)
Thread-879::ERROR::2012-12-30 16:26:01,794::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-907::DEBUG::2012-12-30 16:26:43,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs filer01.qa.lab.tlv.redhat.com:/Dafna /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_Dafna' (cwd None)
Thread-907::ERROR::2012-12-30 16:28:48,460::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-1022::DEBUG::2012-12-30 16:29:53,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs filer01.qa.lab.tlv.redhat.com:/Daffi /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_Daffi' (cwd None)
Thread-1022::ERROR::2012-12-30 16:31:58,620::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-1477::DEBUG::2012-12-30 16:42:38,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t glusterfs filer01.qa.lab.tlv.redhat.com:/daffi /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi' (cwd None)
Thread-1477::ERROR::2012-12-30 16:42:38,246::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-1492::DEBUG::2012-12-30 16:42:58,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t glusterfs filer01.qa.lab.tlv.redhat.com:/Daffi /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_Daffi' (cwd None)
Thread-1492::ERROR::2012-12-30 16:42:59,024::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-1552::DEBUG::2012-12-30 16:44:36,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t glusterfs flier01.qa.lab.tlv.redhat.com:/Daffi /rhev/data-center/mnt/flier01.qa.lab.tlv.redhat.com:_Daffi' (cwd None)
Thread-1552::ERROR::2012-12-30 16:44:36,070::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-1739::DEBUG::2012-12-30 16:49:43,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t glusterfs flier01.qa.lab.tlv.redhat.com:/daffi_test /rhev/data-center/mnt/flier01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)
Thread-1739::ERROR::2012-12-30 16:49:44,027::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-1820::DEBUG::2012-12-30 16:51:58,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs flier01.qa.lab.tlv.redhat.com:/daffi_test /rhev/data-center/mnt/flier01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)
Thread-1820::ERROR::2012-12-30 16:51:58,127::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-1836::DEBUG::2012-12-30 16:52:19,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs flier01.qa.lab.tlv.redhat.com:/daffi_test /rhev/data-center/mnt/flier01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)
Thread-1836::ERROR::2012-12-30 16:52:19,498::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-1947::DEBUG::2012-12-30 16:55:19,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t glusterfs filer01.qa.lab.tlv.redhat.com:/daffi_test /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)
Thread-1947::ERROR::2012-12-30 16:55:19,496::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-2345::DEBUG::2012-12-30 17:06:32,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 filer01.qa.lab.tlv.redhat.com:/daffi_test /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)


Thread-2345::INFO::2012-12-30 17:06:33,009::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '0754c137-6fe1-4927-b9ee-a2576dea93db'}]}

Thread-2402::DEBUG::2012-12-30 17:07:57,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs filer01.qa.lab.tlv.redhat.com:/daffi_test /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)
Thread-2402::ERROR::2012-12-30 17:10:02,816::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-2861::DEBUG::2012-12-30 17:20:44,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 filer01.qa.lab.tlv.redhat.com:/daffi_test /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)
Thread-2861::INFO::2012-12-30 17:20:44,857::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]}

Thread-3154::DEBUG::2012-12-30 17:28:51,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs filer01.qa.lav.tlv.redhat.com:/daffi /rhev/data-center/mnt/filer01.qa.lav.tlv.redhat.com:_daffi' (cwd None)
Thread-3154::ERROR::2012-12-30 17:28:52,029::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-3254::DEBUG::2012-12-30 17:31:50,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs filer01.qa.lab.tlv.redhat.com:/daffi_test /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)
Thread-3254::ERROR::2012-12-30 17:33:55,992::hsm::2210::Storage.HSM::(connectStorageServer) Could not connect to storageServer

Thread-3362::DEBUG::2012-12-30 17:34:52,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 filer01.qa.lab.tlv.redhat.com:/daffi_test /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)
Thread-3362::INFO::2012-12-30 17:34:52,821::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]}

Thread-3375::DEBUG::2012-12-30 17:35:06,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 filer01.qa.lab.tlv.redhat.com:/daffi_test /rhev/data-center/mnt/filer01.qa.lab.tlv.redhat.com:_daffi__test' (cwd None)
Thread-3375::INFO::2012-12-30 17:35:06,750::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '84987064-3130-44f8-9af1-c9f965297440'}]}
Comment 6 Federico Simoncelli 2013-01-21 14:03:07 UTC 
When the administrator wants to connect to an NFS domain (declaring it as a posixfs domain) he should explicitly specify the version (when needed) and all the other required options as well.

Based on the attached logs I can see that the nfs connection (when specified as posixfs) always failed against the gluster server because of a version mismatch (version 4 as default instead of version 3).

I'm strongly against introducing any kind of logic in the posixfs implementation that would try to catch these specific issues. The posixfs domains were introduced to give the flexibility to the administrator to use different types of distributed file systems (samba, etc...) but with this flexibility comes also the responsibility of using the proper configuration/options.

Moving to ON_QA to double check.
Comment 7 Dafna Ron 2013-01-21 15:00:02 UTC 
I tested again. 
it seems that the problem is indeed on gluster side. 
closing the bug