Bug 966555

Summary: [G4S]: gluster-swift does not properly handle the transport endpoint not connected state
Product: [Community] Gluster-Swift Reporter: pushpesh sharma <psharma>
Component: object-serverAssignee: Nobody <nobody>
Status: ASSIGNED --- QA Contact: SATHEESARAN <sasundar>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 1.8.0CC: bugs, david.macdonald, sasundar
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
var logs none

Description pushpesh sharma 2013-05-23 13:28:23 UTC 
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.I ran the functional swift test script on setup, and most of them resulted in "Internal Server Error".
2.On closer inspection of /var/log/messages,I found the 'test' volume is in the transport endpoint not connected state.  
3.Gluster swift is not able to handle this properly and try to mount the volume again, which results in failure to mount the volume. 

Actual results:


May 23 20:39:18 dhcp207-196 container-server 127.0.0.1 - - [23/May/2013:15:09:18 +0000] "HEAD /test/0/AUTH_test/483c5bd53cdb4a8289a239445e21c427" 204 - "tx374b697ba1694b98a172f93632edc278" "-" "-" 0.0026
May 23 20:39:18 dhcp207-196 object-server STDOUT: ERROR:root:Close failed on 7 err: Transport endpoint is not connected#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/gluster/swift/common/fs_utils.py", line 100, in do_close#012    os.close(fd)#012OSError: [Errno 107] Transport endpoint is not connected (txn: tx374b697ba1694b98a172f93632edc278)
May 23 20:39:18 dhcp207-196 object-server STDOUT: ERROR:root:Unlink failed on /mnt/gluster-object/test/483c5bd53cdb4a8289a239445e21c427/.04afdc1f967e4f0abdec5a4eed7611f6.26a0638f626f5f07cb7529dd4fb95522 err: Transport endpoint is not connected#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/gluster/swift/common/fs_utils.py", line 110, in do_unlink#012    os.unlink(path)#012OSError: [Errno 107] Transport endpoint is not connected: '/mnt/gluster-object/test/483c5bd53cdb4a8289a239445e21c427/.04afdc1f967e4f0abdec5a4eed7611f6.26a0638f626f5f07cb7529dd4fb95522' (txn: tx374b697ba1694b98a172f93632edc278)


May 23 20:39:34 dhcp207-196 account-server STDOUT: ERROR:root:Busy wait for mount timed out for mount /mnt/gluster-object/test (txn: tx56cc65ab103d43ef97f57f050f4baae3)
May 23 20:39:34 dhcp207-196 account-server STDOUT: ERROR:root:Mount failed glusterfs: mount -t glusterfs localhost:test /mnt/gluster-object/test (txn: tx56cc65ab103d43ef97f57f050f4baae3)
May 23 20:39:34 dhcp207-196 account-server 127.0.0.1 - - [23/May/2013:15:09:34 +0000] "GET /test/0/AUTH_test" 507 - "tx56cc65ab103d43ef97f57f050f4baae3" "-" "-" 10.1274 ""

Expected results:


Additional info:
Comment 1 pushpesh sharma 2013-05-23 13:39:08 UTC 
Created attachment 752222 [details]
var logs
Comment 2 Junaid 2013-05-24 07:42:58 UTC 
Peter/Luis,

Do you think we should handle the error "Transport end-point not connected" error specially and call umount?