Bug 960961 - G4S: Nested directories and files are not listed with GET calls which results in failure of some swift upstream functional test cases
G4S: Nested directories and files are not listed with GET calls which results...
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: object-storage (Show other bugs)
3.3.1
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Luis Pabón
pushpesh sharma
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-08 07:55 EDT by pushpesh sharma
Modified: 2016-11-08 17:24 EST (History)
4 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-24 13:43:57 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description pushpesh sharma 2013-05-08 07:55:34 EDT
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:(via curl) 

1. make some dir structure (all PUT calls results in success)
#curl -v -X PUT -T proxy-server.conf -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/dir1/test

#curl -v -X PUT -T proxy-server.conf -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/dir2/test
#curl -v -X PUT -T proxy-server.conf -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/dir3/test

#curl -v -X PUT -T proxy-server.conf -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/dir4/test

#curl -v -X PUT -T proxy-server.conf -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/dir5/test

#curl -v -X PUT -T proxy-server.conf -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/test

#curl -v -X PUT -T proxy-server.conf -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/dir5/dir/test

#curl -v -X PUT -T proxy-server.conf -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/dir5/dir/dir/dir/dir/dir/test

#curl -v -X PUT -T proxy-server.conf -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/dir5/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/dir/test


2. start listing directories 
a.) List root directory(container): success 
#curl -v -X GET -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2?delimiter=/

* About to connect() to 10.65.207.5 port 443 (#0)
*   Trying 10.65.207.5...
* connected
* Connected to 10.65.207.5 (10.65.207.5) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* 	subject: O=Default Company Ltd,L=Default City,C=XX
* 	start date: May 08 15:24:51 2013 GMT
* 	expire date: Jun 07 15:24:51 2013 GMT
* 	common name: (nil)
* 	issuer: O=Default Company Ltd,L=Default City,C=XX
> GET /v1/AUTH_test2/dir2?delimiter=/ HTTP/1.1
> User-Agent: curl/7.27.0
> Host: 10.65.207.5
> Accept: */*
> X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963
> 
< HTTP/1.1 200 OK
< X-Container-Object-Count: 28
< X-Timestamp: 1
< X-Container-Bytes-Used: 10926
< Accept-Ranges: bytes
< Content-Length: 30
< Content-Type: text/plain; charset=utf-8
< Date: Wed, 08 May 2013 17:07:59 GMT
< 
dir1
dir2
dir3
dir4
dir5
test
* Connection #0 to host 10.65.207.5 left intact
* Closing connection #0


b.) List nested directories

# curl -v -X GET -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/dir5/?delimiter=/

* About to connect() to 10.65.207.5 port 443 (#0)
*   Trying 10.65.207.5...
* connected
* Connected to 10.65.207.5 (10.65.207.5) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* 	subject: O=Default Company Ltd,L=Default City,C=XX
* 	start date: May 08 15:24:51 2013 GMT
* 	expire date: Jun 07 15:24:51 2013 GMT
* 	common name: (nil)
* 	issuer: O=Default Company Ltd,L=Default City,C=XX
> GET /v1/AUTH_test2/dir2/dir5/?delimiter=/ HTTP/1.1
> User-Agent: curl/7.27.0
> Host: 10.65.207.5
> Accept: */*
> X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963
> 
< HTTP/1.1 503 Internal Server Error
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Date: Wed, 08 May 2013 17:11:18 GMT
< 
* Connection #0 to host 10.65.207.5 left intact
* Closing connection #0

c.) try again with empty nested directory

# curl -v -X GET -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/dir1/?delimiter=/
* About to connect() to 10.65.207.5 port 443 (#0)
*   Trying 10.65.207.5...
* connected
* Connected to 10.65.207.5 (10.65.207.5) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* 	subject: O=Default Company Ltd,L=Default City,C=XX
* 	start date: May 08 15:24:51 2013 GMT
* 	expire date: Jun 07 15:24:51 2013 GMT
* 	common name: (nil)
* 	issuer: O=Default Company Ltd,L=Default City,C=XX
> GET /v1/AUTH_test2/dir2/dir1/?delimiter=/ HTTP/1.1
> User-Agent: curl/7.27.0
> Host: 10.65.207.5
> Accept: */*
> X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963
> 
< HTTP/1.1 503 Internal Server Error
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Date: Wed, 08 May 2013 17:12:33 GMT
< 
* Connection #0 to host 10.65.207.5 left intact
* Closing connection #0
  

Actual results:(Of Swift Functional tests)

FAIL: testContainerListing (test.functional.tests.TestContainerPaths)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/psharma/git/swift/test/functional/tests.py", line 805, in testContainerListing
    self.assertEquals(files, self.env.stored_files)
AssertionError: Lists differ: ['dir1', 'dir1/file2', 'dir1/s... != ['/dir1/', '/dir1/file2', '/di...

First differing element 0:
dir1
/dir1/

Second list contains 17 additional elements.
First extra element 22:
dir1/subdir+with{whatever/

Diff is 1441 characters long. Set self.maxDiff to None to see it.
-------------------- >> begin captured logging << --------------------
swiftclient: DEBUG: REQ: curl -i https://10.65.207.9:443/v1/AUTH_test -X GET

swiftclient: DEBUG: RESP STATUS: 200

--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: testStructure (test.functional.tests.TestContainerPaths)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/psharma/git/swift/test/functional/tests.py", line 820, in testStructure
    assert_listing('/', ['/dir1/', '/dir2/', '/file1', '/file A'])
  File "/home/psharma/git/swift/test/functional/tests.py", line 818, in assert_listing
    self.assertEquals(sorted(list, cmp=locale.strcoll), files)
AssertionError: Lists differ: ['/dir1/', '/dir2/', '/file A'... != []

First list contains 4 additional elements.
First extra element 0:
/dir1/

- ['/dir1/', '/dir2/', '/file A', '/file1']
+ []

======================================================================
FAIL: testTraverseContainer (test.functional.tests.TestContainerPaths)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/psharma/git/swift/test/functional/tests.py", line 778, in testTraverseContainer
    self.assert_(file in found_dirs)
AssertionError: False is not true





Expected results:

These cases should pass 

Additional info:
Comment 1 pushpesh sharma 2013-05-08 07:56:48 EDT
Corresponding test code of these swift functional test is as follows:-

class TestContainerPaths(Base):
    env = TestContainerPathsEnv
    set_up = False

    def testTraverseContainer(self):
        found_files = []
        found_dirs = []

        def recurse_path(path, count=0):
            if count > 10:
                raise ValueError('too deep recursion')

            for file in self.env.container.files(parms={'path': path}):
                self.assert_(file.startswith(path))
                if file.endswith('/'):
                    recurse_path(file, count + 1)
                    found_dirs.append(file)
                else:
                    found_files.append(file)

        recurse_path('')
        for file in self.env.stored_files:
            if file.startswith('/'):
                self.assert_(file not in found_dirs)
                self.assert_(file not in found_files)
            elif file.endswith('/'):
                self.assert_(file in found_dirs)
                self.assert_(file not in found_files)
            else:
                self.assert_(file in found_files)
                self.assert_(file not in found_dirs)

        found_files = []
        found_dirs = []
        recurse_path('/')
        for file in self.env.stored_files:
            if not file.startswith('/'):
                self.assert_(file not in found_dirs)
                self.assert_(file not in found_files)
            elif file.endswith('/'):
                self.assert_(file in found_dirs)
                self.assert_(file not in found_files)
            else:
                self.assert_(file in found_files)
                self.assert_(file not in found_dirs)

    def testContainerListing(self):
        for format in (None, 'json', 'xml'):
            files = self.env.container.files(parms={'format': format})

            if isinstance(files[0], dict):
                files = [str(x['name']) for x in files]

            self.assertEquals(files, self.env.stored_files)

        for format in ('json', 'xml'):
            for file in self.env.container.files(parms={'format': format}):
                self.assert_(int(file['bytes']) >= 0)
                self.assert_('last_modified' in file)
                if file['name'].endswith('/'):
                    self.assertEquals(file['content_type'],
                                      'application/directory')

    def testStructure(self):
        def assert_listing(path, list):
            files = self.env.container.files(parms={'path': path})
            self.assertEquals(sorted(list, cmp=locale.strcoll), files)
        if not normalized_urls:
            assert_listing('/', ['/dir1/', '/dir2/', '/file1', '/file A'])
            assert_listing('/dir1',
                           ['/dir1/file2', '/dir1/subdir1/', '/dir1/subdir2/'])
            assert_listing('/dir1/',
                           ['/dir1/file2', '/dir1/subdir1/', '/dir1/subdir2/'])
            assert_listing('/dir1/subdir1',
                           ['/dir1/subdir1/subsubdir2/', '/dir1/subdir1/file2',
                            '/dir1/subdir1/file3', '/dir1/subdir1/file4',
                            '/dir1/subdir1/subsubdir1/'])
            assert_listing('/dir1/subdir2', [])
            assert_listing('', ['file1', 'dir1/', 'dir2/'])
        else:
            assert_listing('', ['file1', 'dir1/', 'dir2/', 'file A'])
        assert_listing('dir1', ['dir1/file2', 'dir1/subdir1/',
                                'dir1/subdir2/', 'dir1/subdir with spaces/',
                                'dir1/subdir+with{whatever/'])
        assert_listing('dir1/subdir1',
                       ['dir1/subdir1/file4', 'dir1/subdir1/subsubdir2/',
                        'dir1/subdir1/file2', 'dir1/subdir1/file3',
                        'dir1/subdir1/subsubdir1/'])
        assert_listing('dir1/subdir1/subsubdir1',
                       ['dir1/subdir1/subsubdir1/file7',
                        'dir1/subdir1/subsubdir1/file5',
                        'dir1/subdir1/subsubdir1/file8',
                        'dir1/subdir1/subsubdir1/file6'])
        assert_listing('dir1/subdir1/subsubdir1/',
                       ['dir1/subdir1/subsubdir1/file7',
                        'dir1/subdir1/subsubdir1/file5',
                        'dir1/subdir1/subsubdir1/file8',
                        'dir1/subdir1/subsubdir1/file6'])
        assert_listing('dir1/subdir with spaces/',
                       ['dir1/subdir with spaces/file B'])
Comment 2 pushpesh sharma 2013-05-08 07:57:28 EDT
[root@dhcp207-5 swift]# rpm -qa |grep swift
glusterfs-swift-3.3.1-13.fc18.noarch
glusterfs-swift-container-3.3.1-13.fc18.noarch
glusterfs-swift-account-3.3.1-13.fc18.noarch
glusterfs-swift-object-3.3.1-13.fc18.noarch
glusterfs-swift-proxy-3.3.1-13.fc18.noarch
python-swiftclient-1.2.0-3.fc18.noarch
Comment 3 Luis Pabón 2013-06-27 00:57:42 EDT
Fixed by http://review.gluster.org/#/c/5246/
Comment 4 pushpesh sharma 2013-07-24 08:08:22 EDT
$ curl -v -X PUT -H 'X-Auth-Token: AUTH_tk44b09617995d4305a19676857a86b86e' -H 'Content-Type: application/directory' -H 'Content-Length: 0' -k http://10.65.207.210:8080/v1/AUTH_test/container/dir1-l0
* About to connect() to 10.65.207.210 port 8080 (#0)
*   Trying 10.65.207.210...
* Connected to 10.65.207.210 (10.65.207.210) port 8080 (#0)
> PUT /v1/AUTH_test/container/dir1-l0 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.65.207.210:8080
> Accept: */*
> X-Auth-Token: AUTH_tk44b09617995d4305a19676857a86b86e
> Content-Type: application/directory
> Content-Length: 0
> 
< HTTP/1.1 201 Created
< Last-Modified: Wed, 24 Jul 2013 12:02:58 GMT
< Content-Length: 0
< Etag: d41d8cd98f00b204e9800998ecf8427e
< Content-Type: text/html; charset=UTF-8
< Date: Wed, 24 Jul 2013 12:02:58 GMT
< 
* Connection #0 to host 10.65.207.210 left intact

$ curl -v -X PUT -T ~/dummy_files/1k.txt -H 'X-Auth-Token: AUTH_tk44b09617995d4305a19676857a86b86e' -k http://10.65.207.210:8080/v1/AUTH_test/container/dir1-l0/file.txt* About to connect() to 10.65.207.210 port 8080 (#0)
*   Trying 10.65.207.210...
* Connected to 10.65.207.210 (10.65.207.210) port 8080 (#0)
> PUT /v1/AUTH_test/container/dir1-l0/file.txt HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.65.207.210:8080
> Accept: */*
> X-Auth-Token: AUTH_tk44b09617995d4305a19676857a86b86e
> Content-Length: 1024
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Last-Modified: Wed, 24 Jul 2013 12:04:33 GMT
< Content-Length: 0
< Etag: 568ee44fdfc5763002295ac7f732aada
< Content-Type: text/html; charset=UTF-8
< Date: Wed, 24 Jul 2013 12:04:33 GMT
< 
* Connection #0 to host 10.65.207.210 left intact
[psharma@dhcp193-181 ~]$ curl -v -X GET -H 'X-Auth-Token: AUTH_tk44b09617995d4305a19676857a86b86e' -k http://10.65.207.210:8080/v1/AUTH_test/container/dir1-l0/file.txt
* About to connect() to 10.65.207.210 port 8080 (#0)
*   Trying 10.65.207.210...
* Connected to 10.65.207.210 (10.65.207.210) port 8080 (#0)
> GET /v1/AUTH_test/container/dir1-l0/file.txt HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.65.207.210:8080
> Accept: */*
> X-Auth-Token: AUTH_tk44b09617995d4305a19676857a86b86e
> 
< HTTP/1.1 200 OK
< Content-Length: 1024
< Accept-Ranges: bytes
< Last-Modified: Wed, 24 Jul 2013 12:04:33 GMT
< Etag: 568ee44fdfc5763002295ac7f732aada
< X-Timestamp: 1374667473.06849
< Content-Type: text/plain
< Date: Wed, 24 Jul 2013 12:05:12 GMT
< 
11:41:11.270 DEBUG [10] org.openqa.jetty.util.Container - Starting org.openqa.jetty.jetty.servlet.ServletHandler@16ba8602
11:41:11.270 DEBUG [10] org.openqa.jetty.jetty.servlet.AbstractSessionManager - New random session seed
11:41:11.310 DEBUG [11] org.openqa.jetty.jetty.servlet.AbstractSessionManager - Session scavenger period = 30s
11:41:11.310 DEBUG [10] org.openqa.jetty.jetty.servlet.Holder - Started holder of class org.openqa.selenium.remote.server.DriverServlet
11:41:11.311 INFO [10] org.openqa.jetty.util.Container - Started org.openqa.jetty.jetty.servlet.ServletHandler@16ba8602
11:41:11.311 INFO [10] org.openqa.jetty.util.Container - Started HttpContext[/wd,/wd]
11:41:11.314 INFO [10] org.openqa.jetty.http.SocketListener - Started SocketListener on 0.0.0.0:4444
jdhkjhdskjahdskjhadkhdsjh11:41:11.314 INFO [10] org.openqa.jetty.util.Container - Started org.openqa.jetty.jetty.Server@7e0c2ff5
sdhlkjsahdlkjhsakjdhsakjh11:41:44.432 INFO [12] org.openqa.selenium.server.SeleniumServer - Shutting down..dkashk.
* Connection #0 to host 10.65.207.210 left intact

Note You need to log in before you can comment on or make changes to this bug.