Problem: Currently, for container and object listing, gluster-swift does a lot of unnecessary stat() and getxattr() even though that information is never gonna be consumed by clients: Fix: Fetch metadata (xattr) for containers in an account ONLY when the client asks for it (using content-type indicating JSON or XML response). This avoids a lot of unnecessarry stat() and getxattr() calls whose results would anyways be unused. The performance gain is obvious in this case. Some history: https://bugzilla.redhat.com/show_bug.cgi?id=911448
Upstream changes on review: http://review.gluster.org/#/c/13497/ http://review.gluster.org/#/c/13573/
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Downstream changes (MERGED): https://code.engineering.redhat.com/gerrit/#/c/71560/ https://code.engineering.redhat.com/gerrit/#/c/71561/
├── c1 │ ├── file1 │ └── o1 ├── c1o1 ├── c2 ├── container1 │ ├── file1 │ ├── file2 │ ├── file3 │ ├── file4 │ ├── file5 │ └── file6 ├── container2 │ ├── file1 │ ├── file2 │ ├── file3 │ ├── file4 │ ├── file5 │ └── file6 ├── container3 │ ├── file3 │ ├── file4 │ ├── file5 │ └── file6 ├── container4 │ ├── file3 │ └── file5 └── container5 ├── file3 ├── file4 └── file5 curl -i http://localhost:8080/v1/AUTH_test/ -X GET HTTP/1.1 200 OK Content-Length: 66 X-Account-Container-Count: 8 Accept-Ranges: bytes X-Account-Object-Count: 0 X-Bytes-Used: 0 X-Timestamp: 1462793677.99757 X-Object-Count: 0 X-Account-Bytes-Used: 0 X-Type: Account Content-Type: text/plain; charset=utf-8 X-Container-Count: 8 X-Trans-Id: txbf76bcae56db4096bd922-005731ccfa Date: Tue, 10 May 2016 11:58:50 GMT c1 c1o1 c2 container1 container2 container3 container4 container5 ps aux | grep account root 19050 0.0 0.2 271472 21112 ? Ss Apr22 0:00 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf root 19073 0.0 0.2 272116 18648 ? S Apr22 0:06 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf [root@dhcp46-62 test]# strace -ff -p 19073 Process 19073 attached restart_syscall(<... resuming interrupted call ...>) = 1 accept(4, {sa_family=AF_INET, sin_port=htons(42070), sin_addr=inet_addr("127.0.0.1")}, [16]) = 6 fcntl(6, F_GETFL) = 0x2 (flags O_RDWR) fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0 fcntl(6, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK) fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0 accept(4, 0x7ffe99d528f0, [16]) = -1 EAGAIN (Resource temporarily unavailable) recvfrom(6, "GET /test/0/AUTH_test HTTP/1.1\r\n"..., 8192, 0, NULL, NULL) = 276 getsockname(6, {sa_family=AF_INET, sin_port=htons(6012), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0 lstat("/mnt/gluster-object/test", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lstat("/mnt/gluster-object/test/..", {st_mode=S_IFDIR|0755, st_size=63, ...}) = 0 stat("/mnt/gluster-object/test", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 getxattr("/mnt/gluster-object/test", "user.swift.metadata", 0x0, 0) = 172 getxattr("/mnt/gluster-object/test", "user.swift.metadata", "{"X-Bytes-Used":[0,0],"X-Timestamp":["1462793677.99757",0],"X-Object-Count":[0,0],"X-Type":["Account",0],"X-PUT-Timestamp":["1461656152.85326",0],"X-Container-Count":[8,0]}", 172) = 172 stat("/mnt/gluster-object/test", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/mnt/gluster-object/test", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 openat(AT_FDCWD, "/mnt/gluster-object/test", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 10 getdents(10, /* 11 entries */, 32768) = 312 getdents(10, /* 0 entries */, 32768) = 0 close(10) = 0 stat("/mnt/gluster-object/test/container1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/mnt/gluster-object/test/c2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/mnt/gluster-object/test/container3", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/mnt/gluster-object/test/container4", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/mnt/gluster-object/test/c1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/mnt/gluster-object/test/c1o1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/mnt/gluster-object/test/container2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/mnt/gluster-object/test/container5", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 sendto(6, "HTTP/1.1 200 OK\r\nContent-Length:"..., 425, 0, NULL, 0) = 425 shutdown(6, SHUT_RDWR) = 0 There are only 2 getxattr call on the account and not on each of the containers as it was with previous versions. *********************************************************** ps aux | grep container root 19049 0.0 0.2 278332 21520 ? Ss Apr22 0:00 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf root 19072 0.0 0.2 278916 18964 ? S Apr22 0:06 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf root 20967 0.0 0.0 112644 956 pts/1 S+ 13:07 0:00 grep --color=auto container [root@dhcp46-62 ~]# curl -i http://localhost:8080/v1/AUTH_test/container1 -X GET HTTP/1.1 200 OK Content-Length: 36 X-Container-Object-Count: 6 Accept-Ranges: bytes X-Storage-Policy: Policy-0 X-Container-Bytes-Used: 0 X-Timestamp: 0000000001.00000 Content-Type: text/plain; charset=utf-8 X-Trans-Id: txcb4ccfbd09bb40f1a59ba-005731dd33 Date: Tue, 10 May 2016 13:08:03 GMT file1 file2 file3 file4 file5 file6 recvfrom(5, "GET /test/0/AUTH_test/container1"..., 8192, 0, NULL, NULL) = 297 getsockname(5, {sa_family=AF_INET, sin_port=htons(6011), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0 lstat("/mnt/gluster-object/test", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lstat("/mnt/gluster-object/test/..", {st_mode=S_IFDIR|0755, st_size=63, ...}) = 0 stat("/mnt/gluster-object/test/container1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 getxattr("/mnt/gluster-object/test/container1", "user.swift.metadata", 0x0, 0) = 148 getxattr("/mnt/gluster-object/test/container1", "user.swift.metadata", "{"X-Object-Count":[6,0],"X-Timestamp":["1462799867.42137",0],"X-Type":["container",0],"X-PUT-Timestamp":["1462799867.43499",0],"X-Bytes-Used":[0,0]}", 148) = 148 stat("/mnt/gluster-object/test/container1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/mnt/gluster-object/test/container1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 openat(AT_FDCWD, "/mnt/gluster-object/test/container1", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 6 getdents(6, /* 8 entries */, 32768) = 240 getdents(6, /* 0 entries */, 32768) = 0 close(6) = 0 stat("/mnt/gluster-object/test/container1/file3", {st_mode=S_IFREG|0644, st_size=1048576, ...}) = 0 stat("/mnt/gluster-object/test/container1/file4", {st_mode=S_IFREG|0644, st_size=1048576, ...}) = 0 stat("/mnt/gluster-object/test/container1/file1", {st_mode=S_IFREG|0755, st_size=1073741824, ...}) = 0 stat("/mnt/gluster-object/test/container1/file2", {st_mode=S_IFREG|0644, st_size=1073741824, ...}) = 0 stat("/mnt/gluster-object/test/container1/file5", {st_mode=S_IFREG|0644, st_size=1048576, ...}) = 0 stat("/mnt/gluster-object/test/container1/file6", {st_mode=S_IFREG|0644, st_size=1048576, ...}) = 0 poll([{fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout) sendto(5, "HTTP/1.1 200 OK\r\nContent-Length:"..., 504, 0, NULL, 0) = 504 shutdown(5, SHUT_RDWR) = 0 close(5) = 0 poll([{fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 60000 Verified with account and container listing.There are less getxattr calls. There is no extensive perf testing done. marking the BZ verified as per output of strace.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2016:1289