Bug 1312810

Summary: [RHEL-7] gluster-swift [PERF]: Account and container listings fetch metadata from FS even when it's not needed
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Prashanth Pai <ppai>
Component: gluster-swiftAssignee: Prashanth Pai <ppai>
Status: CLOSED ERRATA QA Contact: surabhi <sbhaloth>
Severity: medium Docs Contact:
Priority: high    
Version: rhgs-3.1CC: nlevinki, rcyriac, rhinduja, rhs-bugs, thiago
Target Milestone: ---Keywords: FutureFeature, ZStream
Target Release: RHGS 3.1.3   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: swiftonfile-2.3.0-1 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 1328446 (view as bug list) Environment:
Last Closed: 2016-06-23 05:31:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1299184, 1328446    

Description Prashanth Pai 2016-02-29 09:52:05 UTC 
Problem:
Currently, for container and object listing, gluster-swift does a lot of unnecessary stat() and getxattr() even though that information is never gonna be consumed by clients:

Fix:
Fetch metadata (xattr) for containers in an account ONLY when the client
asks for it (using content-type indicating JSON or XML response).
This avoids a lot of unnecessarry stat() and getxattr() calls whose
results would anyways be unused. The performance gain is obvious in this
case.

Some history:
https://bugzilla.redhat.com/show_bug.cgi?id=911448
Comment 3 Prashanth Pai 2016-03-10 09:13:28 UTC 
Upstream changes on review:
http://review.gluster.org/#/c/13497/
http://review.gluster.org/#/c/13573/
Comment 4 Mike McCune 2016-03-28 23:21:47 UTC 
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Comment 5 Prashanth Pai 2016-04-06 15:23:15 UTC 
Downstream changes (MERGED):
https://code.engineering.redhat.com/gerrit/#/c/71560/
https://code.engineering.redhat.com/gerrit/#/c/71561/
Comment 6 surabhi 2016-05-10 13:10:03 UTC 
├── c1
│   ├── file1
│   └── o1
├── c1o1
├── c2
├── container1
│   ├── file1
│   ├── file2
│   ├── file3
│   ├── file4
│   ├── file5
│   └── file6
├── container2
│   ├── file1
│   ├── file2
│   ├── file3
│   ├── file4
│   ├── file5
│   └── file6
├── container3
│   ├── file3
│   ├── file4
│   ├── file5
│   └── file6
├── container4
│   ├── file3
│   └── file5
└── container5
    ├── file3
    ├── file4
    └── file5


curl -i http://localhost:8080/v1/AUTH_test/ -X GET


HTTP/1.1 200 OK
Content-Length: 66
X-Account-Container-Count: 8
Accept-Ranges: bytes
X-Account-Object-Count: 0
X-Bytes-Used: 0
X-Timestamp: 1462793677.99757
X-Object-Count: 0
X-Account-Bytes-Used: 0
X-Type: Account
Content-Type: text/plain; charset=utf-8
X-Container-Count: 8
X-Trans-Id: txbf76bcae56db4096bd922-005731ccfa
Date: Tue, 10 May 2016 11:58:50 GMT

c1
c1o1
c2
container1
container2
container3
container4
container5


 ps aux | grep account
root     19050  0.0  0.2 271472 21112 ?        Ss   Apr22   0:00 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
root     19073  0.0  0.2 272116 18648 ?        S    Apr22   0:06 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf

[root@dhcp46-62 test]# strace -ff -p 19073
Process 19073 attached
restart_syscall(<... resuming interrupted call ...>) = 1
accept(4, {sa_family=AF_INET, sin_port=htons(42070), sin_addr=inet_addr("127.0.0.1")}, [16]) = 6
fcntl(6, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
fcntl(6, F_GETFL)                       = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
accept(4, 0x7ffe99d528f0, [16])         = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(6, "GET /test/0/AUTH_test HTTP/1.1\r\n"..., 8192, 0, NULL, NULL) = 276
getsockname(6, {sa_family=AF_INET, sin_port=htons(6012), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
lstat("/mnt/gluster-object/test", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/mnt/gluster-object/test/..", {st_mode=S_IFDIR|0755, st_size=63, ...}) = 0
stat("/mnt/gluster-object/test", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getxattr("/mnt/gluster-object/test", "user.swift.metadata", 0x0, 0) = 172
getxattr("/mnt/gluster-object/test", "user.swift.metadata", "{"X-Bytes-Used":[0,0],"X-Timestamp":["1462793677.99757",0],"X-Object-Count":[0,0],"X-Type":["Account",0],"X-PUT-Timestamp":["1461656152.85326",0],"X-Container-Count":[8,0]}", 172) = 172
stat("/mnt/gluster-object/test", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/mnt/gluster-object/test", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/mnt/gluster-object/test", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 10
getdents(10, /* 11 entries */, 32768)   = 312
getdents(10, /* 0 entries */, 32768)    = 0
close(10)                               = 0
stat("/mnt/gluster-object/test/container1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/mnt/gluster-object/test/c2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/mnt/gluster-object/test/container3", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/mnt/gluster-object/test/container4", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/mnt/gluster-object/test/c1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/mnt/gluster-object/test/c1o1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/mnt/gluster-object/test/container2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/mnt/gluster-object/test/container5", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
sendto(6, "HTTP/1.1 200 OK\r\nContent-Length:"..., 425, 0, NULL, 0) = 425
shutdown(6, SHUT_RDWR)                  = 0

There are only 2 getxattr call on the account and not on each of the containers as it was with previous versions.

***********************************************************
ps aux | grep container
root     19049  0.0  0.2 278332 21520 ?        Ss   Apr22   0:00 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
root     19072  0.0  0.2 278916 18964 ?        S    Apr22   0:06 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
root     20967  0.0  0.0 112644   956 pts/1    S+   13:07   0:00 grep --color=auto container

[root@dhcp46-62 ~]# curl -i http://localhost:8080/v1/AUTH_test/container1 -X GET
HTTP/1.1 200 OK
Content-Length: 36
X-Container-Object-Count: 6
Accept-Ranges: bytes
X-Storage-Policy: Policy-0
X-Container-Bytes-Used: 0
X-Timestamp: 0000000001.00000
Content-Type: text/plain; charset=utf-8
X-Trans-Id: txcb4ccfbd09bb40f1a59ba-005731dd33
Date: Tue, 10 May 2016 13:08:03 GMT

file1
file2
file3
file4
file5
file6


recvfrom(5, "GET /test/0/AUTH_test/container1"..., 8192, 0, NULL, NULL) = 297
getsockname(5, {sa_family=AF_INET, sin_port=htons(6011), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
lstat("/mnt/gluster-object/test", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/mnt/gluster-object/test/..", {st_mode=S_IFDIR|0755, st_size=63, ...}) = 0
stat("/mnt/gluster-object/test/container1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getxattr("/mnt/gluster-object/test/container1", "user.swift.metadata", 0x0, 0) = 148
getxattr("/mnt/gluster-object/test/container1", "user.swift.metadata", "{"X-Object-Count":[6,0],"X-Timestamp":["1462799867.42137",0],"X-Type":["container",0],"X-PUT-Timestamp":["1462799867.43499",0],"X-Bytes-Used":[0,0]}", 148) = 148
stat("/mnt/gluster-object/test/container1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/mnt/gluster-object/test/container1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/mnt/gluster-object/test/container1", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 6
getdents(6, /* 8 entries */, 32768)     = 240
getdents(6, /* 0 entries */, 32768)     = 0
close(6)                                = 0
stat("/mnt/gluster-object/test/container1/file3", {st_mode=S_IFREG|0644, st_size=1048576, ...}) = 0
stat("/mnt/gluster-object/test/container1/file4", {st_mode=S_IFREG|0644, st_size=1048576, ...}) = 0
stat("/mnt/gluster-object/test/container1/file1", {st_mode=S_IFREG|0755, st_size=1073741824, ...}) = 0
stat("/mnt/gluster-object/test/container1/file2", {st_mode=S_IFREG|0644, st_size=1073741824, ...}) = 0
stat("/mnt/gluster-object/test/container1/file5", {st_mode=S_IFREG|0644, st_size=1048576, ...}) = 0
stat("/mnt/gluster-object/test/container1/file6", {st_mode=S_IFREG|0644, st_size=1048576, ...}) = 0
poll([{fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
sendto(5, "HTTP/1.1 200 OK\r\nContent-Length:"..., 504, 0, NULL, 0) = 504
shutdown(5, SHUT_RDWR)                  = 0
close(5)                                = 0
poll([{fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 60000


Verified with account and container listing.There are less getxattr calls. There is no extensive perf testing done. marking the BZ verified as per output of strace.
Comment 8 errata-xmlrpc 2016-06-23 05:31:07 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2016:1289