Bug 1317790

Summary: Cache swift xattrs
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Prashanth Pai <ppai>
Component: fuseAssignee: Bug Updates Notification Mailing List <rhs-bugs>
Status: CLOSED ERRATA QA Contact: surabhi <sbhaloth>
Severity: low Docs Contact:
Priority: high    
Version: rhgs-3.1CC: asrivast, bugs, chrisw, csaba, ndevos, nlevinki, rcyriac, rhinduja, sasundar
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.1.3   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: glusterfs-3.7.9-2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1317788 Environment:
Last Closed: 2016-06-23 05:10:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1317785, 1317788    
Bug Blocks: 1299184, 1314171, 1328444    
Attachments:
Description Flags
pcap to show the reduced xattr calls while PUT and Delete none

Description Prashanth Pai 2016-03-15 08:40:25 UTC 
+++ This bug was initially created as a clone of Bug #1317788 +++

+++ This bug was initially created as a clone of Bug #1317785 +++

Description of problem:

gluster-swift extensively uses "user.swift.metadata" xattr in it's functionality. This is not cached and hence the performance is usually terrible.

Every operation (GET,PUT,POST,DELETE,HEAD) in gluster-swift issues getxattr() as follows:

getxattr("/mnt/gluster-object/test/c1/o8", "user.swift.metadata", 0x0, 0) = 190
getxattr("/mnt/gluster-object/test/c1/o8", "user.swift.metadata", "{"Content-Length":"11","ETag":"5eb63bbbe01eeed093cb22bb8f5acdc3","X-Timestamp":"1458021187.45380","X-Object-Type":"file","X-Type":"Object","Content-Type":"application/x-www-form-urlencoded"}", 190) = 190

The first getxattr() is to get the size of xattr value and the second getxattr() is to fetch the actual value itself.

The entire xattr information is sent thrice by brick to FUSE mount. Once on lookup, again on the first getxattr() to get size and then again on second gexattr() call. These three network calls can be reduced to just one if the xattr information is cached when it's fetched for the first time.

--- Additional comment from Prashanth Pai on 2016-03-15 04:34 EDT ---

Attached tcpdump trace of DELETE object operation from gluster-swift
Comment 3 Prashanth Pai 2016-03-15 09:24:04 UTC 
Upstream change on review: http://review.gluster.org/13735
Comment 5 Mike McCune 2016-03-28 23:26:37 UTC 
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Comment 9 surabhi 2016-05-09 14:19:14 UTC 
Performed GET,PUT and DELETE operations and saw only one getxattr calls happening for the specific operations which may improve the performance. Currently verified on the basis os reduced xattr calls. Will take up the basic perf in regression cycle. marking the BZ verified.

[pid 19074] open("/mnt/gluster-object/test/c1/o1", O_RDONLY|O_CLOEXEC) = 10
[pid 19074] fstat(10, {st_mode=S_IFREG|0755, st_size=8, ...}) = 0
[pid 19074] fgetxattr(10, "user.swift.metadata", 0x0, 0) = 178
[pid 19074] fgetxattr(10, "user.swift.metadata", "{"Content-Length":8,"ETag":"15fe514867dd5b4a1abf91ea35ff9e22","X-Object-Type":"file","X-Timestamp":"1462801876.06361","X-Type":"Object","Content-Type":"application/octet-stream"}", 178) = 178
[pid 19074] close(10)                   = 0
[pid 19074] setxattr("/mnt/gluster-object/test/c1/o1", "user.swift.metadata", "{"Content-Length":8,"ETag":"15fe514867dd5b4a1abf91ea35ff9e22","X-Timestamp":"1462802038.60981","X-Object-Type":"file","X-Type":"Object","Content-Type":"application/octet-stream","X-Object-Meta-Hello":"World"}", 208, 0) = 0
[pid 19074] poll([{fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP}, {fd=13, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 2, 0) = 0 (Timeout)
[pid 19074] sendto(6, "HTTP/1.1 202 Accepted\r\nContent-L"..., 217, 0, NULL, 0) = 217
[pid 19074] shutdown(6, SHUT_RDWR)      = 0
[pid 19074] close(6)                    = 0
[pid 19074] poll([{fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP}, {fd=13, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 2, 60000
Comment 10 surabhi 2016-05-10 11:26:09 UTC 
Created attachment 1155681 [details]
pcap to show the reduced xattr calls while PUT and Delete
Comment 12 errata-xmlrpc 2016-06-23 05:10:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240