Bug 1351637

Summary: [RHEL-7] Metadata validation always fails for marker dir objects during GET
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Prashanth Pai <ppai>
Component: gluster-swiftAssignee: Prashanth Pai <ppai>
Status: CLOSED ERRATA QA Contact: surabhi <sbhaloth>
Severity: low Docs Contact:
Priority: low    
Version: rhgs-3.1CC: amukherj, rcyriac, rhinduja, rhs-bugs, thiago
Target Milestone: Development Freeze   
Target Release: RHGS 3.2.0   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: swiftonfile-2.3.0-4.el7rhgs Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1378421 (view as bug list) Environment:
Last Closed: 2017-03-23 05:13:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1351522, 1378421    

Description Prashanth Pai 2016-06-30 13:02:10 UTC 
Description of problem:
For marker directory objects, the validate function always returns False. This is because we compare the stored Content-Length (always zero) and the st_size from stat (always 4096 for directories). As this comparison fails every time, validation is unsuccessful.


Version-Release number of selected component (if applicable):
RHGS 3.1.3
swiftonfile-2.3.0-3.el7rhgs


How reproducible:
Always. Do a GET request on a marker dir object and observe strace output.

Steps to Reproduce:
Create marker dir object:
curl -i http://localhost:8080/v1/AUTH_test/c1/dirobject -X PUT -H 'Content-Type: application/directory' -H 'Content-Length: 0'
HTTP/1.1 201 Created
Last-Modified: Thu, 30 Jun 2016 12:50:59 GMT
Content-Length: 0
Etag: d41d8cd98f00b204e9800998ecf8427e
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txf93b00f66fd542eb91c13-00577515b2
Date: Thu, 30 Jun 2016 12:50:58 GMT

[pid  1427] getxattr("/mnt/gluster-object/test/c1/dirobject", "user.swift.metadata", 0x0, 0) = -1 ENOENT (No such file or directory)
[pid  1427] open("/dev/urandom", O_RDONLY|O_CLOEXEC) = 7
[pid  1427] fcntl(7, F_GETFD)           = 0x1 (flags FD_CLOEXEC)
[pid  1427] fcntl(7, F_SETFD, FD_CLOEXEC) = 0
[pid  1427] getuid()                    = 0
[pid  1427] getppid()                   = 1398
[pid  1427] read(7, "\272\1\354\210Dlh\323$\335^<\373Z\36\223", 16) = 16
[pid  1427] close(7)                    = 0
[pid  1427] gettid()                    = 1427
[pid  1427] open("/mnt/gluster-object/test/c1/.dirobject.740831a72d3a4baaba34788f757c0976", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0777) = 7
[pid  1427] futex(0x7f24bc0014f0, FUTEX_WAKE_PRIVATE, 1) = 1
[pid  1427] poll([{fd=9, events=POLLIN|POLLPRI|POLLERR|POLLHUP}, {fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 2, 60000 <unfinished ...>
[pid  1457] <... futex resumed> )       = 0
[pid  1457] mkdir("/mnt/gluster-object/test/c1/dirobject", 0777) = 0
[pid  1457] stat("/mnt/gluster-object/test/c1/dirobject", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid  1457] setxattr("/mnt/gluster-object/test/c1/dirobject", "user.swift.metadata", "{\"Content-Length\":\"0\",\"ETag\":\"d41d8cd98f00b204e9800998ecf8427e\",\"X-Timestamp\":\"1467291058.49158\",\"X-Object-Type\":\"marker_dir\",\"X-Type\":\"Object\",\"Content-Type\":\"application/directory\"}", 183, 0) = 0
[pid  1457] sendto(10, " ", 1, 0, NULL, 0) = 1
[pid  1427] <... poll resumed> )        = 1 ([{fd=9, revents=POLLIN}])
[pid  1427] recvfrom(9, " ", 1, 0, NULL, NULL) = 1
[pid  1457] futex(0x7f24bc0014f0, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid  1427] recvfrom(9, 0x213d8c4, 1, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid  1427] close(7)                    = 0
[pid  1427] unlink("/mnt/gluster-object/test/c1/.dirobject.740831a72d3a4baaba34788f757c0976") = 0

Do a GET on marker dir object:

root# curl -i http://localhost:8080/v1/AUTH_test/c1/dirobject -X GET
HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Last-Modified: Thu, 30 Jun 2016 12:50:59 GMT
Etag: d41d8cd98f00b204e9800998ecf8427e
X-Timestamp: 1467291058.57983
Content-Type: application/directory
X-Trans-Id: tx9ee33b8ea36d4d84bdeee-00577515dd
Date: Thu, 30 Jun 2016 12:51:41 GMT

[pid  1427] open("/mnt/gluster-object/test/c1/dirobject", O_RDONLY|O_CLOEXEC) = 7
[pid  1427] fstat(7, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid  1427] fgetxattr(7, "user.swift.metadata", 0x0, 0) = 183
[pid  1427] fgetxattr(7, "user.swift.metadata", "{\"Content-Length\":\"0\",\"ETag\":\"d41d8cd98f00b204e9800998ecf8427e\",\"X-Timestamp\":\"1467291058.49158\",\"X-Object-Type\":\"marker_dir\",\"X-Type\":\"Object\",\"Content-Type\":\"application/directory\"}", 183) = 183
[pid  1427] fsetxattr(7, "user.swift.metadata", "{\"Content-Length\":0,\"ETag\":\"d41d8cd98f00b204e9800998ecf8427e\",\"X-Object-Type\":\"dir\",\"X-Timestamp\":\"1467291058.57983\",\"X-Type\":\"Object\",\"Content-Type\":\"application/directory\"}", 174, 0) = 0
[pid  1427] close(7)                    = 0

Actual results:
Validation fails and metadata is over-written unnecessarily.


Expected results:
Validation should not fail and there should be no fsetxattr() call.
Comment 2 Prashanth Pai 2016-07-28 13:29:07 UTC 
Upstream patch: http://review.gluster.org/#/c/14423/
Comment 4 Prashanth Pai 2016-09-13 07:43:31 UTC 
Upstream change has been merged.
Comment 5 Prashanth Pai 2016-09-19 11:30:02 UTC 
Downstream changes have been submitted and merged.

rhel7 - https://code.engineering.redhat.com/gerrit/85042
rhel6 - https://code.engineering.redhat.com/gerrit/85059
Comment 9 surabhi 2016-12-15 10:43:16 UTC 
Verified on marker-objects, on doing a GET on marker object there is not setxattr call for such objects.

curl -i http://localhost:8080/v1/AUTH_test/dir1/dirobject2 -X GET

strace -ff -p 11860,11883

[pid 11883] lstat("/mnt/gluster-object/test", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 11883] lstat("/mnt/gluster-object/test/..", {st_mode=S_IFDIR|0755, st_size=63, ...}) = 0
[pid 11883] open("/mnt/gluster-object/test/dir1/dirobject", O_RDONLY|O_CLOEXEC) = 7
[pid 11883] fstat(7, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 11883] fgetxattr(7, "user.swift.metadata", 0x0, 0) = 183
[pid 11883] fgetxattr(7, "user.swift.metadata", "{"Content-Length":"0","ETag":"d41d8cd98f00b204e9800998ecf8427e","X-Timestamp":"1481798400.00158","X-Object-Type":"marker_dir","X-Type":"Object","Content-Type":"application/directory"}", 183) = 183
[pid 11883] close(7)                    = 0
[pid 11883] sendto(5, "HTTP/1.1 200 OK\r\nContent-Length:"..., 289, 0, NULL, 0) = 289
[pid 11883] shutdown(5, SHUT_RDWR)      = 0
[pid 11883] close(5)                    = 0
[pid 11883] poll([{fd=12, events=POLLIN|POLLPRI|POLLERR|POLLHUP}, {fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 2, 60000) = 1 ([{fd=4, revents=POLLIN}])
[pid 11883] accept(4, {sa_family=AF_INET, sin_port=htons(37488), sin_addr=inet_addr("127.0.0.1")}, [16]) = 5
[pid 11883] fcntl(5, F_GETFL)           = 0x2 (flags O_RDWR)
[pid 11883] fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 11883] fcntl(5, F_GETFL)           = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 11883] fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 11883] accept(4, 0x7ffda869aa90, [16]) = -1 EAGAIN (Resource temporarily unavailable)
[pid 11883] recvfrom(5, "GET /test/0/AUTH_test/dir1/dirob"..., 65536, 0, NULL, NULL) = 342
[pid 11883] getsockname(5, {sa_family=AF_INET, sin_port=htons(6010), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
[pid 11883] lstat("/mnt/gluster-object/test", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 11883] lstat("/mnt/gluster-object/test/..", {st_mode=S_IFDIR|0755, st_size=63, ...}) = 0
[pid 11883] open("/mnt/gluster-object/test/dir1/dirobject2", O_RDONLY|O_CLOEXEC) = 7
[pid 11883] fstat(7, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 11883] fgetxattr(7, "user.swift.metadata", 0x0, 0) = 183
[pid 11883] fgetxattr(7, "user.swift.metadata", "{"Content-Length":"0","ETag":"d41d8cd98f00b204e9800998ecf8427e","X-Timestamp":"1481798452.64525","X-Object-Type":"marker_dir","X-Type":"Object","Content-Type":"application/directory"}", 183) = 183
[pid 11883] close(7)                    = 0
[pid 11883] sendto(5, "HTTP/1.1 200 OK\r\nContent-Length:"..., 289, 0, NULL, 0) = 289
[pid 11883] shutdown(5, SHUT_RDWR)      = 0
[pid 11883] close(5)                    = 0
[pid 11883] poll([{fd=12, events=POLLIN|POLLPRI|POLLERR|POLLHUP}, {fd=4, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 2, 60000

moving the BZ to verified.
Comment 11 errata-xmlrpc 2017-03-23 05:13:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0488.html