Created attachment 882643 [details] Program to reproduce the issue Description of problem: Meatadata operation on a fd fails, if the file is unlinked. Version-Release number of selected component (if applicable): Mainline How reproducible: Always Steps to Reproduce: 1. Open a file and obtain fd 2. Unlink the file 3. Try to perform metadata operations on the fd, like fchmod, fchown, fsetxattr... 4. Metadata operations fail with ENOENT Please find attached program to reproduce the issue.
(06:44:35 PM) bfoster: pranithk: have you identified where the ENOENT originates from? (06:45:07 PM) bfoster: I don't see it being set in that codepath, but I could certainly be missing something or looking at the wrong call, etc. (06:45:27 PM) dougw left the room (quit: Ping timeout: 600 seconds). (06:46:20 PM) pranithk: bfoster: This is what I see in strace... (06:46:23 PM) pranithk: bfoster: open("a.txt", O_RDWR|O_CREAT, 03777747144327610) = 3 (06:46:23 PM) pranithk: unlink("a.txt") = 0 (06:46:23 PM) pranithk: write(3, "abc", 3) = 3 (06:46:23 PM) pranithk: fchmod(3, 0700) = -1 ENOENT (No such file or directory) (06:46:44 PM) ansubram [~anands.redhat.com] entered the room. (06:46:47 PM) pranithk: bfoster: Now tell me what exactly I need to do to give you more information (06:47:23 PM) gdubreui is now known as gdubreui|zZZz (06:47:34 PM) skoduri left the room (quit: Ping timeout: 612 seconds). (06:47:42 PM) dahorak|mtg is now known as dahorak (06:47:53 PM) bfoster: ok... the next question is whether that error originates from fuse, glusterfs or xfs (06:47:59 PM) gdubreui|zZZz left the room (quit: Quit: Leaving). (06:48:03 PM) bfoster: you receive the request in gluster, yes? (06:48:24 PM) pranithk: bfoster: Here is what happens (06:49:15 PM) pranithk: bfoster: The call is taken as setattr on inode instead of fd. But the unlink would have removed the file, so setattr wouldn't find the file so it fails (06:50:36 PM) marek is now known as mfis_marek (06:50:42 PM) pranithk: bfoster: Write was succeeding, problem is only with metadata updates, i.e. setattr, setxattr, removexattr. It may exist even for getattr/xattr but I didn;t check that (06:50:48 PM) mfis_marek is now known as marek (06:51:29 PM) bfoster: ok, so the error originates in glusterfs because it received a request on an inode that doesn't exist (06:51:34 PM) bfoster: but is open while unlinked (06:51:40 PM) bfoster: so there is an fd somewhere (06:51:44 PM) pranithk: bfoster: Exactly (06:51:51 PM) pranithk: bfoster: yes yes (06:52:26 PM) bfoster: hmm, ok. that is more clear (06:53:06 PM) gunnage [~Ritz.redhat.com] entered the room. (06:54:52 PM) bfoster: yeah, the best fix might be to stuff the fd into the attr (06:54:56 PM) eharney [~eharney.redhat.com] entered the room. (06:55:02 PM) bfoster: perhaps just an interface mismatch between fuse and traditional fs' (06:55:06 PM) gdubreui [~gdubreui.redhat.com] entered the room. (06:55:23 PM) bfoster: this works on xfs because it only cares about the inode, the inode stays around because the open file keeps it referenced (06:55:34 PM) bfoster: which glusterfs presumably doesn't do (06:56:24 PM) bfoster: so I take it as it relies on the client fd to reference the open fd on the brick (06:56:55 PM) pranithk: bfoster: yes sir (06:57:15 PM) bfoster: pranithk: can you file a bug that explains this? (06:57:47 PM) pranithk: bfoster: There is a bug already : https://bugzilla.redhat.com/show_bug.cgi?id=1084422 (06:57:48 PM) unifiedbot: [1084422] Metadata operations on a fd (whose file is unlinked) fails [GlusterFS] [vbellur:NEW] (06:58:05 PM) msivanes [~msivanes.49.116] entered the room. (06:58:20 PM) pranithk: bfoster: Shall I copy paste part of this conversation as a comment? (06:58:24 PM) jrankin [~jrankin.redhat.com] entered the room. (06:58:36 PM) bfoster: sure, good idea (06:58:37 PM) surabhi left the room (quit: Ping timeout: 612 seconds). (06:58:37 PM) VivekAgarwal left the room (quit: Ping timeout: 612 seconds). (06:58:54 PM) pranithk: bfoster: So you are saying fuse has to send the fd down so that it can work? (06:59:06 PM) esalexa is now known as esalexa|mtg (06:59:20 PM) vkoppad left the room (quit: Remote host closed the connection). (07:00:10 PM) bfoster: pranithk: I'm guessing, I need to take a closer look
Comment above is the chat Brian and I had about this issue. The issue is that gluster depends on client fd to be passed down to perform the operations where as setattr is sent on an inode from vfs to fuse and since gluster doesn't have any reference to inode once unlink happens, this issue is seen. I will have one more conversation with brian to find what needs to be fixed.
REVIEW: http://review.gluster.org/7928 (tests: Add fops sanity) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)
COMMIT: http://review.gluster.org/7928 committed in master by Vijay Bellur (vbellur) ------ commit 254a5e484b01f946beff3f51f9ade38f4cdfe51b Author: Pranith Kumar K <pkarampu> Date: Wed Apr 9 08:43:45 2014 +0530 tests: Add fops sanity Original-Authors: MS Vishwanath and Sachidananda URS Change-Id: I7a76b74fc4de24a2d7ec5932ff2c47754abf0521 BUG: 1084422 Signed-off-by: Pranith Kumar K <pkarampu> Reviewed-on: http://review.gluster.org/7928 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/8146 (tests: attr/xattr.h doesn't exist without libattr-devel fix it) posted (#1) for review on master by Harshavardhana (harsha)
REVIEW: http://review.gluster.org/8146 (tests: attr/xattr.h doesn't exist without libattr-devel fix it) posted (#2) for review on master by Harshavardhana (harsha)
COMMIT: http://review.gluster.org/8146 committed in master by Harshavardhana (harsha) ------ commit a98c4f8bb0517e961a30acdd3d0dce95aa68b96a Author: Harshavardhana <harsha> Date: Sun Jun 22 14:35:23 2014 -0700 tests: attr/xattr.h doesn't exist without libattr-devel fix it Use sys/xattr.h - glibc provided rather than external libs Change-Id: Iacf80c1089f11a5a9b46d24e2a62e41fa0c4f5ae BUG: 1084422 Signed-off-by: Harshavardhana <harsha> Reviewed-on: http://review.gluster.org/8146 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: M S Vishwanath Bhat <vbhat>
This bug still needs one more fix to send the fd-flags from vfs layer so that fuse can use it. Moving it back to ASSIGNED state.
A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED. Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html [2] http://supercolony.gluster.org/pipermail/gluster-users/
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report. glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html [2] http://supercolony.gluster.org/mailman/listinfo/gluster-users