Bug 765161 (GLUSTER-3429)

Summary: [glusterfs-3.2.3qa1]: rm -rf parallely from 2 clients given ENOMEM
Product: [Community] GlusterFS Reporter: Raghavendra Bhat <rabhat>
Component: quotaAssignee: Raghavendra Bhat <rabhat>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: pre-releaseCC: gluster-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: 3.2.3 Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Raghavendra Bhat 2011-08-17 10:27:10 UTC
rm -rf parallely from 2 clients gives ENOMEM. But from single client it continues. Configuration is single client-server model without any clustering xlators.

[2011-08-17 14:56:57.733089] D [inodelk.c:293:__inode_unlock_lock] 0-vol-locks:  Matching lock found for unlock
[2011-08-17 14:56:57.733114] D [marker-quota.c:1184:quota_inodelk_cbk] 0-vol-marker: inodelk released on /
[2011-08-17 14:56:57.740990] D [marker-quota.c:1981:inspect_file_xattr] 0-vol-marker: size=4096 contri=4096
[2011-08-17 14:56:57.741202] D [marker-quota.c:1981:inspect_file_xattr] 0-vol-marker: size=4096 contri=4096
[2011-08-17 14:56:57.741462] E [posix.c:652:posix_stat] 0-vol-posix: lstat on /okpa/linux-3.0.1/fs/freevxfs/Makefile failed: No such file or directory
[2011-08-17 14:56:57.741492] I [server3_1-fops.c:964:server_unlink_cbk] 0-vol-server: 515808: UNLINK /okpa/linux-3.0.1/fs/freevxfs/Makefile (798179) ==> -1 (Cannot allocate memo
ry)

It is because if this piece of code.

int32_t
marker_unlink_stat_cbk (call_frame_t *frame, void *cookie, xlator_t *this,
                        int32_t op_ret, int32_t op_errno, struct iatt *buf)
{
	marker_local_t *local = NULL;

	if (op_ret < 0) {
	        goto err;
	}

	local = frame->local;
	if (local == NULL) {
                goto err;
	}

	local->ia_nlink = buf->ia_nlink;

        STACK_WIND (frame, marker_unlink_cbk, FIRST_CHILD(this),
	            FIRST_CHILD(this)->fops->unlink, &local->loc);
	return 0;
err:
        STACK_UNWIND_STRICT (unlink, frame, -1, ENOMEM, NULL, NULL);

	return 0;
}

Situation is: 

Marker sends stat first before sending the unlink. By the time stat reaches posix if the file is deleted by another client, then stat fails with op_errno set to ENOENT. But in the corresponding call back function of marker (i.e. the above funtion marker_unlink_stat_cbk), we check for (op_ret < 0) and if so we unwind by sending ENOMEM as the error to the above xlators which results in the application(rm -rf in this case) also getting ENOMEM.

Comment 1 Anand Avati 2011-08-20 04:09:37 UTC
CHANGE: http://review.gluster.com/252 (Change-Id: I8266754d4b53b525b13fed2383cf95d77bab47ba) merged in release-3.2 by Vijay Bellur (vijay)

Comment 2 Anand Avati 2011-08-20 04:14:39 UTC
CHANGE: http://review.gluster.com/253 (Marker sends stat first, before sending the unlink. By the time stat reaches) merged in release-3.2 by Vijay Bellur (vijay)

Comment 3 Raghavendra Bhat 2011-08-29 02:44:30 UTC
Its fixed now. Now rm -rf from 2 clients does not get ENOMEM, as marker sends the errno sent by posix itself to the above xlators.