Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1225919

Summary:	Log EEXIST errors in DEBUG level in fops MKNOD and MKDIR
Product:	[Community] GlusterFS	Reporter:	Krutika Dhananjay <kdhananj>
Component:	protocol	Assignee:	Krutika Dhananjay <kdhananj>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.7.0	CC:	bugs, gluster-bugs
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.7.1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1221889
Clones:	1227174 (view as bug list)		Environment:
Last Closed:	2015-06-02 08:03:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1221889
Bug Blocks:	1227174

Description Krutika Dhananjay 2015-05-28 13:04:44 UTC

+++ This bug was initially created as a clone of Bug #1221889 +++

Description of problem:
Log mknod failure due to EEXIST in DEBUG level in protocol/client.

Without this, there would be excessive logging of EEXIST errors from protocol/client,server in logs of sharded volumes when multiple attempts are made from the same mount to create /.shard or the individual block shards as part of inode write fops.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Krutika Dhananjay on 2015-05-15 05:31:34 EDT ---

Here is an excerpt from the logs when dd is executed on the mount (which possibly writes through multiple threads):

<client-logs>
...
...
[2015-05-15 09:19:21.890807] W [client-rpc-fops.c:308:client3_3_mkdir_cbk] 0-dis-client-0: remote operation failed: File exists. Path: /.shard
[2015-05-15 09:19:21.890958] W [client-rpc-fops.c:308:client3_3_mkdir_cbk] 0-dis-client-0: remote operation failed: File exists. Path: /.shard
[2015-05-15 09:19:21.891049] W [client-rpc-fops.c:308:client3_3_mkdir_cbk] 0-dis-client-0: remote operation failed: File exists. Path: /.shard
[2015-05-15 09:19:21.891148] W [client-rpc-fops.c:308:client3_3_mkdir_cbk] 0-dis-client-0: remote operation failed: File exists. Path: /.shard
[2015-05-15 09:19:21.893121] W [client-rpc-fops.c:308:client3_3_mkdir_cbk] 0-dis-client-0: remote operation failed: File exists. Path: /.shard
[2015-05-15 09:19:21.893277] W [client-rpc-fops.c:308:client3_3_mkdir_cbk] 0-dis-client-0: remote operation failed: File exists. Path: /.shard
[2015-05-15 09:19:21.893375] I [dht-layout.c:697:dht_layout_normalize] 0-dis-dht: Found anomalies in /.shard (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2015-05-15 09:19:21.893502] I [dht-layout.c:697:dht_layout_normalize] 0-dis-dht: Found anomalies in /.shard (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2015-05-15 09:19:21.893575] I [dht-layout.c:697:dht_layout_normalize] 0-dis-dht: Found anomalies in /.shard (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2015-05-15 09:19:21.893703] I [dht-layout.c:697:dht_layout_normalize] 0-dis-dht: Found anomalies in /.shard (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2015-05-15 09:19:21.893820] I [MSGID: 109036] [dht-common.c:6692:dht_log_new_layout_for_dir_selfheal] 0-dis-dht: Setting layout of /.shard with [Subvol_name: dis-client-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], 
[2015-05-15 09:19:21.895055] I [dht-layout.c:697:dht_layout_normalize] 0-dis-dht: Found anomalies in /.shard (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2015-05-15 09:19:21.895924] I [dht-layout.c:697:dht_layout_normalize] 0-dis-dht: Found anomalies in /.shard (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2015-05-15 09:19:22.175998] W [client-rpc-fops.c:241:client3_3_mknod_cbk] 0-dis-client-0: remote operation failed: File exists. Path: /.shard/72c4bc4a-ff12-4e2f-b231-992a64cdde82.1
[2015-05-15 09:19:22.176119] W [client-rpc-fops.c:241:client3_3_mknod_cbk] 0-dis-client-0: remote operation failed: File exists. Path: /.shard/72c4bc4a-ff12-4e2f-b231-992a64cdde82.1
[2015-05-15 09:19:22.451790] W [client-rpc-fops.c:241:client3_3_mknod_cbk] 0-dis-client-0: remote operation failed: File exists. Path: /.shard/72c4bc4a-ff12-4e2f-b231-992a64cdde82.2
[2015-05-15 09:19:22.451908] W [client-rpc-fops.c:241:client3_3_mknod_cbk] 0-dis-client-0: remote operation failed: File exists. Path: /.shard/72c4bc4a-ff12-4e2f-b231-992a64cdde82.2
...
...
</client-logs>

<brick-logs>
...
...
[2015-05-15 09:19:21.890271] I [MSGID: 115056] [server-rpc-fops.c:487:server_mkdir_cbk] 0-dis-server: 31: MKDIR /.shard (00000000-0000-0000-0000-000000000001/.shard) ==> (File exists) [File exists]
[2015-05-15 09:19:21.890662] I [MSGID: 115056] [server-rpc-fops.c:487:server_mkdir_cbk] 0-dis-server: 32: MKDIR (null) (00000000-0000-0000-0000-000000000001/.shard) ==> (File exists) [File exists]
[2015-05-15 09:19:21.890832] I [MSGID: 115056] [server-rpc-fops.c:487:server_mkdir_cbk] 0-dis-server: 33: MKDIR (null) (00000000-0000-0000-0000-000000000001/.shard) ==> (File exists) [File exists]
[2015-05-15 09:19:21.891038] I [MSGID: 115056] [server-rpc-fops.c:487:server_mkdir_cbk] 0-dis-server: 36: MKDIR (null) (00000000-0000-0000-0000-000000000001/.shard) ==> (File exists) [File exists]
[2015-05-15 09:19:21.892857] I [MSGID: 115056] [server-rpc-fops.c:487:server_mkdir_cbk] 0-dis-server: 39: MKDIR (null) (00000000-0000-0000-0000-000000000001/.shard) ==> (File exists) [File exists]
[2015-05-15 09:19:21.893040] I [MSGID: 115056] [server-rpc-fops.c:487:server_mkdir_cbk] 0-dis-server: 42: MKDIR (null) (00000000-0000-0000-0000-000000000001/.shard) ==> (File exists) [File exists]
[2015-05-15 09:19:22.175831] I [MSGID: 115057] [server-rpc-fops.c:535:server_mknod_cbk] 0-dis-server: 176: MKNOD (null) (be318638-e8a0-4c6d-977d-7a937aa84806/72c4bc4a-ff12-4e2f-b231-992a64cdde82.1) ==> (File exists) [File exists]
[2015-05-15 09:19:22.175966] I [MSGID: 115057] [server-rpc-fops.c:535:server_mknod_cbk] 0-dis-server: 177: MKNOD (null) (be318638-e8a0-4c6d-977d-7a937aa84806/72c4bc4a-ff12-4e2f-b231-992a64cdde82.1) ==> (File exists) [File exists]
[2015-05-15 09:19:22.451713] I [MSGID: 115057] [server-rpc-fops.c:535:server_mknod_cbk] 0-dis-server: 277: MKNOD (null) (be318638-e8a0-4c6d-977d-7a937aa84806/72c4bc4a-ff12-4e2f-b231-992a64cdde82.2) ==> (File exists) [File exists]
[2015-05-15 09:19:22.451822] I [MSGID: 115057] [server-rpc-fops.c:535:server_mknod_cbk] 0-dis-server: 278: MKNOD (null) (be318638-e8a0-4c6d-977d-7a937aa84806/72c4bc4a-ff12-4e2f-b231-992a64cdde82.2) ==> (File exists) [File exists]
...
...
</brick-logs>

--- Additional comment from Krutika Dhananjay on 2015-05-15 05:54:58 EDT ---

Patch posted at http://review.gluster.org/#/c/10791/

--- Additional comment from Anand Avati on 2015-05-28 01:40:41 EDT ---

COMMIT: http://review.gluster.org/10791 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit f92a415dd4fdd34c97fa6af4af528e1b50a4265f
Author: Krutika Dhananjay <kdhananj>
Date:   Wed May 13 14:25:44 2015 +0200

    protocol/client,server: Move EEXIST logs in mkdir and mknod to DEBUG level
    
    Change-Id: I0481258de8da36cbee7c046f53b20359badaf064
    BUG: 1221889
    Signed-off-by: Krutika Dhananjay <kdhananj>
    Reviewed-on: http://review.gluster.org/10791
    Tested-by: Gluster Build System <jenkins.com>
    Tested-by: NetBSD Build System
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>

Comment 1 Anand Avati 2015-05-28 13:21:38 UTC

REVIEW: http://review.gluster.org/10973 (protocol/client,server: Move EEXIST logs in mkdir and mknod to DEBUG level) posted (#1) for review on release-3.7 by Krutika Dhananjay (kdhananj)

Comment 2 Anand Avati 2015-05-29 02:24:53 UTC

COMMIT: http://review.gluster.org/10973 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu) 
------
commit 5364a02d2dbb116952086958010ec508956b259b
Author: Krutika Dhananjay <kdhananj>
Date:   Wed May 13 14:25:44 2015 +0200

    protocol/client,server: Move EEXIST logs in mkdir and mknod to DEBUG level
    
            Backport of: http://review.gluster.org/10791
    
    Change-Id: I096296f0b97f62f49577ca698ae34e28cce4a4b4
    BUG: 1225919
    Signed-off-by: Krutika Dhananjay <kdhananj>
    Reviewed-on: http://review.gluster.org/10973
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Gluster Build System <jenkins.com>

Comment 3 Niels de Vos 2015-06-02 08:03:47 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.1, please reopen this bug report.

glusterfs-3.7.1 has been announced on the Gluster Packaging mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.packaging/1
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user