Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1562699

Summary:	./tests/basic/afr/afr-no-fsync.t fails on jenkins slaves
Product:	[Community] GlusterFS	Reporter:	Ravishankar N <ravishankar>
Component:	project-infrastructure	Assignee:	bugs <bugs>
Status:	CLOSED NOTABUG	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	high
Version:	mainline	CC:	bugs, gluster-infra, nigelb
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-04-12 05:45:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ravishankar N 2018-04-02 07:18:40 UTC

Description of problem:

afr-no-fsync.t fails only on the jenkins slaves.

Runs:
https://build.gluster.org/job/centos7-regression/513/consoleFull
https://build.gluster.org/job/centos7-regression/575/console

Failure:
00:16:24 dd: error writing ‘/mnt/glusterfs/0/a’: Cannot allocate memory
00:16:24 dd: closing output file ‘/mnt/glusterfs/0/a’: Cannot allocate memory
00:16:24 not ok 9 , LINENUM:17

The test passes locally on fedora and on the softserve centos 7 VMS.


Snippent from the gluster fuse mount log:
[2018-04-02 05:02:21.642563] E [MSGID: 133007] [shard.c:2051:shard_common_lookup_shards] 0-patchy-shard: Inode path failed on 111eafd5-7f94-40ce-a73b-9dd3d23c2edb.1, base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb
[2018-04-02 05:02:21.642666] E [MSGID: 133010] [shard.c:1944:shard_common_lookup_shards_cbk] 0-patchy-shard: Lookup on shard 1 failed. Base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb [Cannot allocate memory]
[2018-04-02 05:02:21.643269] E [MSGID: 133007] [shard.c:2051:shard_common_lookup_shards] 0-patchy-shard: Inode path failed on 111eafd5-7f94-40ce-a73b-9dd3d23c2edb.1, base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb
[2018-04-02 05:02:21.643311] E [MSGID: 133010] [shard.c:1944:shard_common_lookup_shards_cbk] 0-patchy-shard: Lookup on shard 1 failed. Base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb [Cannot allocate memory]
[2018-04-02 05:02:21.643381] I [MSGID: 101016] [glusterfs3.h:738:dict_to_xdr] 0-dict: key 'trusted.glusterfs.shard.file-size' is would not be sent on wire in future [Invalid argument]
The message "I [MSGID: 101016] [glusterfs3.h:738:dict_to_xdr] 0-dict: key 'trusted.glusterfs.shard.file-size' is would not be sent on wire in future [Invalid argument]" repeated 2 times between [2018-04-02 05:02:21.643381] and [2018-04-02 05:02:21.643534]
[2018-04-02 05:02:21.644186] E [MSGID: 133007] [shard.c:2051:shard_common_lookup_shards] 0-patchy-shard: Inode path failed on 111eafd5-7f94-40ce-a73b-9dd3d23c2edb.1, base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb
[2018-04-02 05:02:21.644241] I [MSGID: 101016] [glusterfs3.h:738:dict_to_xdr] 0-dict: key 'trusted.glusterfs.shard.file-size' is would not be sent on wire in future [Invalid argument]
[2018-04-02 05:02:21.644243] E [MSGID: 133010] [shard.c:1944:shard_common_lookup_shards_cbk] 0-patchy-shard: Lookup on shard 1 failed. Base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb [Cannot allocate memory]
[2018-04-02 05:02:21.644341] I [MSGID: 101016] [glusterfs3.h:738:dict_to_xdr] 0-dict: key 'trusted.glusterfs.shard.file-size' is would not be sent on wire in future [Invalid argument]
[2018-04-02 05:02:21.644432] I [MSGID: 101016] [glusterfs3.h:738:dict_to_xdr] 0-dict: key 'trusted.glusterfs.shard.file-size' is would not be sent on wire in future [Invalid argument]
[2018-04-02 05:02:21.644353] E [MSGID: 133007] [shard.c:2051:shard_common_lookup_shards] 0-patchy-shard: Inode path failed on 111eafd5-7f94-40ce-a73b-9dd3d23c2edb.1, base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb
[2018-04-02 05:02:21.644875] E [MSGID: 133010] [shard.c:1944:shard_common_lookup_shards_cbk] 0-patchy-shard: Lookup on shard 1 failed. Base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb [Cannot allocate memory]
[2018-04-02 05:02:21.645096] E [MSGID: 133007] [shard.c:2051:shard_common_lookup_shards] 0-patchy-shard: Inode path failed on 111eafd5-7f94-40ce-a73b-9dd3d23c2edb.1, base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb



Can it be checked if this is an infra issue? If not, I would like to request a jenkins slave to debug the issue.

Comment 1 Nigel Babu 2018-04-02 08:20:49 UTC

Is it failing on only one node?

Comment 2 Ravishankar N 2018-04-02 08:22:26 UTC

No, I see that the 2 failed runs were on  builder102 and  builder106 respectively.

Comment 3 Ravishankar N 2018-04-02 10:58:10 UTC

I think I goofed up. I had left a a line commented in the .t, which was causing it to succeed. I'm now able to repro the failure locally. Please feel free to close the bug. Apologies!