Description of problem: afr-no-fsync.t fails only on the jenkins slaves. Runs: https://build.gluster.org/job/centos7-regression/513/consoleFull https://build.gluster.org/job/centos7-regression/575/console Failure: 00:16:24 dd: error writing ‘/mnt/glusterfs/0/a’: Cannot allocate memory 00:16:24 dd: closing output file ‘/mnt/glusterfs/0/a’: Cannot allocate memory 00:16:24 not ok 9 , LINENUM:17 The test passes locally on fedora and on the softserve centos 7 VMS. Snippent from the gluster fuse mount log: [2018-04-02 05:02:21.642563] E [MSGID: 133007] [shard.c:2051:shard_common_lookup_shards] 0-patchy-shard: Inode path failed on 111eafd5-7f94-40ce-a73b-9dd3d23c2edb.1, base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb [2018-04-02 05:02:21.642666] E [MSGID: 133010] [shard.c:1944:shard_common_lookup_shards_cbk] 0-patchy-shard: Lookup on shard 1 failed. Base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb [Cannot allocate memory] [2018-04-02 05:02:21.643269] E [MSGID: 133007] [shard.c:2051:shard_common_lookup_shards] 0-patchy-shard: Inode path failed on 111eafd5-7f94-40ce-a73b-9dd3d23c2edb.1, base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb [2018-04-02 05:02:21.643311] E [MSGID: 133010] [shard.c:1944:shard_common_lookup_shards_cbk] 0-patchy-shard: Lookup on shard 1 failed. Base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb [Cannot allocate memory] [2018-04-02 05:02:21.643381] I [MSGID: 101016] [glusterfs3.h:738:dict_to_xdr] 0-dict: key 'trusted.glusterfs.shard.file-size' is would not be sent on wire in future [Invalid argument] The message "I [MSGID: 101016] [glusterfs3.h:738:dict_to_xdr] 0-dict: key 'trusted.glusterfs.shard.file-size' is would not be sent on wire in future [Invalid argument]" repeated 2 times between [2018-04-02 05:02:21.643381] and [2018-04-02 05:02:21.643534] [2018-04-02 05:02:21.644186] E [MSGID: 133007] [shard.c:2051:shard_common_lookup_shards] 0-patchy-shard: Inode path failed on 111eafd5-7f94-40ce-a73b-9dd3d23c2edb.1, base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb [2018-04-02 05:02:21.644241] I [MSGID: 101016] [glusterfs3.h:738:dict_to_xdr] 0-dict: key 'trusted.glusterfs.shard.file-size' is would not be sent on wire in future [Invalid argument] [2018-04-02 05:02:21.644243] E [MSGID: 133010] [shard.c:1944:shard_common_lookup_shards_cbk] 0-patchy-shard: Lookup on shard 1 failed. Base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb [Cannot allocate memory] [2018-04-02 05:02:21.644341] I [MSGID: 101016] [glusterfs3.h:738:dict_to_xdr] 0-dict: key 'trusted.glusterfs.shard.file-size' is would not be sent on wire in future [Invalid argument] [2018-04-02 05:02:21.644432] I [MSGID: 101016] [glusterfs3.h:738:dict_to_xdr] 0-dict: key 'trusted.glusterfs.shard.file-size' is would not be sent on wire in future [Invalid argument] [2018-04-02 05:02:21.644353] E [MSGID: 133007] [shard.c:2051:shard_common_lookup_shards] 0-patchy-shard: Inode path failed on 111eafd5-7f94-40ce-a73b-9dd3d23c2edb.1, base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb [2018-04-02 05:02:21.644875] E [MSGID: 133010] [shard.c:1944:shard_common_lookup_shards_cbk] 0-patchy-shard: Lookup on shard 1 failed. Base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb [Cannot allocate memory] [2018-04-02 05:02:21.645096] E [MSGID: 133007] [shard.c:2051:shard_common_lookup_shards] 0-patchy-shard: Inode path failed on 111eafd5-7f94-40ce-a73b-9dd3d23c2edb.1, base file gfid = 111eafd5-7f94-40ce-a73b-9dd3d23c2edb Can it be checked if this is an infra issue? If not, I would like to request a jenkins slave to debug the issue.
Is it failing on only one node?
No, I see that the 2 failed runs were on builder102 and builder106 respectively.
I think I goofed up. I had left a a line commented in the .t, which was causing it to succeed. I'm now able to repro the failure locally. Please feel free to close the bug. Apologies!