Bug 1156022

Summary:	creating hardlinks with perl fails randomly
Product:	[Community] GlusterFS	Reporter:	m.vernimmen
Component:	core	Assignee:	bugs <bugs>
Status:	CLOSED EOL	QA Contact:
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	3.5.2	CC:	bugs, m.vernimmen, vshankar
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-06-17 15:57:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description m.vernimmen 2014-10-23 12:15:31 UTC

Description of problem:
While running BackupPC 3.3.0-2 on CentOS 6.5 where the data is stored on a gluster mountpoint consisting of 2 nodes with 2 bricks each, in a distributed setup, sometimes the creation of hardlinks fail.
When changing the perl 'link($from, $to)' style command to a shell command 'ln $source $target', the links are created fine. Since the problem does not occur when the data is stored on ext4, to me this seems like a problem of glusterfs.

Version-Release number of selected component (if applicable):
BackupPC-3.3.0-2.el6.x86_64
glusterfs-cli-3.5.2-1.el6.x86_64
glusterfs-api-3.5.2-1.el6.x86_64
glusterfs-server-3.5.2-1.el6.x86_64
glusterfs-libs-3.5.2-1.el6.x86_64
glusterfs-3.5.2-1.el6.x86_64
glusterfs-fuse-3.5.2-1.el6.x86_64

How reproducible:
run backuppc on a distributed gluster setup. possibly the same behaviour can be reproduced by creating a random long list of hardlinks in a nested directory structure.

Steps to Reproduce:
1. set up 2 gluster nodes
2. mount gluster as the storage location for backuppc
3. run a number of backups for a compressed pool

Actual results:
in /var/log/messages
Oct 23 13:12:32 md002 backuppc: 2014-10-23 13:12:27 BackupPC_link got error -3 when calling MakeFileLink(/var/lib/BackupPC/pc/md001/587/f%2fexport/fopsview/fvar/frrd/fad008_fhlive02%252Ea/fCheck%2520Network%2520I%252FO%253A%2520bond0/ftxbitsec/fvalue.rrd, 926a46e5d8b938243c1bc55960115d2d, 1)
Oct 23 13:12:32 md002 backuppc: 2014-10-23 13:12:27 BackupPC_link got error -3 when calling MakeFileLink(/var/lib/BackupPC/pc/md001/587/f%2fexport/fopsview/fvar/frrd/fad008_fhlive02%252Ea/fCheck%2520Network%2520I%252FO%253A%2520bond0/ftxbitsec/attrib, 7042031bf46ad1178cdbcc33684a87ab, 1)
Oct 23 13:12:32 md002 backuppc: 2014-10-23 13:12:27 BackupPC_link got error -3 when calling MakeFileLink(/var/lib/BackupPC/pc/md001/587/f%2fexport/fopsview/fvar/frrd/fad008_fhlive02%252Ea/fCheck%2520Network%2520I%252FO%253A%2520eth2/frxbitsec/fvalue.rrd, 7739740ecf6d9ea7bc35b11fe3017dfa, 1)
Oct 23 13:12:32 md002 backuppc: 2014-10-23 13:12:27 BackupPC_link got error -3 when calling MakeFileLink(/var/lib/BackupPC/pc/md001/587/f%2fexport/fopsview/fvar/frrd/fad008_fhlive02%252Ea/fCheck%2520Network%2520I%252FO%253A%2520eth2/frxbitsec/attrib, 22133a9045b317f7e3830383dc1f0ea8, 1)

Expected results:
no errors in /var/log/messages

Additional info:
I did not find any related information in the glusterfs logs. since I have now a workaround implemented, the problem hopefully won't return.
I noticed the same hardlink would fail consistently to be made, while others are created fine.

Comment 1 m.vernimmen 2014-10-23 12:16:00 UTC

a rebalance was running in the background.

Comment 2 Soumya Koduri 2016-01-05 12:25:37 UTC

Meanwhile could you try running these tests on the latest build and check if you still see the issue

Comment 3 Niels de Vos 2016-06-17 15:57:15 UTC

This bug is getting closed because the 3.5 is marked End-Of-Life. There will be no further updates to this version. Please open a new bug against a version that still receives bugfixes if you are still facing this issue in a more current release.