Bug 1242708

Summary:	fuse/fuse_thread_proc : The fuse_graph_sync function cannot be handled in time after we fix-layout.
Product:	[Community] GlusterFS	Reporter:	wangzhen <linux_wz>
Component:	fuse	Assignee:	wangzhen <linux_wz>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs, csaba, linux_wz, nbalacha, ndevos, ravishankar, rgowdapp
Target Milestone:	---	Keywords:	Triaged
Target Release:	---	Flags:	linux_wz: needinfo-
Hardware:	i386
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-09-01 13:30:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description wangzhen 2015-07-14 02:27:12 UTC

Description of problem:
After we add-brick and fix-layout,the function fuse_thread_proc is still blocked in readv,The old graph will be executed first.The first time we create a file on the client, we will get an incorrect hash volume.


Version-Release number of selected component (if applicable):
all

Steps to Reproduce:
1.[node1]# gluster volume info
  Volume Name: dht_vol
  Type: Distribute
  Volume ID: f99cc88b-9558-4286-a56e-3bcb7d057035
  Status: Started
  Number of Bricks: 1
  Transport-type: tcp
  Bricks:
  Brick1: 10.10.21.150:/root/sdb
  Options Reconfigured:
  performance.readdir-ahead: on
2.[node3]# mount.glusterfs node1:/dht_vol /mnt/gluster/
3.[node1]# gluster volume add-brick dht_vol 10.10.21.150:/root/sdc force
  [node1]# gluster volume rebalance dht_vol fix-layout start
4.[node3]# touch 0
5.[node1]# ls  sd*
  sdb:

  sdc:
  0
6.[node3]# rm 0 -rf; touch 0
7.[root@node1 ~]# ls  sd*
  sdb:
  0
  sdc:

Actual results:
step 3-4: the first time we create a file, 0 is distributed in sdc.
step 5-6: but the second time,file 0 is distributed in sdb

Expected results:
step 5-6: We hope file 0 should belong to the same brick

Additional info:
I have a solution for this problem.we can move the function fuse_graph_sync back
to readv .I've tested it, it's right.But I don't know if it will bring other problems. Waiting for your answer. Thank you.

Comment 1 Anand Avati 2015-07-15 10:17:05 UTC

REVIEW: http://review.gluster.org/11672 (fuse: Get an incorrect hash volume after we add-brick) posted (#2) for review on master by wangzhen (linux_wz)

Comment 2 Anand Avati 2015-07-16 01:21:08 UTC

REVIEW: http://review.gluster.org/11672 (fuse: Get an incorrect hash volume after we add-brick) posted (#3) for review on master by wangzhen (linux_wz)

Comment 3 Anand Avati 2015-07-20 02:59:54 UTC

REVIEW: http://review.gluster.org/11672 (fuse: Get an incorrect hash volume after we add-brick) posted (#4) for review on master by wangzhen (linux_wz)

Comment 4 Anand Avati 2015-07-22 03:47:22 UTC

REVIEW: http://review.gluster.org/11734 (fuse: Get an incorrect hash volume after we add-brick) posted (#1) for review on master by wangzhen (linux_wz)

Comment 5 Raghavendra G 2015-07-29 04:32:29 UTC

In the above tests (comment 1), we are not waiting till fix-layout is complete before creating file in step 4. Can you please re-run the tests with creating the file once fix-layout is complete? You can monitor the status of rebalance using:
 #gluster volume rebalance dht_vol status
My suspicion is that file got created in step 4 before rebalance was able to fix the layout.

Comment 6 Vijay Bellur 2015-10-19 05:49:52 UTC

REVIEW: http://review.gluster.org/11672 (fuse: Get an incorrect hash volume after we add-brick) posted (#5) for review on master by wangzhen (linux_wz)

Comment 7 Vijay Bellur 2015-10-19 05:50:09 UTC

REVIEW: http://review.gluster.org/11672 (fuse: Get an incorrect hash volume after we add-brick) posted (#6) for review on master by wangzhen (linux_wz)

Comment 8 Kaleb KEITHLEY 2015-10-22 17:00:12 UTC

changing version to mainline in order to retire pre-release.

If you know the appropriate, correct version for this bug, please
set it.

Comment 9 Kaleb KEITHLEY 2015-10-22 17:06:51 UTC

changing version to mainline in order to retire pre-release.

If you know the appropriate, correct version for this bug, please
set it.

Comment 10 Kaleb KEITHLEY 2015-10-22 17:11:18 UTC

changing version to mainline in order to retire pre-release.

If you know the appropriate, correct version for this bug, please
set it.

Comment 11 Vijay Bellur 2015-11-19 15:45:49 UTC

REVIEW: http://review.gluster.org/11672 (fuse: Get an incorrect hash volume after we add-brick) posted (#7) for review on master by Niels de Vos (ndevos)

Comment 12 Mike McCune 2016-03-28 22:55:34 UTC

This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 13 Nithya Balachandran 2017-01-31 04:49:57 UTC

The DHT code determines the brick on which to create the file based on the where the hash of the file name falls according to the layout of the parent directory. This may change based on whether the bricks topology has been updated and whether a rebalance/fix-layout has been run.


That being said, it seems odd that the layout has changed from the new brick to the old. This may have been because of the way the older dht layout code worked. Recent changes where the layout will not be changed if it is already complete and spans the bricks will probably prevent this behaviour from showing up.

Comment 14 Raghavendra G 2017-02-01 05:15:54 UTC

Fixed by https://review.gluster.org/14763