Bug 1158435 - DHT: Rebalance + Quota - On a quota enabled volume renames from multiple mounts with rebalance creates lots of hard links on the mount point
Summary: DHT: Rebalance + Quota - On a quota enabled volume renames from multiple moun...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: quota
Version: 2.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: Vijaikumar Mallikarjuna
QA Contact: Amit Chaurasia
URL:
Whiteboard:
Depends On: 1157705
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-10-29 11:37 UTC by shylesh
Modified: 2016-09-17 12:36 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-01-07 05:32:46 UTC
Embargoed:


Attachments (Terms of Use)

Description shylesh 2014-10-29 11:37:14 UTC
Description of problem:
Renaming files from multiple mount points while rebalance in progress creates huge number of hardlinks, quota has to be enabled on the volume.

Version-Release number of selected component (if applicable):
3.4.0.69rhs-1.el6rhs.x86_64

How reproducible:
Always

Steps to Reproduce:
1.created a 6x2 dist-rep volume
2.created 100 files
3.started renaming from 2 mount points 
4. start rebalance , ran rebalance couple of times

Actual results:
After rebalance and renaming is over 30000 files were there on the mount point and each file had around 400+ hard links.

 [root@rhs-client4 mnt]# gluster v info test
 
Volume Name: test
Type: Distributed-Replicate
Volume ID: f0ceb33d-4102-4c71-aefd-651b2527f6c1
Status: Started
Number of Bricks: 15 x 2 = 30
Transport-type: tcp
Bricks:
Brick1: rhs-client4.lab.eng.blr.redhat.com:/home/test0
Brick2: rhs-client39.lab.eng.blr.redhat.com:/home/test1
Brick3: rhs-gp-srv2.lab.eng.blr.redhat.com:/home/test2
Brick4: rhs-client4.lab.eng.blr.redhat.com:/home/test3
Brick5: rhs-client39.lab.eng.blr.redhat.com:/home/test4
Brick6: rhs-gp-srv2.lab.eng.blr.redhat.com:/home/test5
Brick7: rhs-client4.lab.eng.blr.redhat.com:/home/test6
Brick8: rhs-client39.lab.eng.blr.redhat.com:/home/test7
Brick9: rhs-gp-srv2.lab.eng.blr.redhat.com:/home/test8
Brick10: rhs-client4.lab.eng.blr.redhat.com:/home/test9
Brick11: rhs-client39.lab.eng.blr.redhat.com:/home/test10
Brick12: rhs-gp-srv2.lab.eng.blr.redhat.com:/home/test11
Brick13: rhs-client4.lab.eng.blr.redhat.com:/home/test12
Brick14: rhs-client39.lab.eng.blr.redhat.com:/home/test13
Brick15: rhs-gp-srv2.lab.eng.blr.redhat.com:/home/test14
Brick16: rhs-client4.lab.eng.blr.redhat.com:/home/test15
Brick17: rhs-client39.lab.eng.blr.redhat.com:/home/test16
Brick18: rhs-gp-srv2.lab.eng.blr.redhat.com:/home/test17
Brick19: rhs-client4.lab.eng.blr.redhat.com:/home/test18
Brick20: rhs-client39.lab.eng.blr.redhat.com:/home/test19
Brick21: rhs-gp-srv2.lab.eng.blr.redhat.com:/home/test20
Brick22: rhs-client4.lab.eng.blr.redhat.com:/home/test21
Brick23: rhs-client39.lab.eng.blr.redhat.com:/home/test22
Brick24: rhs-gp-srv2.lab.eng.blr.redhat.com:/home/test23
Brick25: rhs-client4.lab.eng.blr.redhat.com:/home/test24
Brick26: rhs-client39.lab.eng.blr.redhat.com:/home/test25
Brick27: rhs-client4.lab.eng.blr.redhat.com:/home/test26
Brick28: rhs-client39.lab.eng.blr.redhat.com:/home/test27
Brick29: rhs-client4.lab.eng.blr.redhat.com:/home/test28
Brick30: rhs-client39.lab.eng.blr.redhat.com:/home/test29
Options Reconfigured:
features.quota: on



Additional info:

[2014-10-28 09:26:09.816663] W [posix.c:1266:posix_unlink] 0-test-posix: modification of parent gfid xattr failed (path:/home/test10/renames/f97-516 gfid:5fa4d205-071b-4f50-9019-008389bd119c)
             [2014-10-28 09:26:09.816723] E [server-rpc[2014-10-28 09:26:09.816663] W [posix.c:1266:posix_unlink] 0-test-posix: modification of parent gfid xattr failed (path:/home/test10/renames/f97-516 gfid:5fa4d205-071b-4f50-9019-008389bd119c)
             [2014-10-28 09:26:09.816723] E [server-rpc-fops.c:1111:server_unlink_cbk] 0-test-server: 182512: UNLINK /renames/f97-516 (6562fd3a-d1d3-4aba-959a-55b88e2152e7/f97-516) ==      (Success)
             [2014-10-28 09:26:10.069936] E [posix-helpers.c:831:posix_handle_pair] 0-test-posix: /home/test10/renames/f13-517: key:trusted.glusterfs.dht.linkto error:File exists
             [2014-10-28 09:26:10.069977] E [posix.c:920:posix_mknod] 0-test-posix: setting xattrs on /home/test10/renames/f13-517 failed (File exists)
             [2014-10-28 09:26:10.072734] W [quota.c:1860:quota_rename_cbk] 0-test-quota: dentry corresponding to the path just renamed (name:f13-517) is not present
             [2014-10-28 09:26:10.074540] I [server-rpc-fops.c:581:server_mknod_cbk] 0-test-server: 260921: MKNOD (null) (6562fd3a-d1d3-4aba-959a-55b88e2152e7/f13-518) ==      (File exists)
             [2014-10-28 09:26:10.903803] W [posix.c:1266:posix_unlink] 0-test-posix: modification of parent gfid xattr failed (path:/home/test10/renames/f66-517 gfid:142232eb-12c9-4d22-9d1a-f20089f897a0)
             [2014-10-28 09:26:10.903861] E [server-rpc-fops.c:1111:server_unlink_cbk] 0-test-server: 182717: UNLINK /renames/f66-517 (6562fd3a-d1d3-4aba-959a-55b88e2152e7/f66-517) ==      (Success)
             [2014-10-28 09:26:11.148741] W [posix.c:1266:posix_unlink] 0-test-posix: modification of parent gfid xattr failed (path:/home/test10/renames/f83-517 gfid:96d87103-5daa-4c93-9224-0f3113e0b799)
             There?
             ping-fops.c:1111:server_unlink_cbk] 0-test-server: 182512: UNLINK /renames/f97-516 (6562fd3a-d1d3-4aba-959a-55b88e2152e7/f97-516) ==      (Success)
             [2014-10-28 09:26:10.069936] E [posix-helpers.c:831:posix_handle_pair] 0-test-posix: /home/test10/renames/f13-517: key:trusted.glusterfs.dht.linkto error:File exists
             [2014-10-28 09:26:10.069977] E [posix.c:920:posix_mknod] 0-test-posix: setting xattrs on /home/test10/renames/f13-517 failed (File exists)
             [2014-10-28 09:26:10.072734] W [quota.c:1860:quota_rename_cbk] 0-test-quota: dentry corresponding to the path just renamed (name:f13-517) is not present
             [2014-10-28 09:26:10.074540] I [server-rpc-fops.c:581:server_mknod_cbk] 0-test-server: 260921: MKNOD (null) (6562fd3a-d1d3-4aba-959a-55b88e2152e7/f13-518) ==      (File exists)
             [2014-10-28 09:26:10.903803] W [posix.c:1266:posix_unlink] 0-test-posix: modification of parent gfid xattr failed (path:/home/test10/renames/f66-517 gfid:142232eb-12c9-4d22-9d1a-f20089f897a0)
             [2014-10-28 09:26:10.903861] E [server-rpc-fops.c:1111:server_unlink_cbk] 0-test-server: 182717: UNLINK /renames/f66-517 (6562fd3a-d1d3-4aba-959a-55b88e2152e7/f66-517) ==      (Success)
             [2014-10-28 09:26:11.148741] W [posix.c:1266:posix_unlink] 0-test-posix: modification of parent gfid xattr failed (path:/home/test10/renames/f83-517 gfid:96d87103-5daa-4c93-9224-0f3113e0b799)

Comment 3 Vijaikumar Mallikarjuna 2015-01-06 16:36:38 UTC
Patch 'https://code.engineering.redhat.com/gerrit/#/c/35825/' fixes the issue


Note You need to log in before you can comment on or make changes to this bug.