Bug 990534
| Summary: | ENOENT errors while deleting directory after add-brick and rebalance | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Anush Shetty <ashetty> |
| Component: | distribute | Assignee: | Nithya Balachandran <nbalacha> |
| Status: | CLOSED DUPLICATE | QA Contact: | storage-qa-internal <storage-qa-internal> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 2.1 | CC: | chrisw, csaba, rhs-bugs, spalai |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-11-27 10:45:49 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Sosreports: 10.70.37.49 (RHSClient was mounted on this machine): http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/990534/sosreport-10.70.37.49-20130731173032-0997.tar.xz 10.70.37.120: http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/990534/sosreport-10.70.37.120-20130731173453-eb6d.tar.xz 10.70.37.132; http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/990534/sosreport-10.70.37.132-20130731173744-75c9.tar.xz *** This bug has been marked as a duplicate of bug 1115367 *** |
Description of problem: While deleting a directory on fuse mount, we see ENOENT errors. The issue persisted for a couple of "rm -rf" attempts and later we are able to successfuly delete the directory. Version-Release number of selected component (if applicable): glusterfs-3.4.0.14rhs-1.el6rhs.x86_64 How reproducible: Saw this issue once Steps to Reproduce: 1. Created a distributed-replicate volume. 2. Created a directory with 1000 files. 3. Added 2 bricks and executed rebalance 4. Tried deleting the directory Actual results: Saw ENOENT errors while deleting the directory for a couple of times. # tail /var/log/glusterfs/mnt-gluster.log [2013-07-31 11:35:56.974726] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-acl-client-3: remote operation failed: No such file or directory. Path: /files (f423eb7e-ee33-4a05-a36e-c11cad25dd3d) [2013-07-31 11:35:56.981423] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-acl-client-2: remote operation failed: No such file or directory. Path: /files (f423eb7e-ee33-4a05-a36e-c11cad25dd3d) [2013-07-31 11:35:56.981689] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-acl-client-3: remote operation failed: No such file or directory. Path: /files (f423eb7e-ee33-4a05-a36e-c11cad25dd3d) [2013-07-31 11:35:57.093259] W [client-rpc-fops.c:1631:client3_3_entrylk_cbk] 4-acl-client-3: remote operation failed: No such file or directory [2013-07-31 11:35:57.093859] W [client-rpc-fops.c:1631:client3_3_entrylk_cbk] 4-acl-client-2: remote operation failed: No such file or directory [2013-07-31 11:35:57.095903] W [client-rpc-fops.c:1631:client3_3_entrylk_cbk] 4-acl-client-2: remote operation failed: No such file or directory [2013-07-31 11:35:57.096430] W [client-rpc-fops.c:1631:client3_3_entrylk_cbk] 4-acl-client-3: remote operation failed: No such file or directory [2013-07-31 11:35:57.097184] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-acl-client-3: remote operation failed: No such file or directory [2013-07-31 11:35:57.097370] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-acl-client-2: remote operation failed: No such file or directory [2013-07-31 11:35:57.097419] W [fuse-bridge.c:1673:fuse_unlink_cbk] 0-glusterfs-fuse: 184019: RMDIR() /files => -1 (No such file or directory) Expected results: Repeated attempts to remove the directory is an unacceptable behaviour. Additional info: # gluster volume info acl Volume Name: acl Type: Distributed-Replicate Volume ID: 4c8234b4-8d74-4048-97a9-0d2f519996d1 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.70.37.49:/glance2/s1 Brick2: 10.70.37.120:/glance2/s2 Brick3: 10.70.37.132:/glance2/s3_new Brick4: 10.70.37.120:/glance2/s4_new Options Reconfigured: diagnostics.client-log-level: INFO diagnostics.brick-log-level: INFO performance.stat-prefetch: off # gluster volume status acl Status of volume: acl Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.37.49:/glance2/s1 49153 Y 13328 Brick 10.70.37.120:/glance2/s2 49153 Y 13083 Brick 10.70.37.132:/glance2/s3_new 49156 Y 18104 Brick 10.70.37.120:/glance2/s4_new 49157 Y 11545 NFS Server on localhost 2049 Y 22750 Self-heal Daemon on localhost N/A Y 22286 NFS Server on 10.70.37.120 2049 Y 11844 Self-heal Daemon on 10.70.37.120 N/A Y 11564 NFS Server on 10.70.37.208 2049 Y 17905 Self-heal Daemon on 10.70.37.208 N/A Y 17633 NFS Server on 10.70.37.132 2049 Y 18358 Self-heal Daemon on 10.70.37.132 N/A Y 18123 Task ID Status ---- -- ------ Rebalance 0821e548-9f64-4426-8da6-69ecadd88b18 3 # df -h 10.70.37.49:acl 100G 112M 100G 1% /mnt/gluster