Created attachment 791627 [details] sosreports Description of problem: Not able to delete directories after upgrade. This happens intermittently, had not observed in previous upgrade exercises. rm -rf exits with error message: rm: cannot remove `sync-2.1/2/linux-3.10.9/drivers/atm': Directory not empty And rsync throws errors like: rsync: mkstemp "/mnt/replicate/sync-2.1/2/linux-3.10.9/drivers/auxdisplay/.Kconfig.hjyl3R" failed: No such file or directory (2) rsync: mkstemp "/mnt/replicate/sync-2.1/2/linux-3.10.9/drivers/auxdisplay/.Makefile.0jeGqB" failed: No such file or directory (2) rsync: mkstemp "/mnt/replicate/sync-2.1/2/linux-3.10.9/drivers/auxdisplay/.cfag12864b.c.W20pOk" failed: No such file or directory (2) NOTE: There were not disconnects or server downs. And strangely no `E' or `W' level logs in servers or clients. Version-Release number of selected component (if applicable): glusterfs 3.4.0.24rhs built on Aug 27 2013 08:08:42 How reproducible: Intermittent Steps to Reproduce: 1. Install U5 create lots of files and directories 2. Upgrade from U5 to Big Bend release 3.4.0.24 or latest 3. Try to remove the files and directories, create new files using rsync. Actual results: rm exits with failure rsync intermittently fails Expected results: There should not be any failures. Additional info: [root@Anshi-1 ~]# gluster vol info Volume Name: upgrade Type: Replicate Volume ID: a4a20b8c-e6fa-4b2a-b69c-106aebc5db2f Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 10.70.37.75:/rhs/brick1/rep-0 Brick2: 10.70.37.116:/rhs/brick1/rep-0 Options Reconfigured: performance.read-ahead: on performance.write-behind: on performance.flush-behind: on Attaching sosreports and client logs. Log snippet from client: [2013-08-29 08:39:14.489896] I [dht-layout.c:633:dht_layout_normalize] 0-upgrade-dht: found anomalies in /2/linux-3.10.9/arch/powerpc/configs/44x. holes=1 overlaps=0 missing=0 down=0 misc=0 [2013-08-29 08:39:14.504621] I [dht-layout.c:633:dht_layout_normalize] 0-upgrade-dht: found anomalies in /2/linux-3.10.9/arch/powerpc/configs/52xx. holes=1 overlaps=0 missing=0 down=0 misc=0 [2013-08-29 08:39:14.516102] I [dht-layout.c:633:dht_layout_normalize] 0-upgrade-dht: found anomalies in /2/linux-3.10.9/arch/powerpc/configs/83xx. holes=1 overlaps=0 missing=0 down=0 misc=0 [2013-08-29 08:39:14.529906] I [dht-layout.c:633:dht_layout_normalize] 0-upgrade-dht: found anomalies in /2/linux-3.10.9/arch/powerpc/configs/85xx. holes=1 overlaps=0 missing=0 down=0 misc=0 [2013-08-29 08:39:14.544157] I [dht-layout.c:633:dht_layout_normalize] 0-upgrade-dht: found anomalies in /2/linux-3.10.9/arch/powerpc/configs/86xx. holes=1 overlaps=0 missing=0 down=0 misc=0