Bug 1002419

Summary: Unable to remove directories after upgrade from U5 to 2.1
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Sachidananda Urs <surs>
Component: glusterfsAssignee: Bug Updates Notification Mailing List <rhs-bugs>
Status: CLOSED EOL QA Contact: storage-qa-internal <storage-qa-internal>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.1CC: rhs-bugs, spalai, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-27 10:50:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sosreports none

Description Sachidananda Urs 2013-08-29 07:29:30 UTC
Created attachment 791627 [details]
sosreports

Description of problem:
Not able to delete directories after upgrade. This happens intermittently, had not observed in previous upgrade exercises.

rm -rf exits with error message:
rm: cannot remove `sync-2.1/2/linux-3.10.9/drivers/atm': Directory not empty

And rsync throws errors like:

rsync: mkstemp "/mnt/replicate/sync-2.1/2/linux-3.10.9/drivers/auxdisplay/.Kconfig.hjyl3R" failed: No such file 
or directory (2)
rsync: mkstemp "/mnt/replicate/sync-2.1/2/linux-3.10.9/drivers/auxdisplay/.Makefile.0jeGqB" failed: No such file
 or directory (2)
rsync: mkstemp "/mnt/replicate/sync-2.1/2/linux-3.10.9/drivers/auxdisplay/.cfag12864b.c.W20pOk" failed: No such 
file or directory (2)

NOTE: There were not disconnects or server downs. And strangely no `E' or `W' level logs in servers or clients.

Version-Release number of selected component (if applicable):
glusterfs 3.4.0.24rhs built on Aug 27 2013 08:08:42

How reproducible:
Intermittent 

Steps to Reproduce:
1. Install U5 create lots of files and directories
2. Upgrade from U5 to Big Bend release 3.4.0.24 or latest
3. Try to remove the files and directories, create new files using rsync.

Actual results:
rm exits with failure
rsync intermittently fails

Expected results:
There should not be any failures.

Additional info:

[root@Anshi-1 ~]# gluster vol info
 
Volume Name: upgrade
Type: Replicate
Volume ID: a4a20b8c-e6fa-4b2a-b69c-106aebc5db2f
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.70.37.75:/rhs/brick1/rep-0
Brick2: 10.70.37.116:/rhs/brick1/rep-0
Options Reconfigured:
performance.read-ahead: on
performance.write-behind: on
performance.flush-behind: on


Attaching sosreports and client logs. Log snippet from client:

[2013-08-29 08:39:14.489896] I [dht-layout.c:633:dht_layout_normalize] 0-upgrade-dht: found anomalies in /2/linux-3.10.9/arch/powerpc/configs/44x. holes=1 overlaps=0 missing=0 down=0 misc=0
[2013-08-29 08:39:14.504621] I [dht-layout.c:633:dht_layout_normalize] 0-upgrade-dht: found anomalies in /2/linux-3.10.9/arch/powerpc/configs/52xx. holes=1 overlaps=0 missing=0 down=0 misc=0
[2013-08-29 08:39:14.516102] I [dht-layout.c:633:dht_layout_normalize] 0-upgrade-dht: found anomalies in /2/linux-3.10.9/arch/powerpc/configs/83xx. holes=1 overlaps=0 missing=0 down=0 misc=0
[2013-08-29 08:39:14.529906] I [dht-layout.c:633:dht_layout_normalize] 0-upgrade-dht: found anomalies in /2/linux-3.10.9/arch/powerpc/configs/85xx. holes=1 overlaps=0 missing=0 down=0 misc=0
[2013-08-29 08:39:14.544157] I [dht-layout.c:633:dht_layout_normalize] 0-upgrade-dht: found anomalies in /2/linux-3.10.9/arch/powerpc/configs/86xx. holes=1 overlaps=0 missing=0 down=0 misc=0