Bug 762885 (GLUSTER-1153) - Rename fails, errors in logfile
Summary: Rename fails, errors in logfile
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: GLUSTER-1153
Product: GlusterFS
Classification: Community
Component: replicate
Version: 3.0.5
Hardware: All
OS: Linux
urgent
high
Target Milestone: ---
Assignee: Pavan Vilas Sondur
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-07-16 05:54 UTC by Sachidananda Urs
Modified: 2015-12-01 16:45 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:

Attachments (Terms of Use)
Ruby file to rename (1.88 KB, application/x-ruby)
2010-07-16 02:56 UTC, Sachidananda Urs 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Description Sachidananda Urs 2010-07-16 02:56:14 UTC 
Created attachment 253 [details]
up2date config file used
Comment 1 Shehjar Tikoo 2010-07-16 05:41:25 UTC 
What version? mainline or a 3.0.x? Please set it in the field above.
Comment 2 Sachidananda Urs 2010-07-16 05:54:11 UTC 
Rename fails in a replicate setup. I have a attached a sample script to reproduce the issue.

The script is creating a temp file, then using a unix rename 
http://linux.die.net/man/2/rename to rename the original file to 
[filename] to [filename].orig.

The behaviour we are getting out of the gfs mount is not consistent.

It appears that after issuing the rename a subsequent check (after a 2 
second sleep) still returns no change from the rename, the change 
sometimes apears sometimes doesn't.
I would expect that once a rename is issued any calls to the gluster 
drive would return the result post rename not a random chance of pre or 
post update.

Regarding the script in summary:

1) Create tempfile,
2) Write contents of realfile to tempfile,
3) Add tags to tempfile,
4) Rename tempfile to realfile using rename(2)
ERROR: The rename(2) system call fails because it thinks realfile 
exists even after it has unlinked it.

Because part way through rename(2),
a) a stat is done for realfile,
b) it is either there or it isn't,
c) if it does exist, it is unlinked,
d) CRASH because the next operation uses a non-synced GFS table and 
finds that even after the unlink in 'c', the file still exists.
(The reverse can be true of course - 'c' doesn't unlink because the 
stat call says it's not there, but then the internal rename inside 
rename(2) then crashes because it is now there)

Log:

[2010-07-13 04:08:49] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3 
failed: File exists
[2010-07-13 04:08:49] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3 
failed: File exists
[2010-07-13 04:20:23] E [posix.c:2068:posix_link] posix1: link 
/Assets/UMGI/03.ForIngest/2010-03-30/3000000142305/00602527348339/01_002_Fever(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-30/3000000142305/00602527348339/01_002_Fever(Cascada)_256.mp3 
failed: File exists
[2010-07-13 04:20:23] E [posix.c:2068:posix_link] posix1: link 
/Assets/UMGI/03.ForIngest/2010-03-30/3000000142305/00602527348339/01_002_Fever(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-30/3000000142305/00602527348339/01_002_Fever(Cascada)_256.mp3 
failed: File exists
[2010-07-14 09:03:50] E [posix.c:1304:posix_mknod] posix3: mknod on 
/Assets/UMGI/03.ForIngest/2010-05-11/3000000420613/00602527361307/01_002_Pyromania(Cascada)_256.mp3 
failed: File exists
[2010-07-15 00:12:13] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3 
failed: File exists
[2010-07-15 00:12:13] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
failed: File exists


[root@gfs01 guvera]# tail -f /var/log/glusterfs/gfs-export.log
[2010-07-15 00:10:27] E [posix.c:1304:posix_mknod] posix2: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00050087152451/01_013_Alice_Reprise_3(Danny_Elfman)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:10:57] E [posix.c:1304:posix_mknod] posix3: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753229507/01_007_All_The_Things_You_Are(Frank_Chacksfield_And_His_Orchestra_and_Chorus_Artist)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:11:07] E [posix.c:1304:posix_mknod] posix2: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254264/01_002_Sixteen(Hugh_Coltman)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:11:43] E [posix.c:2068:posix_link] posix1: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254271/01_007_A_Gozar_El_Tumbao(Orlando_Cachaito_Lopez)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254271/01_007_A_Gozar_El_Tumbao(Orlando_Cachaito_Lopez)_256.mp3 
failed: File exists
[2010-07-15 00:11:43] E [posix.c:1304:posix_mknod] posix3: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254271/01_007_A_Gozar_El_Tumbao(Orlando_Cachaito_Lopez)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:12:04] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254431/01_006_Fantasia_For_Clarinet(Kocani_Orkestar)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:12:13] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3 
failed: File exists
[2010-07-15 00:12:13] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:12:25] E [posix.c:1304:posix_mknod] posix4: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753257289/01_002_To_Love_You_All_Over_Again(Madeleine_Peyroux)_256.mp3 
failed: File exists
[2010-07-15 00:31:27] E [posix.c:654:posix_setattr] posix2: setattr 
(lstat) on 
/exports/sdd1/gfs/Assets/UMGI/03.ForIngest/2010-04-16/3000000218748/00602498289457/04_003_Bossa_Nova(Friedrich_Gulda)_256.mp3.orig 
failed: No such file or directory

[root@gfs02 sdc2]# tail -f /var/log/glusterfs/gfs-export.log
[2010-07-15 00:04:51] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05050159933525/01_003_Theres_No_Me_Without_You(Marcia_Griffiths)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05050159933525/01_003_Theres_No_Me_Without_You(Marcia_Griffiths)_256.mp3 
failed: File exists
[2010-07-15 00:05:04] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05050159935826/01_003_Buttoo(Gregory_Isaacs)_256.mp3 
failed: File exists
[2010-07-15 00:05:40] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05060078524858/01_001_Inside_Out(The_Somebodys_Feat_Rachael_Bell)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05060078524858/01_001_Inside_Out(The_Somebodys_Feat_Rachael_Bell)_256.mp3 
failed: File exists
[2010-07-15 00:06:37] E [posix.c:1304:posix_mknod] posix4: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00011661327726/01_004_Think_About_The_Good_Times(Nathan_and_the_Zydeco_Cha_Chas)_256.mp3 
failed: File exists
[2010-07-15 00:07:41] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783164/01_001_Get_Off_On_The_Pain(Gary_Allan)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783164/01_001_Get_Off_On_The_Pain(Gary_Allan)_256.mp3 
failed: File exists
[2010-07-15 00:07:54] E [posix.c:2068:posix_link] posix4: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783614/01_003_Mentiras(Cucu_Diamantes)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783614/01_003_Mentiras(Cucu_Diamantes)_256.mp3 
failed: File exists
[2010-07-15 00:08:00] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003784048/01_001_Cup_Of_Tea_Song(Judie_Tzuke)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003784048/01_001_Cup_Of_Tea_Song(Judie_Tzuke)_256.mp3 
failed: File exists
[2010-07-15 00:10:57] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753229507/01_007_All_The_Things_You_Are(Frank_Chacksfield_And_His_Orchestra_and_Chorus_Artist)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753229507/01_007_All_The_Things_You_Are(Frank_Chacksfield_And_His_Orchestra_and_Chorus_Artist)_256.mp3 
failed: File exists
[2010-07-15 00:11:07] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254264/01_002_Sixteen(Hugh_Coltman)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254264/01_002_Sixteen(Hugh_Coltman)_256.mp3 
failed: File exists
[2010-07-15 00:12:04] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254431/01_006_Fantasia_For_Clarinet(Kocani_Orkestar)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254431/01_006_Fantasia_For_Clarinet(Kocani_Orkestar)_256.mp3 
failed: File exists
[2010-07-15 00:40:20] E [posix.c:654:posix_setattr] posix4: setattr 
(lstat) on 
/exports/sdc2/gfs/Assets/UMGI/03.ForIngest/2010-04-18/3000000219380/00028947584889/01_016_Down_Over_Lake_Victoria_(from_The_Last_King_of_Scotland)(Orchestra)_256.mp3.orig 
failed: No such file or directory

[root@gfs03 guvera]# tail -f /var/log/glusterfs/gfs-export.log
[2010-07-15 00:10:27] E [posix.c:1304:posix_mknod] posix2: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00050087152451/01_013_Alice_Reprise_3(Danny_Elfman)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:10:57] E [posix.c:1304:posix_mknod] posix3: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753229507/01_007_All_The_Things_You_Are(Frank_Chacksfield_And_His_Orchestra_and_Chorus_Artist)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:11:07] E [posix.c:1304:posix_mknod] posix2: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254264/01_002_Sixteen(Hugh_Coltman)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:11:43] E [posix.c:2068:posix_link] posix1: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254271/01_007_A_Gozar_El_Tumbao(Orlando_Cachaito_Lopez)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254271/01_007_A_Gozar_El_Tumbao(Orlando_Cachaito_Lopez)_256.mp3 
failed: File exists
[2010-07-15 00:11:43] E [posix.c:1304:posix_mknod] posix3: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254271/01_007_A_Gozar_El_Tumbao(Orlando_Cachaito_Lopez)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:12:04] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254431/01_006_Fantasia_For_Clarinet(Kocani_Orkestar)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:12:13] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:12:13] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3 
failed: File exists
[2010-07-15 00:12:25] E [posix.c:1304:posix_mknod] posix4: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753257289/01_002_To_Love_You_All_Over_Again(Madeleine_Peyroux)_256.mp3 
failed: File exists
[2010-07-15 00:31:27] E [posix.c:654:posix_setattr] posix2: setattr 
(lstat) on 
/exports/sdd1/gfs/Assets/UMGI/03.ForIngest/2010-04-16/3000000218748/00602498289457/04_003_Bossa_Nova(Friedrich_Gulda)_256.mp3.orig 
failed: No such file or directory



[root@gfs04 ~]# tail -f /var/log/glusterfs/gfs-export.log
[2010-07-15 00:04:51] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05050159933525/01_003_Theres_No_Me_Without_You(Marcia_Griffiths)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05050159933525/01_003_Theres_No_Me_Without_You(Marcia_Griffiths)_256.mp3 
failed: File exists
[2010-07-15 00:05:04] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05050159935826/01_003_Buttoo(Gregory_Isaacs)_256.mp3 
failed: File exists
[2010-07-15 00:05:40] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05060078524858/01_001_Inside_Out(The_Somebodys_Feat_Rachael_Bell)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05060078524858/01_001_Inside_Out(The_Somebodys_Feat_Rachael_Bell)_256.mp3 
failed: File exists
[2010-07-15 00:06:37] E [posix.c:1304:posix_mknod] posix4: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00011661327726/01_004_Think_About_The_Good_Times(Nathan_and_the_Zydeco_Cha_Chas)_256.mp3 
failed: File exists
[2010-07-15 00:07:41] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783164/01_001_Get_Off_On_The_Pain(Gary_Allan)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783164/01_001_Get_Off_On_The_Pain(Gary_Allan)_256.mp3 
failed: File exists
[2010-07-15 00:07:54] E [posix.c:2068:posix_link] posix4: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783614/01_003_Mentiras(Cucu_Diamantes)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783614/01_003_Mentiras(Cucu_Diamantes)_256.mp3 
failed: File exists
[2010-07-15 00:08:00] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003784048/01_001_Cup_Of_Tea_Song(Judie_Tzuke)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003784048/01_001_Cup_Of_Tea_Song(Judie_Tzuke)_256.mp3 
failed: File exists
[2010-07-15 00:10:57] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753229507/01_007_All_The_Things_You_Are(Frank_Chacksfield_And_His_Orchestra_and_Chorus_Artist)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753229507/01_007_All_The_Things_You_Are(Frank_Chacksfield_And_His_Orchestra_and_Chorus_Artist)_256.mp3 
failed: File exists
[2010-07-15 00:11:07] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254264/01_002_Sixteen(Hugh_Coltman)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254264/01_002_Sixteen(Hugh_Coltman)_256.mp3 
failed: File exists
[2010-07-15 00:12:04] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254431/01_006_Fantasia_For_Clarinet(Kocani_Orkestar)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254431/01_006_Fantasia_For_Clarinet(Kocani_Orkestar)_256.mp3 
failed: File exists
[2010-07-15 00:40:20] E [posix.c:654:posix_setattr] posix4: setattr 
(lstat) on 
/exports/sdc1/gfs/Assets/UMGI/03.ForIngest/2010-04-18/3000000219380/00028947584889/01_016_Down_Over_Lake_Victoria_(from_The_Last_King_of_Scotland)(Orchestra)_256.mp3.orig 
failed: No such file or directory
Comment 3 Sachidananda Urs 2010-07-16 06:35:44 UTC 
(In reply to comment #2)
> What version? mainline or a 3.0.x? Please set it in the field above.

Version is 3.0.5. It is not available in the field above.
Comment 4 Vijay Bellur 2010-08-24 03:10:45 UTC 
Sac, Is this reproducible now?
Comment 5 Sachidananda Urs 2010-08-24 05:04:27 UTC 
(In reply to comment #4)
> Sac, Is this reproducible now?

No, I think you can mark this as fixed.
Note You need to log in before you can comment on or make changes to this bug.