Bug 762885 (GLUSTER-1153)

Summary:

Rename fails, errors in logfile

Product:

[Community] GlusterFS

Reporter:

Sachidananda Urs <sac>

Component:

replicate

Assignee:

Pavan Vilas Sondur <pavan>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Severity:

high

Docs Contact:

Priority:

urgent

Version:

3.0.5

CC:

chida, gluster-bugs, leo, shehjart, tejas, vijay

Target Milestone:

---

Target Release:

---

Hardware:

All

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Attachments:

Description	Flags
Ruby file to rename	none

Description Sachidananda Urs 2010-07-16 02:56:14 UTC

Created attachment 253 [details]
up2date config file used

Comment 1 Shehjar Tikoo 2010-07-16 05:41:25 UTC

What version? mainline or a 3.0.x? Please set it in the field above.

Comment 2 Sachidananda Urs 2010-07-16 05:54:11 UTC

Rename fails in a replicate setup. I have a attached a sample script to reproduce the issue.

The script is creating a temp file, then using a unix rename 
http://linux.die.net/man/2/rename to rename the original file to 
[filename] to [filename].orig.

The behaviour we are getting out of the gfs mount is not consistent.

It appears that after issuing the rename a subsequent check (after a 2 
second sleep) still returns no change from the rename, the change 
sometimes apears sometimes doesn't.
I would expect that once a rename is issued any calls to the gluster 
drive would return the result post rename not a random chance of pre or 
post update.

Regarding the script in summary:

1) Create tempfile,
2) Write contents of realfile to tempfile,
3) Add tags to tempfile,
4) Rename tempfile to realfile using rename(2)
ERROR: The rename(2) system call fails because it thinks realfile 
exists even after it has unlinked it.

Because part way through rename(2),
a) a stat is done for realfile,
b) it is either there or it isn't,
c) if it does exist, it is unlinked,
d) CRASH because the next operation uses a non-synced GFS table and 
finds that even after the unlink in 'c', the file still exists.
(The reverse can be true of course - 'c' doesn't unlink because the 
stat call says it's not there, but then the internal rename inside 
rename(2) then crashes because it is now there)

Log:

[2010-07-13 04:08:49] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3 
failed: File exists
[2010-07-13 04:08:49] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3 
failed: File exists
[2010-07-13 04:20:23] E [posix.c:2068:posix_link] posix1: link 
/Assets/UMGI/03.ForIngest/2010-03-30/3000000142305/00602527348339/01_002_Fever(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-30/3000000142305/00602527348339/01_002_Fever(Cascada)_256.mp3 
failed: File exists
[2010-07-13 04:20:23] E [posix.c:2068:posix_link] posix1: link 
/Assets/UMGI/03.ForIngest/2010-03-30/3000000142305/00602527348339/01_002_Fever(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-30/3000000142305/00602527348339/01_002_Fever(Cascada)_256.mp3 
failed: File exists
[2010-07-14 09:03:50] E [posix.c:1304:posix_mknod] posix3: mknod on 
/Assets/UMGI/03.ForIngest/2010-05-11/3000000420613/00602527361307/01_002_Pyromania(Cascada)_256.mp3 
failed: File exists
[2010-07-15 00:12:13] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3 
failed: File exists
[2010-07-15 00:12:13] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
failed: File exists


[root@gfs01 guvera]# tail -f /var/log/glusterfs/gfs-export.log
[2010-07-15 00:10:27] E [posix.c:1304:posix_mknod] posix2: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00050087152451/01_013_Alice_Reprise_3(Danny_Elfman)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:10:57] E [posix.c:1304:posix_mknod] posix3: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753229507/01_007_All_The_Things_You_Are(Frank_Chacksfield_And_His_Orchestra_and_Chorus_Artist)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:11:07] E [posix.c:1304:posix_mknod] posix2: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254264/01_002_Sixteen(Hugh_Coltman)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:11:43] E [posix.c:2068:posix_link] posix1: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254271/01_007_A_Gozar_El_Tumbao(Orlando_Cachaito_Lopez)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254271/01_007_A_Gozar_El_Tumbao(Orlando_Cachaito_Lopez)_256.mp3 
failed: File exists
[2010-07-15 00:11:43] E [posix.c:1304:posix_mknod] posix3: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254271/01_007_A_Gozar_El_Tumbao(Orlando_Cachaito_Lopez)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:12:04] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254431/01_006_Fantasia_For_Clarinet(Kocani_Orkestar)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:12:13] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3 
failed: File exists
[2010-07-15 00:12:13] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:12:25] E [posix.c:1304:posix_mknod] posix4: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753257289/01_002_To_Love_You_All_Over_Again(Madeleine_Peyroux)_256.mp3 
failed: File exists
[2010-07-15 00:31:27] E [posix.c:654:posix_setattr] posix2: setattr 
(lstat) on 
/exports/sdd1/gfs/Assets/UMGI/03.ForIngest/2010-04-16/3000000218748/00602498289457/04_003_Bossa_Nova(Friedrich_Gulda)_256.mp3.orig 
failed: No such file or directory

[root@gfs02 sdc2]# tail -f /var/log/glusterfs/gfs-export.log
[2010-07-15 00:04:51] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05050159933525/01_003_Theres_No_Me_Without_You(Marcia_Griffiths)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05050159933525/01_003_Theres_No_Me_Without_You(Marcia_Griffiths)_256.mp3 
failed: File exists
[2010-07-15 00:05:04] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05050159935826/01_003_Buttoo(Gregory_Isaacs)_256.mp3 
failed: File exists
[2010-07-15 00:05:40] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05060078524858/01_001_Inside_Out(The_Somebodys_Feat_Rachael_Bell)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05060078524858/01_001_Inside_Out(The_Somebodys_Feat_Rachael_Bell)_256.mp3 
failed: File exists
[2010-07-15 00:06:37] E [posix.c:1304:posix_mknod] posix4: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00011661327726/01_004_Think_About_The_Good_Times(Nathan_and_the_Zydeco_Cha_Chas)_256.mp3 
failed: File exists
[2010-07-15 00:07:41] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783164/01_001_Get_Off_On_The_Pain(Gary_Allan)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783164/01_001_Get_Off_On_The_Pain(Gary_Allan)_256.mp3 
failed: File exists
[2010-07-15 00:07:54] E [posix.c:2068:posix_link] posix4: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783614/01_003_Mentiras(Cucu_Diamantes)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783614/01_003_Mentiras(Cucu_Diamantes)_256.mp3 
failed: File exists
[2010-07-15 00:08:00] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003784048/01_001_Cup_Of_Tea_Song(Judie_Tzuke)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003784048/01_001_Cup_Of_Tea_Song(Judie_Tzuke)_256.mp3 
failed: File exists
[2010-07-15 00:10:57] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753229507/01_007_All_The_Things_You_Are(Frank_Chacksfield_And_His_Orchestra_and_Chorus_Artist)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753229507/01_007_All_The_Things_You_Are(Frank_Chacksfield_And_His_Orchestra_and_Chorus_Artist)_256.mp3 
failed: File exists
[2010-07-15 00:11:07] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254264/01_002_Sixteen(Hugh_Coltman)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254264/01_002_Sixteen(Hugh_Coltman)_256.mp3 
failed: File exists
[2010-07-15 00:12:04] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254431/01_006_Fantasia_For_Clarinet(Kocani_Orkestar)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254431/01_006_Fantasia_For_Clarinet(Kocani_Orkestar)_256.mp3 
failed: File exists
[2010-07-15 00:40:20] E [posix.c:654:posix_setattr] posix4: setattr 
(lstat) on 
/exports/sdc2/gfs/Assets/UMGI/03.ForIngest/2010-04-18/3000000219380/00028947584889/01_016_Down_Over_Lake_Victoria_(from_The_Last_King_of_Scotland)(Orchestra)_256.mp3.orig 
failed: No such file or directory

[root@gfs03 guvera]# tail -f /var/log/glusterfs/gfs-export.log
[2010-07-15 00:10:27] E [posix.c:1304:posix_mknod] posix2: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00050087152451/01_013_Alice_Reprise_3(Danny_Elfman)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:10:57] E [posix.c:1304:posix_mknod] posix3: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753229507/01_007_All_The_Things_You_Are(Frank_Chacksfield_And_His_Orchestra_and_Chorus_Artist)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:11:07] E [posix.c:1304:posix_mknod] posix2: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254264/01_002_Sixteen(Hugh_Coltman)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:11:43] E [posix.c:2068:posix_link] posix1: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254271/01_007_A_Gozar_El_Tumbao(Orlando_Cachaito_Lopez)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254271/01_007_A_Gozar_El_Tumbao(Orlando_Cachaito_Lopez)_256.mp3 
failed: File exists
[2010-07-15 00:11:43] E [posix.c:1304:posix_mknod] posix3: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254271/01_007_A_Gozar_El_Tumbao(Orlando_Cachaito_Lopez)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:12:04] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254431/01_006_Fantasia_For_Clarinet(Kocani_Orkestar)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:12:13] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
failed: File exists
[2010-07-15 00:12:13] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753255926/01_001_Evacuate_The_Dancefloor(Cascada)_256.mp3 
failed: File exists
[2010-07-15 00:12:25] E [posix.c:1304:posix_mknod] posix4: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753257289/01_002_To_Love_You_All_Over_Again(Madeleine_Peyroux)_256.mp3 
failed: File exists
[2010-07-15 00:31:27] E [posix.c:654:posix_setattr] posix2: setattr 
(lstat) on 
/exports/sdd1/gfs/Assets/UMGI/03.ForIngest/2010-04-16/3000000218748/00602498289457/04_003_Bossa_Nova(Friedrich_Gulda)_256.mp3.orig 
failed: No such file or directory



[root@gfs04 ~]# tail -f /var/log/glusterfs/gfs-export.log
[2010-07-15 00:04:51] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05050159933525/01_003_Theres_No_Me_Without_You(Marcia_Griffiths)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05050159933525/01_003_Theres_No_Me_Without_You(Marcia_Griffiths)_256.mp3 
failed: File exists
[2010-07-15 00:05:04] E [posix.c:1304:posix_mknod] posix1: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05050159935826/01_003_Buttoo(Gregory_Isaacs)_256.mp3 
failed: File exists
[2010-07-15 00:05:40] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05060078524858/01_001_Inside_Out(The_Somebodys_Feat_Rachael_Bell)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949553/05060078524858/01_001_Inside_Out(The_Somebodys_Feat_Rachael_Bell)_256.mp3 
failed: File exists
[2010-07-15 00:06:37] E [posix.c:1304:posix_mknod] posix4: mknod on 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00011661327726/01_004_Think_About_The_Good_Times(Nathan_and_the_Zydeco_Cha_Chas)_256.mp3 
failed: File exists
[2010-07-15 00:07:41] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783164/01_001_Get_Off_On_The_Pain(Gary_Allan)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783164/01_001_Get_Off_On_The_Pain(Gary_Allan)_256.mp3 
failed: File exists
[2010-07-15 00:07:54] E [posix.c:2068:posix_link] posix4: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783614/01_003_Mentiras(Cucu_Diamantes)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003783614/01_003_Mentiras(Cucu_Diamantes)_256.mp3 
failed: File exists
[2010-07-15 00:08:00] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003784048/01_001_Cup_Of_Tea_Song(Judie_Tzuke)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00044003784048/01_001_Cup_Of_Tea_Song(Judie_Tzuke)_256.mp3 
failed: File exists
[2010-07-15 00:10:57] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753229507/01_007_All_The_Things_You_Are(Frank_Chacksfield_And_His_Orchestra_and_Chorus_Artist)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753229507/01_007_All_The_Things_You_Are(Frank_Chacksfield_And_His_Orchestra_and_Chorus_Artist)_256.mp3 
failed: File exists
[2010-07-15 00:11:07] E [posix.c:2068:posix_link] posix3: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254264/01_002_Sixteen(Hugh_Coltman)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254264/01_002_Sixteen(Hugh_Coltman)_256.mp3 
failed: File exists
[2010-07-15 00:12:04] E [posix.c:2068:posix_link] posix2: link 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254431/01_006_Fantasia_For_Clarinet(Kocani_Orkestar)_256.mp3.tmp 
to 
/Assets/UMGI/03.ForIngest/2010-03-01/2000000949554/00600753254431/01_006_Fantasia_For_Clarinet(Kocani_Orkestar)_256.mp3 
failed: File exists
[2010-07-15 00:40:20] E [posix.c:654:posix_setattr] posix4: setattr 
(lstat) on 
/exports/sdc1/gfs/Assets/UMGI/03.ForIngest/2010-04-18/3000000219380/00028947584889/01_016_Down_Over_Lake_Victoria_(from_The_Last_King_of_Scotland)(Orchestra)_256.mp3.orig 
failed: No such file or directory

Comment 3 Sachidananda Urs 2010-07-16 06:35:44 UTC

(In reply to comment #2)
> What version? mainline or a 3.0.x? Please set it in the field above.

Version is 3.0.5. It is not available in the field above.

Comment 4 Vijay Bellur 2010-08-24 03:10:45 UTC

Sac, Is this reproducible now?

Comment 5 Sachidananda Urs 2010-08-24 05:04:27 UTC

(In reply to comment #4)
> Sac, Is this reproducible now?

No, I think you can mark this as fixed.