Bug 489222 - unable to see/create non-ASCII utf-8 filenames in vfat USB stricks
Summary: unable to see/create non-ASCII utf-8 filenames in vfat USB stricks
Alias: None
Product: Fedora
Classification: Fedora
Component: DeviceKit-disks
Version: rawhide
Hardware: All
OS: Linux
Target Milestone: ---
Assignee: David Zeuthen
QA Contact: Fedora Extras Quality Assurance
Depends On:
Blocks: F11Blocker, F11FinalBlocker
TreeView+ depends on / blocked
Reported: 2009-03-08 19:23 UTC by Muayyad Alsadi
Modified: 2009-04-09 19:47 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 470592
Last Closed: 2009-04-08 21:54:46 UTC
Type: ---

Attachments (Terms of Use)
patch to that effect (706 bytes, patch)
2009-03-09 09:14 UTC, Yanko Kaneti
no flags Details | Diff

Description Muayyad Alsadi 2009-03-08 19:23:39 UTC
+++ This bug was initially created as a clone of Bug #470592 +++

Description of problem:
fedora has dropped a patch from kernel that forces IOCHARSET=utf-8
to make people with non-UTF-8 locales happy

but it makes people with UTF-8 locales sad
when I put my USB stick and copy/paste files with non-ASCII charts
or when I chdir there, I got errors

if you really want to release fedora with such kerenl you should add a hal fdi configuration file that pass utf8 to mount

just like what was done before that patch is introduced

as kernel docs (Documentation/filesystems/vfat.txt) there is a safe utf8
by using utf8 option instead of iocharset=utf8


Version-Release number of selected component (if applicable):
fedora 10

How reproducible:

Steps to Reproduce:
[alsadi@pc1 ALSADI]$ touch "سلام"
touch: setting times of `سلام': No such file or directory
[alsadi@pc1 ALSADI]$ touch peace
[alsadi@pc1 ALSADI]$ mkdir "الإسلام"
mkdir: cannot create directory `الإسلام': Invalid argument
Expected results:
no seeing the errors and having the files/dirs created with non-ASCII chars

Additional info:
having non-UTF-8 filesystem encoding could bug many poorly written GTK+ application

--- Additional comment from wwoods on 2008-11-07 16:42:26 EDT ---

*** Bug 470086 has been marked as a duplicate of this bug. ***

--- Additional comment from wwoods on 2008-11-07 16:48:11 EDT ---

Forcing utf8 in the kernel was the wrong idea - see bug 454013 for an example of the problems that caused. Further it made Fedora behave differently from all other Linux distributions when mounting vfat volumes, which is bad.

So dropping that kernel patch was probably the right decision, but it seems that we still need 'utf8' when mounting vfat for a lot of locales. So something in userspace needs to enable utf8 when needed. Unfortunately figuring out *when* it's needed is really hard. 

To maintain compatibility with older Fedora releases we could probably have HAL suggest use that flag by default - as with bug 144600. In a future release we might somehow decide that based on the current locale.

--- Additional comment from alsadi on 2008-11-10 12:33:00 EDT ---

from bug 454013 
> ru_RU.cp1251 or ru_RU.koi8r
you should flag that bug as "non-BUG" becayse
the default locale in fedora for Russian locales is UTF-8 just look at this:

grep '^ru' /usr/share/system-config-language/locale-list
ru_RU.UTF-8 utf8 latarcyrheb-sun16 Russian  -  Русский
ru_UA.UTF-8 utf8 latarcyrheb-sun16 Russian (Ukraine)

so it's his own problem, he should edit his hal files to do his customizations

utf-8 is universal, forcing latin1 or whatever should not be done in an international project as fedora

so if you want to go upstream, then you should move the patch as a hal configuration

and those who want to use non-international encoding let them use some /etc/hal/fdi/policy/ file

if you insist on not making utf8 the default on hal level at least make a package having utf-8 fdi file for hal and make that package a default in the comps file for each *-support group

be aware of the following facts
grep 'utf8' /usr/share/system-config-language/locale-list | wc -l
wc -l /usr/share/system-config-language/locale-list
143 /usr/share/system-config-language/locale-list

which mean you are breaking more than 95% of the locales by not making utf8 tobe the default

my suggestion is doing the opposite having the default to be the default as a fdi hal file
and having 2 packages (8859-15 and 8859-2) for all the 7 non-utf-8 locales and make that one a default or optional in the comps file

grep -v 'utf8' /usr/share/system-config-language/locale-list
af_ZA 8859-15 lat0-sun16 Afrikaans (South Africa)
bs_BA 8859-2 lat2-sun16 Bosnian (Bosnia and Herzegowina)
br_FR 8859-15 lat0-sun16 Breton (France)
oc_FR 8859-15 lat0-sun16 Occitan (France)
tl_PH 8859-15 lat0-sun16 Tagalog (Philippines)
uz_UZ 8859-15 lat0-sun16 Uzbek (Uzbekistan)
wa_BE@euro 8859-15 lat0-sun16 Walloon (Belgium)

--- Additional comment from wwoods on 2008-11-10 16:20:36 EDT ---

After much discussion we've decided to restore the kernel patch that enables 'utf8' by default on vfat, at least for F10 final:

* Mon Nov 10 2008 Dave Jones <davej>
- Make UTF-8 the default for FAT/VFAT filesystems again.

F11 will not include this patch, so this bug will need to be fixed in userspace (HAL etc.) in Rawhide.

Moving to F11Blocker.

--- Additional comment from fedora-triage-list on 2008-11-26 00:00:44 EDT ---

This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:

--- Additional comment from alsadi on 2008-11-26 17:35:32 EDT ---

I guess it won't be in hal only but also to DeviceKit which is planed for F11

Comment 1 Yanko Kaneti 2009-03-09 09:14:32 UTC
Created attachment 334479 [details]
patch to that effect

This should be a F11 blocker.

Comment 2 Matthias Clasen 2009-03-24 16:34:13 UTC
This has been fixed upstream, we'll get it with the next update.

Comment 3 Muayyad Alsadi 2009-03-24 17:01:32 UTC
good news

we need similar one for hal, I guess it would be just an fdi policy file

Comment 4 Matthias Clasen 2009-04-07 04:39:30 UTC
Should be fixed in DeviceKit-disks-004 in rawhide.

Comment 5 Yanko Kaneti 2009-04-07 06:27:39 UTC
I tried this and it doesn't work for me. The upstream fix adds  "iocharset=utf8,codepage=437" which behaves differently than just "utf8". Don't know why but it does. Just adding "utf8" to 
"iocharset=utf8,codepage=437" also doesn't work. So if I patch out the newly added ones and add only "utf8" then everyting works as before. 

The option that was on by default in the kernel of previous Fedoras  was specifically "utf8", where did the "iocharset=utf8,codepage=437" idea even come from ?

Comment 6 Kevin DeKorte 2009-04-07 18:52:58 UTC
DeviceKit-003 breaks loading of my ipod with libgpod.

Prior to this fix going in, everything worked correctly.

Comment 7 Kevin DeKorte 2009-04-07 19:06:04 UTC
libgpod reports the filename as

but when I try and load it from the ipod the filesystem name is


Rawhide mount reports
/dev/sdb1 on /media/IPODVIDEO2 type vfat (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=500)

F10 mount reports
/dev/sdb1 on /media/IPODVIDEO2 type vfat (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=500)

The device works properly on F10, but is broken in rawhide

Comment 8 David Zeuthen 2009-04-07 19:12:59 UTC
(In reply to comment #7)
> libgpod reports the filename as
> file:///media/IPODVIDEO2/iPod_Control/Music/F29/QCAR.mp3
> but when I try and load it from the ipod the filesystem name is
> /media/IPODVIDEO2/iPod_Control/Music/f29/qcar.mp3
> Rawhide mount reports
> /dev/sdb1 on /media/IPODVIDEO2 type vfat
> (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=500)
> F10 mount reports
> /dev/sdb1 on /media/IPODVIDEO2 type vfat
> (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=500)
> The device works properly on F10, but is broken in rawhide  

Please paste the appropriate lines from from /proc/self/mountinfo since mount(8) just prints out /etc/mtab (which doesn't show all the options used).

FWIW, it's not unlikely we want to change the default of "iocharset=utf8,codepage=437" for vfat to something else (to match F10 behavior and not break things like libgpod). However before that we need to know exactly what that "something else" should be.

Comment 9 Kevin DeKorte 2009-04-07 19:25:57 UTC
I remounted the ipod without the iocharset,codepage options and it returned to working like before.

Comment 10 Kevin DeKorte 2009-04-07 19:38:05 UTC
The issue is that fopen should work on FAT/VFAT no matter the case of the filename, as long as the actual name is right, so using the utf8 option is probably incorrect for vfat formatted devices, since vfat should not be case sensitive. If there is a way to make utf8 and case insensitivity work, I'm all for it.

Comment 11 Will Woods 2009-04-07 19:51:57 UTC
The kernel docs specifically say:

  NOTE: "iocharset=utf8" is not recommended. If unsure,
  you should consider the following option [utf8] instead.


I think the correct behavior here is to leave off iocharset & codepage and either:

a) Just use 'utf8' by default everywhere (matches old behavior), or
b) have something in the desktop environment (gnome-mount?) choose either 'utf8' or an appropriate iocharset/codepage, depending on the user's locale.

..but I don't know what component should make that choice. 

Probably easiest just to use 'utf8' everywhere by default, and document how you can override that, for the benefit those few people still using non-UTF8 data (e.g. bug 454013 and the 7 locales mentioned previously).

Comment 12 Kevin DeKorte 2009-04-07 19:58:56 UTC
(In reply to comment #7)
> libgpod reports the filename as
> file:///media/IPODVIDEO2/iPod_Control/Music/F29/QCAR.mp3
> but when I try and load it from the ipod the filesystem name is
> /media/IPODVIDEO2/iPod_Control/Music/f29/qcar.mp3
> Rawhide mount reports
> /dev/sdb1 on /media/IPODVIDEO2 type vfat
> (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=500)
> F10 mount reports
> /dev/sdb1 on /media/IPODVIDEO2 type vfat
> (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=500)
> The device works properly on F10, but is broken in rawhide  

I pasted the incorrect rawhide mount information, it should have been

 /dev/sdb1 on /media/IPODVIDEO2 type vfat (rw,nosuid,nodev,uhelper=devkit,uid=500,gid=500,shortname=lower,dmask=0077,iocharset=utf8,codepage=437)

As you can see it includes the iochars and codepage options that F10 does not

Comment 13 Muayyad Alsadi 2009-04-07 20:24:01 UTC
as for Comment #10 yes that is solved by passing utf8 not iocharset=utf8
passing iocharset=utf8 is not recommended

please try passing utf8 not iocharset=utf8
and it's the default since F7, if you have problem with it it would appear before this

Comment 14 David Zeuthen 2009-04-07 20:36:16 UTC
It seems that the proposal is that we should use


to match F10 and earlier releases. I'd appreciate if someone can check if that makes libgpod work. Thanks.

Comment 15 Kevin DeKorte 2009-04-07 20:51:34 UTC
Yes, that appears to work for me...

/dev/sdb1 on /media type vfat (rw,nosuid,nodev,uid=500,gid=500,shortname=lower,dmask=0077,utf8=true)

Comment 16 Will Woods 2009-04-07 21:30:01 UTC
Reopening until we're sure this is set straight. Or at least firmly crooked.

For the record, 'shortname=lower' is the documented default, and 'utf8' is the documented version of the mount option, so an equivalent option string would be: 'rw,nosuid,nodev,uid=500,gid=500,dmask=0077,utf8'

Comment 17 Kevin DeKorte 2009-04-08 18:56:31 UTC
Any ETA for a new DeviceKit-disks rpm in rawhide?

Comment 18 David Zeuthen 2009-04-08 21:54:46 UTC
DeviceKit-disks-004-0.5.20090408git.fc11 will use utf8=1,shortname=lower by default.


Comment 19 Kevin DeKorte 2009-04-09 19:47:49 UTC
Running with DeviceKit-disks-004-0.6.20090408git.fc11.x86_64 from koji and my ipod with libgpod seems to be working correctly again.

Note You need to log in before you can comment on or make changes to this bug.