This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 489222 - unable to see/create non-ASCII utf-8 filenames in vfat USB stricks
unable to see/create non-ASCII utf-8 filenames in vfat USB stricks
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: DeviceKit-disks (Show other bugs)
rawhide
All Linux
low Severity medium
: ---
: ---
Assigned To: David Zeuthen
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks: F11Blocker/F11FinalBlocker
  Show dependency treegraph
 
Reported: 2009-03-08 15:23 EDT by Muayyad Alsadi
Modified: 2009-04-09 15:47 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 470592
Environment:
Last Closed: 2009-04-08 17:54:46 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
patch to that effect (706 bytes, patch)
2009-03-09 05:14 EDT, Yanko Kaneti
no flags Details | Diff

  None (edit)
Description Muayyad Alsadi 2009-03-08 15:23:39 EDT
+++ This bug was initially created as a clone of Bug #470592 +++

Description of problem:
fedora has dropped a patch from kernel that forces IOCHARSET=utf-8
to make people with non-UTF-8 locales happy
https://www.redhat.com/archives/fedora-kernel-list/2008-October/msg00089.html

but it makes people with UTF-8 locales sad
when I put my USB stick and copy/paste files with non-ASCII charts
or when I chdir there, I got errors

if you really want to release fedora with such kerenl you should add a hal fdi configuration file that pass utf8 to mount

just like what was done before that patch is introduced
https://bugzilla.redhat.com/show_bug.cgi?id=144600

as kernel docs (Documentation/filesystems/vfat.txt) there is a safe utf8
by using utf8 option instead of iocharset=utf8

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/vfat.txt;h=3a5ddc96901a665e3801d8ebf80c817e2efb5fc2;hb=HEAD

Version-Release number of selected component (if applicable):
fedora 10
kernel-2.6.27.4-68.fc10.i686
hal-0.5.12-9.20081027git.fc10.i386

How reproducible:
always

Steps to Reproduce:
[alsadi@pc1 ALSADI]$ touch "سلام"
touch: setting times of `سلام': No such file or directory
[alsadi@pc1 ALSADI]$ touch peace
[alsadi@pc1 ALSADI]$ mkdir "الإسلام"
mkdir: cannot create directory `الإسلام': Invalid argument
  
Expected results:
no seeing the errors and having the files/dirs created with non-ASCII chars

Additional info:
having non-UTF-8 filesystem encoding could bug many poorly written GTK+ application
http://library.gnome.org/devel/gtk/2.14/GtkFileChooser.html#gtkfilechooser-encodings

--- Additional comment from wwoods@redhat.com on 2008-11-07 16:42:26 EDT ---

*** Bug 470086 has been marked as a duplicate of this bug. ***

--- Additional comment from wwoods@redhat.com on 2008-11-07 16:48:11 EDT ---

Forcing utf8 in the kernel was the wrong idea - see bug 454013 for an example of the problems that caused. Further it made Fedora behave differently from all other Linux distributions when mounting vfat volumes, which is bad.

So dropping that kernel patch was probably the right decision, but it seems that we still need 'utf8' when mounting vfat for a lot of locales. So something in userspace needs to enable utf8 when needed. Unfortunately figuring out *when* it's needed is really hard. 

To maintain compatibility with older Fedora releases we could probably have HAL suggest use that flag by default - as with bug 144600. In a future release we might somehow decide that based on the current locale.

--- Additional comment from alsadi@ojuba.org on 2008-11-10 12:33:00 EDT ---

from bug 454013 
> ru_RU.cp1251 or ru_RU.koi8r
you should flag that bug as "non-BUG" becayse
the default locale in fedora for Russian locales is UTF-8 just look at this:

grep '^ru' /usr/share/system-config-language/locale-list
ru_RU.UTF-8 utf8 latarcyrheb-sun16 Russian  -  Русский
ru_UA.UTF-8 utf8 latarcyrheb-sun16 Russian (Ukraine)

so it's his own problem, he should edit his hal files to do his customizations

utf-8 is universal, forcing latin1 or whatever should not be done in an international project as fedora

so if you want to go upstream, then you should move the patch as a hal configuration

and those who want to use non-international encoding let them use some /etc/hal/fdi/policy/ file

if you insist on not making utf8 the default on hal level at least make a package having utf-8 fdi file for hal and make that package a default in the comps file for each *-support group

be aware of the following facts
grep 'utf8' /usr/share/system-config-language/locale-list | wc -l
136
wc -l /usr/share/system-config-language/locale-list
143 /usr/share/system-config-language/locale-list

which mean you are breaking more than 95% of the locales by not making utf8 tobe the default

my suggestion is doing the opposite having the default to be the default as a fdi hal file
and having 2 packages (8859-15 and 8859-2) for all the 7 non-utf-8 locales and make that one a default or optional in the comps file

grep -v 'utf8' /usr/share/system-config-language/locale-list
af_ZA 8859-15 lat0-sun16 Afrikaans (South Africa)
bs_BA 8859-2 lat2-sun16 Bosnian (Bosnia and Herzegowina)
br_FR 8859-15 lat0-sun16 Breton (France)
oc_FR 8859-15 lat0-sun16 Occitan (France)
tl_PH 8859-15 lat0-sun16 Tagalog (Philippines)
uz_UZ 8859-15 lat0-sun16 Uzbek (Uzbekistan)
wa_BE@euro 8859-15 lat0-sun16 Walloon (Belgium)

--- Additional comment from wwoods@redhat.com on 2008-11-10 16:20:36 EDT ---

After much discussion we've decided to restore the kernel patch that enables 'utf8' by default on vfat, at least for F10 final:

* Mon Nov 10 2008 Dave Jones <davej@redhat.com> 2.6.27.5-93
- Make UTF-8 the default for FAT/VFAT filesystems again.

F11 will not include this patch, so this bug will need to be fixed in userspace (HAL etc.) in Rawhide.

Moving to F11Blocker.

--- Additional comment from fedora-triage-list@redhat.com on 2008-11-26 00:00:44 EDT ---


This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

--- Additional comment from alsadi@ojuba.org on 2008-11-26 17:35:32 EDT ---

I guess it won't be in hal only but also to DeviceKit which is planed for F11
Comment 1 Yanko Kaneti 2009-03-09 05:14:32 EDT
Created attachment 334479 [details]
patch to that effect

This should be a F11 blocker.
Comment 2 Matthias Clasen 2009-03-24 12:34:13 EDT
This has been fixed upstream, we'll get it with the next update.
Comment 3 Muayyad Alsadi 2009-03-24 13:01:32 EDT
good news

we need similar one for hal, I guess it would be just an fdi policy file
Comment 4 Matthias Clasen 2009-04-07 00:39:30 EDT
Should be fixed in DeviceKit-disks-004 in rawhide.
Comment 5 Yanko Kaneti 2009-04-07 02:27:39 EDT
I tried this and it doesn't work for me. The upstream fix adds  "iocharset=utf8,codepage=437" which behaves differently than just "utf8". Don't know why but it does. Just adding "utf8" to 
"iocharset=utf8,codepage=437" also doesn't work. So if I patch out the newly added ones and add only "utf8" then everyting works as before. 

The option that was on by default in the kernel of previous Fedoras  was specifically "utf8", where did the "iocharset=utf8,codepage=437" idea even come from ?
Comment 6 Kevin DeKorte 2009-04-07 14:52:58 EDT
DeviceKit-003 breaks loading of my ipod with libgpod.

Prior to this fix going in, everything worked correctly.
Comment 7 Kevin DeKorte 2009-04-07 15:06:04 EDT
libgpod reports the filename as
file:///media/IPODVIDEO2/iPod_Control/Music/F29/QCAR.mp3

but when I try and load it from the ipod the filesystem name is

/media/IPODVIDEO2/iPod_Control/Music/f29/qcar.mp3

Rawhide mount reports
/dev/sdb1 on /media/IPODVIDEO2 type vfat (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=500)

F10 mount reports
/dev/sdb1 on /media/IPODVIDEO2 type vfat (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=500)

The device works properly on F10, but is broken in rawhide
Comment 8 David Zeuthen 2009-04-07 15:12:59 EDT
(In reply to comment #7)
> libgpod reports the filename as
> file:///media/IPODVIDEO2/iPod_Control/Music/F29/QCAR.mp3
> 
> but when I try and load it from the ipod the filesystem name is
> 
> /media/IPODVIDEO2/iPod_Control/Music/f29/qcar.mp3
> 
> Rawhide mount reports
> /dev/sdb1 on /media/IPODVIDEO2 type vfat
> (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=500)
> 
> F10 mount reports
> /dev/sdb1 on /media/IPODVIDEO2 type vfat
> (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=500)
> 
> The device works properly on F10, but is broken in rawhide  

Please paste the appropriate lines from from /proc/self/mountinfo since mount(8) just prints out /etc/mtab (which doesn't show all the options used).

FWIW, it's not unlikely we want to change the default of "iocharset=utf8,codepage=437" for vfat to something else (to match F10 behavior and not break things like libgpod). However before that we need to know exactly what that "something else" should be.
Comment 9 Kevin DeKorte 2009-04-07 15:25:57 EDT
I remounted the ipod without the iocharset,codepage options and it returned to working like before.
Comment 10 Kevin DeKorte 2009-04-07 15:38:05 EDT
The issue is that fopen should work on FAT/VFAT no matter the case of the filename, as long as the actual name is right, so using the utf8 option is probably incorrect for vfat formatted devices, since vfat should not be case sensitive. If there is a way to make utf8 and case insensitivity work, I'm all for it.
Comment 11 Will Woods 2009-04-07 15:51:57 EDT
The kernel docs specifically say:

  NOTE: "iocharset=utf8" is not recommended. If unsure,
  you should consider the following option [utf8] instead.

(http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/vfat.txt;hb=HEAD#l55)

I think the correct behavior here is to leave off iocharset & codepage and either:

a) Just use 'utf8' by default everywhere (matches old behavior), or
b) have something in the desktop environment (gnome-mount?) choose either 'utf8' or an appropriate iocharset/codepage, depending on the user's locale.

..but I don't know what component should make that choice. 

Probably easiest just to use 'utf8' everywhere by default, and document how you can override that, for the benefit those few people still using non-UTF8 data (e.g. bug 454013 and the 7 locales mentioned previously).
Comment 12 Kevin DeKorte 2009-04-07 15:58:56 EDT
(In reply to comment #7)
> libgpod reports the filename as
> file:///media/IPODVIDEO2/iPod_Control/Music/F29/QCAR.mp3
> 
> but when I try and load it from the ipod the filesystem name is
> 
> /media/IPODVIDEO2/iPod_Control/Music/f29/qcar.mp3
> 
> Rawhide mount reports
> /dev/sdb1 on /media/IPODVIDEO2 type vfat
> (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=500)
> 
> F10 mount reports
> /dev/sdb1 on /media/IPODVIDEO2 type vfat
> (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=500)
> 
> The device works properly on F10, but is broken in rawhide  

I pasted the incorrect rawhide mount information, it should have been

 /dev/sdb1 on /media/IPODVIDEO2 type vfat (rw,nosuid,nodev,uhelper=devkit,uid=500,gid=500,shortname=lower,dmask=0077,iocharset=utf8,codepage=437)

As you can see it includes the iochars and codepage options that F10 does not
Comment 13 Muayyad Alsadi 2009-04-07 16:24:01 EDT
as for Comment #10 yes that is solved by passing utf8 not iocharset=utf8
passing iocharset=utf8 is not recommended

please try passing utf8 not iocharset=utf8
and it's the default since F7, if you have problem with it it would appear before this
Comment 14 David Zeuthen 2009-04-07 16:36:16 EDT
It seems that the proposal is that we should use

 utf8=true,shortname=lower

to match F10 and earlier releases. I'd appreciate if someone can check if that makes libgpod work. Thanks.
Comment 15 Kevin DeKorte 2009-04-07 16:51:34 EDT
Yes, that appears to work for me...

/dev/sdb1 on /media type vfat (rw,nosuid,nodev,uid=500,gid=500,shortname=lower,dmask=0077,utf8=true)
Comment 16 Will Woods 2009-04-07 17:30:01 EDT
Reopening until we're sure this is set straight. Or at least firmly crooked.

For the record, 'shortname=lower' is the documented default, and 'utf8' is the documented version of the mount option, so an equivalent option string would be: 'rw,nosuid,nodev,uid=500,gid=500,dmask=0077,utf8'
Comment 17 Kevin DeKorte 2009-04-08 14:56:31 EDT
Any ETA for a new DeviceKit-disks rpm in rawhide?
Comment 18 David Zeuthen 2009-04-08 17:54:46 EDT
DeviceKit-disks-004-0.5.20090408git.fc11 will use utf8=1,shortname=lower by default.

http://koji.fedoraproject.org/koji/taskinfo?taskID=1286129
Comment 19 Kevin DeKorte 2009-04-09 15:47:49 EDT
Running with DeviceKit-disks-004-0.6.20090408git.fc11.x86_64 from koji and my ipod with libgpod seems to be working correctly again.

Note You need to log in before you can comment on or make changes to this bug.