Bug 470592 - unable to see/create non-ASCII utf-8 filenames in vfat USB stricks
unable to see/create non-ASCII utf-8 filenames in vfat USB stricks
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: hal (Show other bugs)
rawhide
All Linux
medium Severity medium
: ---
: ---
Assigned To: Richard Hughes
Fedora Extras Quality Assurance
:
: 470086 (view as bug list)
Depends On:
Blocks: F11Blocker/F11FinalBlocker
  Show dependency treegraph
 
Reported: 2008-11-07 16:15 EST by Muayyad Alsadi
Modified: 2013-01-09 23:54 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 489222 (view as bug list)
Environment:
Last Closed: 2009-04-13 20:39:32 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Muayyad Alsadi 2008-11-07 16:15:54 EST
Description of problem:
fedora has dropped a patch from kernel that forces IOCHARSET=utf-8
to make people with non-UTF-8 locales happy
https://www.redhat.com/archives/fedora-kernel-list/2008-October/msg00089.html

but it makes people with UTF-8 locales sad
when I put my USB stick and copy/paste files with non-ASCII charts
or when I chdir there, I got errors

if you really want to release fedora with such kerenl you should add a hal fdi configuration file that pass utf8 to mount

just like what was done before that patch is introduced
https://bugzilla.redhat.com/show_bug.cgi?id=144600

as kernel docs (Documentation/filesystems/vfat.txt) there is a safe utf8
by using utf8 option instead of iocharset=utf8

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/vfat.txt;h=3a5ddc96901a665e3801d8ebf80c817e2efb5fc2;hb=HEAD

Version-Release number of selected component (if applicable):
fedora 10
kernel-2.6.27.4-68.fc10.i686
hal-0.5.12-9.20081027git.fc10.i386

How reproducible:
always

Steps to Reproduce:
[alsadi@pc1 ALSADI]$ touch "سلام"
touch: setting times of `سلام': No such file or directory
[alsadi@pc1 ALSADI]$ touch peace
[alsadi@pc1 ALSADI]$ mkdir "الإسلام"
mkdir: cannot create directory `الإسلام': Invalid argument
  
Expected results:
no seeing the errors and having the files/dirs created with non-ASCII chars

Additional info:
having non-UTF-8 filesystem encoding could bug many poorly written GTK+ application
http://library.gnome.org/devel/gtk/2.14/GtkFileChooser.html#gtkfilechooser-encodings
Comment 1 Will Woods 2008-11-07 16:42:26 EST
*** Bug 470086 has been marked as a duplicate of this bug. ***
Comment 2 Will Woods 2008-11-07 16:48:11 EST
Forcing utf8 in the kernel was the wrong idea - see bug 454013 for an example of the problems that caused. Further it made Fedora behave differently from all other Linux distributions when mounting vfat volumes, which is bad.

So dropping that kernel patch was probably the right decision, but it seems that we still need 'utf8' when mounting vfat for a lot of locales. So something in userspace needs to enable utf8 when needed. Unfortunately figuring out *when* it's needed is really hard. 

To maintain compatibility with older Fedora releases we could probably have HAL suggest use that flag by default - as with bug 144600. In a future release we might somehow decide that based on the current locale.
Comment 3 Muayyad Alsadi 2008-11-10 12:33:00 EST
from bug 454013 
> ru_RU.cp1251 or ru_RU.koi8r
you should flag that bug as "non-BUG" becayse
the default locale in fedora for Russian locales is UTF-8 just look at this:

grep '^ru' /usr/share/system-config-language/locale-list
ru_RU.UTF-8 utf8 latarcyrheb-sun16 Russian  -  Русский
ru_UA.UTF-8 utf8 latarcyrheb-sun16 Russian (Ukraine)

so it's his own problem, he should edit his hal files to do his customizations

utf-8 is universal, forcing latin1 or whatever should not be done in an international project as fedora

so if you want to go upstream, then you should move the patch as a hal configuration

and those who want to use non-international encoding let them use some /etc/hal/fdi/policy/ file

if you insist on not making utf8 the default on hal level at least make a package having utf-8 fdi file for hal and make that package a default in the comps file for each *-support group

be aware of the following facts
grep 'utf8' /usr/share/system-config-language/locale-list | wc -l
136
wc -l /usr/share/system-config-language/locale-list
143 /usr/share/system-config-language/locale-list

which mean you are breaking more than 95% of the locales by not making utf8 tobe the default

my suggestion is doing the opposite having the default to be the default as a fdi hal file
and having 2 packages (8859-15 and 8859-2) for all the 7 non-utf-8 locales and make that one a default or optional in the comps file

grep -v 'utf8' /usr/share/system-config-language/locale-list
af_ZA 8859-15 lat0-sun16 Afrikaans (South Africa)
bs_BA 8859-2 lat2-sun16 Bosnian (Bosnia and Herzegowina)
br_FR 8859-15 lat0-sun16 Breton (France)
oc_FR 8859-15 lat0-sun16 Occitan (France)
tl_PH 8859-15 lat0-sun16 Tagalog (Philippines)
uz_UZ 8859-15 lat0-sun16 Uzbek (Uzbekistan)
wa_BE@euro 8859-15 lat0-sun16 Walloon (Belgium)
Comment 4 Will Woods 2008-11-10 16:20:36 EST
After much discussion we've decided to restore the kernel patch that enables 'utf8' by default on vfat, at least for F10 final:

* Mon Nov 10 2008 Dave Jones <davej@redhat.com> 2.6.27.5-93
- Make UTF-8 the default for FAT/VFAT filesystems again.

F11 will not include this patch, so this bug will need to be fixed in userspace (HAL etc.) in Rawhide.

Moving to F11Blocker.
Comment 5 Bug Zapper 2008-11-26 00:00:44 EST
This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 6 Muayyad Alsadi 2008-11-26 17:35:32 EST
I guess it won't be in hal only but also to DeviceKit which is planed for F11
Comment 7 Kevin Kofler 2009-03-08 15:30:07 EDT
FWIW, KDE 4's Solid simply passes utf8 where available and iocharset=utf8 otherwise (unless that too is unavailable) to HAL when it mounts volumes. See StorageAccess::callHalVolumeMount() in http://websvn.kde.org/branches/KDE/4.2/kdelibs/solid/solid/backends/hal/halstorageaccess.cpp?revision=918779&view=markup for details. (And yes, it still uses HAL, not DeviceKit. Volunteers to write a Solid DeviceKit backend welcome.)
Comment 8 Muayyad Alsadi 2009-03-08 16:28:54 EDT
it was just a guess that DeviceKit-disks is involved,
it's important to keep DeviceKit aware of that as hal could be dropped after some long time.

if KDE relies on hal and gnome in device kit and xfce on whatever
this mean that we should fix it in both to keep the system consistent
Comment 9 Matthias Clasen 2009-03-24 00:45:46 EDT
I don't think there is any hal bug here. hal and DeviceKit-disks are just the mechanisms that are used for mounting. As Kevin says above, kde is using hal and is already passing the utf8 mount options. gnome is using DeviceKit-disks in F11, and the issue of making it pass utf8 mount options is tracked in bug 490347.
Comment 10 Muayyad Alsadi 2009-03-24 12:51:55 EDT
the solution should be centralized

I have mentioned some scenario of failure in ML if we let this go per application

let me mentioned one

someone is using kde, which uses HAL and it passes utf8 option
he created a file there

he logged off and log into gnome for example
gnome does not rely on hal so when he tries to open his files

a more tricky is a less famous WM or DE like lxde or Window Maker

he logged off and log into lxde for example

the more tricky part isn't this it's the opposite case


now we have several layers of complexity
and a very likely inconsistency and corruption 
as the default encoding assumed by the kernel can't be validated as opposed to utf-8
and writing in one encoding and reading in another will for sure case problems

what would we tell most of the users as a reply "how to set utf-8 as default encoding for all DEs and WMs just as it was in previous releases of Fedora ?" 

no matter what solution you take it should be centralized
preferably in the kernel or at least in system wide policy used by all the user land tools
Comment 11 Matthias Clasen 2009-03-24 14:45:21 EDT
...it should be centralized preferably in the kernel...

I hear you. But the kernel guys disagreed...
Comment 12 Jesse Keating 2009-04-13 20:39:32 EDT
This is "fixed" in our main desktops (KDE/Gnome) so I'm going to consider it fixed for F11.  This bug entry is a bit of a disaster and probably shouldn't stay open so I'm closing it too.  Further discussion about this should happen on list.
Comment 13 Muayyad Alsadi 2009-04-14 10:46:58 EDT
> This is "fixed" in our main desktops (KDE/Gnome)
I guess regarding DeviceKit, yes it's solved and fixed
https://bugzilla.redhat.com/show_bug.cgi?id=489222

but I did not see any hal system-wide policy here
so it's not yet fixed
I call for having a small fdi policy file setting it for whatever desktop relying on HAL containing something like:
<device>
<match key="block.is_volume" bool="true">
<match key="volume.fstype" string="vfat">
<merge key="volume.policy.mount_option.utf8=true"
type="bool">true</merge>
</match>
</match>
</device>

having it fixed in both DeviceKit and hal then I can say we have it almost solved and proudly flag this bug as closed
please reconsider keeping it open until we having fdi file like the above one in hal

> I hear you. But the kernel guys disagreed...  

because they are stupid and close minded
Linux is supposed to be international
Comment 14 Kevin Kofler 2009-04-14 13:10:48 EDT
It's also solved and fixed in KDE, they pass the utf8 option to HAL as per https://bugs.kde.org/show_bug.cgi?id=161673 .

Note You need to log in before you can comment on or make changes to this bug.