Bug 490347

oops! sorry.

$ rpm -q DeviceKit-disks gnome-mount
DeviceKit-disks-003-6.fc11.i586
gnome-mount-0.8-5.fc11.i586

Non-ascii files and directorys name is broken in vfat partion.

LANG=ko_KR.UTF-8. 

$mount 
[skip]
/dev/sda7 on /media/1C49-4FD3 type vfat (rw,nosuid,nodev,uhelper=devkit,uid=500,gid=500,shortname=lower,dmask=0077)

This is a gvfs bug since 

 - DeviceKit-disks is just a mechanism like mount(1)

 - Mounting is now done through gvfs, not gnome-mount

with the move from HAL/gnome-mount to DeviceKit-disks/gvfs. But I'm keeping the discussion here for now.

FWIW, we have talked on the gvfs-list about introducing mount options at the gvfs level but haven't yet identified exactly *where* it is needed when it comes to handling file systems on block devices. 

The only real use case for mount options seems to be character encoding... but even there I'm not sure why mount options would be needed (I added it to gnome-mount without really thinking why it would be a good idea); I mean, ideally we wouldn't need to set mount options *at all*...

So, I'm certainly no expert when it comes to handling character set encoding but as far as I'm concerned, see

http://en.wikipedia.org/wiki/Long_filename

VFAT is using Unicode for file names, specifically each file name can be 255 UTF-16 characters. So if the on-disk contents are in unicode the question is why this isn't working for you...

You mention two mount options

 1. iocharset=utf8

and

 2. codepage=949

Let's look at the one at a time; from the mount man page and the 'fat' section (that also applies to 'vfat')

       codepage=value
              Sets the codepage for converting to shortname characters on  FAT
              and VFAT filesystems. By default, codepage 437 is used.

I'm pretty sure that 8.3 file names are totally ignored so I don't think this option is interesting at all.

       iocharset=value
              Character set to use for converting between 8 bit characters and
              16 bit Unicode characters. The default is iso8859-1.  Long file-
              names are stored on disk in Unicode format.

I'm baffled why the default is iso8859-1 when *everything* else on Linux uses UTF-8 these days. So it seems that all that is needed to fix this problem is to use iocharset=utf8 by default. But now I wonder what would break if we did this...

Sangu, any chance you test if mounting (by hand) with only iocharset=utf8 fixes the problem for you? Thanks.

Ah, also see bug 489222 and bug 454013.

> So it seems that all that is needed to fix this problem is to
> use iocharset=utf8 by default. But now I wonder what would break if we did
> this...

In a recent thread on fedora-devel-list, Kevin Kofler claimed that this is exactly what KDE does nowadays.

(In reply to comment #2)
> > Sangu, any chance you test if mounting (by hand) with only iocharset=utf8 fixes
> the problem for you? Thanks.  
#  mount -o iocharset=utf8,umask=002 /dev/sda7 /mnt

Non-ascii files name and directorys name are no problem(readable). 

# cat /proc/mounts
[...]
/dev/sda7 /mnt vfat rw,fmask=0002,dmask=0002,allow_utime=0020,codepage=cp437,iocharset=utf8 0 0

(In reply to comment #2)
> VFAT is using Unicode for file names, specifically each file name can be 255
> UTF-16 characters. So if the on-disk contents are in unicode the question is
> why this isn't working for you...

How about the on-disk contents are created when mounted by non-unicode environment?

AFAIK, if contents are created by linux at unicode environment, it should be no problem if you mount it via VFAT again. However, not all users create data and access vfat filesystem solely on linux, precisely with utf-8.

> I'm baffled why the default is iso8859-1 when *everything* else on Linux uses
> UTF-8 these days. So it seems that all that is needed to fix this problem is to
> use iocharset=utf8 by default. 

Ask a silly question, is utf-8 a perfect superset of iso8859-1?

IMHO, using iso8859-1 as default is for *fallback* reason, as encoding of a storage should be unknown except some detection algorithms are executed to detect that (just like how firefox detect the encoding of a page by reading early part of the html source).

=====

If bug of gvfs is what the source of the problem, I guess this is a regression and might be inappropriate to let it go into coming release. I suggest to reassign this bug report to gvfs component and request that to be blocker of beta or GA.

> How about the on-disk contents are created when mounted by non-unicode
> environment?

This bug is not about content at all, it only concerns filenames.

> Ask a silly question, is utf-8 a perfect superset of iso8859-1?

The question does not really make sense. If you are talking about character sets, then yes, Unicode is a superset of iso8859-1. If you are talking about encoding, then no. A valid iso8859-1 character sequence is not generally valid utf-8.

Lets move this bug where it belongs.

(In reply to comment #7)
> > How about the on-disk contents are created when mounted by non-unicode
> > environment?
> 
> This bug is not about content at all, it only concerns filenames.

How about the filenames of data that were created when mounted by non-unicode environment?

> 
> > Ask a silly question, is utf-8 a perfect superset of iso8859-1?
> 
> The question does not really make sense. If you are talking about character
> sets, then yes, Unicode is a superset of iso8859-1. If you are talking about
> encoding, then no. A valid iso8859-1 character sequence is not generally valid
> utf-8.  

All I would like to express is unicode is not covering certain characters of regional charsets such as CNS11643 http://en.wikipedia.org/wiki/CNS_11643 , and wondering if users who has FAT storage device originally run in localized win32 are supported. They should have filenames named in regional languages and encoding.

(In reply to comment #2)
> This is a gvfs bug since 
> 
>  - DeviceKit-disks is just a mechanism like mount(1)
> 
>  - Mounting is now done through gvfs, not gnome-mount
> 
> with the move from HAL/gnome-mount to DeviceKit-disks/gvfs. But I'm keeping the
> discussion here for now.
> 
> FWIW, we have talked on the gvfs-list about introducing mount options at the
> gvfs level but haven't yet identified exactly *where* it is needed when it
> comes to handling file systems on block devices. 
> 
> The only real use case for mount options seems to be character encoding... but
> even there I'm not sure why mount options would be needed (I added it to
> gnome-mount without really thinking why it would be a good idea); I mean,
> ideally we wouldn't need to set mount options *at all*...
> 

We need to set options as long as Fedora continues to break existing behaviour through updates (through upgrades is OK, of course).
vfat used to be mounted with shortname=lower until last week or so. Means that, e.g., everyone rsyncing their memory cards over had lower case file names. Now we insert our cards with a few more pictures taken and rsync with shortname=mixed, the new default...

So, please, do whatever for upgrades, but on updates don't change defaults unless
- they can be overriden and
- there is some notice about breaking existing behaviour.

First, this bug is really about DeviceKit-disks choosing non-sensical mount options. As it turns out this is fixed in upstream git

 http://cgit.freedesktop.org/DeviceKit/DeviceKit-disks/commit/?id=5c82f144da2cabb5750c716c9ff731b1b3c24372

already and this fix will be in Rawhide shortly. The problem, in a nutshell, was that the kernel people (for reasons unbeknownst to me) decided to remove the utf8 flag. So now we pass sane defaults (including utf8) and everything will work just fine in Fedora. As such, I'm moving this bug back to DeviceKit-disks and will close it with resolution RAWHIDE once the fix is in.

The other part of this bug a discussion of whether exposing mount options to end users is an useful thing to do. My view is that it is not. So the replacement for gnome-mount/HAL, namely gvfs/DeviceKit-disks, will not support that.

Either way, if people want mount options the DeviceKit-disks interface supports that. There are two ways to go about this

 1. use /etc/fstab
    - to cope with removable media, use the /dev/disk/by-* symlinks

 2. modify the user of the DeviceKit-disks API to pass the mount options
    you want

If you think gvfs and GNOME should support mount options the same way gnome-mount did (e.g. with gconf keys and UI dialogs), the best place to state your case about it is the upstream bug tracker

 http://bugzilla.gnome.org/enter_bug.cgi?product=gvfs

and we can have a discussion about it there. Please include a clear rationale why you think mount options for block devices are useful.

What is not going to help your cause is

 - complaining that gnome-mount/HAL used to support this and gvfs/DeviceKit-
   disk don't; we simply don't guarantee that kind of interface stability 
   between major releases of Fedora

 - filing enhancement requests in a downstream bugzilla

Thanks.

(In reply to comment #11)
> First, this bug is really about DeviceKit-disks choosing non-sensical mount
> options. As it turns out this is fixed in upstream git
> 
> 
> http://cgit.freedesktop.org/DeviceKit/DeviceKit-disks/commit/?id=5c82f144da2cabb5750c716c9ff731b1b3c24372
> 
> already and this fix will be in Rawhide shortly. The problem, in a nutshell,
> was that the kernel people (for reasons unbeknownst to me) decided to remove
> the utf8 flag. So now we pass sane defaults (including utf8) and everything
> will work just fine in Fedora. As such, I'm moving this bug back to
> DeviceKit-disks and will close it with resolution RAWHIDE once the fix is in.
> 
> The other part of this bug a discussion of whether exposing mount options to
> end users is an useful thing to do. My view is that it is not. So the
> replacement for gnome-mount/HAL, namely gvfs/DeviceKit-disks, will not support
> that.
> 
> Either way, if people want mount options the DeviceKit-disks interface supports
> that. There are two ways to go about this
> 
>  1. use /etc/fstab
>     - to cope with removable media, use the /dev/disk/by-* symlinks
> 
>  2. modify the user of the DeviceKit-disks API to pass the mount options
>     you want
> 
> If you think gvfs and GNOME should support mount options the same way
> gnome-mount did (e.g. with gconf keys and UI dialogs), the best place to state
> your case about it is the upstream bug tracker
> 
>  http://bugzilla.gnome.org/enter_bug.cgi?product=gvfs
> 
> and we can have a discussion about it there. Please include a clear rationale
> why you think mount options for block devices are useful.
> 
> What is not going to help your cause is
> 
>  - complaining that gnome-mount/HAL used to support this and gvfs/DeviceKit-
>    disk don't; we simply don't guarantee that kind of interface stability 
>    between major releases of Fedora

You're breaking existing installations through update!
As pointed out before, during upgrade this is a non-issue (in the sense that there are no guarantees) but during update it should a big no-no. (I'm getting the impression the F10 updates cycle is pretty bad at this, compared to earlier versions.)

> 
>  - filing enhancement requests in a downstream bugzilla
> 
> Thanks.

(In reply to comment #12)
> You're breaking existing installations through update!
> As pointed out before, during upgrade this is a non-issue (in the sense that
> there are no guarantees) but during update it should a big no-no. (I'm getting
> the impression the F10 updates cycle is pretty bad at this, compared to earlier
> versions.)

FWIW, I disagree with using the word "breakage" for this change.

FYI, this change only happens when updating to F11 so everything is fine. I agree if a change like this were to affect people on a stable release like F10 it would be bad (we generally don't rev or swap out major subsystems in a stable series). But that is not the case.

(In reply to comment #13)
> (In reply to comment #12)
> > You're breaking existing installations through update!
> > As pointed out before, during upgrade this is a non-issue (in the sense that
> > there are no guarantees) but during update it should a big no-no. (I'm getting
> > the impression the F10 updates cycle is pretty bad at this, compared to earlier
> > versions.)
> 
> FWIW, I disagree with using the word "breakage" for this change.
> 

If, for example, I rsync memory cards over and suddenly all files are in upper case instead of lower, keeping rsync from detecting duplicates, it's hard not to call it breakage.

> FYI, this change only happens when updating to F11 so everything is fine. I
> agree if a change like this were to affect people on a stable release like F10
> it would be bad (we generally don't rev or swap out major subsystems in a
> stable series). But that is not the case.  

I have an F10 with updates and no rawhide packages. The behaviour with respect to shortname=lower/mixed changed about a week ago through updates. Otherwise I wouldn't complain.

Michael

DeviceKit-disks-004 has the fixed mentioned in comment 11