Bug 467629 - NTFS volume skips file names with non-ASCII characters with NTFS-3G
NTFS volume skips file names with non-ASCII characters with NTFS-3G
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: ntfs-3g (Show other bugs)
9
All Linux
medium Severity high
: ---
: ---
Assigned To: Tom "spot" Callaway
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-10-19 12:56 EDT by Juhani Jaakola
Modified: 2008-10-30 08:56 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-10-21 15:51:00 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Juhani Jaakola 2008-10-19 12:56:33 EDT
Description of problem:
If I mount an external USB hard disk with a NTFS file system on it, I can't see those files which have non-ASCII characters in their file name. Instead I get this in /var/log/messages:

ntfs-3g[5836]: Skipping unrepresentable filename (inode 85): Invalid or incomplete multibyte or wide character

This occurs on a system which I upgraded from Fedora 7 to Fedora 9 using the Fedora 9 DVD. With Fedora 7 there were no problems.

Page http://www.ntfs-3g.org/support.html#locale claims that this is not a NTFS-3G issue but an OS issue... I don't care who to blaim, I just want a working system like I had with Fedora 7!

Adding an entry to /etc/fstab is not an option because I want this to work with my external USB drive.

I'd like to use English as my user interface language but I need to be able to handle file names with Finnish (or German or Swedish) special characters.

Version-Release number of selected component (if applicable):
ntfs-3g-1.2918-1.fc9.i386
hal-0.5.11-2.fc9.i386

How reproducible:
Always

Steps to Reproduce:
1. Plug in an external USB hard disk with a NTFS formatted volume
2. Try to find files with non-ASCII names (umlauts etc)
3.
  
Actual results:
Files with non-ASCII characters are missing. Error messages in /var/log/messages.

Expected results:
Files with non-ASCII characters should be visible (just as in Fedora 7)

Additional info:
Here is an excerpt from /var/log/messages from Fedora 7:
ntfs-3g[29774]: Version 1.2506 integrated FUSE 27
ntfs-3g[29774]: Mounted /dev/sdd1 (Read-Write, label "1TB080928", NTFS 3.1)
ntfs-3g[29774]: Cmdline options: rw,nosuid,nodev,uhelper=hal,locale=en_US.UTF-8
ntfs-3g[29774]: Mount options: rw,nosuid,nodev,uhelper=hal,silent,allow_other,nonempty,relatime,fsname=/dev/sdd1,blkdev,blksize=4096

Here is an excerpt from /var/log/messages from Fedora 9:
ntfs-3g[5836]: Version 1.2918 integrated FUSE 27
ntfs-3g[5836]: Mounted /dev/sdd1 (Read-Write, label "1TB080928", NTFS 3.1)
ntfs-3g[5836]: Cmdline options: rw,nosuid,nodev,uhelper=hal
ntfs-3g[5836]: Mount options: rw,nosuid,nodev,uhelper=hal,silent,allow_other,nonempty,relatime,fsname=/dev/sdd1,blkdev,blksize=4096

Please note that Fedora 7 had "locale=en_US.UTF-8" in Cmdline options but Fedora 9 does not.

$ locale -a | grep en_US
en_US
en_US.iso88591
en_US.iso885915
en_US.utf8

$ echo $LANG
en_US.utf8

By the way, should it be en_US.utf8 or en_US.UTF-8????
Comment 1 Fedora Update System 2008-10-20 14:50:54 EDT
ntfs-3g-1.5012-2.fc8 has been submitted as an update for Fedora 8.
http://admin.fedoraproject.org/updates/ntfs-3g-1.5012-2.fc8
Comment 2 Fedora Update System 2008-10-20 14:51:00 EDT
ntfs-3g-1.5012-2.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/ntfs-3g-1.5012-2.fc9
Comment 3 Tom "spot" Callaway 2008-10-20 14:52:47 EDT
Juhani,

Please try this update and see if it resolves your issue, I tested it in rawhide and it works fine now. If it does, please leave a "works for me" comment at the update page.
Comment 4 Juhani Jaakola 2008-10-20 15:52:39 EDT
Thanks for quick action - but unfortunately it does not help. I installed the package (with rpm -Uvh ~jj/Download/ntfs-3g-1.5012-2.fc9.i386.rpm) and mounted the USB disk again (without reboot). The files with umlauts are still skipped.

Now /var/log/messages show:

ntfs-3g[15786]: Version 1.5012 integrated FUSE 27
ntfs-3g[15786]: Mounted /dev/sdd1 (Read-Write, label "1TB080928", NTFS 3.1)
ntfs-3g[15786]: Cmdline options: rw,nosuid,nodev,uhelper=hal
ntfs-3g[15786]: Mount options: rw,nosuid,nodev,uhelper=hal,silent,allow_other,nonempty,relatime,fsname=/dev/sdd1,blkdev,blksize=4096
hald: mounted /dev/sdd1 on behalf of uid 500
ntfs-3g[15786]: Skipping unrepresentable filename (inode 94): Invalid or incomplete multibyte or wide character

As you can see, the "locale" argument is missing from "Cmdline options".

Could UTF-8 be the default for ntfs-3g mounts? Or must we modify hal (or something else) to get the locale argument added?
Comment 5 Tom "spot" Callaway 2008-10-21 12:53:44 EDT
Okay, so now we're in the situation where I can't reproduce your fault.

I created a new NTFS filesystem on a USB key, then mounted it and ran:

touch κόσμε kosme
touch tämän taman

I ran sync a few times to be sure it wrote to the disk, then pulled the key out, waited for a clean umount, then reinserted it:

Oct 21 12:49:22 localhost ntfs-3g[9812]: Version 1.5012 integrated FUSE 27
Oct 21 12:49:22 localhost ntfs-3g[9812]: Mounted /dev/sdb1 (Read-Write, label "", NTFS 3.1)
Oct 21 12:49:22 localhost ntfs-3g[9812]: Cmdline options: rw,nosuid,nodev,uhelper=hal,locale=en_US.utf8
Oct 21 12:49:22 localhost ntfs-3g[9812]: Mount options: rw,nosuid,nodev,uhelper=hal,silent,allow_other,nonempty,relatime,fsname=/dev/sdb1,blkdev,blksize=4096
Oct 21 12:49:22 localhost hald: mounted /dev/sdb1 on behalf of uid 500

[spot@localhost ~]$ ls -l /media/disk/
total 0
-rwxrwxrwx 1 root root 0 2008-10-20 12:59 kosme
-rwxrwxrwx 1 root root 0 2008-10-21 12:48 taman
-rwxrwxrwx 1 root root 0 2008-10-21 12:48 tämän
-rwxrwxrwx 1 root root 0 2008-10-20 12:59 κόσμε

This is on rawhide x86_64, and it didn't work before I added a hal fdi file to ntfs-3g.

[spot@localhost ~]$ rpm -q hal ntfs-3g
hal-0.5.12-3.20081013git.fc10.x86_64
ntfs-3g-1.5012-2.fc10.x86_64

So, the question is now, what's different about Fedora 9? Well, obviously hal is not as updated, so it could be a bug in hal. Let me try to dig into that.
Comment 6 Juhani Jaakola 2008-10-21 13:15:13 EDT
So you have "locale=en_US.utf8" in "Cmdline options" but I don't...

I don't know details of hal, so I have to ask dumb questions :-) Does the fdi file merely tell that locale is an allowed parameter or does it cause the locale to be read from somewhere during a mount?

From where does hal read the locale? In page http://www.ntfs-3g.org/support.html#locale they claim that reason for troubles can be "The OS configures the setting in a too late stage during the boot process, only after the NTFS volume was already mounted." - or only after the hal daemon was started.

According to that page, another reason might be that the locale is not configured properly so that it is not in the environment that hal runs in.

How can I check that my locale is configured properly? I have an upgraded system, not a fresh install...
Comment 7 Tom "spot" Callaway 2008-10-21 13:46:58 EDT
I'm not a hal expert either. :) The fdi file tells it that locale is an allowed parameter, I'm not sure how it populates that field, but if I had to guess, I'd say it looks at $LANG, which is set in /etc/sysconfig/i18n.

What output do you get from simply running "locale" ?
Comment 8 Tom "spot" Callaway 2008-10-21 14:19:11 EDT
Also, I booted a Fedora 9 LiveCD, then stuck my USB key into that system, and it mounted fine, even without the new .fdi file.

Oct 21 14:15:57 localhost ntfs-3g[3489]: Version 1.2506 integrated FUSE 27
Oct 21 14:15:57 localhost ntfs-3g[3489]: Mounted /dev/sdd1 (Read-Write, label "", NTFS 3.1)
Oct 21 14:15:57 localhost ntfs-3g[3489]: Cmdline options: rw,nosuid,nodev,uhelper=hal,locale=en_US.utf8
Oct 21 14:15:57 localhost ntfs-3g[3489]: Mount options:
rw,nosuid,nodev,uhelper=hal,silent,allow_other,nonempty,relatime,fsname=/dev/sdd1,blkdev,blksize=4096
Oct 21 14:15:57 localhost hald: mounted /dev/sdd1 on behalf of uid 500
Oct 21 14:15:57 localhost gnome-keyring-daemon[3276]: adding removable
location: volume_uuid_14FB38BB5BEC10EB at /media/disk

hal-0.5.11-0.7.rc2.fc9.x86_64
ntfs-3g-1.2506-1.fc9.x86_64
Comment 9 Juhani Jaakola 2008-10-21 14:48:46 EDT
Here is my locale:

$ locale
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=

But I added "locale=en_US.utf8" as value of key /system/storage/default_options/ntfs-3g/mount_options in gconf-editor (I use Gnome) and after that I can see "locale=en_US.utf8" in "Cmdline options" in /var/log/messages! Now I can see files with umlauts in their names! So this is a workaround for this problem!

If the LiveCD works the problem must be my setup. It can't be the gconf-editor trick. Perhaps the upgrade caused this.

I can live with the above workaround, but of course I'd like to know what's wrong with my locale settings... I guess the locale is not set when hald starts.
Comment 10 Tom "spot" Callaway 2008-10-21 15:51:00 EDT
That key is set to "locale=" on my rawhide system. I wish I knew what was wrong with your locale settings, but I honestly don't know. It gets set very early on.

Closing out this one as RAWHIDE.
Comment 11 Fedora Update System 2008-10-29 15:34:33 EDT
ntfs-3g-1.5012-3.fc8 has been submitted as an update for Fedora 8.
http://admin.fedoraproject.org/updates/ntfs-3g-1.5012-3.fc8
Comment 12 Fedora Update System 2008-10-29 15:34:42 EDT
ntfs-3g-1.5012-3.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/ntfs-3g-1.5012-3.fc9
Comment 13 Fedora Update System 2008-10-30 08:48:29 EDT
ntfs-3g-1.5012-3.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 14 Fedora Update System 2008-10-30 08:56:32 EDT
ntfs-3g-1.5012-3.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.