Bug 486619 - ntfs-3g mount in /etc/fstab freezes boot....
Summary: ntfs-3g mount in /etc/fstab freezes boot....
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: mdadm
Version: rawhide
Hardware: All
OS: Linux
low
high
Target Milestone: ---
Assignee: Doug Ledford
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-02-20 17:24 UTC by Tom London
Modified: 2009-04-06 20:34 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-04-01 22:04:49 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
photo of screen hanging on boot with ntfs mount in /etc/fstab (209.67 KB, image/jpeg)
2009-02-20 17:25 UTC, Tom London
no flags Details
screenshot of boot freeze with ntfs-3g mount (185.31 KB, image/jpeg)
2009-03-27 00:12 UTC, Tom London
no flags Details

Description Tom London 2009-02-20 17:24:07 UTC
Description of problem:
After this morning's updates, my system failed to boot on the latest kernel (kernel-2.6.29-0.137.rc5.git4.fc11.x86_64) or the previous one (kernel-2.6.29-0.134.rc5.git2.fc11.x86_64).

Consistently hung at "Mounting local filesystems:"  (see attached photo of screen).

After some fear and anxiety, I managed to boot "readonlyroot", remount / "rw", and edit /etc/rc.d/rc.sysinit commenting out the lines that do the "local filesystem mounts".

System then booted fine, but it appeared (via "ps algx") that the mount of the ntfs partition was hung.  In fact, trying "ls /media/disk" hung the console.

I rebooted single user, edited out the /etc/fstab entry, and restored /etc/rc.d/rc.sysinit to its pristine state.  Rebooting was fine........

Here is the (now) commented line from /etc/fstab:

#LABEL=WinXP             /mnt/winXP              ntfs-3g  defaults       0 0

Now, after the system boots up graphicall, I see the partitions mounted: 
/dev/sda1 on /media/disk type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096)

and can "ls /media/disk" without issue:

[tbl@tlondon ~]$ ls /media/disk&
[2] 5196
[tbl@tlondon ~]$ AUTOEXEC.BAT            hiberfil.sys  Program Files
boot.ini                IO.SYS        SWTOOLS
Config.Msi              MSDOS.SYS     System Volume Information
CONFIG.SYS              NTDETECT.COM  TrackitAudit.id
Documents and Settings  ntldr         WINDOWS
DRIVERS                 pagefile.sys

[2]+  Done                    ls --color=auto /media/disk
[tbl@tlondon ~]$ 

I didn't see ntfs-3g updated today.... 


Version-Release number of selected component (if applicable):
kernel-2.6.29-0.134.rc5.git2.fc11.x86_64
kernel-2.6.29-0.124.rc5.fc11.x86_64
ntfs-3g-2009.2.1-2.fc11.x86_64
kernel-2.6.29-0.137.rc5.git4.fc11.x86_64


How reproducible:
Freezes every boot.

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Tom London 2009-02-20 17:25:02 UTC
Created attachment 332735 [details]
photo of screen hanging on boot with ntfs mount in /etc/fstab

Comment 2 Tom London 2009-02-20 22:17:37 UTC
If I turn off 'mdmonitor' (via 'chkconfig mdmonitor off'), the system boots just fine with the ntfs-3g partition in /etc/fstab.

Moving this from ntfs-3g to mdadm.....

Comment 3 Doug Ledford 2009-02-23 14:40:26 UTC
Can you re-enable mdmonitor and see if the problem returns?  The reason I ask is because the point at which your machine is hanging is *before* mdmonitor is even started, so whether on or off, if your machine reliably hangs at that point, it's never even running mdmonitor, so one would think that mdmonitor can't be the catalyst to the hang.

Comment 4 Tom London 2009-02-23 15:18:39 UTC
Damn.  Of course not.....

I did 'chkconfig --level 2345 mdmonitor' and rebooted; of course it works now.

Not sure what changed.  Sigh.  This failed every boot (10 times in a row....).

Looks like same kernel, same ntfs-3g, same mdadm.

So, this froze until I edited these lines from /etc/rc.d/rc.sysinit:


# Mount all other filesystems (except for NFS and /proc, which is already
# mounted). Contrary to standard usage,
# filesystems are NOT unmounted in single user mode.
if [ "$READONLY" != "yes" ] ; then
	action $"Mounting local filesystems: " mount -a -t nonfs,nfs4,smbfs,ncpfs,cifs,gfs,gfs2 -O no_netdev
else
	action $"Mounting local filesystems: " mount -a -n -t nfs4,smbfs,ncpfs,cifs,gfs,gfs2 -O no_netdev
fi

With these commented, the system booted up to runlevel 3 fine, but something else was mounting the ntfs-3g partition (/dev/sda1) from /etc/fstab: there was an 'ntfs-3g' process running, and an attempt to "ls /mnt/winXP" hung the terminal.

I then commented out the line in /etc/fstab and rebooted just fine; /dev/sda1 got mounted just fine as /media/WinXP).

I thought I saw an mdadm process in the bootchart, and I saw the thread in fedora-test referring to mdadm, so I turned that off, restored fstab, and the reboot worked just fine (with /dev/sda1 mounted as /mnt/winXP).

Sorry for pointing a finger in the wrong direction here....

I remain "dazed and confused"......

Comment 5 Tom London 2009-03-04 22:34:02 UTC
This happened again after the "big rawhide update" this past weekend.

I recovered as described above, by booting "readonlyroot", remounting / as r/w, and editing out the ntfs mount in /etc/fstab.

Since I couldn't localize it further than that, I've continued to boot with that entry commented out.

With new devkit-disk, this now mounts when I login, btw.

Comment 6 Andreas Simon 2009-03-20 12:28:51 UTC
I can confirm this, the same happens here.

This is my fstab:

/dev/vg0/fedora         /                       ext4    defaults        1 1
/dev/vg0/scratch        /scratch                ext3    defaults        1 2
/dev/vg0/data           /home                   ext3    defaults        1 2
/dev/vg0/games          /games                  ext3    defaults        1 2
/dev/vg0/music          /mnt/music              ext3    defaults        1 2
UUID=6a780285-f5e7-492b-b5ba-2ce7774b1534 /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
UUID=bcd904af-68f5-4dcf-95c6-99ee989bb809 swap                    swap    defaults        0 0
/dev/mapper/isw_bgjcdcfheh_volume0p1	/mnt/windows/C	ntfs-3g	defaults	0 0
/dev/mapper/isw_bgjcdcfheh_volume0p2	/mnt/windows/Games	ntfs-3g	defaults	0 0
/dev/mapper/isw_bgjcdcfheh_volume0p3	/mnt/windows/Stuff	ntfs-3g	defaults	0 0


The boot also hangs at "Mounting local file systems:". After that message a warning "All config files need .conf: /etc/modprobe.d/floppy-pnp" is printed for every file under /etc/modprobe.d/ .
Then the system stays dead. In this state even Control-Alt-Backspace is not able to reboot the system.

If I change to nfts-3g entries in /etc/fstab to "noauto" the system boots fine.

My NTFS partitions are on a dmraid managed raid0. Mounting them after the system has been fully booted works fine.

Another interesting thing: If I initiate a full relabeling of the filesystems (i.e. touching /.autorelabel) the system boots fine too and mounts the ntfs paritions during boot without problems. The next boot will hang again (unless initiating another relabeling or changing the nfts fstab-entries to noauto)

This is with current rawhide:
ntfs-3g-2009.2.1-3.fc11.x86_64
kernel-2.6.29-0.258.rc8.git2.fc11.x86_64
dmraid-1.0.0.rc15-6.fc11.x86_64

Comment 7 Tom "spot" Callaway 2009-03-26 23:10:05 UTC
Hmmm. Does booting with selinux in permissive get it working again? I'm wondering if this is an SELinux issue. I'm trying to reproduce this on my end, but at the moment, I only have a file mount in /etc/fstab as opposed to a partition mount, and I can't seem to trigger any lockups.

There is a new ntfs-3g as well, but they don't list anything in the changelog that leads me to believe that it would resolve this... I'm going to push it anyways because the other fixes are useful, but I didn't want anyone to think I had forgotten about this bug.

Comment 8 Tom London 2009-03-26 23:45:50 UTC
Nope.  Freezes for me both with and without "enforcing=0".

Here is my /etc/fstab when it boots:

[tbl@tlondon ~]$ cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Fri Oct 17 14:03:24 2008
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or vol_id(8) for more info
#
/dev/VolGroup00/LogVol00 /                       ext3    defaults,noatime,nodiratime        1 1
UUID=27db997b-1bda-4e46-8a89-ae5159412a00 /boot  ext3    defaults,noatime,nodiratime        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
tmpfs                   /tmp                    tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0
#LABEL=WinXP             /mnt/winXP              ntfs-3g  defaults       0 0
#LABEL=extra             /mnt/extra              ext3    defaults,noatime,nodiratime       0 0
#/mnt/extra/tbl          /home/tbl/Music/more    none    defaults,bind   0 0
[tbl@tlondon ~]$ 

Deleting the '#' on the line starting with "LABEL=WinXP" makes it freeze.

I can "recover" by booting with "readonlyroot s", and then remounting / with "mount -o remount,rw /", and then editing /etc/fstab to put the '#' back in.

I'm running latest rawhide with an encrypted root partition (I guess swap too).

Froze 4 or 5 consecutive times....

Comment 9 Tom "spot" Callaway 2009-03-27 00:00:42 UTC
Try updating to this:

http://kojipkgs.fedoraproject.org/packages/ntfs-3g/2009.3.8/1.fc11/x86_64/ntfs-3g-2009.3.8-1.fc11.x86_64.rpm

See if it makes any difference?

Also, try changing it to mount by /dev/foo rather than by label? Admittedly, I'm making guesses in the dark here. When I get back into the office next week, I'm going to reinstall a system with an NTFS partition (rather than a file).

e.g.
/home/spot/cvs/ntfs-3g/devel/ntfs.img /mnt/ntfs  ntfs-3g defaults        0 0

Comment 10 Tom London 2009-03-27 00:12:10 UTC
Created attachment 336928 [details]
screenshot of boot freeze with ntfs-3g mount

Nope... ntfs-3g-2009.3.8-1.fc11.x86_64 freezes too.

But, I did get some red "FAILED" indications on the screen.

Sorry for the blurry picture.

Believe the error messages says:

mount: devpts already mounted, or /dev/pts is busy.

Not sure why it appeared twice.

Restoring the comment to the /etc/fstab line restores booting (and removes the devpts error message).

I'll try changing the mount from a "label mount" to a partition mount next.

Comment 11 Tom London 2009-03-27 00:24:34 UTC
Well, a "partition mount" appears to work (at least once!).

Booted up in SELinux enforcing mode.

I did however notice a few "new" AVCs, the first that came out as or before udev started:

Mar 26 17:15:39 tlondon kernel: mount used greatest stack depth: 3088 bytes left
Mar 26 17:15:39 tlondon kernel: type=1400 audit(1238112926.206:4): avc:  denied  { read } for  pid=973 comm="dmesg" name="ld.so.cache" dev=dm-1 ino=28009 scontext=system_u:system_r:dmesg_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file
Mar 26 17:15:39 tlondon kernel: udev: starting version 139

Here is the second:
Mar 26 17:15:39 tlondon kernel: Adding 4063224k swap on /dev/mapper/VolGroup00-LogVol01.  Priority:-1 extents:1 across:4063224k
Mar 26 17:15:39 tlondon kernel: type=1400 audit(1238112936.209:5): avc:  denied  { read } for  pid=2129 comm="dmesg" name="ld.so.cache" dev=dm-1 ino=28009 scontext=system_u:system_r:dmesg_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file
Mar 26 17:15:39 tlondon kernel: platform microcode: firmware: requesting intel-ucode/06-17-06

and the third:

Mar 26 17:15:39 tlondon kernel: Microcode Update Driver: v2.00 <tigran.co.uk>, Peter Oruba
Mar 26 17:15:39 tlondon kernel: type=1400 audit(1238112936.713:6): avc:  denied  { read } for  pid=2191 comm="microcode_ctl" name="ld.so.cache" dev=dm-1 ino=28009 scontext=system_u:system_r:cpucontrol_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file
Mar 26 17:15:39 tlondon kernel: Microcode Update Driver: v2.00 removed.

Notwithstanding, system booted and ntfs-3g mounted the fs:
[root@tlondon ~]# ls /mnt/winXP
73b7774c2054f88ef458b8e2  DRIVERS       pagefile.sys
AUTOEXEC.BAT              hiberfil.sys  Program Files
boot.ini                  IO.SYS        SWTOOLS
Config.Msi                MSDOS.SYS     System Volume Information
CONFIG.SYS                NTDETECT.COM  TrackitAudit.id
Documents and Settings    ntldr         WINDOWS
[root@tlondon ~]# 


This helpful?

Comment 12 Tom "spot" Callaway 2009-03-27 00:42:40 UTC
I don't think the audit messages have anything to do with the lockup. How did you label the ntfs filesystem? Did windows do that?

I'm not sure why it tries to mount devpts twice. Something very strange is happening here.

Comment 13 Pascal Terjan 2009-03-27 13:41:59 UTC
Hello,
we had the same issue on Mandriva but could not often reproduce it. It was triggered by readahead-collector enabling audit.

I can now easily trigger it with :

auditctl -e 1
auditctl -w /etc
mount /dev/sda1 /mnt/windows

Comment 14 Tom London 2009-03-27 15:39:54 UTC
Interesting.....

Well, in (partial?) support of #13, I just got a freeze with a "partition mount" too, so the issue must be something "underlying".

Comment 15 Pascal Terjan 2009-03-27 18:50:37 UTC
I think I understood the problem and sent the explanation upstream

mount.ntfs-3g calls mount to update mtab

mount does readlink on mountpoint and when audit is enabled this leads to a fuse request which hangs because mount.ntfs-3g can't reply as it is waiting for mount to return

mount         S 00004cca     0 11821  11819
 c3401dcc 00200082 00004cca 00004cca c3401d7c c0134709 00200086 386e9131
 000003e1 00004cca dfb8d540 dfb5a000 c05ebcd9 c05b7380 df88a490 df88a6f0
 c1409380 00000000 00000000 38734404 000003e1 c3401db0 c3401da8 df88a6f0
Call Trace:
 [<c0134709>] ? release_console_sem+0x199/0x1e0
 [<c014934a>] ? prepare_to_wait+0x3a/0x70
 [<e0d4a77c>] wait_answer_interruptible+0x6c/0xa0 [fuse]
 [<c01490f0>] ? autoremove_wake_function+0x0/0x50
 [<e0d4a9f1>] fuse_request_send+0x181/0x250 [fuse]
 [<e0d4bba5>] ? fuse_get_req+0xc5/0x120 [fuse]
 [<e0d4c403>] fuse_getxattr+0xe3/0x140 [fuse]
 [<e0d4c320>] ? fuse_getxattr+0x0/0x140 [fuse]
 [<c022d940>] get_vfs_caps_from_disk+0x60/0xe0
 [<c01ba5eb>] ? __link_path_walk+0x26b/0xd70
 [<c016cfa7>] audit_copy_inode+0x77/0xb0
 [<c016d410>] __audit_inode+0xf0/0x280
 [<c01bb4da>] do_path_lookup+0x13a/0x1a0
 [<c01bc0aa>] user_path_at+0x4a/0x80
 [<c01c770d>] ? mntput_no_expire+0x1d/0x140
 [<c01c770d>] ? mntput_no_expire+0x1d/0x140
 [<c01b45fe>] sys_readlinkat+0x2e/0x90
 [<c01b4687>] sys_readlink+0x27/0x30
 [<c0103fdb>] sysenter_do_call+0x12/0x2f
 [<c03a0000>] ? serial_pnp_probe+0xb0/0x240

Comment 16 Andreas Simon 2009-03-28 12:53:58 UTC
The new ntfs-3g package from comment #9 doesn't change here anything too.

I am also able to trigger a mount-freeze with the commands from comment #13.

Comment 17 Tom "spot" Callaway 2009-03-28 14:29:04 UTC
Pascal, thanks for the great information. Did you hear anything back from upstream?

Comment 18 Szabolcs Szakacsits 2009-03-28 20:32:15 UTC
Upstream is still discussing the issue with FUSE.

We would like to know why /etc/mtab update needs readlink(mountpoint)? Perhaps mount(8) shouldn't do it for /etc/mtab update. Where could I find the used util-linux-ng mount(8) source code (and all the applied patches)?

The long term solution would be to symlink /etc/mtab to /proc/mounts. There are kernel patches for this for years.

Temporary workaround could be compiling ntfs-3g with ./configure --disable-mtab, or patch ntfs-3g not to wait for /bin/mount -i to return. We are still investigating the later solution (e.g. should an mtab update problem result umount(2) or not?).

Comment 19 Szabolcs Szakacsits 2009-03-30 14:07:48 UTC
You may consider the below patch as a better temporary workaround:
http://ntfs-3g.org/download/mount-readlink-hang-workaround.diff

It should be used with internal fuse-lite (default) and no --disable-mtab
option. --disable-mtab breaks too many things (df, mount, etc).

Comment 20 Fedora Update System 2009-03-30 15:56:36 UTC
ntfs-3g-2009.3.8-2.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/ntfs-3g-2009.3.8-2.fc9

Comment 21 Fedora Update System 2009-03-30 15:56:42 UTC
ntfs-3g-2009.3.8-2.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/ntfs-3g-2009.3.8-2.fc10

Comment 22 Tom "spot" Callaway 2009-03-30 15:57:52 UTC
Here is a testing update with that patch attached, please test it out:

Fedora 9:
https://admin.fedoraproject.org/updates/ntfs-3g-2009.3.8-2.fc9

Fedora 10:
https://admin.fedoraproject.org/updates/ntfs-3g-2009.3.8-2.fc10

Rawhide:
http://koji.fedoraproject.org/koji/buildinfo?buildID=95884

Comment 23 Tom London 2009-03-30 16:23:56 UTC
ntfs-3g-2009.3.8-2.fc11.x86_64 "works for me" (did not freeze boot, at least a couple of times).....

Comment 24 Andreas Simon 2009-03-31 05:49:11 UTC
Works here too, tested ntfs-3g-2009.3.8-2.fc11.x86_64.rpm on current rawhide. Thanks.

Comment 25 Tom London 2009-04-01 22:04:49 UTC
Closing this as fixed in rawhide.

Comment 26 Fedora Update System 2009-04-03 13:35:58 UTC
ntfs-3g-2009.4.4-1.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/ntfs-3g-2009.4.4-1.fc9

Comment 27 Fedora Update System 2009-04-03 13:36:05 UTC
ntfs-3g-2009.4.4-1.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/ntfs-3g-2009.4.4-1.fc10

Comment 28 Fedora Update System 2009-04-06 20:31:52 UTC
ntfs-3g-2009.4.4-1.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 29 Fedora Update System 2009-04-06 20:34:36 UTC
ntfs-3g-2009.4.4-1.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.