Bug 844485 - Cannot create files on ntfs-3g filesystem
Cannot create files on ntfs-3g filesystem
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Kernel Maintainer List
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-30 16:56 EDT by Richard W.M. Jones
Modified: 2012-08-21 11:25 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-08-13 16:11:53 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Richard W.M. Jones 2012-07-30 16:56:41 EDT
Description of problem:

libguestfs: recv_from_daemon: received GUESTFS_LAUNCH_FLAG
libguestfs: [89832ms] appliance is up
libguestfs: trace: launch = 0
libguestfs: trace: part_init "/dev/sda" "mbr"
libguestfs: send_to_daemon: 64 bytes: 00 00 00 3c | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 d0 | 00 00 00 00 | ...
guestfsd: main_loop: new request, len 0x3c
udevadm settle
udevsettle
udevsettle: No such file or directory
parted -s -- /dev/sda mklabel msdos
udevadm settle
udevsettle
udevsettle: No such file or directory
libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 d0 | 00 00 00 01 | 00 12 34 00 | ...
libguestfs: trace: part_init = 0
libguestfs: trace: part_add "/dev/sda" "p" 64 524287
libguestfs: send_to_daemon: 80 bytes: 00 00 00 4c | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 d1 | 00 00 00 00 | ...
guestfsd: main_loop: proc 208 (part_init) took 3.16 seconds
guestfsd: main_loop: new request, len 0x4c
udevadm settle
udevsettle
udevsettle: No such file or directory
parted -s -- /dev/sda mkpart primary 64s 524287s
Warning: The resulting partition is not properly aligned for best performance.
udevadm settle
udevsettle
udevsettle: No such file or directory
libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 d1 | 00 00 00 01 | 00 12 34 01 | ...
libguestfs: trace: part_add = 0
libguestfs: trace: part_add "/dev/sda" "p" 524288 -64
libguestfs: send_to_daemon: 80 bytes: 00 00 00 4c | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 d1 | 00 00 00 00 | ...
guestfsd: main_loop: proc 209 (part_add) took 3.62 seconds
guestfsd: main_loop: new request, len 0x4c
udevadm settle
udevsettle
udevsettle: No such file or directory
parted -s -- /dev/sda mkpart primary 524288s -64s
udevadm settle
udevsettle
udevsettle: No such file or directory
libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 d1 | 00 00 00 01 | 00 12 34 02 | ...
libguestfs: trace: part_add = 0
libguestfs: trace: pwrite_device "/dev/sda" "1234" 440
libguestfs: send_to_daemon: 72 bytes: 00 00 00 44 | 20 00 f5 f5 | 00 00 00 04 | 00 00 01 13 | 00 00 00 00 | ...
guestfsd: main_loop: proc 209 (part_add) took 5.05 seconds
guestfsd: main_loop: new request, len 0x44
libguestfs: recv_from_daemon: 44 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 01 13 | 00 00 00 01 | 00 12 34 03 | ...
libguestfs: trace: pwrite_device = 4
libguestfs: trace: mkfs "ntfs" "/dev/sda1"
libguestfs: send_to_daemon: 84 bytes: 00 00 00 50 | 20 00 f5 f5 | 00 00 00 04 | 00 00 01 16 | 00 00 00 00 | ...
guestfsd: main_loop: proc 275 (pwrite_device) took 0.05 seconds
guestfsd: main_loop: new request, len 0x50
mkfs -t ntfs -Q /dev/sda1
libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 01 16 | 00 00 00 01 | 00 12 34 04 | ...
libguestfs: trace: mkfs = 0
libguestfs: trace: mkfs "ntfs" "/dev/sda2"
libguestfs: send_to_daemon: 84 bytes: 00 00 00 50 | 20 00 f5 f5 | 00 00 00 04 | 00 00 01 16 | 00 00 00 00 | ...
guestfsd: main_loop: proc 278 (mkfs) took 4.63 seconds
guestfsd: main_loop: new request, len 0x50
mkfs -t ntfs -Q /dev/sda2
libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 01 16 | 00 00 00 01 | 00 12 34 05 | ...
libguestfs: trace: mkfs = 0
libguestfs: trace: mount_options "" "/dev/sda2" "/"
libguestfs: send_to_daemon: 72 bytes: 00 00 00 44 | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 4a | 00 00 00 00 | ...
guestfsd: main_loop: proc 278 (mkfs) took 3.74 seconds
guestfsd: main_loop: new request, len 0x44
mount -o  /dev/sda2 /sysroot/
[   97.401403] fuse init (API version 7.19)
libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 4a | 00 00 00 01 | 00 12 34 06 | ...
libguestfs: trace: mount_options = 0
libguestfs: trace: mkdir_p "/Windows/System32/Config"
libguestfs: send_to_daemon: 72 bytes: 00 00 00 44 | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 21 | 00 00 00 00 | ...
guestfsd: main_loop: proc 74 (mount_options) took 4.58 seconds
guestfsd: main_loop: new request, len 0x44
libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 21 | 00 00 00 01 | 00 12 34 07 | ...
libguestfs: trace: mkdir_p = 0
libguestfs: trace: upload "./guest-aux/windows-software" "/Windows/System32/Config/SOFTWARE"
libguestfs: send_to_daemon: 84 bytes: 00 00 00 50 | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 42 | 00 00 00 00 | ...
guestfsd: main_loop: proc 33 (mkdir_p) took 0.17 seconds
libguestfs: send_to_daemon: 8204 bytes: 00 00 20 08 | 00 00 00 00 | 00 00 20 00 | 72 65 67 66 | 01 01 00 00 | ...
libguestfs: send_to_daemon: 4108 bytes: 00 00 10 08 | 00 00 00 00 | 00 00 10 00 | 68 62 69 6e | 00 10 00 00 | ...
libguestfs: send_to_daemon: 12 bytes: 00 00 00 08 | 00 00 00 00 | 00 00 00 00 | 
guestfsd: main_loop: new request, len 0x50
libguestfs: recv_from_daemon: received GUESTFS_CANCEL_FLAG
guestfsd: receive_file: reading length word
guestfsd: receive_file: got chunk: cancel = 0x0, len = 8192, buf = 0x4cef820
guestfsd: receive_file: reading length word
guestfsd: receive_file: got chunk: cancel = 0x0, len = 4096, buf = 0x4cee820
guestfsd: receive_file: reading length word
guestfsd: receive_file: got chunk: cancel = 0x0, len = 0, buf = (nil)
guestfsd: receive_file: end of file, leaving function
guestfsd: error: /Windows/System32/Config/SOFTWARE: No such file or directory
libguestfs: recv_from_daemon: 116 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 42 | 00 00 00 01 | 00 12 34 08 | ...
libguestfs: trace: upload = -1 (error)
*stdin*:22: libguestfs: error: upload: /Windows/System32/Config/SOFTWARE: No such file or directory
libguestfs: trace: close
libguestfs: closing guestfs handle 0x1d98ad0 (state 2)
libguestfs: trace: shutdown
libguestfs: trace: internal_autosync
libguestfs: send_to_daemon: 44 bytes: 00 00 00 28 | 20 00 f5 f5 | 00 00 00 04 | 00 00 01 1a | 00 00 00 00 | ...
guestfsd: main_loop: proc 66 (upload) took 0.05 seconds
guestfsd: main_loop: new request, len 0x28
umount /sysroot
fsync /dev/sda
libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 01 1a | 00 00 00 01 | 00 12 34 09 | ...
libguestfs: trace: internal_autosync = 0
libguestfs: sending SIGTERM to process 22474
libguestfs: trace: shutdown = 0
/builddir/build/BUILD/libguestfs-1.19.27/run: command failed with exit code 1
make[2]: Leaving directory `/builddir/build/BUILD/libguestfs-1.19.27/tests/guests'
make[1]: Leaving directory `/builddir/build/BUILD/libguestfs-1.19.27/tests/guests'
RPM build errors:
make[2]: *** [windows.img] Error 1
make[1]: *** [check-am] Error 2
make: *** [check-recursive] Error 1

Version-Release number of selected component (if applicable):

libguestfs 1.19.27

How reproducible:

Unknown, but seen several times.

Steps to Reproduce:
1. make check
Comment 1 Richard W.M. Jones 2012-07-30 16:57:09 EDT
Full build log:
http://kojipkgs.fedoraproject.org//work/tasks/3338/4343338/build.log
Comment 2 Richard W.M. Jones 2012-08-01 08:52:29 EDT
Finally I can reproduce this one, in a Rawhide guest.

By updating components one at a time I determined that what
cause the bug is updating to kernel 3.6.0-0.rc0.git4.1.fc18.x86_64.

Here is a reproducer:

guestfish -xv -N fs:ntfs -m /dev/sda1 \
  mkdir-p /Windows/System32/Config : \
  upload tests/guests/guest-aux/windows-software /Windows/System32/Config/SOFTWARE

Setting the environment variable FEBOOTSTRAP_KERNEL
before the build makes the bug go away if you use

export FEBOOTSTRAP_KERNEL=/boot/vmlinuz-3.5.0-1.fc18.x86_64

and reappear reliably if you use:

export FEBOOTSTRAP_KERNEL=/boot/vmlinuz-3.6.0-0.rc0.git4.1.fc18.x86_64

I wonder if the new kernel has broken FUSE support in some way?
Comment 3 Richard W.M. Jones 2012-08-01 08:56:01 EDT
Here's an even simpler reproducer which shows that simply
creating a directory and writing to a file fails:

FEBOOTSTRAP_KERNEL=/boot/vmlinuz-3.6.0-0.rc0.git4.1.fc18.x86_64 \
guestfish -xv -N fs:ntfs -m /dev/sda1 mkdir /foo : write /foo/bar data

The error is:

libguestfs: error: write: open: /foo/bar: No such file or directory
Comment 4 Richard W.M. Jones 2012-08-01 09:02:48 EDT
Some variations on a theme.

The directory exists and can be listed with "ls -l":

$ guestfish -N fs:ntfs -m /dev/sda1 mkdir /foo : ll /
total 8
drwxrwxrwx  1 root root 4096 Aug  1 13:56 .
drwxr-xr-x 23 1000 1000 4096 Aug  1 13:56 ..
drwxrwxrwx  1 root root    0 Aug  1 13:56 foo

$ guestfish -N fs:ntfs -m /dev/sda1 mkdir /foo : ll /foo
total 4
drwxrwxrwx 1 root root    0 Aug  1 13:56 .
drwxrwxrwx 1 root root 4096 Aug  1 13:56 ..

But a simple 'touch' of any file in there fails:

$ guestfish -N fs:ntfs -m /dev/sda1 mkdir /foo : touch /foo/bar
libguestfs: error: touch: open: /foo/bar: No such file or directory

[Here is the code that the 'touch' command executes:
https://github.com/libguestfs/libguestfs/blob/499497fab03716278cb5214a8aa89a35618e06c6/daemon/file.c#L32
The error is at line 69]

Adding a 10 second sleep before the touch does not help, so it's
not a race condition or timing related.

Adding sync before the touch doesn't help either.

'touch' in the root directory of the NTFS filesystem also fails.

$ guestfish -N fs:ntfs -m /dev/sda1 touch /foo
libguestfs: error: touch: open: /foo: No such file or directory

Using the regular echo bash-builtin also fails:

$ guestfish -N fs:ntfs -m /dev/sda1 debug sh "echo > /sysroot/foo"
libguestfs: error: debug: /bin/sh: /sysroot/foo: No such file or directory
Comment 5 Richard W.M. Jones 2012-08-01 09:47:00 EDT
Upstream report of the same thing:
http://sourceforge.net/mailarchive/message.php?msg_id=29617641
Comment 6 Richard W.M. Jones 2012-08-01 18:22:08 EDT
git bisect:

c8ccbe032feb127a977c66865cb63d72d9a6e08b is the first bad commit
commit c8ccbe032feb127a977c66865cb63d72d9a6e08b
Author: Miklos Szeredi <mszeredi@suse.cz>
Date:   Tue Jun 5 15:10:22 2012 +0200

    fuse: implement i_op->atomic_open()
    
    Add an ->atomic_open implementation which replaces the atomic open+create
    operation implemented via ->create.  No functionality is changed.
    
    Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
    Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>

:040000 040000 7a000897ad0d1524a7e1cb187bc77d92b8397d56 38f02e55f70b009f8b1149cdd75f571c72406fe4 M	fs
Comment 7 Richard W.M. Jones 2012-08-03 14:30:30 EDT
Miklos posted the following fix:

http://sourceforge.net/mailarchive/message.php?msg_id=29630599

and I have tested this patch on top of linux.git and confirmed
that it does indeed fix this bug.
Comment 8 Josh Boyer 2012-08-10 10:01:54 EDT
Believe Miklos wound up submitting a larger series to fix this upstream:

http://thread.gmane.org/gmane.linux.file-systems/66445/focus=1339000

We'll pick that up through Linus' tree
Comment 9 Richard W.M. Jones 2012-08-10 17:42:01 EDT
(In reply to comment #8)
> Believe Miklos wound up submitting a larger series to fix this upstream:
> 
> http://thread.gmane.org/gmane.linux.file-systems/66445/focus=1339000
> 
> We'll pick that up through Linus' tree

I applied these patches on top of kernel-3.6.0-0.rc1.git3.2
and they fix all[*] of the problems with ntfs-3g and FUSE
that I found.

[*] There are several problems, not just this bug.
Comment 10 Josh Boyer 2012-08-13 09:37:36 EDT
(In reply to comment #9)
> (In reply to comment #8)
> > Believe Miklos wound up submitting a larger series to fix this upstream:
> > 
> > http://thread.gmane.org/gmane.linux.file-systems/66445/focus=1339000
> > 
> > We'll pick that up through Linus' tree
> 
> I applied these patches on top of kernel-3.6.0-0.rc1.git3.2
> and they fix all[*] of the problems with ntfs-3g and FUSE
> that I found.
> 
> [*] There are several problems, not just this bug.

Linus is on vacation and those patches are taking their sweet time getting into his tree.  I'll pull them into F18/rawhide this afternoon.

Thanks much for reporting and testing on this one.
Comment 11 Josh Boyer 2012-08-13 14:59:17 EDT
OK, committed.
Comment 12 Josh Boyer 2012-08-13 16:11:53 EDT
And building.  Should be in tomorrow's F18/rawhide.
Comment 13 Richard W.M. Jones 2012-08-21 11:25:19 EDT
FYI all 4 patches went into Linus's kernel.

Note You need to log in before you can comment on or make changes to this bug.