Description of problem: libguestfs: recv_from_daemon: received GUESTFS_LAUNCH_FLAG libguestfs: [89832ms] appliance is up libguestfs: trace: launch = 0 libguestfs: trace: part_init "/dev/sda" "mbr" libguestfs: send_to_daemon: 64 bytes: 00 00 00 3c | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 d0 | 00 00 00 00 | ... guestfsd: main_loop: new request, len 0x3c udevadm settle udevsettle udevsettle: No such file or directory parted -s -- /dev/sda mklabel msdos udevadm settle udevsettle udevsettle: No such file or directory libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 d0 | 00 00 00 01 | 00 12 34 00 | ... libguestfs: trace: part_init = 0 libguestfs: trace: part_add "/dev/sda" "p" 64 524287 libguestfs: send_to_daemon: 80 bytes: 00 00 00 4c | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 d1 | 00 00 00 00 | ... guestfsd: main_loop: proc 208 (part_init) took 3.16 seconds guestfsd: main_loop: new request, len 0x4c udevadm settle udevsettle udevsettle: No such file or directory parted -s -- /dev/sda mkpart primary 64s 524287s Warning: The resulting partition is not properly aligned for best performance. udevadm settle udevsettle udevsettle: No such file or directory libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 d1 | 00 00 00 01 | 00 12 34 01 | ... libguestfs: trace: part_add = 0 libguestfs: trace: part_add "/dev/sda" "p" 524288 -64 libguestfs: send_to_daemon: 80 bytes: 00 00 00 4c | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 d1 | 00 00 00 00 | ... guestfsd: main_loop: proc 209 (part_add) took 3.62 seconds guestfsd: main_loop: new request, len 0x4c udevadm settle udevsettle udevsettle: No such file or directory parted -s -- /dev/sda mkpart primary 524288s -64s udevadm settle udevsettle udevsettle: No such file or directory libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 d1 | 00 00 00 01 | 00 12 34 02 | ... libguestfs: trace: part_add = 0 libguestfs: trace: pwrite_device "/dev/sda" "1234" 440 libguestfs: send_to_daemon: 72 bytes: 00 00 00 44 | 20 00 f5 f5 | 00 00 00 04 | 00 00 01 13 | 00 00 00 00 | ... guestfsd: main_loop: proc 209 (part_add) took 5.05 seconds guestfsd: main_loop: new request, len 0x44 libguestfs: recv_from_daemon: 44 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 01 13 | 00 00 00 01 | 00 12 34 03 | ... libguestfs: trace: pwrite_device = 4 libguestfs: trace: mkfs "ntfs" "/dev/sda1" libguestfs: send_to_daemon: 84 bytes: 00 00 00 50 | 20 00 f5 f5 | 00 00 00 04 | 00 00 01 16 | 00 00 00 00 | ... guestfsd: main_loop: proc 275 (pwrite_device) took 0.05 seconds guestfsd: main_loop: new request, len 0x50 mkfs -t ntfs -Q /dev/sda1 libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 01 16 | 00 00 00 01 | 00 12 34 04 | ... libguestfs: trace: mkfs = 0 libguestfs: trace: mkfs "ntfs" "/dev/sda2" libguestfs: send_to_daemon: 84 bytes: 00 00 00 50 | 20 00 f5 f5 | 00 00 00 04 | 00 00 01 16 | 00 00 00 00 | ... guestfsd: main_loop: proc 278 (mkfs) took 4.63 seconds guestfsd: main_loop: new request, len 0x50 mkfs -t ntfs -Q /dev/sda2 libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 01 16 | 00 00 00 01 | 00 12 34 05 | ... libguestfs: trace: mkfs = 0 libguestfs: trace: mount_options "" "/dev/sda2" "/" libguestfs: send_to_daemon: 72 bytes: 00 00 00 44 | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 4a | 00 00 00 00 | ... guestfsd: main_loop: proc 278 (mkfs) took 3.74 seconds guestfsd: main_loop: new request, len 0x44 mount -o /dev/sda2 /sysroot/ [ 97.401403] fuse init (API version 7.19) libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 4a | 00 00 00 01 | 00 12 34 06 | ... libguestfs: trace: mount_options = 0 libguestfs: trace: mkdir_p "/Windows/System32/Config" libguestfs: send_to_daemon: 72 bytes: 00 00 00 44 | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 21 | 00 00 00 00 | ... guestfsd: main_loop: proc 74 (mount_options) took 4.58 seconds guestfsd: main_loop: new request, len 0x44 libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 21 | 00 00 00 01 | 00 12 34 07 | ... libguestfs: trace: mkdir_p = 0 libguestfs: trace: upload "./guest-aux/windows-software" "/Windows/System32/Config/SOFTWARE" libguestfs: send_to_daemon: 84 bytes: 00 00 00 50 | 20 00 f5 f5 | 00 00 00 04 | 00 00 00 42 | 00 00 00 00 | ... guestfsd: main_loop: proc 33 (mkdir_p) took 0.17 seconds libguestfs: send_to_daemon: 8204 bytes: 00 00 20 08 | 00 00 00 00 | 00 00 20 00 | 72 65 67 66 | 01 01 00 00 | ... libguestfs: send_to_daemon: 4108 bytes: 00 00 10 08 | 00 00 00 00 | 00 00 10 00 | 68 62 69 6e | 00 10 00 00 | ... libguestfs: send_to_daemon: 12 bytes: 00 00 00 08 | 00 00 00 00 | 00 00 00 00 | guestfsd: main_loop: new request, len 0x50 libguestfs: recv_from_daemon: received GUESTFS_CANCEL_FLAG guestfsd: receive_file: reading length word guestfsd: receive_file: got chunk: cancel = 0x0, len = 8192, buf = 0x4cef820 guestfsd: receive_file: reading length word guestfsd: receive_file: got chunk: cancel = 0x0, len = 4096, buf = 0x4cee820 guestfsd: receive_file: reading length word guestfsd: receive_file: got chunk: cancel = 0x0, len = 0, buf = (nil) guestfsd: receive_file: end of file, leaving function guestfsd: error: /Windows/System32/Config/SOFTWARE: No such file or directory libguestfs: recv_from_daemon: 116 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 00 42 | 00 00 00 01 | 00 12 34 08 | ... libguestfs: trace: upload = -1 (error) *stdin*:22: libguestfs: error: upload: /Windows/System32/Config/SOFTWARE: No such file or directory libguestfs: trace: close libguestfs: closing guestfs handle 0x1d98ad0 (state 2) libguestfs: trace: shutdown libguestfs: trace: internal_autosync libguestfs: send_to_daemon: 44 bytes: 00 00 00 28 | 20 00 f5 f5 | 00 00 00 04 | 00 00 01 1a | 00 00 00 00 | ... guestfsd: main_loop: proc 66 (upload) took 0.05 seconds guestfsd: main_loop: new request, len 0x28 umount /sysroot fsync /dev/sda libguestfs: recv_from_daemon: 40 bytes: 20 00 f5 f5 | 00 00 00 04 | 00 00 01 1a | 00 00 00 01 | 00 12 34 09 | ... libguestfs: trace: internal_autosync = 0 libguestfs: sending SIGTERM to process 22474 libguestfs: trace: shutdown = 0 /builddir/build/BUILD/libguestfs-1.19.27/run: command failed with exit code 1 make[2]: Leaving directory `/builddir/build/BUILD/libguestfs-1.19.27/tests/guests' make[1]: Leaving directory `/builddir/build/BUILD/libguestfs-1.19.27/tests/guests' RPM build errors: make[2]: *** [windows.img] Error 1 make[1]: *** [check-am] Error 2 make: *** [check-recursive] Error 1 Version-Release number of selected component (if applicable): libguestfs 1.19.27 How reproducible: Unknown, but seen several times. Steps to Reproduce: 1. make check
Full build log: http://kojipkgs.fedoraproject.org//work/tasks/3338/4343338/build.log
Finally I can reproduce this one, in a Rawhide guest. By updating components one at a time I determined that what cause the bug is updating to kernel 3.6.0-0.rc0.git4.1.fc18.x86_64. Here is a reproducer: guestfish -xv -N fs:ntfs -m /dev/sda1 \ mkdir-p /Windows/System32/Config : \ upload tests/guests/guest-aux/windows-software /Windows/System32/Config/SOFTWARE Setting the environment variable FEBOOTSTRAP_KERNEL before the build makes the bug go away if you use export FEBOOTSTRAP_KERNEL=/boot/vmlinuz-3.5.0-1.fc18.x86_64 and reappear reliably if you use: export FEBOOTSTRAP_KERNEL=/boot/vmlinuz-3.6.0-0.rc0.git4.1.fc18.x86_64 I wonder if the new kernel has broken FUSE support in some way?
Here's an even simpler reproducer which shows that simply creating a directory and writing to a file fails: FEBOOTSTRAP_KERNEL=/boot/vmlinuz-3.6.0-0.rc0.git4.1.fc18.x86_64 \ guestfish -xv -N fs:ntfs -m /dev/sda1 mkdir /foo : write /foo/bar data The error is: libguestfs: error: write: open: /foo/bar: No such file or directory
Some variations on a theme. The directory exists and can be listed with "ls -l": $ guestfish -N fs:ntfs -m /dev/sda1 mkdir /foo : ll / total 8 drwxrwxrwx 1 root root 4096 Aug 1 13:56 . drwxr-xr-x 23 1000 1000 4096 Aug 1 13:56 .. drwxrwxrwx 1 root root 0 Aug 1 13:56 foo $ guestfish -N fs:ntfs -m /dev/sda1 mkdir /foo : ll /foo total 4 drwxrwxrwx 1 root root 0 Aug 1 13:56 . drwxrwxrwx 1 root root 4096 Aug 1 13:56 .. But a simple 'touch' of any file in there fails: $ guestfish -N fs:ntfs -m /dev/sda1 mkdir /foo : touch /foo/bar libguestfs: error: touch: open: /foo/bar: No such file or directory [Here is the code that the 'touch' command executes: https://github.com/libguestfs/libguestfs/blob/499497fab03716278cb5214a8aa89a35618e06c6/daemon/file.c#L32 The error is at line 69] Adding a 10 second sleep before the touch does not help, so it's not a race condition or timing related. Adding sync before the touch doesn't help either. 'touch' in the root directory of the NTFS filesystem also fails. $ guestfish -N fs:ntfs -m /dev/sda1 touch /foo libguestfs: error: touch: open: /foo: No such file or directory Using the regular echo bash-builtin also fails: $ guestfish -N fs:ntfs -m /dev/sda1 debug sh "echo > /sysroot/foo" libguestfs: error: debug: /bin/sh: /sysroot/foo: No such file or directory
Upstream report of the same thing: http://sourceforge.net/mailarchive/message.php?msg_id=29617641
git bisect: c8ccbe032feb127a977c66865cb63d72d9a6e08b is the first bad commit commit c8ccbe032feb127a977c66865cb63d72d9a6e08b Author: Miklos Szeredi <mszeredi> Date: Tue Jun 5 15:10:22 2012 +0200 fuse: implement i_op->atomic_open() Add an ->atomic_open implementation which replaces the atomic open+create operation implemented via ->create. No functionality is changed. Signed-off-by: Miklos Szeredi <mszeredi> Signed-off-by: Al Viro <viro.org.uk> :040000 040000 7a000897ad0d1524a7e1cb187bc77d92b8397d56 38f02e55f70b009f8b1149cdd75f571c72406fe4 M fs
Miklos posted the following fix: http://sourceforge.net/mailarchive/message.php?msg_id=29630599 and I have tested this patch on top of linux.git and confirmed that it does indeed fix this bug.
Believe Miklos wound up submitting a larger series to fix this upstream: http://thread.gmane.org/gmane.linux.file-systems/66445/focus=1339000 We'll pick that up through Linus' tree
(In reply to comment #8) > Believe Miklos wound up submitting a larger series to fix this upstream: > > http://thread.gmane.org/gmane.linux.file-systems/66445/focus=1339000 > > We'll pick that up through Linus' tree I applied these patches on top of kernel-3.6.0-0.rc1.git3.2 and they fix all[*] of the problems with ntfs-3g and FUSE that I found. [*] There are several problems, not just this bug.
(In reply to comment #9) > (In reply to comment #8) > > Believe Miklos wound up submitting a larger series to fix this upstream: > > > > http://thread.gmane.org/gmane.linux.file-systems/66445/focus=1339000 > > > > We'll pick that up through Linus' tree > > I applied these patches on top of kernel-3.6.0-0.rc1.git3.2 > and they fix all[*] of the problems with ntfs-3g and FUSE > that I found. > > [*] There are several problems, not just this bug. Linus is on vacation and those patches are taking their sweet time getting into his tree. I'll pull them into F18/rawhide this afternoon. Thanks much for reporting and testing on this one.
OK, committed.
And building. Should be in tomorrow's F18/rawhide.
FYI all 4 patches went into Linus's kernel.