Created attachment 1559102 [details] detail log of virt-inspector Description of problem: After we install 4 window 10 enterprise with version 1809,inspect os failed for two of them , for other inspect os run successfully the window 10 VM only has one disk, and the partition /dev/sda2 is the root partition i mean the C: drive, but seems libguestfs failed to detect it and i compared the log from success and failed inspect, for successfull inspect the command " is-file /Window/System32/cmd.exe" return true, but for failed one the command "is-file /Window/System32/cmd.exe " return false follow is some piece of failed inspect log, libguestfs: recv_from_daemon: received GUESTFS_LAUNCH_FLAG libguestfs: appliance is up guestfsd: <= list_partitions (0x8) request length 40 bytes commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle -E /dev/sda calling: settle commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle -E /dev/sdb calling: settle guestfsd: => list_partitions (0x8) took 0.00 secs guestfsd: <= vfs_type (0xc6) request length 56 bytes commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle -E /dev/sda1 calling: settle command: blkid '-c' '/dev/null' '-o' 'value' '-s' 'TYPE' '/dev/sda1' command: blkid returned 0 command: blkid: stdout: ntfs guestfsd: => vfs_type (0xc6) took 0.01 secs guestfsd: <= vfs_type (0xc6) request length 56 bytes commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle -E /dev/sda2 calling: settle command: blkid '-c' '/dev/null' '-o' 'value' '-s' 'TYPE' '/dev/sda2' command: blkid returned 0 command: blkid: stdout: ntfs guestfsd: => vfs_type (0xc6) took 0.00 secs guestfsd: <= vfs_type (0xc6) request length 56 bytes commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle -E /dev/sda3 calling: settle command: blkid '-c' '/dev/null' '-o' 'value' '-s' 'TYPE' '/dev/sda3' command: blkid returned 0 command: blkid: stdout: ntfs guestfsd: => vfs_type (0xc6) took 0.00 secs guestfsd: <= inspect_os (0x1e0) request length 40 bytes commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle -E /dev/sda calling: settle commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle -E /dev/sdb calling: settle commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle -E /dev/sda calling: settle commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle -E /dev/sdb calling: settle command: sfdisk '--help' command: sfdisk returned 0 command: sfdisk: stdout: Usage: sfdisk [options] <device> [...] Options: -s, --show-size list size of a partition -c, --id change or print partition Id --change-id change Id --print-id print Id -l, --list list partitions of each device -d, --dump idem, but in a format suitable for later input -i, --increment number cylinders etc. from 1 instead of from 0 -u, --unit <letter> units to be used; <letter> can be one of S (sectors), C (cylinders), B (blocks), or M (MB) -1, --one-only reserved option that does nothing currently -T, --list-types list the known partition types -D, --DOS for DOS-compatibility: waste a little space -E, --DOS-extended DOS extended partition compatibility -R, --re-read make the kernel reread the partition table -N <number> change only the partition with this <number> -n do not actually write to disk -O <file> save the sectors that will be overwritten to <file> -I <file> restore sectors from <file> -V, --verify check that the listed partitions are reasonable -v, --version display version information and exit -h, --help display this help text and exit Dangerous options: -f, --force disable all consistency checking --no-reread do not check whether the partition is in use -q, --quiet suppress warning messages -L, --Linux do not complain about things irrelevant for Linux -g, --show-geometry print the kernel's idea of the geometry -G, --show-pt-geometry print geometry guessed from the partition table -A, --activate[=<device>] activate bootable flag -U, --unhide[=<dev>] set partition unhidden -x, --show-extended also list extended partitions in the output, or expect descriptors for them in the input --leave-last do not allocate the last cylinder --IBM same as --leave-last --in-order partitions are in order --not-in-order partitions are not in order --inside-outer all logicals inside outermost extended --not-inside-outer not all logicals inside outermost extended --nested every partition is disjoint from all others --chained like nested, but extended partitions may lie outside --onesector partitions are mutually disjoint Override the detected geometry using: -C, --cylinders <number> set the number of cylinders to use -H, --heads <number> set the number of heads to use -S, --sectors <number> set the number of sectors to use commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle calling: settle command: sfdisk '--print-id' '/dev/sda' '1' [ 1.690630] sda: sda1 sda2 sda3 command: sfdisk returned 0 command: sfdisk: stdout: 7 commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle calling: settle command: blkid '-c' '/dev/null' '-o' 'value' '-s' 'TYPE' '/dev/sda1' command: blkid returned 0 command: blkid: stdout: ntfs commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle calling: settle command: sfdisk '--print-id' '/dev/sda' '2' [ 1.731749] sda: sda1 sda2 sda3 command: sfdisk returned 0 command: sfdisk: stdout: 7 commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle calling: settle command: blkid '-c' '/dev/null' '-o' 'value' '-s' 'TYPE' '/dev/sda2' command: blkid returned 0 command: blkid: stdout: ntfs commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle calling: settle command: sfdisk '--print-id' '/dev/sda' '3' [ 1.768274] sda: sda1 sda2 sda3 command: sfdisk returned 0 command: sfdisk: stdout: 27 commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle calling: settle command: blkid '-c' '/dev/null' '-o' 'value' '-s' 'TYPE' '/dev/sda3' command: blkid returned 0 command: blkid: stdout: ntfs command: lvm 'lvs' '--help' command: lvm returned 0 command: lvm: stdout: lvs - Display information about logical volumes lvs \t[ -H|--history ] \t[ -a|--all ] \t[ -o|--options String ] \t[ -S|--select String ] \t[ -O|--sort String ] \t[ --segments ] \t[ --aligned ] \t[ --binary ] \t[ --configreport log|vg|lv|pv|pvseg|seg ] \t[ --foreign ] \t[ --ignorelockingfailure ] \t[ --ignoreskippedcluster ] \t[ --logonly ] \t[ --nameprefixes ] \t[ --noheadings ] \t[ --nolocking ] \t[ --nosuffix ] \t[ --readonly ] \t[ --reportformat basic|json ] \t[ --rows ] \t[ --separator String ] \t[ --shared ] \t[ --trustcache ] \t[ --unbuffered ] \t[ --units r|R|h|H|b|B|s|S|k|K|m|M|g|G|t|T|p|P|e|E ] \t[ --unquoted ] \t[ COMMON_OPTIONS ] \t[ VG|LV|Tag ... ] Common options for lvm: \t[ -d|--debug ] \t[ -h|--help ] \t[ -q|--quiet ] \t[ -v|--verbose ] \t[ -y|--yes ] \t[ -t|--test ] \t[ --commandprofile String ] \t[ --config String ] \t[ --driverloaded y|n ] \t[ --lockopt String ] \t[ --longhelp ] \t[ --profile String ] \t[ --version ] Use --longhelp to show all options and advanced commands. command: lvm 'lvs' '-o' 'vg_name,lv_name' '-S' 'lv_role=public && lv_skip_activation!=yes' '--noheadings' '--separator' '/' command: lvm returned 0 check_for_filesystem_on: /dev/sda1 (ntfs) command: mount '-o' 'ro' '/dev/sda1' '/sysroot//' [ 1.859047] fuse init (API version 7.22) command: mount returned 0 chroot: /sysroot: running 'is_file: /grub/menu.lst' chroot: /sysroot: running 'is_file: /grub/grub.conf' chroot: /sysroot: running 'is_file: /grub2/grub.cfg' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_file: /hurd/console' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /bin' chroot: /sysroot: running 'is_dir: /log' chroot: /sysroot: running 'case_sensitive_path: /windows' chroot: /sysroot: running 'case_sensitive_path: /windows/system32' chroot: /sysroot: running 'case_sensitive_path: /winnt' chroot: /sysroot: running 'case_sensitive_path: /winnt/system32' chroot: /sysroot: running 'case_sensitive_path: /win32' chroot: /sysroot: running 'case_sensitive_path: /win32/system32' chroot: /sysroot: running 'case_sensitive_path: /win' chroot: /sysroot: running 'case_sensitive_path: /win/system32' chroot: /sysroot: running 'case_sensitive_path: /reactos' chroot: /sysroot: running 'case_sensitive_path: /reactos/system32' chroot: /sysroot: running 'case_sensitive_path: /boot.ini' chroot: /sysroot: running 'get_windows_systemroot_from_boot_ini' chroot: /sysroot: running 'case_sensitive_path: /System Volume Information' chroot: /sysroot: running 'is_dir: /System Volume Information' chroot: /sysroot: running 'case_sensitive_path: /Program Files' chroot: /sysroot: running 'is_dir: /Program Files' chroot: /sysroot: running 'case_sensitive_path: /System Volume Information' chroot: /sysroot: running 'is_dir: /System Volume Information' check_filesystem: /dev/sda1 matched Windows volume without installed applications command: umount '/sysroot' command: umount returned 0 check_for_filesystem_on: /dev/sda2 (ntfs) command: mount '-o' 'ro' '/dev/sda2' '/sysroot//' command: mount returned 0 chroot: /sysroot: running 'is_file: /grub/menu.lst' chroot: /sysroot: running 'is_file: /grub/grub.conf' chroot: /sysroot: running 'is_file: /grub2/grub.cfg' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_file: /hurd/console' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /bin' chroot: /sysroot: running 'is_dir: /log' chroot: /sysroot: running 'case_sensitive_path: /windows' chroot: /sysroot: running 'case_sensitive_path: /Windows/system32' chroot: /sysroot: running 'is_dir: /Windows/System32' chroot: /sysroot: running 'case_sensitive_path: /Windows/system32/config' chroot: /sysroot: running 'is_dir: /Windows/System32/config' chroot: /sysroot: running 'case_sensitive_path: /Windows/system32/cmd.exe' chroot: /sysroot: running 'is_file: /Windows/System32/cmd.exe' chroot: /sysroot: running 'case_sensitive_path: /winnt' chroot: /sysroot: running 'case_sensitive_path: /winnt/system32' chroot: /sysroot: running 'case_sensitive_path: /win32' chroot: /sysroot: running 'case_sensitive_path: /win32/system32' chroot: /sysroot: running 'case_sensitive_path: /win' chroot: /sysroot: running 'case_sensitive_path: /win/system32' chroot: /sysroot: running 'case_sensitive_path: /reactos' chroot: /sysroot: running 'case_sensitive_path: /reactos/system32' chroot: /sysroot: running 'case_sensitive_path: /boot.ini' chroot: /sysroot: running 'get_windows_systemroot_from_boot_ini' chroot: /sysroot: running 'case_sensitive_path: /System Volume Information' chroot: /sysroot: running 'is_dir: /System Volume Information' chroot: /sysroot: running 'case_sensitive_path: /Program Files' chroot: /sysroot: running 'is_dir: /Program Files' check_filesystem: /dev/sda2 matched Windows volume with installed applications command: umount '/sysroot' command: umount returned 0 check_for_filesystem_on: /dev/sda3 (ntfs) command: mount '-o' 'ro' '/dev/sda3' '/sysroot//' command: mount returned 0 chroot: /sysroot: running 'is_file: /grub/menu.lst' chroot: /sysroot: running 'is_file: /grub/grub.conf' chroot: /sysroot: running 'is_file: /grub2/grub.cfg' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_file: /hurd/console' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /etc' chroot: /sysroot: running 'is_dir: /bin' chroot: /sysroot: running 'is_dir: /log' chroot: /sysroot: running 'case_sensitive_path: /windows' chroot: /sysroot: running 'case_sensitive_path: /windows/system32' chroot: /sysroot: running 'case_sensitive_path: /winnt' chroot: /sysroot: running 'case_sensitive_path: /winnt/system32' chroot: /sysroot: running 'case_sensitive_path: /win32' chroot: /sysroot: running 'case_sensitive_path: /win32/system32' chroot: /sysroot: running 'case_sensitive_path: /win' chroot: /sysroot: running 'case_sensitive_path: /win/system32' chroot: /sysroot: running 'case_sensitive_path: /reactos' chroot: /sysroot: running 'case_sensitive_path: /reactos/system32' chroot: /sysroot: running 'case_sensitive_path: /boot.ini' chroot: /sysroot: running 'get_windows_systemroot_from_boot_ini' chroot: /sysroot: running 'case_sensitive_path: /System Volume Information' chroot: /sysroot: running 'is_dir: /System Volume Information' chroot: /sysroot: running 'case_sensitive_path: /Program Files' chroot: /sysroot: running 'is_dir: /Program Files' chroot: /sysroot: running 'case_sensitive_path: /System Volume Information' chroot: /sysroot: running 'is_dir: /System Volume Information' check_filesystem: /dev/sda3 matched Windows volume without installed applications command: umount '/sysroot' command: umount returned 0 inspect_os: fses: fs: /dev/sda1 (ntfs) role: other fs: /dev/sda2 (ntfs) role: other fs: /dev/sda3 (ntfs) role: other inspect_get_roots: roots: guestfsd: => inspect_os (0x1e0) took 0.72 secs <?xml version="1.0"?> <operatingsystems/> libguestfs: closing guestfs handle 0x557d33b728c0 (state 2) guestfsd: <= internal_autosync (0x11a) request length 40 bytes umount-all: /proc/mounts: fsname=rootfs dir=/ type=rootfs opts=rw freq=0 passno=0 umount-all: /proc/mounts: fsname=proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0 umount-all: /proc/mounts: fsname=/dev/root dir=/ type=ext2 opts=rw,noatime freq=0 passno=0 umount-all: /proc/mounts: fsname=/proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0 umount-all: /proc/mounts: fsname=/sys dir=/sys type=sysfs opts=rw,relatime freq=0 passno=0 umount-all: /proc/mounts: fsname=tmpfs dir=/run type=tmpfs opts=rw,nosuid,relatime,size=97332k,mode=755 freq=0 passno=0 umount-all: /proc/mounts: fsname=/dev dir=/dev type=devtmpfs opts=rw,relatime,size=240200k,nr_inodes=60050,mode=755 freq=0 passno=0 umount-all: /proc/mounts: fsname=/dev/pts dir=/dev/pts type=devpts opts=rw,relatime,mode=600,ptmxmode=000 freq=0 passno=0 umount-all: /proc/mounts: fsname=shmfs dir=/dev/shm type=tmpfs opts=rw,relatime freq=0 passno=0 commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle -E /dev/sda calling: settle fsync /dev/sda commandrvf: stdout=n stderr=y flags=0x0 commandrvf: udevadm --debug settle -E /dev/sdb calling: settle guestfsd: => internal_autosync (0x11a) took 0.03 secs libguestfs: sending SIGTERM to process 92935 libguestfs: qemu maxrss 170928K libguestfs: command: run: rm libguestfs: command: run: \ -rf /tmp/libguestfs5SW5lm libguestfs: command: run: rm libguestfs: command: run: \ -rf /tmp/libguestfsCa6SbS Version-Release number of selected component (if applicable): libguestfs-1.38.2-12.el7_6.2.x86_64 How reproducible: Steps to Reproduce: 1.install win10 wiht 1809 build 2.run virt-inspector 3. Actual results: failed to found os information Expected results: should get the os information Additional info:
It's pretty strange. The obvious question ... > for successfull inspect the command " is-file /Window/System32/cmd.exe" return true, > but for failed one the command "is-file /Window/System32/cmd.exe " return false does /Windows/System32/cmd.exe exist?
Of course Yes, it/s wired, we install the vm use same ISO, and both the VM works well when i try to use guestfish to debug, i mkdir the mount point and mount the /dev/sda2, and use ls to list the cmd.exe, i can see the file, but there are "unsupported reparse point" after the file name, maybe this is the root cause for inspect failed, but for another win10 VMDK, it works well we want to figure out is there any difference for window feature or bugs in libguestfs Thanks
OK I know the problem. This is caused by the guest / filesystem using NTFS compression, a new feature in Windows 10 which ntfs-3g has trouble with. I had a discussion with Nikolay about this, reproduced in part below: 05:48 < StenaviN> Jimmy_Grover: there are different kind of compression in Windows :-( 05:49 < StenaviN> Read this one: http://jp-andre.pagesperso-orange.fr/compression.html 05:50 < StenaviN> "unsupported reparse point" means exactly that ntfs-3g cannot read it due to missing plugin 05:51 < StenaviN> you can check a reparse point type using the following command: 05:51 < StenaviN> getfattr -h -e hex -n system.ntfs_reparse_data file-or-folder 05:52 < StenaviN> or using "debug sh" under guestfish: 05:52 < StenaviN> debug sh "getfattr -h -e hex -n system.ntfs_reparse_data /sysroot/dedup1.txt" 05:53 < StenaviN> getfattr: Removing leading '/' from absolute path names 05:53 < StenaviN> # file: sysroot/dedup1.txt 05:53 < StenaviN> system.ntfs_reparse_data=0x130000807c00000001027c000000000000000040000000000000000000000000032be97b92848a41af78a4c12b7eb3f340004000400000004ec01f2b9f0ed401020000000000010000f00d00010000000100000008001000c8920d000000000070b59586dbd7132f56a811074be4aaa6aa452e3700745a15b0707350f4c5b87b9aa36aa6 05:54 < StenaviN> "system.ntfs_reparse_data=0x13000080" - is deduplication signature introduced in Win2012 05:54 < StenaviN> and you need corresponding ntfs-3g plugin to read such a files 05:56 < StenaviN> if you see "system.ntfs_reparse_data=0x17000080" - it is a System compression 05:57 < StenaviN> also take a look here: https://bugzilla.redhat.com/show_bug.cgi?id=1557796 05:57 < StenaviN> another one: https://bugs.launchpad.net/ubuntu/+source/ntfs-3g/+bug/1728354 05:58 < StenaviN> Microsoft article, if you wish: https://blogs.windows.com/windowsexperience/2015/03/16/how-windows-10-achieves-its-compact-footprint/ 05:59 < StenaviN> rwmjones: it might be a good idea to include ntfs-3g plugins by-default while building libguestfs appliance if they exist on original system 06:01 < StenaviN> or at least provide them in pre-built libguestfs appliance (http://download.libguestfs.org/binaries/appliance/)
Yes, i can understand, do you mean the NTFS compression is automatically enable or need we manually set, how can i disable it or detect it before inspect os Thanks
I haven't really looked into it in much detail, but read what Nikolay says and see the links above.
Yes, i verified, the VM compress the system file, i use compact /compactos:query, to get the result, after use compact /compactos:never, then the virt-inspector run successfully. so my question is does we will support compressed system file for window in future libguestfs release? because the compress system file is automatically behavior Thanks
It's an ntfs-3g problem and has to be solved there. As I say I've not looked into this very much, all current information is in comment 3.
In Fedora I believe we may be able to solve this by simply adding a dependency on this package: https://src.fedoraproject.org/rpms/ntfs-3g-system-compression
The way to reproduce this bug is: (1) Install Windows 10 as a guest. (I used Windows 2016 Server) (2) In the guest run the following command: compact /compactos:always This will take several minutes to complete, and at the end print something like: 27527 files within 10759 directories were compressed. 4,104,020,580 total bytes of data are stored in 2,753,420,944 bytes. The compression ratio is 1.5 to 1. (3) Shut down the Windows guest. (4) Attempt to convert the Windows guest, eg: $ virt-v2v -i disk windows-10.0-server-compact.img -o null [ 0.0] Opening the source -i disk windows-10.0-server-compact.img [ 0.0] Creating an overlay to protect the source from being modified [ 0.1] Opening the overlay [ 8.6] Inspecting the overlay virt-v2v: error: libguestfs error: inspect_os: lstat: /Windows/System32/cmd.exe: Input/output error
With patched libguestfs, the conversion step succeeds: $ ~/d/libguestfs/run ~/d/virt-v2v/run virt-v2v -i disk windows-10.0-server-compact.img -o null [ 0.0] Opening the source -i disk windows-10.0-server-compact.img [ 0.0] Creating an overlay to protect the source from being modified [ 0.1] Opening the overlay [ 7.3] Inspecting the overlay [ 9.0] Checking for sufficient free disk space in the guest [ 9.0] Estimating space required on target for each disk [ 9.0] Converting Windows Server 2016 Standard to run on KVM virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing. Firstboot scripts may conflict with PnP. virt-v2v: warning: QEMU Guest Agent MSI not found on tools ISO/directory. You may want to install the guest agent manually after conversion. virt-v2v: warning: there are no virtio drivers available for this version of Windows (10.0 x86_64 Server Core). virt-v2v looks for drivers in /usr/share/virtio-win The guest will be configured to use slower emulated devices. virt-v2v: This guest does not have virtio drivers installed. [ 9.9] Mapping filesystem data to avoid copying unused and blank areas [ 11.5] Closing the overlay [ 11.8] Assigning disks to buses [ 11.8] Checking if the guest needs BIOS or UEFI to boot [ 11.8] Initializing the target -o null [ 11.8] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw) (100.00/100%) [ 13.6] Creating output metadata [ 13.6] Finishing off
https://www.redhat.com/archives/libguestfs/2020-January/msg00223.html
Upstream in libguestfs commit e6764a5415b198a9e563473fd9b41077b9e71970. I'm closing this as it is the upstream bug. NB: We do NOT need a downstream libguestfs bug for this, because RHEL uses libguestfs-winsupport and we have added the compression code to that.