Description of problem: I end up with this exception when running supermin as part of virt-customize or libguestfs Version-Release number of selected component (if applicable): supermin-5.3.2-4.fc37.x86_64 libguestfs-1.49.4-1.fc37.x86_64 guestfs-tools-1.49.4-1.fc37.x86_64 all layered on Fedora Silverblue 37.20220905.n.0 How reproducible: Steps to Reproduce: 1. download qcow2 image 2. a) LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 virt-customize -v -x -d rhel9-tmp --root-password password:networkmanager --ssh-inject USE b) LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 guestfish --rw -d rhel9-tmp Actual results: ... supermin: reading the supermin appliance supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib64/guestfs/supermin.d/zz-packages-xfs type uncompressed packages supermin: mapping package names to installed packages supermin: resolving full list of package dependencies supermin: build: 266 packages, including dependencies supermin: internal error: Not_found exception was thrown libguestfs: error: /usr/bin/supermin exited with error status 1, see debug messages above libguestfs: trace: launch = -1 (error) Expected results: commands either run successfully or print more info about exception to give clue what to do next Additional info:
After reboot, it works. It seems like some issue with layering or still experimental 'rpm-ostree ex apply-live'. Sorry for the noise.
We shouldn't ever be throwing Not_found exceptions. If it happens again please set: export OCAMLRUNPARAM=b which will print the stack trace.
OK, got it again on fully running system, OCAMLPARAM=b seem to have no effect on output... (I'll attach output separately)
Ah yes, part of the problem is that we collect these "should never be thrown" exceptions and print an error message: https://github.com/libguestfs/supermin/blob/e3cb3fb2c3e6c29d57db953099878813538281e6/src/supermin.ml#L293 and the catch + print hides the original trace. I should really fix that ...
Can you try this package please (it should install fine on F37): https://koji.fedoraproject.org/koji/taskinfo?taskID=91744393 It contains the following patches which sure display accurate stack traces, no need to set any environment variables: https://github.com/libguestfs/supermin/commit/5d3d21b985b328317537e3d9a540840c5dade940 https://github.com/libguestfs/supermin/commit/ae7151cd943907a6044d9e1b82df87a2f805507b
Fixed build: https://koji.fedoraproject.org/koji/taskinfo?taskID=91744680
FEDORA-2022-5ad8724136 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-5ad8724136
FEDORA-2022-5ad8724136 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.
Hm, I got the error without backtrace again: Called from Supermin in file "supermin.ml", line 296, characters 4-11 supermin: internal error: Not_found exception was thrown libguestfs: trace: launch = -1 (error) virt-customize: error: libguestfs error: /usr/bin/supermin exited with error status 1, see debug messages above (with caveat that this is on silverblue proper after rpm-ostree install https://koji....rpm followed by rpm-ostree ex apply-live --allow replace. Upgraded supermin in toolbx container didn't encounter an error). I'll keep you updated once I manage to hit it again.
I've checked every place that supermin calls a standard library that could throw Not_found and that isn't surrounded by a try..with to catch the exception, and the only one I could find is: https://github.com/libguestfs/supermin/blob/ae7151cd943907a6044d9e1b82df87a2f805507b/src/format_ext2_initrd.ml#L192 (This is a bug which I'll fix later) Can you see what the /lib/modules/<kernel>/modules.dep file contains on this system? Also it may be useful to run libguestfs-test-tool under "strace -f -o 1024" and capture all of that output (which may be very large) when the bug occurs. It should tell us what supermin is doing when it fails.
Created attachment 1910448 [details] modules.dep (In reply to Richard W.M. Jones from comment #11) > I've checked every place that supermin calls a standard library > that could throw Not_found and that isn't surrounded by a try..with > to catch the exception, and the only one I could find is: > > https://github.com/libguestfs/supermin/blob/ > ae7151cd943907a6044d9e1b82df87a2f805507b/src/format_ext2_initrd.ml#L192 > > (This is a bug which I'll fix later) > > Can you see what the /lib/modules/<kernel>/modules.dep file contains > on this system? attached. There is caveat I have my system in 'apply-live' state again. > > Also it may be useful to run libguestfs-test-tool under "strace -f -o 1024" > and capture all of that output (which may be very large) when the bug > occurs. It should tell us what supermin is doing when it fails. I'll try that if/when I encounter it next time.
modules.dep looks normal, as in there are no lines without ":" which might trigger that code to throw a Not_found exception. I'm not sure what this "apply-live" stuff means. Do you have a reliable way to reproduce the bug or does it happen at random?
(In reply to Richard W.M. Jones from comment #13) > Do you have > a reliable way to reproduce the bug or does it happen at random? It seems more random. It looks there are more odds or less odds of happening based on circumstances but the only reliable pattern is that once this starts happening, it won't cease anytime soon. However even within the same boot, the next working day, it may work again. > I'm not sure what this "apply-live" stuff means. My system is Fedora Silverblue which is image-based. There is "base image" on top which are "overlayed packages" installed including guestfs-tools & supermin. The "correct" way to get the system to new state is reboot. However, as of rpm-ostree in Fedora 37, there is option to "apply-live" the changes, which I presume is some sort of switching root or mounting overlay FS on-the-fly. End result should be the same like the reboot at least as read-only parts of VFS are concerned but the changes to /etc or /var (and /usr/local, /opt) may end up different. As far as I understand.
I've installed Fedora Silverblue 36 in a VM. Any tips on how to install libguestfs? (It says "dnf: command not found")
I didn't hit this since update to F38 versions of components - closing.