Description of problem: $ guestfish Welcome to guestfish, the libguestfs filesystem interactive shell for editing virtual machine filesystems. Type: 'help' for help on commands 'man' to read the manual 'quit' to quit the shell ><fs> add-drive /dev/null ><fs> run libguestfs: error: could not create appliance through libvirt: internal error process exited while connecting to monitor: 2012-10-11 13:56:56.893+0000: 12185: debug : virFileClose:72 : Closed fd 31 2012-10-11 13:56:56.893+0000: 12185: debug : virFileClose:72 : Closed fd 66 2012-10-11 13:56:56.906+0000: 12185: debug : virFileClose:72 : Closed fd 3 qemu-kvm: -drive file=/var/tmp/.guestfs-1000/root.11693,if=none,id=drive-scsi0-0-1-0,format=raw,cache=unsafe: could not open disk image /var/tmp/.guestfs-1000/root.11693: No such file or directory [code=1 domain=10] However the file does exist: $ ll /var/tmp/.guestfs-1000 -Z -rwxr-xr-x. rjones rjones unconfined_u:object_r:user_tmp_t:s0 checksum -rw-r--r--. rjones rjones system_u:object_r:virt_content_t:s0 initrd -rw-r--r--. rjones rjones system_u:object_r:virt_content_t:s0 initrd.11693 -rw-r--r--. rjones rjones system_u:object_r:virt_content_t:s0 kernel -rw-r--r--. rjones rjones system_u:object_r:virt_content_t:s0 kernel.11693 -rw-r--r--. rjones rjones system_u:object_r:svirt_image_t:s0 root -rw-r--r--. rjones rjones system_u:object_r:svirt_image_t:s0 root.11693 Version-Release number of selected component (if applicable): libguestfs-1.19.48-1.fc18.x86_64 libvirt-0.10.2-3.fc19.x86_64 SELinux is enforcing. How reproducible: 100% Steps to Reproduce: 1. See above.
It works when you run the same (guestfish) command as root.
I don't think this is SELinux. Why does it say "No such file or directory" (vs. "Permission denied"). That doesn't sound like an SELinux bug, but something to do with libguestfs not creating the file before qemu starts. Similar to bug 864871?
Also, setting SELinux to permissive does NOT make this bug go away, so it does sound like it's nothing to do with SELinux.
FWIW this was "fixed" by running: killall libvirtd (as non-root) to get rid of the per-user libvirtd. Who knows ... libvirt was not updated recently.
My theory is that the old libvirtd was probably started by some test job, so it had TMPDIR set to a directory that no longer exists. When qemu tries to create snapshots (even though we are setting <qemu:env>) for some reason it's still seeing TMPDIR set to the non-existent directory. Here is a reproducer: killall libvirtd mkdir /tmp/foo TMPDIR=/tmp/foo guestfish -a /dev/null run # successful rm -rf /tmp/foo guestfish -a /dev/null run # fails with error above Reassigning to libvirt, because the per-user libvirtd should not be sensitive to $TMPDIR that happened to be used in one caller.
libvirt-0.10.2-3.fc19.x86_64
OK, I get it. We only set <qemu:env> TMPDIR *if* TMPDIR is actually set in the libguestfs environment. So in fact we were not setting it in this case. In any case, it's still a bug in libvirt that we need to set TMPDIR at all. It should ignore TMPDIR from the caller's environment.
The libguestfs bug is fixed in commit 67e9572286f98b830bef64cfcae5a072bf3a1882. However this is still a libvirt bug. All non-root callers must specify <qemu:env> TMPDIR, because if they don't they risk getting a random TMPDIR from another process, and if that random directory has been deleted then they won't be able to start any transient qemu processes.
I don't think this is a libvirt bug really, although the interaction here is certainly undesirable. If the user has set a custom TMPDIR env for their login session, applications (including libvirtd) should be honouring that variable. The problem arising here is that libguestfs is setting a customized TMPDIR of its own. We can't distinguish between a user specified TMPDIR that we should honour, vs a libguestfs transient TMPDIR that we should ignore. The only solution I see here is to stop using auto-start of libvirtd from libguestfs, and have libguestfs launch libvirtd explicitly (if not already running of course) with TMPDIR cleared (or the user's TMPDIR intact). This can be done using a URI parameter qemu:///session?autostart=0 or by setting LIBVIRT_AUTOSTART=0 env variable. We probably want the env var, so that this works when the user provides a custom URI. It could then try to open the connection and if this fails, then spawn libvirtd & retry. Alternatively could libguestfs avoid setting TMPDIR at all, and simply pass its desired directory as a parameter to mkstemp() and similar calls, and set TMPDIR only for child apps it might spawn ?
I think using autostart=0 is a good idea. I didn't know that option existed. However I still think this is a libvirt bug. The problem is that environment variables from one process leak into another process. It's unexpected that setting TMPDIR in one process should affect all future processes where TMPDIR may not be set at all.
Dan says: (In reply to Daniel Berrange from comment #9) > I don't think this is a libvirt bug really, although the interaction here is > certainly undesirable. If the user has set a custom TMPDIR env for their > login session, applications (including libvirtd) should be honouring that > variable. > Rich says: > However I still think this is a libvirt bug. The problem is > that environment variables from one process leak into another > process. It's unexpected that setting TMPDIR in one process > should affect all future processes where TMPDIR may not be > set at all. It sounds like the only way libvirt could 'fix' this is to completely ignore the environments TMPDIR, which sounds unexpected to me. I don't really see this as a libvirt bug. Rich, is this still relevant?
Actually this seems to just be the same as bug 856619 *** This bug has been marked as a duplicate of bug 856619 ***
Yes this is still a bug and yes I agree it's a particular case of the more general design flaw / bug 856619.