Hide Forgot
Description of problem: After some operation (have not been able to narrow down to what) /tmp/.guestfs0/* becomes unreadable with "Permission denied". After this happens, imagefactory refuses to start. If /tmp/.guestfs0 is moved away, factory will start again. Will update with a consistent reproducer once it is found. Version-Release number of selected component (if applicable): libvirt-0.9.4-23.el6_2.6.x86_64 imagefactory-1.0.0rc8-1.el6.noarch libguestfs-1.7.17-26.el6.x86_64 How reproducible: Random (so far)
Possible reproducer: restart libvirtd while imagefactory is running Try to restart imagefactory. It doesn't restart. running with 'LIBGUESTFS_DEBUG=1 imagefactory --rest --debug --foreground' gives this: 2012-03-08 11:17:34,185 INFO root thread(MainThread) Message: Launching in foreground... new guestfs handle 0x1541050 [00000ms] febootstrap-supermin-helper --verbose -f checksum '/usr/lib64/guestfs/supermin.d' x86_64 supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = (null), initrd = (null), appliance = (null) supermin helper [00000ms] inputs[0] = /usr/lib64/guestfs/supermin.d checking modpath /lib/modules/2.6.32-220.el6.x86_64 is a directory picked vmlinuz-2.6.32-220.el6.x86_64 because modpath /lib/modules/2.6.32-220.el6.x86_64 exists supermin helper [00000ms] finished creating kernel supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/base.img supermin helper [00001ms] visiting /usr/lib64/guestfs/supermin.d/daemon.img supermin helper [00001ms] visiting /usr/lib64/guestfs/supermin.d/hostfiles supermin helper [00086ms] finished creating appliance [00092ms] begin testing qemu features [00124ms] finished testing qemu features [00125ms] /usr/libexec/qemu-kvm \ -drive file=/dev/zero,if=virtio \ -nodefconfig \ -enable-kvm \ -nodefaults \ -nographic \ -m 500 \ -no-reboot \ -device virtio-serial \ -serial stdio \ -chardev socket,path=/tmp/libguestfs4WZxBf/guestfsd.sock,id=channel0 \ -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \ -kernel /tmp/.guestfs-0/kernel.4047 \ -initrd /tmp/.guestfs-0/initrd.4047 \ -append 'panic=1 console=ttyS0 udevtimeout=300 noapic acpi=off printk.time=1 cgroup_disable=memory selinux=0 guestfs_verbose=1 TERM=xterm ' \ -drive file=/tmp/.guestfs-0/root.4047,snapshot=on,if=virtio,cache=unsafe accept_from_daemon: 0x1541050 g->state = 1 recv_from_daemon: 0x1541050 g->state = 1, size_rtn = 0x7fff0f3989fc, buf_rtn = 0x7fff0f3989c0 qemu: could not load kernel '/tmp/.guestfs-0/kernel.4047': Permission denied child_cleanup: 0x1541050: child process died Traceback (most recent call last): File "/usr/bin/imagefactory", line 221, in <module> sys.exit(Application().main()) File "/usr/bin/imagefactory", line 152, in main self._init_guestfs() File "/usr/bin/imagefactory", line 217, in _init_guestfs g.launch() File "/usr/lib/python2.6/site-packages/guestfs.py", line 152, in launch return libguestfsmod.launch (self._o) RuntimeError: unexpected end of file when reading from daemon. See earlier debug messages. Or you can run 'libguestfs-test-tool' and post the complete output into a bug report or message to the libguestfs mailing list. closing guestfs handle 0x1541050 (state 0) [root@qeblade33 tmp]# ls -ldZ .guestfs-0/ drwx------. root root unconfined_u:object_r:tmp_t:s0 .guestfs-0/ [root@qeblade33 tmp]# ls -laZ .guestfs-0/ drwx------. root root unconfined_u:object_r:tmp_t:s0 . drwxrwxrwt. root root system_u:object_r:tmp_t:s0 .. -rwx------. root root unconfined_u:object_r:tmp_t:s0 checksum -rw-r--r--. root root unconfined_u:object_r:tmp_t:s0 initrd -rw-r--r--. root root unconfined_u:object_r:tmp_t:s0 initrd.4047 lrwxrwxrwx. root root unconfined_u:object_r:tmp_t:s0 kernel -> /boot/vmlinuz-2.6.32-220.el6.x86_64 lrwxrwxrwx. root root unconfined_u:object_r:tmp_t:s0 kernel.4047 -> /boot/vmlinuz-2.6.32-220.el6.x86_64 -rw-r--r--. root root unconfined_u:object_r:tmp_t:s0 root -rw-r--r--. root root unconfined_u:object_r:tmp_t:s0 root.4047
This miniroot linking process has suddenly become problematic. Bizarre. This directory and the links and files inside of it are created incidentally as part our use of libguestfs. In theory, the directory should stick around as long as a factory process is running as root, and the links should stick around as long as the PID they are associated with is alive. I need to punt this one over to Richard Jones for some feedback. My best wild guess is that this is SELinux related. Can you attach the audit log from this system?
Created attachment 568693 [details] Audit log
Notice the kernel is unreadable. Now the kernel is the first thing that qemu tries to read, so it could simply be that. Or it could be that the kernel is actually unreadable. Could you check its permissions (in /boot)?
Maybe it is an SELinux thing after all. Here are the AVCs: type=AVC msg=audit(1331223396.667:51131): avc: denied { read } for pid=3990 comm="qemu-kvm" name="kernel.3980" dev=sdb1 ino=22216730 scontext=unconfined_u:system_r:qemu_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:tmp_t:s0 tclass=lnk_file type=AVC msg=audit(1331223434.277:51134): avc: denied { read } for pid=4039 comm="qemu-kvm" name="kernel.4029" dev=sdb1 ino=22216730 scontext=unconfined_u:system_r:qemu_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:tmp_t:s0 tclass=lnk_file type=AVC msg=audit(1331223454.362:51135): avc: denied { read } for pid=4060 comm="qemu-kvm" name="kernel.4047" dev=sdb1 ino=22216730 scontext=unconfined_u:unconfined_r:qemu_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:tmp_t:s0 tclass=lnk_file
The only selinux / libguestfs bugs I'm aware of are these two, and neither looks like the problem you've got here: https://bugzilla.redhat.com/show_bug.cgi?id=729365 https://bugzilla.redhat.com/show_bug.cgi?id=670258 There is one suggestion here you could try: https://bugzilla.redhat.com/show_bug.cgi?id=670258#c3 # setsebool -P allow_unconfined_qemu_transition 0
moving to 1.1 to recheck then
Moved to on_qa for retest. I don't think we should consider for z.
As we have not seen this again and cannot reproduce I am going to close this.