Description of problem: I see stuff like this in my testbed logs: 2010-06-21 07:56:13 starting: ./scripts/xen/bootup-kvm rhel4u4i error: Failed to start domain rhel4u4i error: cannot create pipe: Too many open files That bootup-kvm script simply ssh's to the host named "xen" and does a virsh start command (along with a bunch of error checking). If I go to the host and look at /proc/<libvirtd pid>/fd I see 1018 open file descriptors, most pointing to log files for vms that are no longer running. Sometimes VMs crash under test, or the network goes screwy, and the testbed has to resort to "virsh destroy" to make the VM go away. Since the host has been up for about 12 days, I suspect it is the virsh destroy that leaks the descriptor (but haven't tried to prove it). Version-Release number of selected component (if applicable): libvirt-client-0.7.7-4.fc13.x86_64 libvirt-0.7.7-4.fc13.x86_64 libvirt-python-0.7.7-4.fc13.x86_64 How reproducible: Seems likely to be reproducible, but I haven't tried restarting everything and waiting another 12 days yet :-). Steps to Reproduce: 1. constant starting and shutting down of many different KVM machines 2. eventually libvirtd runs out of file descriptors. 3. Actual results: Out of file descriptors Expected results: Clean up after machines after shutdown or destruction Additional info: As long as everything is hosed now, I'm going to go ahead and get the latest updates and reboot my host machines, so perhaps a newer libvirt will fix this.
Created attachment 425608 [details] the ls -l of libvirtd's /proc/pid/fd directory
Well, I updated and still see these versions on the system, so I guess I already had the latest: libvirt-client-0.7.7-4.fc13.x86_64 libvirt-0.7.7-4.fc13.x86_64 libvirt-python-0.7.7-4.fc13.x86_64 I've just rebooted to get new kernel active, and before ever starting a VM, I see these file descriptors: lr-x------ 1 root root 64 Jun 21 08:34 0 -> /dev/null l-wx------ 1 root root 64 Jun 21 08:34 1 -> /dev/null lrwx------ 1 root root 64 Jun 21 08:34 10 -> socket:[11232] lrwx------ 1 root root 64 Jun 21 08:34 11 -> socket:[11228] lrwx------ 1 root root 64 Jun 21 08:34 12 -> socket:[11246] l-wx------ 1 root root 64 Jun 21 08:34 13 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 14 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 15 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 16 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 17 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 18 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 19 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 2 -> /dev/null l-wx------ 1 root root 64 Jun 21 08:34 20 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 21 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 22 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 23 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 24 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 25 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 26 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 27 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 28 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 29 -> /proc/mtrr lr-x------ 1 root root 64 Jun 21 08:34 3 -> pipe:[11222] l-wx------ 1 root root 64 Jun 21 08:34 30 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 31 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 32 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 33 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 34 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 35 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 36 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 37 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 38 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 39 -> /proc/mtrr lrwx------ 1 root root 64 Jun 21 08:34 4 -> socket:[11236] l-wx------ 1 root root 64 Jun 21 08:34 40 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 41 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 42 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 43 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 44 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 45 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 46 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 47 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 48 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 49 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 5 -> pipe:[11222] l-wx------ 1 root root 64 Jun 21 08:34 50 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 51 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 52 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 53 -> /proc/mtrr l-wx------ 1 root root 64 Jun 21 08:34 54 -> /proc/mtrr lr-x------ 1 root root 64 Jun 21 08:34 55 -> inotify lr-x------ 1 root root 64 Jun 21 08:34 6 -> pipe:[11223] l-wx------ 1 root root 64 Jun 21 08:34 7 -> pipe:[11223] lrwx------ 1 root root 64 Jun 21 08:34 8 -> socket:[11224] lrwx------ 1 root root 64 Jun 21 08:34 9 -> socket:[11226] I don't have any idea what /proc/mtrr is, but does it really need 42 copies?
I think /proc/mtrr is used by netcf, which is used by libvirt. Is netcf fully updated? Laine, see the previous comment, user is reporting a ton of open /proc/mtrr FDs.
I just did a yum update, so everything should be up to date. I'm showing netcf-libs-0.1.6-1.fc13.x86_64 But the /proc/mtrr FDs don't bother me as much as the nearly 1000 qemu log files that are shown left open in the attachment showing the original ls -l of libvirtd's file descriptors.
I just did an experiment: virsh start <machine> wait a few seconds virsh destroy <machine> Every time you repeat that sequence, a new file descriptor appears in the libvirtd process pointing at the qemu log file for <machine>. If you do the exact same machine 10 times in a row, you'll get 10 new file descriptors for log files for the same machine.
Sweet, thanks for narrowing it down Tom. Sounds like this commit might fix the issue: http://libvirt.org/git/?p=libvirt.git;a=commit;h=caad0a878337b52c4453444bb227b74cce6e6d5f
Just to let you know, the newest update: libvirt-0.8.2-1.fc13.x86_64 does seem to fix the primary problem - no more log file descriptor leaks (there are still lots of /proc/mtrr instances, but they don't seem to be leaking, there are just lots of them).
Cool, since 0.8.2 is in stable now, we can close this. I think the /proc/mtrr instances have something to do with netcf and the apps they launch.