Red Hat Bugzilla – Bug 790958
multiprovider build error: RuntimeError: link: /tmp/.guestfs-0/kernel /tmp/.guestfs-0/kernel.10139: File exists
Last modified: 2012-06-20 03:00:35 EDT
+++ This bug was initially created as a clone of Bug #790721 +++ So, this error seems to originate in the code that creates hard links for launching the appliance VM that libguestfs uses to do its job. The relevant code is here: https://github.com/libguestfs/libguestfs/blob/master/src/appliance.c The comments at the top make it sound as if this activity _should_ be thread/concurrency safe. However, maybe not. Guestfs is using a read lock on the checksum file to avoid walking on itself when creating the links. Oz did something similar to avoid having two concurrent Oz instances download the same ISO file into the same canonical location. The issue with Oz, which may come into play here, is that we (Factory) are a single process multi-threaded application and multiple threads of the same process seem to be able to acquire a lock on the same file at the same time without error. So, perhaps we have two threads doing g.launch() at the same time. Both libguestfs instances try, and succeeded, in acquiring a read lock on the checksum file (as they are both part of the same process), one creates the kernel.$PID symlink first and the second one fails because the link is already there. Am going to ask Mr. Jones to weigh in on my largely uninformed speculation above to see if there's any sense in it. If this is indeed the case we may be able to work around the race in the short term by launching and then stopping a dummy libguestfs instance during Factory startup, before we start the server and "go multithreaded". --- Additional comment from rjones@redhat.com on 2012-02-15 14:34:17 EST --- Patch posted: https://www.redhat.com/archives/libguestfs/2012-February/msg00068.html
Upstream commit is https://github.com/libguestfs/libguestfs/commit/afed7e493dcd594620f19b93e9fb73e58553f60a
Requesting 6.2.z as this is a blocker for Cloudforms.
Per Hugh, imcleod did a workaround for 6.2 in Factory. Removing zstream request, fix will hit just 6.3.
Reproduced on libguestfs-1.16.5-1.el6.x86_64 and verified on libguestfs-1.16.12-1.el6.x86_64. Reproduce steps: excuet reproducer like following: #cat test.py #!/usr/bin/python import os import threading import guestfs def factory(): g = guestfs.GuestFS() g.set_qemu("/bin/true") g.add_drive("/dev/null") g.launch() g.sync() g.umount_all() def main(): threads = [] for thread_is in range(0,20): threads.append(threading.Thread(target=factory)) for thread in threads: thread.start() for thread in threads: thread.join() if __name__ == "__main__": main() # python test.py Get error like following: Exception in thread Thread-18: Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "test.py", line 10, in factory g.launch() File "/usr/lib/python2.6/site-packages/guestfs.py", line 244, in launch return libguestfsmod.launch (self._o) RuntimeError: link: /var/tmp/.guestfs-0/root /var/tmp/.guestfs-0/root.7816: File exists Verify Step: [root@localhost ~]# python test.py Only get errors like following, no File exists errors. Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "test.py", line 10, in factory g.launch() File "/usr/lib/python2.6/site-packages/guestfs.py", line 244, in launch return libguestfsmod.launch (self._o) RuntimeError: child process died unexpectedly According to test result above, change this bug into VERIFIED.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Cause: libguestfs appliance building was not thread-safe when two threads *in the same program* called 'guestfs_launch' at the same time. (Note this didn't affect multiple processes using libguestfs, only multiple threads in the same process) Consequence: Multi-threaded programs using multiple libguestfs handles would rarely give unexpected errors. Fix: libguestfs appliance building and 'guestfs_launch' is now thread safe. Result: libguestfs can be used from multi-threaded programs with more than one libguestfs handle.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2012-0774.html