Bug 790958 - multiprovider build error: RuntimeError: link: /tmp/.guestfs-0/kernel /tmp/.guestfs-0/kernel.10139: File exists
Summary: multiprovider build error: RuntimeError: link: /tmp/.guestfs-0/kernel /tmp/.g...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libguestfs
Version: 6.3
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 790721
Blocks: 790528
TreeView+ depends on / blocked
 
Reported: 2012-02-15 19:35 UTC by Richard W.M. Jones
Modified: 2012-06-20 07:00 UTC (History)
17 users (show)

Fixed In Version: libguestfs-1.16.6-1.el6
Doc Type: Bug Fix
Doc Text:
Cause: libguestfs appliance building was not thread-safe when two threads *in the same program* called 'guestfs_launch' at the same time. (Note this didn't affect multiple processes using libguestfs, only multiple threads in the same process) Consequence: Multi-threaded programs using multiple libguestfs handles would rarely give unexpected errors. Fix: libguestfs appliance building and 'guestfs_launch' is now thread safe. Result: libguestfs can be used from multi-threaded programs with more than one libguestfs handle.
Clone Of: 790721
Environment:
Last Closed: 2012-06-20 07:00:35 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2012:0774 normal SHIPPED_LIVE Low: libguestfs security, bug fix, and enhancement update 2012-06-19 19:29:50 UTC

Description Richard W.M. Jones 2012-02-15 19:35:52 UTC
+++ This bug was initially created as a clone of Bug #790721 +++

So, this error seems to originate in the code that creates hard links for launching the appliance VM that libguestfs uses to do its job.  The relevant code is here:

https://github.com/libguestfs/libguestfs/blob/master/src/appliance.c

The comments at the top make it sound as if this activity _should_ be thread/concurrency safe.

However, maybe not.  

Guestfs is using a read lock on the checksum file to avoid walking on itself when creating the links.  Oz did something similar to avoid having two concurrent Oz instances download the same ISO file into the same canonical location.

The issue with Oz, which may come into play here, is that we (Factory) are a single process multi-threaded application and multiple threads of the same process seem to be able to acquire a lock on the same file at the same time without error.

So, perhaps we have two threads doing g.launch() at the same time.  Both libguestfs instances try, and succeeded, in acquiring a read lock on the checksum file (as they are both part of the same process), one creates the kernel.$PID symlink first and the second one fails because the link is already there.

Am going to ask Mr. Jones to weigh in on my largely uninformed speculation above to see if there's any sense in it.

If this is indeed the case we may be able to work around the race in the short term by launching and then stopping a dummy libguestfs instance during Factory startup, before we start the server and "go multithreaded".

--- Additional comment from rjones@redhat.com on 2012-02-15 14:34:17 EST ---

Patch posted:
https://www.redhat.com/archives/libguestfs/2012-February/msg00068.html

Comment 1 Richard W.M. Jones 2012-02-17 15:59:40 UTC
Upstream commit is
https://github.com/libguestfs/libguestfs/commit/afed7e493dcd594620f19b93e9fb73e58553f60a

Comment 2 Hugh Brock 2012-02-20 19:04:30 UTC
Requesting 6.2.z as this is a blocker for Cloudforms.

Comment 3 Denise Dumas 2012-02-23 23:15:22 UTC
Per Hugh, imcleod did a workaround for 6.2 in Factory. Removing zstream request, fix will hit just 6.3.

Comment 6 Yuyu Zhou 2012-03-26 13:39:04 UTC
Reproduced on libguestfs-1.16.5-1.el6.x86_64 and verified on libguestfs-1.16.12-1.el6.x86_64.

Reproduce steps:
excuet reproducer like following:
#cat test.py
#!/usr/bin/python
import os
import threading
import guestfs

def factory():
    g = guestfs.GuestFS()
    g.set_qemu("/bin/true")
    g.add_drive("/dev/null")
    g.launch()
    g.sync()
    g.umount_all()

def main():
    threads = []
    for thread_is in range(0,20):
        threads.append(threading.Thread(target=factory))

    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()

if __name__ == "__main__":
    main()

# python test.py
Get error like following:
Exception in thread Thread-18:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "test.py", line 10, in factory
    g.launch()
  File "/usr/lib/python2.6/site-packages/guestfs.py", line 244, in launch
    return libguestfsmod.launch (self._o)
RuntimeError: link: /var/tmp/.guestfs-0/root /var/tmp/.guestfs-0/root.7816: File exists

Verify Step:
[root@localhost ~]# python test.py 
Only get errors like following, no File exists errors.
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "test.py", line 10, in factory
    g.launch()
  File "/usr/lib/python2.6/site-packages/guestfs.py", line 244, in launch
    return libguestfsmod.launch (self._o)
RuntimeError: child process died unexpectedly

According to test result above, change this bug into VERIFIED.

Comment 7 Richard W.M. Jones 2012-04-26 13:40:44 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause:
libguestfs appliance building was not thread-safe when two threads *in the same program* called 'guestfs_launch' at the same time.  (Note this didn't affect multiple processes using libguestfs, only multiple threads in the same process)

Consequence:
Multi-threaded programs using multiple libguestfs handles would rarely give unexpected errors.

Fix:
libguestfs appliance building and 'guestfs_launch' is now thread safe.

Result:
libguestfs can be used from multi-threaded programs with more than one libguestfs handle.

Comment 9 errata-xmlrpc 2012-06-20 07:00:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0774.html


Note You need to log in before you can comment on or make changes to this bug.