Bug 718714

Since rebooting my builder a few days ago, I haven't seen a single instance of this problem, so I suspect it was a kernel issue.

However, one change I'd suggest is that "Namespace unshare failed" become a fatal error as failure there inevitably results in a failed build.

Yeah, sounds like a good idea. I'll raise an exception when the unshare fails.

(In reply to comment #1)
> Since rebooting my builder a few days ago, I haven't seen a single instance of
> this problem, so I suspect it was a kernel issue.

Hmm, spoke too soon. Just had another one.

(In reply to comment #2)
> Yeah, sounds like a good idea. I'll raise an exception when the unshare fails.

Please make this exception cause mock to exit with a specific exit code that's different from other error cases - I know this one is retry-able and can get my builder script to retry the build if it detects the right exit code.

mock-1.1.12-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/mock-1.1.12-1.el6

mock-1.1.12-1.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/mock-1.1.12-1.fc14

mock-1.0.19-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/mock-1.0.19-1.el5

mock-1.1.12-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/mock-1.1.12-1.fc15

With 1.1.12, it's not exiting when the namespace unshare failure happens but instead when a subsequent failure (which is probably a result of the unshare failure) happens. The exit status is 1, which is indistinguishable from other failure modes.

$ mock -r city-fan-fedora-10-i386 /home/paul/cfo-build/perl-version/trunk/perl-version-0.92-1.fc15.src.rpm
INFO: mock.py version 1.1.12 starting...
State Changed: init plugins
INFO: selinux enabled
State Changed: start
INFO: Namespace unshare failed.
ERROR: Cannot find/open srpm: /home/paul/cfo-build/perl-version/trunk/perl-version-0.92-1.fc15.src.rpm. Error: Unable to open file
WARNING: unable to delete selinux filesystems (/tmp/mock-selinux-plugin.6Q7Myj): [Errno 2] No such file or directory: '/tmp/mock-selinux-plugin.6Q7Myj'

ugh, I did all the hard work (creating an Exception, etc) and failed to actually exit in the except. Sigh...

Lemme whip you up a patch.

Please apply this set of changes to mock.py to see if it does what you want.

diff --git a/py/mock.py b/py/mock.py
index f2c9e55..268b727 100755
--- a/py/mock.py
+++ b/py/mock.py
@@ -691,8 +691,11 @@ def main(ret):
     # New namespace starting from here
     try:
         mock.util.unshare(mock.util.CLONE_NEWNS)
+    except UnshareFailed, e:
+        log.error("Namespace unshare failed.")
+        sys.exit(e.resultcode)
     except:
-        log.info("Namespace unshare failed.")
+        raise
 
     # set personality (ie. setarch)
     if config_opts['internal_setarch']:

That didn't work:

$ mock -r city-fan-fedora-rawhide-i686 --disable-plugin=tmpfs /home/paul/cfo-build/perl-BerkeleyDB/trunk/perl-BerkeleyDB-0.48-2.fc15.src.rpm
INFO: mock.py version 1.1.12 starting...
State Changed: init plugins
INFO: selinux enabled
State Changed: start
ERROR: global name 'UnshareFailed' is not defined
Traceback (most recent call last):
  File "/usr/sbin/mock", line 491, in <module>
  File "/usr/sbin/mock", line 697, in main
NameError: global name 'UnshareFailed' is not defined
WARNING: unable to delete selinux filesystems (/tmp/mock-selinux-plugin.I2t01s): [Errno 2] No such file or directory: '/tmp/mock-selinux-plugin.I2t01s'

So on line 697 of /usr/sbin/mock I tried changing 'UnshareFailed' to 'mock.exception.UnshareFailed' and then I got this:

$ mock -r city-fan-redhat-9-i386 /home/paul/cfo-build/perl-NetAddr-IP/trunk/perl-NetAddr-IP-4.044-2.fc15.src.rpm
INFO: mock.py version 1.1.12 starting...
State Changed: init plugins
INFO: selinux enabled
State Changed: start
ERROR: global name 'UnshareFailed' is not defined
Traceback (most recent call last):
  File "/usr/sbin/mock", line 491, in <module>
  File "/usr/sbin/mock", line 696, in main
  File "<peak.util.decorators.rewrap wrapping mock.util.unshare at 0x01CA3AA0>", line 3, in unshare
    def unshare(flags): return __decorated(flags)
  File "/usr/lib/python2.7/site-packages/mock/trace_decorator.py", line 70, in trace
  File "/usr/lib/python2.7/site-packages/mock/util.py", line 186, in unshare
NameError: global name 'UnshareFailed' is not defined
WARNING: unable to delete selinux filesystems (/tmp/mock-selinux-plugin.sTh2LE): [Errno 2] No such file or directory: '/tmp/mock-selinux-plugin.sTh2LE'

So it looks like it needs slightly more tweaking yet.

Yeah, you could say it needs more tweaking :)

Try this one:

diff --git a/py/mock.py b/py/mock.py
index f2c9e55..f8a4d0f 100755
--- a/py/mock.py
+++ b/py/mock.py
@@ -691,8 +691,11 @@ def main(ret):
     # New namespace starting from here
     try:
         mock.util.unshare(mock.util.CLONE_NEWNS)
+    except mock.exception.UnshareFailed, e:
+        log.error("Namespace unshare failed.")
+        sys.exit(e.resultcode)
     except:
-        log.info("Namespace unshare failed.")
+        raise
 
     # set personality (ie. setarch)
     if config_opts['internal_setarch']:
diff --git a/py/mock/util.py b/py/mock/util.py
index 5fb0f18..33c5f44 100644
--- a/py/mock/util.py
+++ b/py/mock/util.py
@@ -184,7 +184,7 @@ def unshare(flags):
     try:
         res = _libc.unshare(flags)
         if res:
-            raise UnshareFailed(os.strerror(_errno.value))
+            raise mock.exception.UnshareFailed(os.strerror(_errno.value))
     except AttributeError, e:
         pass

Yes, that's better:

$ mock -r city-fan-fedora-1-i386 /home/paul/cfo-build/perl-NetAddr-IP/trunk/perl-NetAddr-IP-4.044-2.fc15.src.rpm
INFO: mock.py version 1.1.12 starting...
State Changed: init plugins
INFO: selinux enabled
State Changed: start
ERROR: Namespace unshare failed.
WARNING: unable to delete selinux filesystems (/tmp/mock-selinux-plugin.VL2LSu): [Errno 2] No such file or directory: '/tmp/mock-selinux-plugin.VL2LSu'

This time it exited with the right exit code (undocumented by the way) and my build script was able to retry it straight away, with success. Now if only we could get rid of that warning from the mock-selinux-plugin...

See if this does the trick:

diff --git a/py/mock/plugins/selinux.py b/py/mock/plugins/selinux.py
index 0f5bad1..814919f 100644
--- a/py/mock/plugins/selinux.py
+++ b/py/mock/plugins/selinux.py
@@ -67,11 +67,12 @@ class SELinux(object):
 
     decorate(traceLog())
     def _selinuxAtExit(self):
-        try:
-            os.unlink(self.filesystems)
-        except OSError, e:
-            getLog().warning("unable to delete selinux filesystems (%s): %s" % 
-            pass
+        if os.path.exists(self.filesystems):
+            try:
+                os.unlink(self.filesystems)
+            except OSError, e:
+                getLog().warning("unable to delete selinux filesystems (%s): %s
+                pass
 
     decorate(traceLog())
     def _selinuxPreYumHook(self):

That diff looks a bit mangled so I went with this:

--- /usr/lib/python2.7/site-packages/mock/plugins/selinux.py.orig	2011-07-26 20:30:18.000000000 +0100
+++ /usr/lib/python2.7/site-packages/mock/plugins/selinux.py	2011-07-27 19:08:54.403560565 +0100
@@ -50,7 +50,7 @@
             rootObj.addHook("preyum", self._selinuxPreYumHook)
             rootObj.addHook("postyum", self._selinuxPostYumHook)
         else:
-            getLog().warn("selinux: 'yum' does not support '--setopt' option")
+            getLog().warning("selinux: 'yum' does not support '--setopt' option")
 
     decorate(traceLog())
     def _selinuxCreateFauxFilesystems(self):
@@ -68,11 +68,12 @@
 
     decorate(traceLog())
     def _selinuxAtExit(self):
-        try:
-            os.unlink(self.filesystems)
-        except OSError, e:
-            getLog().warning("unable to delete selinux filesystems (%s): %s" % (self.filesystems, e))
-            pass
+        if os.path.exists(self.filesystems):
+            try:
+                os.unlink(self.filesystems)
+            except OSError, e:
+                getLog().warning("unable to delete selinux filesystems (%s): %s" % (self.filesystems, e))
+                pass
 
     decorate(traceLog())
     def _selinuxPreYumHook(self):

That had the desired effect:

$ mock -r city-fan-fedora-10-x86_64 /home/paul/cfo-build/perl-JSON-XS/trunk/perl-JSON-XS-2.31-1.fc15.src.rpm
INFO: mock.py version 1.1.12 starting...
State Changed: init plugins
INFO: selinux enabled
State Changed: start
ERROR: Namespace unshare failed.

Great - right exit code too! I think it just needs documenting now in mock/exception.py.

Created attachment 515583 [details]
fix to allow retries for namespace 'unshare(2)' failures

proposed patch to allow retries when unshare(2) fails

Package mock-1.1.12-1.el6:
* should fix your issue,
* was pushed to the Fedora EPEL 6 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing mock-1.1.12-1.el6'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/mock-1.1.12-1.el6
then log in and leave karma (feedback).

(In reply to comment #17)
> Created attachment 515583 [details]
> fix to allow retries for namespace 'unshare(2)' failures
> 
> proposed patch to allow retries when unshare(2) fails

Patch looks good.

Just one query: should getLog().warn in the SELinux plugin be getLog().warning?

(In reply to comment #19)
> (In reply to comment #17)
> > Created attachment 515583 [details]
> > fix to allow retries for namespace 'unshare(2)' failures
> > 
> > proposed patch to allow retries when unshare(2) fails
> 
> Patch looks good.
> 
> Just one query: should getLog().warn in the SELinux plugin be getLog().warning?

Well, that getLog class in trace_decorator.py is kinda magic in that it does late bindings and other weird mojo. Both .warn and .warning will work and will generate a warning entry.

I'll change it to warning (as well as one in the root_cache plugin) just for consistency's sake.

(In reply to comment #20)
> > Just one query: should getLog().warn in the SELinux plugin be getLog().warning?
> 
> Well, that getLog class in trace_decorator.py is kinda magic in that it does
> late bindings and other weird mojo. Both .warn and .warning will work and will
> generate a warning entry.
> 
> I'll change it to warning (as well as one in the root_cache plugin) just for
> consistency's sake.

OK, thanks. I have seen some tracebacks with getlog in them (didn't prevent the build working so I wasn't too bothered) and wondered if that was related - apparently not then. Let's see if I can reproduce one...

mock-1.1.12-1.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

mock-1.1.12-1.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

Reopening pending release of version including the patches discussed earlier.

I haven't seen any getlog tracebacks over the last week or so, so I'll open a fresh bug if I spot one.

mock-1.1.12-1.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.

mock-1.0.19-1.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.