Bug 609201

Summary: building perl-TermReadKey hangs in mock
Product: [Fedora] Fedora Reporter: Ralf Corsepius <rc040203>
Component: mockAssignee: Clark Williams <williams>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 13CC: dcantrell, herrold, kasal, leonard-rh-bugzilla, lfarkas, mbooth, mebrown, mishu, mmaslano, pampelmuse, paul, ppisar, psabata, slords, williams
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: mock-1.1.10-1.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-03-03 03:18:00 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
perl-TermReadKey 2.30-10.fc14 SRPM none

Description Ralf Corsepius 2010-06-29 16:13:46 UTC
Description of problem:

Building perl-TermReadKey hangs inside of its testsuile when building it in a local mock. It does not hang in Fedora's koji nor when rebuilding the package outside of mock

AFAICT, it hangs inside of its testsuite when trying to access /dev/tty.

In koji it doesn't seem to find /dev/tty:
...
+ make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-Iblib/lib" "-Iblib/arch" -w test.pl
1 .. 8
ok 1
ok 2  # skip /dev/tty is absent
ok 3  # skip /dev/tty is absent
ok 4  # skip /dev/tty is absent
ok 5  # skip /dev/tty is absent
ok 6  # skip /dev/tty is absent
ok 7  # skip /dev/tty is absent
ok 8  # skip /dev/tty is absent
...

In a local mock it finds /dev/tty but seems to have problems in accessing it:
+ make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-Iblib/lib" "-Iblib/arch" -w test.pl
1 .. 8
ok 1
ok 2
ok 3
ok 4
ok 5
ok 6
ok 7  
[hangs]


Version-Release number of selected component (if applicable):
perl-TermReadKey-2.30-10.fc14


How reproducible:
Always

Steps to Reproduce:
1. mock -r fedora-rawhide-x86_64 perl-TermReadKey-2.30-10.fc14.src.rpm
  
Actual results:
Koji builts and built outside of mock succeed.
Builts in a local mock hang.

Expected results:
Deterministic builts.

Additional info:
* I am using fedora 13's mock (mock-1.1.1-1.fc13.noarch)

* One way to circumvent this issue is to add a --with <something> rpm option only run this package's testsuite. This would not hurt much, because building this package in koji already causes building this package to skip 7 out of its 8 tests.

* A better way would be to improve the testsuite or mock (Unfortunately I don't have an idea how to implement this).

Comment 1 Fedora Admin XMLRPC Client 2010-07-02 14:29:54 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 2 Paul Howarth 2010-07-02 15:10:34 UTC
This problem manifested in F-12 too, which I mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=510183#c21 but that seems to have fallen through the cracks.

Comment 3 Petr Šabata 2010-07-07 07:36:31 UTC
This really seems to be a mock issue. Changing the component to mock.

Comment 4 Clark Williams 2010-07-16 13:34:25 UTC
Would you attach the perl-TermReadKey SRPM to the BZ or give me a link to where I can get it?

Comment 5 Ralf Corsepius 2010-07-16 14:32:12 UTC
(In reply to comment #4)
> Would you attach the perl-TermReadKey SRPM to the BZ or give me a link to where
> I can get it?    

You can get it from rawhide rsp. fedora's CVS ;)

Comment 6 Petr Šabata 2010-07-16 14:35:49 UTC
Created attachment 432400 [details]
perl-TermReadKey 2.30-10.fc14 SRPM

Comment 7 Clark Williams 2010-07-16 14:39:52 UTC
thanks Petr

Comment 8 Clark Williams 2010-07-16 18:13:10 UTC
I've duplicated the hang and I'm now wondering what /dev/tty should imply while inside a chroot (since /dev/tty is supposed to be the "current" terminal device). 

Looking at /proc/tty/drivers while the build is hung shows:

# cat /proc/tty/drivers 
/dev/tty             /dev/tty        5       0 system:/dev/tty
/dev/console         /dev/console    5       1 system:console
/dev/ptmx            /dev/ptmx       5       2 system
/dev/vc/0            /dev/vc/0       4       0 system:vtmaster
rfcomm               /dev/rfcomm   216 0-255 serial
serial               /dev/ttyS       4 64-95 serial
pty_slave            /dev/pts      136 0-1048575 pty:slave
pty_master           /dev/ptm      128 0-1048575 pty:master
unknown              /dev/tty        4 1-63 console

I believe that the last line is our /dev/tty from the chroot. Note the fact that it's shown as a console device rather than a system:/dev/tty. I'm not at all sure how we could coerce the special device in the chroot to be wired to a functional device, nor do I know what that device should be. I'm wondering if we should ditch /dev/tty from the chroot altogether (since I'm not sure it has ever worked). 

I need to consult with some folks that are more knowledgeable about the kernel terminal subsystem than I am...

Comment 9 Clark Williams 2010-07-16 19:04:12 UTC
Ok, another theory shot. Looking at proc.txt in the kernel Documentation tree, that "unknown" line in /proc/tty/drivers is normal, so it might not be our tty entry. 

Still digging.

Comment 10 Levente Farkas 2010-11-09 09:29:52 UTC
is there any progress with it?
it seems rhel-6's bash still has the same problems...

Comment 11 Levente Farkas 2010-12-03 13:33:13 UTC
is there any list for mock or where is the mock development happened? mailing list or something like that? or should i ask here?
it seems most of the mock problem in the current release comes from the %check section of the rpmbuild. there are 4,5 different bz about it.
is there any mock option to switch off the rpmbuild check part from the build? it can solve a lots of issue until someone can fix mock and different packages.
thanks.

Comment 12 Leonard den Ottolander 2010-12-14 16:44:31 UTC
Clark, is the 2009-07-08 commit regarding /dev/tty and the use of the 'newinstance' option somehow related to this issue?

Comment 13 Leonard den Ottolander 2010-12-27 22:53:01 UTC
Not creating the node /dev/tty but soft linking it to ptmx seems to make this build succeed (haven't tried the bash build from #613392). Used the RHEL-6 srpm of the same name but disabled the patch that skips the last test (all 8 tests succeed).

--- backend.py.000      2010-11-24 10:49:04.000000000 +0100
+++ backend.py  2010-12-27 23:42:26.000000000 +0100
@@ -363,7 +363,6 @@ class Root(object):
             (stat.S_IFCHR | 0666, os.makedev(1, 5), "dev/zero"),
             (stat.S_IFCHR | 0666, os.makedev(1, 8), "dev/random"),
             (stat.S_IFCHR | 0444, os.makedev(1, 9), "dev/urandom"),
-            (stat.S_IFCHR | 0666, os.makedev(5, 0), "dev/tty"),
             (stat.S_IFCHR | 0600, os.makedev(5, 1), "dev/console"),
         ]
         kver = os.uname()[2]
@@ -392,6 +391,7 @@ class Root(object):
         # symlink it for FC and el6 hosts
         if mock.util.cmpKernelEVR(kver, '2.6.18') > 0:
             os.symlink("/dev/pts/ptmx", self.makeChrootPath("dev/ptmx"))
+            os.symlink("./ptmx", self.makeChrootPath("dev/tty"))
 
         os.umask(prevMask)
 

(Not sure what to do with /dev/tty in older systems. Koji only creates /dev/urandom and none of the others mock does.)

Comment 14 Clark Williams 2011-01-03 16:16:22 UTC
Leonard, thanks for looking into this. I suspect wq'll have to do some magic based on the host OS as to whether to create /dev/tty or link it, since EPEL5 and F{13,14,15} handle tty's differently. 

I'm currently debugging a somewhat random failure in my test suite ('make check') so I'll queue this up and work on it after that.

Comment 15 Clark Williams 2011-01-06 17:53:04 UTC
Ok, I started working on this and noticed that you symlinked "./ptmx" to dev/tty. Being a non-trusting soul, I changed this to /dev/ptmx -> /dev/tty (i.e. uses absolute paths inside the chroot). This doesn't break anything in my regression suite ('make check') and it successfully rebuilds perl-TermReadKey as well. 

Here's the resulting patch:

diff --git a/py/mock/backend.py b/py/mock/backend.py
index 2d71e0a..51b3639 100644
--- a/py/mock/backend.py
+++ b/py/mock/backend.py
@@ -388,10 +388,10 @@ class Root(object):
             (stat.S_IFCHR | 0666, os.makedev(1, 5), "dev/zero"),
             (stat.S_IFCHR | 0666, os.makedev(1, 8), "dev/random"),
             (stat.S_IFCHR | 0444, os.makedev(1, 9), "dev/urandom"),
-            (stat.S_IFCHR | 0666, os.makedev(5, 0), "dev/tty"),
             (stat.S_IFCHR | 0600, os.makedev(5, 1), "dev/console"),
         ]
         kver = os.uname()[2]
+        getLog().debug("kver == %s" % kver)
         # make the device node for el4 and el5
         if mock.util.cmpKernelEVR(kver, '2.6.18') <= 0:
             devFiles.append((stat.S_IFCHR | 0666, os.makedev(5, 2), "dev/ptmx")
@@ -410,13 +410,23 @@ class Root(object):
         os.symlink("/proc/self/fd/1", self.makeChrootPath("dev/stdout"))
         os.symlink("/proc/self/fd/2", self.makeChrootPath("dev/stderr"))
 
-        # don't symlink for RHEL4 systems
+        # if hosted on EL{4,5} create /dev/tty node in chroot
+        if mock.util.cmpKernelEVR(kver, '2.6.19') <= 0:
+            os.mknod(self.makeChrootPath("dev/tty", stat.S_IFCHR|0666, os.maked
+            getLog().debug("created /dev/tty node device node")
+
+        # symlink /dev/fd in the chroot for everything except RHEL4
         if mock.util.cmpKernelEVR(kver, '2.6.9') > 0:
             os.symlink("/proc/self/fd",   self.makeChrootPath("dev/fd"))
 
-        # symlink it for FC and el6 hosts
-        if mock.util.cmpKernelEVR(kver, '2.6.18') > 0:
+        # symlinks for Fedora and el6 hosts
+        if mock.util.cmpKernelEVR(kver, '2.6.19') > 0:
             os.symlink("/dev/pts/ptmx", self.makeChrootPath("dev/ptmx"))
+            if os.path.exists(self.makeChrootPath("dev/tty")):
+                getLog().debug("removed dev/tty device node!")
+                os.remove(self.makeChrootPath("dev/tty"))
+            os.symlink("/dev/ptmx", self.makeChrootPath("dev/tty"))
+            getLog().debug("symlinked dev/tty to ptmx")
 
         os.umask(prevMask)
 
Leonard, does this still work for you? If so I'll queue it up for release with mock-1.1.8

Comment 16 Clark Williams 2011-01-06 18:07:10 UTC
btw, the current mock-1.1.8 candidate is available in my git tree under the 'work' branch and can be fetched like so:

   $ git clone git://git.fedorahosted.org/mock.git
   $ cd mock && git checkout -b origin/work work

This will get you the git tree. To use it do some variant on the following:

   $ sh autogen.sh && ./configure
   $ make install-devel-packages
   $ make rpm

This will generate a mock rpm which will end up in the 'noarch' directory.

If you'd like to run the mock test suite, run:

   $ make check

This will build the mock srpm on all supported configurations (and will take some time, depending on how fast your link to the fedora repos is).

Comment 17 Levente Farkas 2011-01-06 19:54:36 UTC
could you post it here when 1.1.8 will be released? 
thanks

Comment 18 Clark Williams 2011-01-07 01:19:43 UTC
I'll go ahead and kick off a koji build and you can snag it from the testing repo. Should happen tomorrow (Friday).

Comment 19 Levente Farkas 2011-01-07 14:10:49 UTC
1.1.8 still not solve the problem of building bash and zsh on rhel-6 ie. #613392

Comment 20 Paul Howarth 2011-01-07 15:05:37 UTC
I can build perl-TermReadKey on my local mock builder now (F14 x86_64 host) without issues using mock 1.1.8.

Comment 21 Leonard den Ottolander 2011-01-16 11:43:59 UTC
Sorry that I didn't answer sooner. Thanks Paul for testing Clarks patch. That way I don't have to do it :) .

Soon after I posted this patch I found out the build of bash still fails. So even though this symlink seems to be somewhat of an improvement it's not entirely correct.

I'm not sure what the best way forward is here. Perhaps /dev/tty should be ripped out altogether until someone who really understands these matters figures out a way to link /dev/tty inside the chroot correctly.

Comment 22 Fedora Update System 2011-01-18 19:57:46 UTC
mock-1.1.8-1.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/mock-1.1.8-1.fc14

Comment 23 Fedora Update System 2011-01-18 19:58:26 UTC
mock-1.1.8-1.fc13 has been submitted as an update for Fedora 13.
https://admin.fedoraproject.org/updates/mock-1.1.8-1.fc13

Comment 24 Fedora Update System 2011-01-18 20:03:19 UTC
mock-1.0.15-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/mock-1.0.15-1.el5

Comment 25 Fedora Update System 2011-01-19 18:26:04 UTC
mock-1.0.15-1.el5 has been pushed to the Fedora EPEL 5 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update mock'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/mock-1.0.15-1.el5

Comment 26 Fedora Update System 2011-01-21 23:03:13 UTC
mock-1.1.8-1.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 27 Levente Farkas 2011-01-21 23:18:47 UTC
are you sure this bug can be closed by 1.1.8?

Comment 28 Clark Williams 2011-01-23 16:18:12 UTC
No its not

Comment 29 Fedora Update System 2011-02-20 02:25:15 UTC
mock-1.1.9-1.fc13 has been submitted as an update for Fedora 13.
https://admin.fedoraproject.org/updates/mock-1.1.9-1.fc13

Comment 30 Fedora Update System 2011-02-20 02:28:31 UTC
mock-1.0.16-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/mock-1.0.16-1.el5

Comment 31 Fedora Update System 2011-02-20 02:31:28 UTC
mock-1.1.9-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/mock-1.1.9-1.el6

Comment 32 Fedora Update System 2011-02-20 02:34:15 UTC
mock-1.1.9-1.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/mock-1.1.9-1.fc14

Comment 33 Clark Williams 2011-02-20 02:54:09 UTC
I just pushed 1.1.9 to testing and this version doesn't setup /dev/tty for
f13,f14,f15 and el6.

Please give this one a try and see if it helps the hang situation.

Comment 34 Fedora Update System 2011-02-20 05:54:27 UTC
mock-1.0.16-1.el5 has been pushed to the Fedora EPEL 5 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update mock'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/mock-1.0.16-1.el5

Comment 35 Paul Howarth 2011-02-20 10:33:17 UTC
As noted in Comment #20, this was working in mock 1.1.8 as far as perl-Term-ReadKey was concerned (with a F-14 builder). The build still completes OK with 1.1.9, though most of the tests are skipped due to the absence of /dev/tty.

On the other hand, as noted in Bug #672713, perl-Expect failed to build in mock 1.1.8 and this is still the case with 1.1.9:

Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.L1Myox
+ umask 022
+ cd /builddir/build/BUILD
+ cd Expect-1.21
+ unset DISPLAY
+ /usr/bin/make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-Iblib/lib" "-Iblib/arch" test.pl
pty_allocate(nonfatal): posix_openpt(): No such file or directory at /usr/lib64/perl5/vendor_perl/IO/Pty.pm line 24.
pty_allocate(nonfatal): getpt(): No such file or directory at /usr/lib64/perl5/vendor_perl/IO/Pty.pm line 24.
pty_allocate(nonfatal): openpty(): No such file or directory at /usr/lib64/perl5/vendor_perl/IO/Pty.pm line 24.
pty_allocate(nonfatal): open(/dev/ptmx): No such file or directory at /usr/lib64/perl5/vendor_perl/IO/Pty.pm line 24.
1..42
Basic tests...
Cannot open a pty at test.pl line 34
make: *** [test_dynamic] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.L1Myox (%check)

So this change doesn't seem to have bought us anything over what was in 1.1.8 (unless maybe bash now builds - Bug #613392 ?).

Comment 36 Levente Farkas 2011-02-20 10:53:08 UTC
something change since bash finally build with this mock:-) #613392

Comment 37 Fedora Update System 2011-02-20 22:49:48 UTC
mock-1.1.9-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/mock-1.1.9-1.fc15

Comment 38 Christoph Karl 2011-02-27 14:51:29 UTC
Same for Fedora 14.
With "mock-1.1.9-1.fc14.noarch.rpm" bash builds.

Comment 39 Fedora Update System 2011-03-03 03:17:01 UTC
mock-1.1.9-1.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 40 Fedora Update System 2011-03-03 08:23:46 UTC
mock-1.1.9-1.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 41 Fedora Update System 2011-03-03 08:32:49 UTC
mock-1.1.9-1.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 42 Fedora Update System 2011-05-13 20:32:54 UTC
mock-1.1.10-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/mock-1.1.10-1.fc15

Comment 43 Fedora Update System 2011-05-13 20:37:52 UTC
mock-1.1.10-1.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/mock-1.1.10-1.fc14

Comment 44 Fedora Update System 2011-05-13 20:42:08 UTC
mock-1.0.17-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/mock-1.0.17-1.el5

Comment 45 Fedora Update System 2011-05-13 20:46:23 UTC
mock-1.1.10-1.fc13 has been submitted as an update for Fedora 13.
https://admin.fedoraproject.org/updates/mock-1.1.10-1.fc13

Comment 46 Fedora Update System 2011-05-13 20:50:40 UTC
mock-1.1.10-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/mock-1.1.10-1.el6

Comment 47 Fedora Update System 2011-05-19 04:34:23 UTC
mock-1.1.10-1.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 48 Fedora Update System 2011-05-25 02:41:42 UTC
mock-1.1.10-1.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 49 Fedora Update System 2011-05-25 03:16:10 UTC
mock-1.1.10-1.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 50 Fedora Update System 2011-06-02 19:05:56 UTC
mock-1.0.17-1.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 51 Fedora Update System 2011-06-02 19:15:57 UTC
mock-1.1.10-1.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.