Bug 613392 - bash not able to rebuild in mock
bash not able to rebuild in mock
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: mock (Show other bugs)
14
All Linux
low Severity medium
: ---
: ---
Assigned To: Clark Williams
Fedora Extras Quality Assurance
:
Depends On:
Blocks: 660809
  Show dependency treegraph
 
Reported: 2010-07-11 04:08 EDT by Levente Farkas
Modified: 2013-01-10 01:04 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-02-20 05:52:48 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Levente Farkas 2010-07-11 04:08:00 EDT
neither the beta1 nor the beta2 bash can be rebuild in mock. eg on beta2 it's hang forever in this stage:
-------------------------------------------------
robot    26974  0.0  0.0  10868  2956 pts/1    T    Jul10   0:00              \_ rpmbuild -bb --target i686 --nodeps builddir/build/SPECS/bash.spec
robot     6104  0.0  0.0   5016  1004 pts/1    T    Jul10   0:00                  \_ /bin/sh -e /var/tmp/rpm-tmp.PgsDPm
robot     6105  0.0  0.0   2392  1012 pts/1    T    Jul10   0:00                      \_ make check
robot     6137  0.0  0.0   2936   892 pts/1    T    Jul10   0:00                          \_ /bin/sh -c ( cd ./tests && \??PATH=/builddir/build/BUILD/bash-4.1/tests:$P
robot     6138  0.0  0.0   2936   588 pts/1    T    Jul10   0:00                              \_ /bin/sh -c ( cd ./tests && \??PATH=/builddir/build/BUILD/bash-4.1/test
robot     6139  0.0  0.0   2940  1064 pts/1    T    Jul10   0:00                                  \_ /bin/sh run-all
robot     6747  0.0  0.0   2940   992 pts/1    T    Jul10   0:00                                      \_ sh run-execscript
robot     6748  0.0  0.0   2944  1044 pts/1    T    Jul10   0:01                                          \_ /builddir/build/BUILD/bash-4.1/bash ./execscript
robot     6803  0.0  0.0   2992   816 pts/1    T    Jul10   0:00                                              \_ /builddir/build/BUILD/bash-4.1/bash -i ./exec8.sub
-------------------------------------------------
Comment 2 Roman Rakus 2010-07-23 10:16:49 EDT
I can easily rebuild bash with rpmbuild (not in mock). Assigning to mock.
Comment 3 Levente Farkas 2010-07-23 11:28:50 EDT
i can also easily rebuild with rpmbuild, but imho it's still not a mock issue. the bug is in the check part of the spec file while waiting for something which is not available during background/non interactive/chroot shell. so it'd have to fix in bash spec or bash's Makefile's check target.
Comment 4 Roman Rakus 2010-07-26 08:35:07 EDT
In RHEL the official way to (re)build packages is with direct rpmbuild.
I will look deeper what's going on and possibly fix it in Fedora.
The build gangs in the test when executing execscript.
Comment 5 Roman Rakus 2010-07-26 13:54:36 EDT
Bash hangs when run in mock chroot trying to run interactively - opening /dev/tty.
Comment 6 Levente Farkas 2010-07-26 15:03:20 EDT
does it normal to do it during the build?
Comment 7 Roman Rakus 2010-07-29 04:50:00 EDT
Depends on what is "normal". This occurs only in the check routine.
Comment 8 Bug Zapper 2010-07-30 08:30:05 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 14 development cycle.
Changing version to '14'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 9 Roman Rakus 2010-08-10 13:31:58 EDT
Changing component to mock itself. Maybe there can be something changed in mock, so the bash can borrow /dev/tty for a while. If not, there will needed some dirty hack...
Comment 10 Paul Howarth 2010-08-20 11:34:02 EDT
This looks similar to Bug #609201.
Comment 11 Roman Rakus 2010-08-23 03:39:14 EDT
Yep. Similar or same.
Comment 12 Levente Farkas 2010-11-09 04:29:44 EST
is there any progress with it?
it seems rhel-6's bash still has the same problems...
Comment 13 Levente Farkas 2010-12-03 08:32:12 EST
is there any list for mock or where is the mock development happened? mailing list or something like that? or should i ask here?
it seems most of the mock problem in the current release comes from the %check section of the rpmbuild. there are 4,5 different bz about it.
is there any mock option to switch off the rpmbuild check part from the build? it can solve a lots of issue until someone can fix mock and different packages.
thanks.
Comment 14 Clark Williams 2010-12-03 10:09:16 EST
Development discussion usually takes place on the fedora-buildsys-list (but it's been pretty quiet lately. I apologize for the lack of movement on this bug, but I'm coming out of a release cycle so I'll be able to do mock things now. 

I suspect that this and https://bugzilla.redhat.com/show_bug.cgi?id=609201 are both symptoms of tty issues (ugh).
Comment 15 Levente Farkas 2010-12-03 10:46:08 EST
it's not just tty, the same happened with guile, bash, zsh, jna and many others.
they all build with rpmbuild but not in mock because of different reason, but all failed in %check section so it'd be useful feature for mock.
Comment 16 Leonard den Ottolander 2010-12-03 15:07:40 EST
I think you misunderstand what is being said. tty is not a package but the current terminal device. There seems to be in issue where this device is not properly passed to the mock root.

At least the failing of the build of bash is an issue that has to do with this. The execscript test starts an interactive shell (bash -i ./exec8.sub) which is where the build stalls.

By the way, if you chroot to the mock root and su - mockbuild you can complete the build by just starting a
$ rpmbuild -ba bash.spec
Comment 17 Levente Farkas 2010-12-06 11:03:59 EST
no i understand it. i just said that there are many other packages which fail in the %check section with mock-1.1.5 (not just tty related bug) while it was build with mock-0.6.3. so it'd be a useful feature for mock to disable the %check part during build.
Comment 18 Levente Farkas 2010-12-08 03:02:25 EST
half of the rhel-6's perl packages also fail in the %check section (i don't they are tty related or not).
Comment 19 Clark Williams 2011-01-07 11:38:14 EST
I finally got a RHEL6 box setup and duplicated your hang. 

in a separate window I did:

$ ps -Ao state,tty,wchan:20,cmd

and found this group of threads:

T pts/0    signal_stop          /bin/sh -e /var/tmp/rpm-tmp.zAHwbl
T pts/0    signal_stop          make check
T pts/0    signal_stop          /bin/sh -c ( cd ./tests && \??PATH=/builddir/build/BUILD/bash-4.1/tests:$PATH THIS_SH=/builddir/bui
T pts/0    signal_stop          /bin/sh -c ( cd ./tests && \??PATH=/builddir/build/BUILD/bash-4.1/tests:$PATH THIS_SH=/builddir/bui
T pts/0    signal_stop          /bin/sh run-all
T pts/0    signal_stop          sh run-execscript
T pts/0    signal_stop          /builddir/build/BUILD/bash-4.1/bash ./execscript
T pts/0    signal_stop          /builddir/build/BUILD/bash-4.1/bash -i ./exec8.sub


So everything looks like it's stuck in the T state, meaning:

 T    Stopped, either by a job control signal or because it is being traced.

Since the wchan is signal_stop, I'd guess that we somehow got a job control signal and things went bad from there. Not sure why this is hanging, since the same rpm gets built on both f13 and f14, which are relatively new kernels and should have the same pseudo-terminal behavior as the RHEL6 kernel. 

Continuing to debug...
Comment 20 Clark Williams 2011-01-07 13:13:59 EST

After trying to build bash for x86_64 on a RHEL6 system, I did this:

$ sudo ./py/mock.py -r epel-6-x86_64 --shell
mock-chroot> cd /builddir/build/BUILD/bash-4.1
mock-chroot> make check

<lots of output about testing>

run-execscript
warning: the text of a system error message may vary between systems and
warning: produce diff output.
warning: if the text of the error messages concerning `notthere' or
warning: `/tmp/bash-notthere' not being found or `/' being a directory
warning: produce diff output, please do not consider this a test failure
warning: if diff output differing only in the location of the bash
warning: binary appears, please do not consider this a test failure

[1]+  Stopped(SIGTTIN)        make check
mock-chroot> 

So, something in bash's run-execscript is trying to do an interactive read from the terminal. Or at least that's what the system *thinks* is happening. 

Any thoughts?
Comment 21 Levente Farkas 2011-01-07 17:46:06 EST
as i wrote to the epel list 
mock --shell
rpmbuild ...
will work in both zsh and bash case. so it should have to be something with interactive/none interactice shell/terminal settings durng build.
Comment 22 Roman Rakus 2011-01-10 11:45:11 EST
I'm not familiar with python, but I guess the python is blocking /dev/tty. Maybe it is possible to unlock it or share... don't know, just shooting
Comment 23 Leonard den Ottolander 2011-01-16 06:55:49 EST
> So, something in bash's run-execscript is trying to do an interactive read from
> the terminal. Or at least that's what the system *thinks* is happening. 

The trace at the top says it:
tests/execscript calls
bash -i ./exec8.sub

I.e. it uses an interactive bash shell for that last test. Even that symlink from /dev/tty to /dev/ptmx I proposed in bug 609201 doesn't fix this issue. Ripping out /dev/tty from the buildroot altogether does, which is why koji does not fail to build this package.
Comment 24 Clark Williams 2011-01-16 09:29:54 EST
so what about if we remove /dev/tty in the build case and symlink it for the --shell case?
Comment 25 Clark Williams 2011-01-21 18:18:18 EST
Hmmm, now I'm wondering if we just need to nuke /dev/tty from inside the chroot?

Would it make sense to have a config variable 'setup_dev_tty' that's normally False, but can be set to True and would cause mock to setup /dev/tty inside the chroot (in case there's a package out there that desperately needs /dev/tty or checks it for existance, etc.)?
Comment 26 Levente Farkas 2011-01-21 18:20:05 EST
why not always? does it expensive?
Comment 27 Leonard den Ottolander 2011-01-21 23:06:42 EST
Clark, I suggest you ask one of the kernel guys at Red Hat to look into this. Linking /dev/ptmx to /dev/tty as I proposed for bug 609201 seemed a good idea but I can't vouch for the correctness of the approach. Perhaps it is indeed correct and the fact that this bash build fails is a problem with python as Roman suggests but I don't claim to be an authority on these matters so I can't tell you for sure. I just tried something that seemed logical, or at least a path worth exploring. That approach fixed the other issue but not this one, so you should put it to someone with more knowledge of these matters.
Comment 28 Clark Williams 2011-01-23 11:35:57 EST
heh, fact is that I'm a kernel guy at Red Hat :). What I'm *not* is familiar with the tty subsystem.

We need to find someone that knows how the tty system has changed from 2.6.9 to the present. I'll see if I can dig someone up.
Comment 29 Levente Farkas 2011-01-23 12:35:57 EST
another (may be) useful tip, is this working with mock-0.6.x...
Comment 30 Christoph Karl 2011-02-05 06:40:21 EST
Me having the same problem trying to rebuild bash with mock
-) fedora 14 (plus all updates until today)
-) x86_64
-) processes hanging in T state
-) does rebuild with rpmbuild
Comment 31 Christoph Karl 2011-02-05 07:14:21 EST
@ Roman Comment 4

After the discussion on this bug
https://bugzilla.redhat.com/show_bug.cgi?id=432194
I was thinking "mock" is the preferred build environment.

What is the current preferred way to rebuild for Fedora?
Comment 32 Roman Rakus 2011-02-07 10:40:32 EST
(In reply to comment #31)
> @ Roman Comment 4
> 
> After the discussion on this bug
> https://bugzilla.redhat.com/show_bug.cgi?id=432194
> I was thinking "mock" is the preferred build environment.
> 
> What is the current preferred way to rebuild for Fedora?

I don't know. Maybe not any one :)
Koji is using mock for building and most likely it is using some older version. How much builds are failing with newer mock? It make sense to not run tests requiring access to controlling terminals in non-interactive builds. But how to determinate it?
Comment 33 Levente Farkas 2011-02-20 05:52:48 EST
1.1.19 solve it:-)
Comment 34 Christoph Karl 2011-02-27 09:44:08 EST
"mock-1.1.9-1.fc14.noarch.rpm" works for me also.

Note You need to log in before you can comment on or make changes to this bug.