Bug 1398405 - -O breaks recursive parallel builds using GNU make
Summary: -O breaks recursive parallel builds using GNU make
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: make
Version: 24
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Patsy Franklin
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-24 18:00 UTC by Petr Spacek
Modified: 2017-01-09 03:00 UTC (History)
8 users (show)

Fixed In Version: make-1:4.2.1-1.fc26
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-09 03:00:21 UTC


Attachments (Terms of Use)
reproducer.sh (18 bytes, text/plain)
2016-11-24 18:00 UTC, Petr Spacek
no flags Details
Makefile (53 bytes, text/plain)
2016-11-24 18:00 UTC, Petr Spacek
no flags Details
test.src.rpm (5.85 KB, application/octet-stream)
2016-11-24 18:01 UTC, Petr Spacek
no flags Details

Description Petr Spacek 2016-11-24 18:00:25 UTC
Created attachment 1223944 [details]
reproducer.sh

Description of problem:
rpm-build 4.13.0-1 broke recursive parallel builds using GNU make. Minimal reproducer is attached.

This bug manifests itself only if make is called with multiple jobs, e.g. -j16, and when rpmbuild itself is called from parallel make.

Version-Release number of selected component (if applicable):
rpm-build-4.13.0-1.fc24.x86_64 (broken)
rpm-build-4.13.0-0.rc1.27.fc24.x86_64 (latest functional version)


How reproducible:
100 %

Steps to Reproduce:
0. Prepare environment with rpmbuild and GNU make.
1. Download attached reproducer.sh and all other files into one working directory.
2. Run reproducer.sh

Actual results:
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.11YS3o
+ umask 022
+ cd /home/pspacek/rpmbuild/BUILD
+ cp /home/pspacek/rpmbuild/SOURCES/Makefile /home/pspacek/rpmbuild/BUILD/
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.OjLUrR
+ umask 022
+ cd /home/pspacek/rpmbuild/BUILD
+ /usr/bin/make -O -j8
make[1]: Entering directory '/home/pspacek/rpmbuild/BUILD'
sleep "1"
echo "1"
1
make[1]: Leaving directory '/home/pspacek/rpmbuild/BUILD'
make[1]: *** write jobserver: Bad file descriptor.  Stop.
make[1]: *** Waiting for unfinished jobs....
make[1]: Entering directory '/home/pspacek/rpmbuild/BUILD'
sleep "2"
echo "2"
2
make[1]: Leaving directory '/home/pspacek/rpmbuild/BUILD'
make[1]: *** write jobserver: Bad file descriptor.  Stop.
error: Bad exit status from /var/tmp/rpm-tmp.OjLUrR (%build)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.OjLUrR (%build)
Makefile:2: recipe for target 'doit' failed


Expected results:
It should work as it worked before, with rpm-build-4.13.0-0.rc1.27.fc24.x86_64.

Additional information:
This broke FreeIPA build workflow and costs us time becaus we have to run builds sequentially.

Comment 1 Petr Spacek 2016-11-24 18:00:54 UTC
Created attachment 1223945 [details]
Makefile

Comment 2 Petr Spacek 2016-11-24 18:01:33 UTC
Created attachment 1223946 [details]
test.src.rpm

Comment 3 Petr Spacek 2016-11-24 18:04:41 UTC
This is log from RPM transaction which upgrades rpm-build to the broken version:
Packages Altered:
    Upgraded rpm-4.13.0-0.rc1.27.fc24.x86_64                        @fedora
    Upgrade      4.13.0-1.fc24.x86_64                               @updates
    Upgraded rpm-build-4.13.0-0.rc1.27.fc24.x86_64                  @fedora
    Upgrade            4.13.0-1.fc24.x86_64                         @updates
    Upgraded rpm-build-libs-4.13.0-0.rc1.27.fc24.x86_64             @fedora
    Upgrade                 4.13.0-1.fc24.x86_64                    @updates
    Upgraded rpm-devel-4.13.0-0.rc1.27.fc24.x86_64                  @fedora
    Upgrade            4.13.0-1.fc24.x86_64                         @updates
    Upgraded rpm-libs-4.13.0-0.rc1.27.fc24.x86_64                   @fedora
    Upgrade           4.13.0-1.fc24.x86_64                          @updates
    Upgraded rpm-plugin-selinux-4.13.0-0.rc1.27.fc24.x86_64         @fedora
    Upgrade                     4.13.0-1.fc24.x86_64                @updates
    Upgraded rpm-plugin-systemd-inhibit-4.13.0-0.rc1.27.fc24.x86_64 @fedora
    Upgrade                             4.13.0-1.fc24.x86_64        @updates
    Upgraded rpm-python-4.13.0-0.rc1.27.fc24.x86_64                 @fedora
    Upgrade             4.13.0-1.fc24.x86_64                        @updates
    Upgraded rpm-python3-4.13.0-0.rc1.27.fc24.x86_64                @fedora
    Upgrade              4.13.0-1.fc24.x86_64                       @updates

Comment 4 Igor Gnatenko 2016-11-24 18:14:29 UTC
[brain@x1carbon tpm]$ rpmbuild --rebuild ./test-1.0-1.fc24.src.rpm 
Installing ./test-1.0-1.fc24.src.rpm
warning: user pspacek does not exist - using root
warning: group pspacek does not exist - using root
warning: user pspacek does not exist - using root
warning: group pspacek does not exist - using root
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.OKdH5n
+ umask 022
+ cd /home/brain/rpmbuild/BUILD
+ cp /home/brain/rpmbuild/SOURCES/Makefile /home/brain/rpmbuild/BUILD/
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.gm5skv
+ umask 022
+ cd /home/brain/rpmbuild/BUILD
+ /usr/bin/make -O -j4
sleep "1"
echo "1"
1
sleep "2"
echo "2"
2
sleep "3"
echo "3"
3
+ exit 0
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/brain/rpmbuild/BUILDROOT/test-1.0-1.fc26.x86_64
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.wBRgUK
+ umask 022
+ cd /home/brain/rpmbuild/BUILD
+ /usr/bin/rm -rf /home/brain/rpmbuild/BUILDROOT/test-1.0-1.fc26.x86_64
+ exit 0
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.EsHUx0
+ umask 022
+ cd /home/brain/rpmbuild/BUILD
+ exit 0




I don't see any error. Please provide exact reproducer.



Regarding "-O" flag, it is just for synchronizing output. I guess you are doing something wrong in makefile.
"""
       -O[type], --output-sync[=type]
            When running multiple jobs in parallel with -j, ensure the output of each job is collected together rather than interspersed with output from other  jobs.   If
            type  is  not  specified or is target the output from the entire recipe for each target is grouped together.  If type is line the output from each command line
            within a recipe is grouped together.  If type is recurse output from an entire recursive make is grouped together.  If type is none output  synchronization  is
            disabled.
"""

Comment 5 Panu Matilainen 2016-11-25 06:40:56 UTC
Yes I can reproduce this. If make breaks with added -O then it is a bug in make, switching component.

Running without -O I make is warning about the construct, so what you're doing might not be that hot an idea to begin with:

> Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.reWm3s
> + umask 022
> + cd /home/pmatilai/rpmbuild/BUILD
> + /usr/bin/make -j4
> make[1]: warning: -jN forced in submake: disabling jobserver mode.

Anyway, this is TRIVIAL to work around as you're just as free to call "make" in whatever manner you wish from %build as you've always been, so complaining how this is costing you time is a bit thick really.

As in: -O is treated as last-option-wins case so you can just use eg:
"%make_build -Onone" Or -Orecursive which appears to work too in this case. Or just do the old-fashioned thing and call it "make %{?_smp_mflags}" like you had to do with older rpm versions.

Comment 6 Panu Matilainen 2016-11-25 06:47:34 UTC
As for the bug in make, the warning about forced -j and disabling jobserver is ominuously missing when -O is added. And then there's the error about bad file descriptor about jobserver:
make[1]: *** write jobserver: Bad file descriptor.  Stop.

Comment 7 Petr Spacek 2016-11-25 09:14:13 UTC
@Igor, the very exact reproducer is already provided in bug description. To reproduce the issue you have to use script reproducer.sh. It is not sufficient to call rpmbuild --rebuild on the srpm.

"%make_build -Onone" works as workaround. It breaks as soon as I try make -j2 -Oline.

Thank you for debugging this!

Comment 8 Igor Gnatenko 2016-11-25 11:11:49 UTC
[brain@x1carbon tpm]$ curl https://bugzilla.redhat.com/attachment.cgi?id=1223944 -o reproducer.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    18  100    18    0     0     12      0  0:00:01  0:00:01 --:--:--    12
[brain@x1carbon tpm]$ curl https://bugzilla.redhat.com/attachment.cgi?id=1223945 -o Makefile
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    53  100    53    0     0     41      0  0:00:01  0:00:01 --:--:--    41
[brain@x1carbon tpm]$ curl https://bugzilla.redhat.com/attachment.cgi?id=1223946 -o test.src.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5993  100  5993    0     0   4133      0  0:00:01  0:00:01 --:--:--  4135
[brain@x1carbon tpm]$ bash -x reproducer.sh 
+ make -j16 breakit
rpmbuild --rebuild test-1.0-1.fc24.src.rpm
error: cannot open test-1.0-1.fc24.src.rpm: No such file or directory
make: *** [Makefile:2: breakit] Error 1
[brain@x1carbon tpm]$ mv test.src.rpm test-1.0-1.fc24.src.rpm
[brain@x1carbon tpm]$ bash -x reproducer.sh 
+ make -j16 breakit
rpmbuild --rebuild test-1.0-1.fc24.src.rpm
Installing test-1.0-1.fc24.src.rpm
warning: user pspacek does not exist - using root
warning: group pspacek does not exist - using root
warning: user pspacek does not exist - using root
warning: group pspacek does not exist - using root
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.70A2uM
+ umask 022
+ cd /home/brain/rpmbuild/BUILD
+ cp /home/brain/rpmbuild/SOURCES/Makefile /home/brain/rpmbuild/BUILD/
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.yFJEbd
+ umask 022
+ cd /home/brain/rpmbuild/BUILD
+ /usr/bin/make -O -j4
make[1]: Entering directory '/home/brain/rpmbuild/BUILD'
make[1]: warning: -jN forced in submake: disabling jobserver mode.
make[1]: Leaving directory '/home/brain/rpmbuild/BUILD'
make[1]: Entering directory '/home/brain/rpmbuild/BUILD'
sleep "1"
echo "1"
1
make[1]: Leaving directory '/home/brain/rpmbuild/BUILD'
make[1]: Entering directory '/home/brain/rpmbuild/BUILD'
sleep "2"
echo "2"
2
make[1]: Leaving directory '/home/brain/rpmbuild/BUILD'
make[1]: Entering directory '/home/brain/rpmbuild/BUILD'
sleep "3"
echo "3"
3
make[1]: Leaving directory '/home/brain/rpmbuild/BUILD'
+ exit 0
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/brain/rpmbuild/BUILDROOT/test-1.0-1.fc26.x86_64
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.Fei2dM
+ umask 022
+ cd /home/brain/rpmbuild/BUILD
+ /usr/bin/rm -rf /home/brain/rpmbuild/BUILDROOT/test-1.0-1.fc26.x86_64
+ exit 0
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.NUQfll
+ umask 022
+ cd /home/brain/rpmbuild/BUILD
+ exit 0


[brain@x1carbon tpm]$ make --version 
GNU Make 4.2.1
Built for x86_64-redhat-linux-gnu
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Comment 9 Panu Matilainen 2016-11-25 11:15:50 UTC
Looks like its been fixed upstream then, good. Fedora 24 (for which this bug is filed) has make 4.1 though, and with that this is reproducable.

Comment 10 Petr Spacek 2016-11-25 12:04:56 UTC
Okay. I confirm that make-1:4.1-5.fc24.x86_64 is broken and that %make_build works fine with make-1:4.2.1-1.fc26.x86_64.

Thank you for you help, guys!


Note You need to log in before you can comment on or make changes to this bug.