Bug 1199360 - Parallel build of webkitgtk4 fails with cmake 3.2
Summary: Parallel build of webkitgtk4 fails with cmake 3.2
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: cmake
Version: 22
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Orion Poplawski
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-03-06 03:29 UTC by Michael Catanzaro
Modified: 2015-03-18 10:34 UTC (History)
9 users (show)

Fixed In Version: cmake-3.2.1-1.fc22
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-03-18 10:34:15 UTC
Type: Bug


Attachments (Terms of Use)

Description Michael Catanzaro 2015-03-06 03:29:34 UTC
Description of problem: Parallel build of webkitgtk4 fails with cmake 3.2. I wonder if it's a WebKit bug or a CMake bug.


Version-Release number of selected component (if applicable): cmake-3.2.0-0.1.rc1.fc22


How reproducible: ...eh... always?


Steps to Reproduce:
1. Revert the latest commit to webkitgtk4 which disables parallel build
2. fedpkg build webkitgtk4


Actual results: https://kojipkgs.fedoraproject.org//work/tasks/2915/9132915/build.log

The interesting part is this:

[ 15%] cd /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/DerivedSources/WebCore && /usr/bin/perl -I/builddir/build/BUILD/webkitgtk-2.7.91/Source/WebCore/bindings/scripts /builddir/build/BUILD/webkitgtk-2.7.91/Source/WebCore/css/makeprop.pl --defines " ENABLE_3D_RENDERING ENABLE_ACCELERATED_2D_CANVAS ENABLE_ACCESSIBILITY ENABLE_CHANNEL_MESSAGING ENABLE_CONTEXT_MENUS ENABLE_CSS_BOX_DECORATION_BREAK ENABLE_CSS_IMAGE_SET ENABLE_CSS_REGIONS ENABLE_CSS_SELECTORS_LEVEL4 ENABLE_DETAILS_ELEMENT ENABLE_DRAG_SUPPORT ENABLE_FULLSCREEN_API ENABLE_GEOLOCATION ENABLE_ICONDATABASE ENABLE_INPUT_TYPE_COLOR ENABLE_JIT ENABLE_LEGACY_VENDOR_PREFIXES ENABLE_MATHML ENABLE_MEDIA_CONTROLS_SCRIPT ENABLE_MEMORY_SAMPLER ENABLE_METER_ELEMENT ENABLE_MHTML ENABLE_NAVIGATOR_HWCONCURRENCY ENABLE_NETSCAPE_PLUGIN_API ENABLE_NETWORK_PROCESS ENABLE_NOTIFICATIONS ENABLE_PICTURE_SIZES ENABLE_PROMISES ENABLE_REQUEST_ANIMATION_FRAME ENABLE_SATURATED_LAYOUT_ARITHMETIC ENABLE_SPELLCHECK ENABLE_STREAMS_API ENABLE_SVG_FONTS ENABLE_TEMPLATE_ELEMENT ENABLE_TOUCH_EVENTS ENABLE_USERSELECT_ALL ENABLE_USER_MESSAGE_HANDLERS ENABLE_VIDEO ENABLE_VIDEO_TRACK ENABLE_VIEW_MODE_CSS_MEDIA ENABLE_WEB_AUDIO ENABLE_WEB_SOCKETS ENABLE_WEB_TIMING ENABLE_WEBGL ENABLE_XHR_TIMEOUT ENABLE_XSLT ENABLE_PLUGIN_PROCESS_GTK2" --preprocessor "/usr/bin/c++ -E -P -x c++"
/usr/bin/cmake -E cmake_progress_report /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/CMakeFiles 
/usr/bin/cmake -E cmake_progress_report /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/CMakeFiles 
/usr/bin/cmake -E cmake_progress_report /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/CMakeFiles 
/usr/bin/cmake -E cmake_progress_report /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/CMakeFiles 
/usr/bin/cmake -E cmake_progress_report /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/CMakeFiles 
/usr/bin/cmake -E cmake_progress_report /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/CMakeFiles 
[ 15%] [ 15%] [ 15%] [ 15%] [ 15%] Generating ../../DerivedSources/WebCore/SelectorPseudoElementTypeMap.gperf, ../../DerivedSources/WebCore/SelectorPseudoElementTypeMap.cpp
[ 15%] CSSPropertyNames.gperf: The input file is empty!
calling gperf failed: 256 at /builddir/build/BUILD/webkitgtk-2.7.91/Source/WebCore/css/makeprop.pl line 1032.
Source/WebCore/CMakeFiles/WebCore.dir/build.make:155: recipe for target 'DerivedSources/WebCore/CSSPropertyNames.h' failed
make[2]: *** [DerivedSources/WebCore/CSSPropertyNames.h] Error 25
make[2]: *** Waiting for unfinished jobs...

Line 1032 of makeprop.pl:

system("\"$gperf\" --key-positions=\"*\" -D -n -s 2 CSSPropertyNames.gperf --output-file=CSSPropertyNames.cpp") == 0 || die "calling gperf failed: $?";

So I guess CSSPropertyNames.gperf has not been built yet. Oh, but that file is built by makeprop.pl, beginning with line 113:

open GPERF, ">CSSPropertyNames.gperf" || die "Could not open CSSPropertyNames.gperf for writing";

So it doesn't SEEM like a CMake dependency issue... but the script is a mess to read, and I can imagine it not working if CSSPropertyNames.in was not properly generated.

Now, in WebKit/Source/WebCore/CMakeLists.txt we have this:

# Generate CSS property names
add_custom_command(
    OUTPUT ${DERIVED_SOURCES_WEBCORE_DIR}/CSSPropertyNames.in ${DERIVED_SOURCES_WEBCORE_DIR}/CSSPropertyNames.h ${DERIVED_SOURCES_WEBCORE_DIR}/CSSPropertyNames.cpp ${DERIVED_SOURCES_WEBCORE_DIR}/CSSPropertyNames.gperf ${DERIVED_SOURCES_WEBCORE_DIR}/StyleBuilder.cpp ${DERIVED_SOURCES_WEBCORE_DIR}/StylePropertyShorthandFunctions.h ${DERIVED_SOURCES_WEBCORE_DIR}/StylePropertyShorthandFunctions.cpp
    MAIN_DEPENDENCY ${WEBCORE_DIR}/css/makeprop.pl
    DEPENDS ${WebCore_CSS_PROPERTY_NAMES}
    WORKING_DIRECTORY ${DERIVED_SOURCES_WEBCORE_DIR}
    COMMAND ${PERL_EXECUTABLE} -ne "print" ${WebCore_CSS_PROPERTY_NAMES} > ${DERIVED_SOURCES_WEBCORE_DIR}/CSSPropertyNames.in
    COMMAND ${PERL_EXECUTABLE} -I${WEBCORE_DIR}/bindings/scripts ${WEBCORE_DIR}/css/makeprop.pl --defines "${FEATURE_DEFINES_WITH_SPACE_SEPARATOR}" --preprocessor "${CODE_GENERATOR_PREPROCESSOR}"
    VERBATIM)
list(APPEND WebCore_SOURCES ${DERIVED_SOURCES_WEBCORE_DIR}/CSSPropertyNames.cpp)
list(APPEND WebCore_SOURCES ${DERIVED_SOURCES_WEBCORE_DIR}/StyleBuilder.cpp)
list(APPEND WebCore_SOURCES ${DERIVED_SOURCES_WEBCORE_DIR}/StylePropertyShorthandFunctions.cpp)
ADD_SOURCE_WEBCORE_DERIVED_DEPENDENCIES(${WEBCORE_DIR}/css/CSSParser.cpp CSSValueKeywords.h)
ADD_SOURCE_WEBCORE_DERIVED_DEPENDENCIES(${DERIVED_SOURCES_WEBCORE_DIR}/CSSGrammar.cpp CSSPropertyNames.h)

Simple, right? :p

And also:

set(WebCore_CSS_PROPERTY_NAMES
    ${WEBCORE_DIR}/css/CSSPropertyNames.in
)

...looks right? At least, I don't notice anything wrong there. But I do notice higher up in the build log:

[ 15%] [ 15%] [ 15%] Generating ../../DerivedSources/WebCore/CSSPropertyNames.in, ../../DerivedSources/WebCore/CSSPropertyNames.h, ../../DerivedSources/WebCore/CSSPropertyNames.cpp, ../../DerivedSources/WebCore/CSSPropertyNames.gperf, ../../DerivedSources/WebCore/StyleBuilder.cpp, ../../DerivedSources/WebCore/StylePropertyShorthandFunctions.h, ../../DerivedSources/WebCore/StylePropertyShorthandFunctions.cpp
cd /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/DerivedSources/WebCore && /usr/bin/perl -ne print /builddir/build/BUILD/webkitgtk-2.7.91/Source/WebCore/css/CSSPropertyNames.in > /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/DerivedSources/WebCore/CSSPropertyNames.in
Generating ../../DerivedSources/WebCore/CSSPropertyNames.in, ../../DerivedSources/WebCore/CSSPropertyNames.h, ../../DerivedSources/WebCore/CSSPropertyNames.cpp, ../../DerivedSources/WebCore/CSSPropertyNames.gperf, ../../DerivedSources/WebCore/StyleBuilder.cpp, ../../DerivedSources/WebCore/StylePropertyShorthandFunctions.h, ../../DerivedSources/WebCore/StylePropertyShorthandFunctions.cpp
cd /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/DerivedSources/WebCore && /usr/bin/perl -ne print /builddir/build/BUILD/webkitgtk-2.7.91/Source/WebCore/css/CSSPropertyNames.in > /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/DerivedSources/WebCore/CSSPropertyNames.in
Generating ../../DerivedSources/WebCore/CSSPropertyNames.in, ../../DerivedSources/WebCore/CSSPropertyNames.h, ../../DerivedSources/WebCore/CSSPropertyNames.cpp, ../../DerivedSources/WebCore/CSSPropertyNames.gperf, ../../DerivedSources/WebCore/StyleBuilder.cpp, ../../DerivedSources/WebCore/StylePropertyShorthandFunctions.h, ../../DerivedSources/WebCore/StylePropertyShorthandFunctions.cpp
cd /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/DerivedSources/WebCore && /usr/bin/perl -ne print /builddir/build/BUILD/webkitgtk-2.7.91/Source/WebCore/css/CSSPropertyNames.in > /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/DerivedSources/WebCore/CSSPropertyNames.in
/usr/bin/cmake -E cmake_progress_report /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/CMakeFiles 
Generating ../../DerivedSources/WebCore/CSSPropertyNames.in, ../../DerivedSources/WebCore/CSSPropertyNames.h, ../../DerivedSources/WebCore/CSSPropertyNames.cpp, ../../DerivedSources/WebCore/CSSPropertyNames.gperf, ../../DerivedSources/WebCore/StyleBuilder.cpp, ../../DerivedSources/WebCore/StylePropertyShorthandFunctions.h, ../../DerivedSources/WebCore/StylePropertyShorthandFunctions.cpp
/usr/bin/cmake -E cmake_progress_report /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/CMakeFiles 
cd /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/DerivedSources/WebCore && /usr/bin/perl -ne print /builddir/build/BUILD/webkitgtk-2.7.91/Source/WebCore/css/CSSPropertyNames.in > /builddir/build/BUILD/webkitgtk-2.7.91/x86_64-redhat-linux-gnu/DerivedSources/WebCore/CSSPropertyNames.in

Um, that command is run four times instead of once! And that is as far as I've gotten, but I guess that must be a CMake bug.

Comment 1 Ben Boeckel 2015-03-06 15:34:27 UTC
I've heard of this before (but have not seen it myself). The fix before was to not use the same MAIN_DEPENDENCY for multiple add_custom_commands. Looking at the source tree of WebKit though doesn't seem to indicate that makeprop.pl is being used in that way though. Is it possible to get the debug output from make for why it is running the command? Also, how does the Ninja generator fare?

Comment 2 Ben Boeckel 2015-03-06 16:41:19 UTC
Bug reported upstream: http://www.cmake.org/Bug/view.php?id=15434

Comment 3 Michael Catanzaro 2015-03-06 23:40:12 UTC
Thanks for your help Ben! As noted in your upstream bug, it's probably the same issue as http://www.cmake.org/Bug/view.php?id=15116#c38168 which was hopefully fixed earlier today, so I'd rather see if that works before spending more time on this.

Comment 4 Ben Boeckel 2015-03-07 02:52:31 UTC
Well, if you get a chance, it might be worth trying the Ninja generator to help get build times down as well (I'd also be interested in a %cmake_ninja macro for use in specfiles).

You might need to set some pool limits to keep the memory usage down during link steps though. See: http://www.cmake.org/cmake/help/v3.1/prop_gbl/JOB_POOLS.html

Comment 5 Terje Røsten 2015-03-11 16:23:54 UTC
Can we have cmake updated to 3.2.1, will fix this bug (hopefully):

 http://www.kitware.com/blog/home/post/862

Comment 6 Orion Poplawski 2015-03-11 16:28:35 UTC
Building - http://koji.fedoraproject.org/koji/taskinfo?taskID=9205195

Comment 7 Terje Røsten 2015-03-11 16:31:45 UTC
Thanks Orion!

Comment 8 Michael Catanzaro 2015-03-11 19:36:23 UTC
By the way, I've been using the Ninja generator pretty much exclusively for local development of WebKitGTK+ since last summer. It works great.

Comment 9 Tomas Popela 2015-03-12 14:27:18 UTC
(In reply to Orion Poplawski from comment #6)
> Building - http://koji.fedoraproject.org/koji/taskinfo?taskID=9205195

Orion, can you please build it for F22 as well?

Comment 10 Fedora Update System 2015-03-13 16:29:11 UTC
cmake-3.2.1-1.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/cmake-3.2.1-1.fc22

Comment 11 Fedora Update System 2015-03-14 09:21:56 UTC
Package cmake-3.2.1-1.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing cmake-3.2.1-1.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-3892/cmake-3.2.1-1.fc22
then log in and leave karma (feedback).

Comment 12 Fedora Update System 2015-03-18 10:34:15 UTC
cmake-3.2.1-1.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.