Bug 2290987 - ninja always rebuilding because it considers many things "dirty"
Summary: ninja always rebuilding because it considers many things "dirty"
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: ninja-build
Version: 42
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Ben Boeckel
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-06-07 21:53 UTC by Adam Williamson
Modified: 2026-05-06 18:38 UTC (History)
8 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2026-05-06 18:38:59 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github rpm-software-management rpm pull 3160 0 None open Avoid special characters from %{VERSION} in %builddir 2024-06-12 07:41:01 UTC

Description Adam Williamson 2024-06-07 21:53:59 UTC
In Rawhide, ninja seems to be repeating most of the source build steps at %install and %check , when building os-autoinst. Check this log extract:

Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.eekeuV
...
ninja: Entering directory `redhat-linux-build'
[1/16] cd "/builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/redhat-linux-build" && mkdir -p /builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/redhat-linux-build/install-target
[2/16] cd "/builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/redhat-linux-build" && sed -e "s,\$sysdir\ =\ undef\;,\$sysdir\ =\ '/usr/lib/os-autoinst'\;," /builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/cv.pm > /builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/redhat-linux-build/install-target/cv.pm
...
[16/16] : && /usr/bin/g++ -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer  -Wl,-z,relro -Wl,--as-needed  -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -shared  -o ppmclibs/tinycv.so ppmclibs/CMakeFiles/tinycv.dir/tinycv_ast2100.cc.o ppmclibs/CMakeFiles/tinycv.dir/tinycv_impl.cc.o ppmclibs/CMakeFiles/tinycv.dir/tinycv-xs.cpp.o  /usr/lib64/libopencv_imgcodecs.so.4.9.0  /usr/lib64/libopencv_imgproc.so.4.9.0  /usr/lib64/libopencv_core.so.4.9.0 && :
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.1nXusY
...
[1/13] /usr/bin/g++  -isystem /usr/include/opencv4 -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD -MT debugviewer/CMakeFiles/debugviewer.dir/debugviewer.cpp.o -MF debugviewer/CMakeFiles/debugviewer.dir/debugviewer.cpp.o.d -o debugviewer/CMakeFiles/debugviewer.dir/debugviewer.cpp.o -c '/builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/debugviewer/debugviewer.cpp'
[2/13] cd "/builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/redhat-linux-build" && sed -e "s,\$sysdir\ =\ undef\;,\$sysdir\ =\ '/usr/lib/os-autoinst'\;," /builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/cv.pm > /builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/redhat-linux-build/install-target/cv.pm
...
[12/13] : && /usr/bin/g++ -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer  -Wl,-z,relro -Wl,--as-needed  -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -shared  -o ppmclibs/tinycv.so ppmclibs/CMakeFiles/tinycv.dir/tinycv_ast2100.cc.o ppmclibs/CMakeFiles/tinycv.dir/tinycv_impl.cc.o ppmclibs/CMakeFiles/tinycv.dir/tinycv-xs.cpp.o  /usr/lib64/libopencv_imgcodecs.so.4.9.0  /usr/lib64/libopencv_imgproc.so.4.9.0  /usr/lib64/libopencv_core.so.4.9.0 && :
[12/13] cd "/builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/redhat-linux-build" && /usr/bin/cmake -P cmake_install.cmake

That is, it re-does a bunch of the build steps during %install - everything from 1/13 to 12/13 is a source build step that it already did in %build.

This does not happen in f40 builds. We get the same 16 steps in %build, but here's %install:

ninja: Entering directory `redhat-linux-build'
[1/3] cd /builddir/build/BUILD/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/redhat-linux-build && sed -e "s,\$sysdir\ =\ undef\;,\$sysdir\ =\ '/usr/lib/os-autoinst'\;," /builddir/build/BUILD/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/cv.pm > /builddir/build/BUILD/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/redhat-linux-build/install-target/cv.pm
[1/3] cd /builddir/build/BUILD/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/redhat-linux-build && /usr/bin/cmake -P cmake_install.cmake

that is, it only schedules three steps and none of them are source compilation, it goes straight to install (as you'd expect).

So I hacked the build up to fail at the start of %install and run it in mock so I can shell in after and poke around, and...here's Rawhide:

<mock-chroot> sh-5.2# cd /builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/
<mock-chroot> sh-5.2# ninja -C redhat-linux-build -d explain
ninja: Entering directory `redhat-linux-build'
ninja explain: /builddir/build/BUILD/os-autoinst-4.6 has no in-edge and is missing
ninja explain: /builddir/build/BUILD/os-autoinst-4.6 is dirty
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/videoencoder.cpp has no in-edge and is missing
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/videoencoder.cpp is dirty
ninja explain: CMakeFiles/videoencoder.dir/videoencoder.cpp.o is dirty
ninja explain: videoencoder is dirty
ninja explain: recorded mtime of install-target/cv.pm older than most recent input install-target (1717796610887967609 vs 1717796610888967596)
ninja explain: install-target/cv.pm is dirty
ninja explain: CMakeFiles/generate-install-versions is dirty
ninja explain: install-target/cv.pm is dirty
ninja explain: generate-install-versions is dirty
ninja explain: /builddir/build/BUILD/os-autoinst-4.6 is dirty
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/debugviewer/debugviewer.cpp has no in-edge and is missing
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/debugviewer/debugviewer.cpp is dirty
ninja explain: debugviewer/CMakeFiles/debugviewer.dir/debugviewer.cpp.o is dirty
ninja explain: debugviewer/debugviewer is dirty
ninja explain: debugviewer/all is dirty
ninja explain: /builddir/build/BUILD/os-autoinst-4.6 is dirty
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/snd2png/snd2png.cpp has no in-edge and is missing
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/snd2png/snd2png.cpp is dirty
ninja explain: snd2png/CMakeFiles/snd2png.dir/snd2png.cpp.o is dirty
ninja explain: snd2png/snd2png is dirty
ninja explain: snd2png/all is dirty
ninja explain: /builddir/build/BUILD/os-autoinst-4.6 is dirty
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/ppmclibs/tinycv_ast2100.cc has no in-edge and is missing
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/ppmclibs/tinycv_ast2100.cc is dirty
ninja explain: ppmclibs/CMakeFiles/tinycv.dir/tinycv_ast2100.cc.o is dirty
ninja explain: /builddir/build/BUILD/os-autoinst-4.6 is dirty
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/ppmclibs/tinycv_impl.cc has no in-edge and is missing
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/ppmclibs/tinycv_impl.cc is dirty
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/ppmclibs/tinycv.h has no in-edge and is missing
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/ppmclibs/tinycv.h is dirty
ninja explain: ppmclibs/CMakeFiles/tinycv.dir/tinycv_impl.cc.o is dirty
ninja explain: /builddir/build/BUILD/os-autoinst-4.6 is dirty
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/redhat-linux-build/ppmclibs/tinycv-xs.cpp has no in-edge and is missing
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/redhat-linux-build/ppmclibs/tinycv-xs.cpp is dirty
ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/ppmclibs/tinycv.h is dirty
ninja explain: ppmclibs/CMakeFiles/tinycv.dir/tinycv-xs.cpp.o is dirty
ninja explain: ppmclibs/tinycv.so is dirty
ninja explain: ppmclibs/all is dirty
ninja explain: output systemd/all of phony edge with no inputs doesn't exist
ninja explain: systemd/all is dirty

Then it tries to build stuff and fails because RPM_ARCH isn't set.

Here's f40:

<mock-chroot> sh-5.2# cd /builddir/build/BUILD/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7
<mock-chroot> sh-5.2# ninja -C redhat-linux-build -d explain
ninja: Entering directory `redhat-linux-build'
ninja explain: output systemd/all of phony edge with no inputs doesn't exist
ninja explain: systemd/all is dirty
ninja: no work to do.
<mock-chroot> sh-5.2# 

Note: the difference between the build directories - /builddir/build/BUILD/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7 on F40, /builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7 on Rawhide - is because of a change in RPM 4.20 - https://github.com/rpm-software-management/rpm/pull/2885 . Not sure if it has anything to do with this.

Comment 1 Ben Boeckel 2024-06-11 21:39:44 UTC
I don't think there's anything `ninja` can do about this; it sees that paths are not the same and does what it sees as necessary. CMake doesn't create relocatable builds, so this is probably not going to work out well from that side either. One possible solution is to use the `install/fast` target which performs the installation without considering the `all` target (and therefore basically just runs the `install` commands).

Note that CMake does support a "translation map" which preserves some symlink names when `$PWD` is set up properly (as if a shell had `cd`'d there). With that, a stable name could be used to point to the per-build path (but would have to reside outside of it somewhere).

Reassigning to `rpm` for further triage. I suspect some macros may need to change.

Comment 2 Adam Williamson 2024-06-11 22:40:19 UTC
I don't quite get what you mean by "paths are not the same", though. They aren't the same between 40 and 41, but *within a single package build on 41*, the paths should be the same each time ninja is called.

Comment 3 Panu Matilainen 2024-06-12 06:23:26 UTC
The paths in the log are inconsistent, see eg

> <mock-chroot> sh-5.2# cd /builddir/build/BUILD/os-autoinst-4.6^20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/

This is the full path as expected

> <mock-chroot> sh-5.2# ninja -C redhat-linux-build -d explain
> ninja: Entering directory `redhat-linux-build'
> ninja explain: /builddir/build/BUILD/os-autoinst-4.6 has no in-edge and is missing
      
But this looks like the old style build directory all of a sudden. No wonder its missing.

> ninja explain: 20240604git6646558-build/os-autoinst-664655866ebd886b2bb9163bcec0ca6b371cccc7/videoencoder.cpp has no in-edge and is missing

...and where did this come from?

Would've taken me a whole lot more staring at this if it hadn't been for https://github.com/rpm-software-management/rpm/issues/3159 reported overnight: this seems to be the caret (^) in the path messing up ninja. 
%setup always defaulted to %{name}-%{version} directories but that practically never included special characters like that.

There may well be a ninja bug in there, but I also think we'll need to sanitize the builddir path in rpm somehow to avoid such special characters.

Comment 4 Panu Matilainen 2024-06-12 06:36:57 UTC
Also makes me think it'd be good to somehow catch accidental building in %install, but dunno if that's possible in any generic way.

Comment 5 Panu Matilainen 2024-06-12 07:48:37 UTC
From rpm POV, we need to avoid special characters in paths whose names we decide, obviously not everything little scriptlet etc is 100% escaped and quoted. I'll rebuild rpm with the patch from the PR once the python sidetag merge finishes, don't want to disturb that process up any more than we already have.

Reassigning back to ninja though: if a generic build system can't deal with ^ and ~ in path names it seems like a bug to me.

Comment 6 Panu Matilainen 2024-06-12 13:46:02 UTC
FWIW, the rpm side is addressed in https://bodhi.fedoraproject.org/updates/FEDORA-2024-4f6ad64973

Comment 7 Ben Boeckel 2024-06-12 14:10:15 UTC
> Also makes me think it'd be good to somehow catch accidental building in %install, but dunno if that's possible in any generic way.

Could the files (not directories) in the build tree be made immutable? This would allow using the build tree for pre-install prep steps yet avoid touching any files that may have been written during the build.

> Reassigning back to ninja though: if a generic build system can't deal with ^ and ~ in path names it seems like a bug to me.

Indeed; I'll take a look at that.

Comment 8 Adam Williamson 2024-06-12 15:35:41 UTC
> Could the files (not directories) in the build tree be made immutable? This would allow using the build tree for pre-install prep steps yet avoid touching any files that may have been written during the build.

I don't think you could do that universally, no, not without breaking stuff. There are specs that delete things from the source tree after %build and before %install to make sure they don't get installed. It's easier and probably more reliable than installing them then trying to wipe the installed copy.

Comment 9 Aoife Moloney 2025-02-26 13:03:40 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 42 development cycle.
Changing version to 42.

Comment 10 Fedora Release Engineering 2026-05-06 11:42:52 UTC
This message is a reminder that Fedora Linux 42 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 42 on 2026-05-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '42'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 42 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 11 Adam Williamson 2026-05-06 18:38:59 UTC
Checking the most recent os-autoinst build logs it looks like this is fixed.


Note You need to log in before you can comment on or make changes to this bug.