The kernel-debug-devel-5.14.20-300.fc35.x86_64 generate more 100000 error message with hardline error because it try to hardline to the other installed kernel sources: From /var/log/dnf.rpm.log.1: 2021-11-20T09:58:14+0100 INFO hardlink: cannot link ./Documentation/devicetree/bindings/Makefile to /usr/src/kernels/5.14.18-300.fc35.x86_64+debug/./Documentation/devicetree/bindings/Makefile.hardlink-temporary: File exists hardlink: cannot link ./Documentation/devicetree/bindings/Makefile to /usr/src/kernels/5.14.17-301.fc35.x86_64+debug/./Documentation/devicetree/bindings/Makefile.hardlink-temporary: File exists hardlink: cannot link ./Documentation/devicetree/bindings/Makefile to /usr/src/kernels/5.14.16-301.fc35.x86_64+debug/./Documentation/devicetree/bindings/Makefile.hardlink-temporary: File exists hardlink: cannot link /usr/src/kernels/5.14.18-300.fc35.x86_64+debug/./Documentation/devicetree/bindings/Makefile to /usr/src/kernels/5.14.17-301.fc35.x86_64+debug/./Documentation/devicetree/bindings/Makefile.hardlink-temporary: File exists hardlink: cannot link /usr/src/kernels/5.14.18-300.fc35.x86_64+debug/./Documentation/devicetree/bindings/Makefile to /usr/src/kernels/5.14.16-301.fc35.x86_64+debug/./Documentation/devicetree/bindings/Makefile.hardlink-temporary: File exists hardlink: cannot link /usr/src/kernels/5.14.17-301.fc35.x86_64+debug/./Documentation/devicetree/bindings/Makefile to /usr/src/kernels/5.14.16-301.fc35.x86_64+debug/./Documentation/devicetree/bindings/Makefile.hardlink-temporary: File exists -rw-r--r--. 1 root root 12664771 Nov 20 09:58 /var/log/dnf.rpm.log.1 -rw-r--r--. 1 root root 82 Nov 20 09:57 /var/log/dnf.rpm.log.2 -rw-r--r--. 1 root root 12664771 Nov 20 09:57 /var/log/dnf.rpm.log.3 -rw-r--r--. 1 root root 41601 Nov 20 09:57 /var/log/dnf.rpm.log.4 wc -l /var/log/dnf.rpm.log.? 65415 /var/log/dnf.rpm.log.1 1 /var/log/dnf.rpm.log.2 65415 /var/log/dnf.rpm.log.3 533 /var/log/dnf.rpm.log.4 131364 total
On my system, after a `dnf upgrade` that just installed kernel-debug-devel-5.15.7-200.fc35.x86_64: $ cd /usr/src/kernels $ find . -iname '*.hardlink-temporary'|wc -l 22143 (Which then becomes the cause of said 100,000 errors.) It seems like the postinstall scriptlets for kernel-devel and kernel-debug-devel... $ rpm -q --scripts kernel-debug-devel-5.15.7-200.fc35.x86_64 postinstall scriptlet (using /bin/sh): if [ -f /etc/sysconfig/kernel ] then . /etc/sysconfig/kernel || exit $? fi if [ "$HARDLINK" != "no" -a -x /usr/bin/hardlink -a ! -e /run/ostree-booted ] then (cd /usr/src/kernels/5.15.7-200.fc35.x86_64+debug && /usr/bin/find . -type f | while read f; do hardlink -c /usr/src/kernels/*.fc35.*/$f $f > /dev/null done) fi $ rpm -q --scripts kernel-devel-5.15.7-200.fc35.x86_64 postinstall scriptlet (using /bin/sh): if [ -f /etc/sysconfig/kernel ] then . /etc/sysconfig/kernel || exit $? fi if [ "$HARDLINK" != "no" -a -x /usr/bin/hardlink -a ! -e /run/ostree-booted ] then (cd /usr/src/kernels/5.15.7-200.fc35.x86_64 && /usr/bin/find . -type f | while read f; do hardlink -c /usr/src/kernels/*.fc35.*/$f $f > /dev/null done) fi ...Will end up trying to perform effectively the same cleanup (consolidate all copies of $f in all kernel config trees *.fc35.* under /usr/src/kernels/, by converting them to hardlinks), and in the process trample all over each other. Seems like that glob is a dangerous thing to have in two separate scriptlets from the same transaction. They're also both doing an awful lot of "stuff" that could arguably be expressed better as simply: cd /usr/src/kernels && hardlink -q -c *.fc35.* That'll recursively hardlink all identical files in all of the kernel trees, efficiently and without launching thousands of separate hardlink processes. If two files at different paths happen to have identical contents, they'll be consolidated, and that's... probably fine? (A -f flag could always be added to the hardlink call, to ensure they have identical filenames as well. Though, since 90% of the filenames in question are either 'Makefile' or 'Kconfig' that may not actually matter much in practice.)
On the topic of efficiency, manually running the recursive hardlink command I suggested on my own SSD-housed /usr/src/kernels tree takes all of 1.4 seconds to consolidate three kernel-devel trees and three kernel-debug-devel trees: $ rpm -qf *.fc35.* file /usr/src/kernels/5.14.18-300.fc35.x86_64 is not owned by any package kernel-devel-5.15.4-201.fc35.x86_64 kernel-debug-devel-5.15.4-201.fc35.x86_64 kernel-devel-5.15.6-200.fc35.x86_64 kernel-debug-devel-5.15.6-200.fc35.x86_64 kernel-devel-5.15.7-200.fc35.x86_64 kernel-debug-devel-5.15.7-200.fc35.x86_64 $ sudo hardlink -c *.fc35.* Mode: real Files: 105495 Linked: 12666 files Compared: 0 xattrs Compared: 45331 files Saved: 49.4 MiB Duration: 1.374871 seconds Sure, that's reaping the benefits of a hot cache... cold, it might take all of 3 seconds. The current scriptlet takes... oh my, quite a bit longer, even with the same hot cache. $ sudo -s # cd /usr/src/kernels/5.15.7-200.fc35.x86_64 # time (/usr/bin/find . -type f | while read f; do hardlink -c /usr/src/kernels/*.fc35.*/$f $f > /dev/null; done) ( /usr/bin/find . -type f | while read f; do; hardlink -c $f > /dev/null; ; ) 6.59s user 18.33s system 104% cpu 23.764 total
Same happened to me for upgrade to kernel-5.15.8 but not on subsequent upgrade to kernel-5.15.10 Excerpt from dnf.rpm.log Installing : kernel-devel-5.15.8-200.fc35.x86_64 103/212 Running scriptlet: kernel-devel-5.15.8-200.fc35.x86_64 103/212 hardlink: cannot link ./Documentation/Kconfig to /usr/src/kernels/5.15.6-200.fc35.x86_64/./Documentation/Kconfig.hardlink-temporary: File exists hardlink: cannot link ./Documentation/Makefile to /usr/src/kernels/5.15.6-200.fc35.x86_64/./Documentation/Makefile.hardlink-temporary: File exists hardlink: cannot link ./lib/kunit/Kconfig to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/kunit/Kconfig.hardlink-temporary: File exists hardlink: cannot link ./lib/kunit/Makefile to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/kunit/Makefile.hardlink-temporary: File exists hardlink: cannot link ./lib/Kconfig.kfence to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/Kconfig.kfence.hardlink-temporary: File exists hardlink: cannot link ./lib/Kconfig.ubsan to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/Kconfig.ubsan.hardlink-temporary: File exists hardlink: cannot link ./lib/mpi/Makefile to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/mpi/Makefile.hardlink-temporary: File exists hardlink: cannot link ./lib/crypto/Kconfig to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/crypto/Kconfig.hardlink-temporary: File exists
I just upgraded from kernel kernel-5.15.11-200.fc35.x86_64 to kernel-5.15.12-200.fc35.x86_64 and got the same thing. Additionally, dnf hung when doing the final verification. The process status was reported as "disk sleep". This is the second time it's happened and required that I use a kill -9 to recover. I'm not sure this is a related problem but this happened around the same time so it makes me suspicious.
*** Bug 2028156 has been marked as a duplicate of this bug. ***
I just stumbled upon the same (geez when does one *watch* a dnf upgrade) and have find /usr/src/kernels/ -name '*.hardlink-temporary' |wc -l 33228 of these.. Meanwhile of kernels find /usr/src/kernels/ -name '*.hardlink-temporary' |sed -Ee 's#.*/kernels/([^/]+)/.*#\1#' |sort -u 5.14.17-301.fc35.x86_64 5.15.10-200.fc35.x86_64+debug 5.15.12-200.fc35.x86_64+debug 5.15.13-200.fc35.x86_64+debug 5.15.4-201.fc35.x86_64 5.15.6-200.fc35.x86_64
Feels like the best solution might be to throw a quick HARDLINK=no into /etc/sysconfig/kernel, and manage the hardlinks manually. (Or just deal with the kernel trees consuming an extra 100MB or so of disk. Realistically, these days that's nothing, and hardly worth the trouble.) Dnf's post-transaction scripting when it does kernel installs will go _way_ faster, as a bonus. The current scriptlet wastes nearly 30 seconds even on an SSD.
For me, dnf waits for _minutes_ (on an SSD) at the end of a transaction, writing furiously to disk. Some digging around in the process seems to indicate that dnf is writing every single output log line to the history database, one line at a time, each line in one transaction. That, in itself, is maybe not a smart way to do this, but it's definitely made a lot worse by this bug.
Yes, history seems to be problem as well. Already consuming 0.5 GB for me and I don't want to erase it completely: $ du -m /var/lib/dnf/history.sqlite 461 /var/lib/dnf/history.sqlite
I've found the issue: this is caused by what appears to be a bug in hardlink that results in it to leaving .hardlink-temporary files around if a path is specified multiple times: $ echo value >f1 ; echo value >f2 ; echo value >f3 ; stat -c '%n %i' f* f1 35261410 f2 35261411 f3 35261416 $ hardlink -c f1 f2 f3 f3 Mode: real Files: 4 Linked: 3 files Compared: 0 xattrs Compared: 2 files Saved: 12 B Duration: 0.000145 seconds $ stat -c '%n %i' f* f1 35261410 f2 35261410 f3 35261410 f3.hardlink-temporary 35261410 $ rm f* $ echo value >f1 ; echo value >f2 ; echo value >f3 ; stat -c '%n %i' f* f1 35261421 f2 35261423 f3 35261425 $ hardlink -c f1 f2 f3 Mode: real Files: 3 Linked: 2 files Compared: 0 xattrs Compared: 2 files Saved: 12 B Duration: 0.000100 seconds $ stat -c '%n %i' f* f1 35261421 f2 35261421 f3 35261421 The RPM scriptlets are currently passing in the new file twice: hardlink -c /usr/src/kernels/*.fc35.*/$f $f > /dev/null Removing the lone $f from the RPM scriptlets will prevent this from happening moving forward. Additionally, it may be prudent to temporarily add a cleanup step that runs the following command first: /usr/bin/find /usr/src/kernels -type f -name '*.hardlink-temporary' -delete I have reported this bug to upstream at: https://github.com/util-linux/util-linux/issues/1602
The bug causing this has been fixed by upstream in their development branch. In the meantime, we should probably fix the RPM scriptlets, as this is going to continue to cause issues for users.
4 months passed and the error still exists. Please describe the temporary workaround for common users like me. Step by step.
See https://bugzilla.redhat.com/show_bug.cgi?id=2025157#c7
I have reported the bug with the RPM scriptlet to upstream: https://gitlab.com/cki-project/kernel-ark/-/issues/79
(In reply to appdevsw from comment #12) > 4 months passed and the error still exists. > Please describe the temporary workaround for common users like me. Step by > step. 9 months. This happened to me for the first time (AFAIK) a couple of weeks ago, with 5.19.4-100.fc35.x86_64
Fwiw, I ended up running this command after dnf upgrade adding new kernel sources: # Bug leaving *.hardlink-temporary files behind during updates. find /usr/src/kernels/5.1* -name '*.hardlink-temporary' -exec rename -v .hardlink-temporary '' {} \; Which of course takes even more time ... one could omit the -v verbosity parameter for saving a little. Though https://gitlab.com/cki-project/kernel-ark/-/issues/79 says this /usr/bin/find /usr/src/kernels -type f -name '*.hardlink-temporary' -delete would do, I'm not so sure about the *extraneous* mentioned there and kept it cautious. The issue says it was addressed with merge https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1811 3 months ago.
Erm.. rename followed by the -delete of course to remove the extraneous..
(In reply to Eike Rathke from comment #16) [...] > /usr/bin/find /usr/src/kernels -type f -name '*.hardlink-temporary' -delete Not seeing any "*.hardlink-temporary" in /usr/src/kernels (& subfolders), maybe they got nuked when I deleted 5.19.4-100.fc35.x86_64 (etc) after it kernel-panicked on boot?
This message is a reminder that Fedora Linux 35 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '35'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 35 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Fedora Linux 35 entered end-of-life (EOL) status on 2022-12-13. Fedora Linux 35 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora Linux please feel free to reopen this bug against that version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see the version field. If you are unable to reopen this bug, please file a new report against an active release. Thank you for reporting this bug and we are sorry it could not be fixed.