On my system, after a `dnf upgrade` that just installed kernel-debug-devel-5.15.7-200.fc35.x86_64:

$ cd /usr/src/kernels
$ find . -iname '*.hardlink-temporary'|wc -l
22143

(Which then becomes the cause of said 100,000 errors.)

It seems like the postinstall scriptlets for kernel-devel and kernel-debug-devel...

$ rpm -q --scripts kernel-debug-devel-5.15.7-200.fc35.x86_64
postinstall scriptlet (using /bin/sh):
if [ -f /etc/sysconfig/kernel ]
then
    . /etc/sysconfig/kernel || exit $?
fi
if [ "$HARDLINK" != "no" -a -x /usr/bin/hardlink -a ! -e /run/ostree-booted ] 
then
    (cd /usr/src/kernels/5.15.7-200.fc35.x86_64+debug &&
     /usr/bin/find . -type f | while read f; do
       hardlink -c /usr/src/kernels/*.fc35.*/$f $f > /dev/null
     done)
fi

$ rpm -q --scripts kernel-devel-5.15.7-200.fc35.x86_64      
postinstall scriptlet (using /bin/sh):
if [ -f /etc/sysconfig/kernel ]
then
    . /etc/sysconfig/kernel || exit $?
fi
if [ "$HARDLINK" != "no" -a -x /usr/bin/hardlink -a ! -e /run/ostree-booted ] 
then
    (cd /usr/src/kernels/5.15.7-200.fc35.x86_64 &&
     /usr/bin/find . -type f | while read f; do
       hardlink -c /usr/src/kernels/*.fc35.*/$f $f > /dev/null
     done)
fi


...Will end up trying to perform effectively the same cleanup (consolidate all copies of $f in all kernel config trees *.fc35.* under /usr/src/kernels/, by converting them to hardlinks), and in the process trample all over each other. Seems like that glob is a dangerous thing to have in two separate scriptlets from the same transaction. 

They're also both doing an awful lot of "stuff" that could arguably be expressed better as simply:

cd /usr/src/kernels && hardlink -q -c *.fc35.*

That'll recursively hardlink all identical files in all of the kernel trees, efficiently and without launching thousands of separate hardlink processes. If two files at different paths happen to have identical contents, they'll be consolidated, and that's... probably fine? (A -f flag could always be added to the hardlink call, to ensure they have identical filenames as well. Though, since 90% of the filenames in question are either 'Makefile' or 'Kconfig' that may not actually matter much in practice.)

On the topic of efficiency, manually running the recursive hardlink command I suggested on my own SSD-housed /usr/src/kernels tree takes all of 1.4 seconds to consolidate three kernel-devel trees and three kernel-debug-devel trees:

$ rpm -qf *.fc35.*   
file /usr/src/kernels/5.14.18-300.fc35.x86_64 is not owned by any package
kernel-devel-5.15.4-201.fc35.x86_64
kernel-debug-devel-5.15.4-201.fc35.x86_64
kernel-devel-5.15.6-200.fc35.x86_64
kernel-debug-devel-5.15.6-200.fc35.x86_64
kernel-devel-5.15.7-200.fc35.x86_64
kernel-debug-devel-5.15.7-200.fc35.x86_64

$ sudo hardlink -c *.fc35.*
Mode:           real
Files:          105495
Linked:         12666 files
Compared:       0 xattrs
Compared:       45331 files
Saved:          49.4 MiB
Duration:       1.374871 seconds


Sure, that's reaping the benefits of a hot cache... cold, it might take all of 3 seconds.

The current scriptlet takes... oh my, quite a bit longer, even with the same hot cache.

$ sudo -s
# cd /usr/src/kernels/5.15.7-200.fc35.x86_64
# time (/usr/bin/find . -type f | while read f; do hardlink -c /usr/src/kernels/*.fc35.*/$f $f > /dev/null; done)
( /usr/bin/find . -type f | while read f; do; hardlink -c  $f > /dev/null; ; )  6.59s user 18.33s system 104% cpu 23.764 total

Same happened to me for upgrade to kernel-5.15.8 but not on subsequent upgrade to kernel-5.15.10

Excerpt from dnf.rpm.log

Installing : kernel-devel-5.15.8-200.fc35.x86_64 103/212
Running scriptlet: kernel-devel-5.15.8-200.fc35.x86_64 103/212
hardlink: cannot link ./Documentation/Kconfig to /usr/src/kernels/5.15.6-200.fc35.x86_64/./Documentation/Kconfig.hardlink-temporary: File exists
hardlink: cannot link ./Documentation/Makefile to /usr/src/kernels/5.15.6-200.fc35.x86_64/./Documentation/Makefile.hardlink-temporary: File exists
hardlink: cannot link ./lib/kunit/Kconfig to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/kunit/Kconfig.hardlink-temporary: File exists
hardlink: cannot link ./lib/kunit/Makefile to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/kunit/Makefile.hardlink-temporary: File exists
hardlink: cannot link ./lib/Kconfig.kfence to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/Kconfig.kfence.hardlink-temporary: File exists
hardlink: cannot link ./lib/Kconfig.ubsan to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/Kconfig.ubsan.hardlink-temporary: File exists
hardlink: cannot link ./lib/mpi/Makefile to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/mpi/Makefile.hardlink-temporary: File exists
hardlink: cannot link ./lib/crypto/Kconfig to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/crypto/Kconfig.hardlink-temporary: File exists

I just upgraded from kernel kernel-5.15.11-200.fc35.x86_64 to kernel-5.15.12-200.fc35.x86_64 and got the same thing.

Additionally, dnf hung when doing the final verification. The process status was reported as "disk sleep". This is the second time it's happened and required that I use a kill -9 to recover. I'm not sure this is a related problem but this happened around the same time so it makes me suspicious.

*** Bug 2028156 has been marked as a duplicate of this bug. ***

I just stumbled upon the same (geez when does one *watch* a dnf upgrade) and have
find /usr/src/kernels/ -name '*.hardlink-temporary' |wc -l
33228
of these..

Meanwhile of kernels
find /usr/src/kernels/ -name '*.hardlink-temporary' |sed -Ee 's#.*/kernels/([^/]+)/.*#\1#' |sort -u
5.14.17-301.fc35.x86_64
5.15.10-200.fc35.x86_64+debug
5.15.12-200.fc35.x86_64+debug
5.15.13-200.fc35.x86_64+debug
5.15.4-201.fc35.x86_64
5.15.6-200.fc35.x86_64

Feels like the best solution might be to throw a quick

HARDLINK=no

into /etc/sysconfig/kernel, and manage the hardlinks manually. (Or just deal with the kernel trees consuming an extra 100MB or so of disk. Realistically, these days that's nothing, and hardly worth the trouble.)

Dnf's post-transaction scripting when it does kernel installs will go _way_ faster, as a bonus. The current scriptlet wastes nearly 30 seconds even on an SSD.

For me, dnf waits for _minutes_ (on an SSD) at the end of a transaction, writing furiously to disk.

Some digging around in the process seems to indicate that dnf is writing every single output log line to the history database, one line at a time, each line in one transaction. That, in itself, is maybe not a smart way to do this, but it's definitely made a lot worse by this bug.

Yes, history seems to be problem as well. Already consuming 0.5 GB for me and I don't want to erase it completely:

$ du -m /var/lib/dnf/history.sqlite
461     /var/lib/dnf/history.sqlite

I've found the issue: this is caused by what appears to be a bug in hardlink that results in it to leaving .hardlink-temporary files around if a path is specified multiple times:

$ echo value >f1 ; echo value >f2 ; echo value >f3 ; stat -c '%n %i' f*
f1 35261410
f2 35261411
f3 35261416
$ hardlink -c f1 f2 f3 f3
Mode:           real
Files:          4
Linked:         3 files
Compared:       0 xattrs
Compared:       2 files
Saved:          12 B
Duration:       0.000145 seconds
$ stat -c '%n %i' f*
f1 35261410
f2 35261410
f3 35261410
f3.hardlink-temporary 35261410
$ rm f*
$ echo value >f1 ; echo value >f2 ; echo value >f3 ; stat -c '%n %i' f*
f1 35261421
f2 35261423
f3 35261425
$ hardlink -c f1 f2 f3
Mode:           real
Files:          3
Linked:         2 files
Compared:       0 xattrs
Compared:       2 files
Saved:          12 B
Duration:       0.000100 seconds
$ stat -c '%n %i' f*
f1 35261421
f2 35261421
f3 35261421

The RPM scriptlets are currently passing in the new file twice:

hardlink -c /usr/src/kernels/*.fc35.*/$f $f > /dev/null

Removing the lone $f from the RPM scriptlets will prevent this from happening moving forward. Additionally, it may be prudent to temporarily add a cleanup step that runs the following command first:

/usr/bin/find /usr/src/kernels -type f -name '*.hardlink-temporary' -delete

I have reported this bug to upstream at: https://github.com/util-linux/util-linux/issues/1602

The bug causing this has been fixed by upstream in their development branch. In the meantime, we should probably fix the RPM scriptlets, as this is going to continue to cause issues for users.

4 months passed and the error still exists.
Please describe the temporary workaround for common users like me. Step by step.

See https://bugzilla.redhat.com/show_bug.cgi?id=2025157#c7

I have reported the bug with the RPM scriptlet to upstream: https://gitlab.com/cki-project/kernel-ark/-/issues/79