On my system, after a `dnf upgrade` that just installed kernel-debug-devel-5.15.7-200.fc35.x86_64:

$ cd /usr/src/kernels
$ find . -iname '*.hardlink-temporary'|wc -l
22143

(Which then becomes the cause of said 100,000 errors.)

It seems like the postinstall scriptlets for kernel-devel and kernel-debug-devel...

$ rpm -q --scripts kernel-debug-devel-5.15.7-200.fc35.x86_64
postinstall scriptlet (using /bin/sh):
if [ -f /etc/sysconfig/kernel ]
then
    . /etc/sysconfig/kernel || exit $?
fi
if [ "$HARDLINK" != "no" -a -x /usr/bin/hardlink -a ! -e /run/ostree-booted ] 
then
    (cd /usr/src/kernels/5.15.7-200.fc35.x86_64+debug &&
     /usr/bin/find . -type f | while read f; do
       hardlink -c /usr/src/kernels/*.fc35.*/$f $f > /dev/null
     done)
fi

$ rpm -q --scripts kernel-devel-5.15.7-200.fc35.x86_64      
postinstall scriptlet (using /bin/sh):
if [ -f /etc/sysconfig/kernel ]
then
    . /etc/sysconfig/kernel || exit $?
fi
if [ "$HARDLINK" != "no" -a -x /usr/bin/hardlink -a ! -e /run/ostree-booted ] 
then
    (cd /usr/src/kernels/5.15.7-200.fc35.x86_64 &&
     /usr/bin/find . -type f | while read f; do
       hardlink -c /usr/src/kernels/*.fc35.*/$f $f > /dev/null
     done)
fi


...Will end up trying to perform effectively the same cleanup (consolidate all copies of $f in all kernel config trees *.fc35.* under /usr/src/kernels/, by converting them to hardlinks), and in the process trample all over each other. Seems like that glob is a dangerous thing to have in two separate scriptlets from the same transaction. 

They're also both doing an awful lot of "stuff" that could arguably be expressed better as simply:

cd /usr/src/kernels && hardlink -q -c *.fc35.*

That'll recursively hardlink all identical files in all of the kernel trees, efficiently and without launching thousands of separate hardlink processes. If two files at different paths happen to have identical contents, they'll be consolidated, and that's... probably fine? (A -f flag could always be added to the hardlink call, to ensure they have identical filenames as well. Though, since 90% of the filenames in question are either 'Makefile' or 'Kconfig' that may not actually matter much in practice.)

On the topic of efficiency, manually running the recursive hardlink command I suggested on my own SSD-housed /usr/src/kernels tree takes all of 1.4 seconds to consolidate three kernel-devel trees and three kernel-debug-devel trees:

$ rpm -qf *.fc35.*   
file /usr/src/kernels/5.14.18-300.fc35.x86_64 is not owned by any package
kernel-devel-5.15.4-201.fc35.x86_64
kernel-debug-devel-5.15.4-201.fc35.x86_64
kernel-devel-5.15.6-200.fc35.x86_64
kernel-debug-devel-5.15.6-200.fc35.x86_64
kernel-devel-5.15.7-200.fc35.x86_64
kernel-debug-devel-5.15.7-200.fc35.x86_64

$ sudo hardlink -c *.fc35.*
Mode:           real
Files:          105495
Linked:         12666 files
Compared:       0 xattrs
Compared:       45331 files
Saved:          49.4 MiB
Duration:       1.374871 seconds


Sure, that's reaping the benefits of a hot cache... cold, it might take all of 3 seconds.

The current scriptlet takes... oh my, quite a bit longer, even with the same hot cache.

$ sudo -s
# cd /usr/src/kernels/5.15.7-200.fc35.x86_64
# time (/usr/bin/find . -type f | while read f; do hardlink -c /usr/src/kernels/*.fc35.*/$f $f > /dev/null; done)
( /usr/bin/find . -type f | while read f; do; hardlink -c  $f > /dev/null; ; )  6.59s user 18.33s system 104% cpu 23.764 total

Same happened to me for upgrade to kernel-5.15.8 but not on subsequent upgrade to kernel-5.15.10

Excerpt from dnf.rpm.log

Installing : kernel-devel-5.15.8-200.fc35.x86_64 103/212
Running scriptlet: kernel-devel-5.15.8-200.fc35.x86_64 103/212
hardlink: cannot link ./Documentation/Kconfig to /usr/src/kernels/5.15.6-200.fc35.x86_64/./Documentation/Kconfig.hardlink-temporary: File exists
hardlink: cannot link ./Documentation/Makefile to /usr/src/kernels/5.15.6-200.fc35.x86_64/./Documentation/Makefile.hardlink-temporary: File exists
hardlink: cannot link ./lib/kunit/Kconfig to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/kunit/Kconfig.hardlink-temporary: File exists
hardlink: cannot link ./lib/kunit/Makefile to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/kunit/Makefile.hardlink-temporary: File exists
hardlink: cannot link ./lib/Kconfig.kfence to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/Kconfig.kfence.hardlink-temporary: File exists
hardlink: cannot link ./lib/Kconfig.ubsan to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/Kconfig.ubsan.hardlink-temporary: File exists
hardlink: cannot link ./lib/mpi/Makefile to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/mpi/Makefile.hardlink-temporary: File exists
hardlink: cannot link ./lib/crypto/Kconfig to /usr/src/kernels/5.15.6-200.fc35.x86_64/./lib/crypto/Kconfig.hardlink-temporary: File exists

I just upgraded from kernel kernel-5.15.11-200.fc35.x86_64 to kernel-5.15.12-200.fc35.x86_64 and got the same thing.

Additionally, dnf hung when doing the final verification. The process status was reported as "disk sleep". This is the second time it's happened and required that I use a kill -9 to recover. I'm not sure this is a related problem but this happened around the same time so it makes me suspicious.

*** Bug 2028156 has been marked as a duplicate of this bug. ***

I just stumbled upon the same (geez when does one *watch* a dnf upgrade) and have
find /usr/src/kernels/ -name '*.hardlink-temporary' |wc -l
33228
of these..

Meanwhile of kernels
find /usr/src/kernels/ -name '*.hardlink-temporary' |sed -Ee 's#.*/kernels/([^/]+)/.*#\1#' |sort -u
5.14.17-301.fc35.x86_64
5.15.10-200.fc35.x86_64+debug
5.15.12-200.fc35.x86_64+debug
5.15.13-200.fc35.x86_64+debug
5.15.4-201.fc35.x86_64
5.15.6-200.fc35.x86_64

Feels like the best solution might be to throw a quick

HARDLINK=no

into /etc/sysconfig/kernel, and manage the hardlinks manually. (Or just deal with the kernel trees consuming an extra 100MB or so of disk. Realistically, these days that's nothing, and hardly worth the trouble.)

Dnf's post-transaction scripting when it does kernel installs will go _way_ faster, as a bonus. The current scriptlet wastes nearly 30 seconds even on an SSD.

For me, dnf waits for _minutes_ (on an SSD) at the end of a transaction, writing furiously to disk.

Some digging around in the process seems to indicate that dnf is writing every single output log line to the history database, one line at a time, each line in one transaction. That, in itself, is maybe not a smart way to do this, but it's definitely made a lot worse by this bug.

Yes, history seems to be problem as well. Already consuming 0.5 GB for me and I don't want to erase it completely:

$ du -m /var/lib/dnf/history.sqlite
461     /var/lib/dnf/history.sqlite

I've found the issue: this is caused by what appears to be a bug in hardlink that results in it to leaving .hardlink-temporary files around if a path is specified multiple times:

$ echo value >f1 ; echo value >f2 ; echo value >f3 ; stat -c '%n %i' f*
f1 35261410
f2 35261411
f3 35261416
$ hardlink -c f1 f2 f3 f3
Mode:           real
Files:          4
Linked:         3 files
Compared:       0 xattrs
Compared:       2 files
Saved:          12 B
Duration:       0.000145 seconds
$ stat -c '%n %i' f*
f1 35261410
f2 35261410
f3 35261410
f3.hardlink-temporary 35261410
$ rm f*
$ echo value >f1 ; echo value >f2 ; echo value >f3 ; stat -c '%n %i' f*
f1 35261421
f2 35261423
f3 35261425
$ hardlink -c f1 f2 f3
Mode:           real
Files:          3
Linked:         2 files
Compared:       0 xattrs
Compared:       2 files
Saved:          12 B
Duration:       0.000100 seconds
$ stat -c '%n %i' f*
f1 35261421
f2 35261421
f3 35261421

The RPM scriptlets are currently passing in the new file twice:

hardlink -c /usr/src/kernels/*.fc35.*/$f $f > /dev/null

Removing the lone $f from the RPM scriptlets will prevent this from happening moving forward. Additionally, it may be prudent to temporarily add a cleanup step that runs the following command first:

/usr/bin/find /usr/src/kernels -type f -name '*.hardlink-temporary' -delete

I have reported this bug to upstream at: https://github.com/util-linux/util-linux/issues/1602

The bug causing this has been fixed by upstream in their development branch. In the meantime, we should probably fix the RPM scriptlets, as this is going to continue to cause issues for users.

4 months passed and the error still exists.
Please describe the temporary workaround for common users like me. Step by step.

See https://bugzilla.redhat.com/show_bug.cgi?id=2025157#c7

I have reported the bug with the RPM scriptlet to upstream: https://gitlab.com/cki-project/kernel-ark/-/issues/79

(In reply to appdevsw from comment #12)
> 4 months passed and the error still exists.
> Please describe the temporary workaround for common users like me. Step by
> step.

9 months. This happened to me for the first time (AFAIK) a couple of weeks ago, with 5.19.4-100.fc35.x86_64

Fwiw, I ended up running this command after dnf upgrade adding new kernel sources:

# Bug leaving *.hardlink-temporary files behind during updates.
find /usr/src/kernels/5.1* -name '*.hardlink-temporary' -exec rename -v .hardlink-temporary '' {} \;

Which of course takes even more time ... one could omit the -v verbosity parameter for saving a little.

Though https://gitlab.com/cki-project/kernel-ark/-/issues/79 says this

/usr/bin/find /usr/src/kernels -type f -name '*.hardlink-temporary' -delete

would do, I'm not so sure about the *extraneous* mentioned there and kept it cautious.
The issue says it was addressed with merge https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1811 3 months ago.

Erm.. rename followed by the -delete of course to remove the extraneous..

(In reply to Eike Rathke from comment #16)
[...]
> /usr/bin/find /usr/src/kernels -type f -name '*.hardlink-temporary' -delete

Not seeing any "*.hardlink-temporary" in /usr/src/kernels (& subfolders), maybe they got nuked when I deleted 5.19.4-100.fc35.x86_64 (etc) after it kernel-panicked on boot?

This message is a reminder that Fedora Linux 35 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '35'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 35 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Fedora Linux 35 entered end-of-life (EOL) status on 2022-12-13.

Fedora Linux 35 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.