Bug 2251262
| Summary: | kernel install fails with mirror BTRFS /boot partition with `/run/systemd/inaccessible/blk: Failed to create blkid probe: Permission denied` | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Arcadiy Ivanov <arcadiy> |
| Component: | systemd | Assignee: | systemd-maint |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 39 | CC: | acaringi, accounts+fedora, adscvr, airlied, alciregi, bskeggs, dtardon, fedoraproject, filbranden, hdegoede, hpa, jarod, jforbes, josef, kernel-maint, linville, lnykryn, marcel, masami256, mchehab, mh7596, msekleta, nixuser, ptalbert, ryncsn, steved, systemd-maint, yuwatana, zbyszek |
| Target Milestone: | --- | Keywords: | Regression, Upgrades |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | systemd-254.9-1.fc39 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-02-09 01:25:45 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
``` # ls -la total 0 drwxr-xr-x 3 root root 160 Nov 23 10:32 . drwxr-xr-x 24 root root 580 Nov 23 12:38 .. b--------- 1 root root 0, 0 Nov 23 10:32 blk c--------- 1 root root 0, 0 Nov 23 10:32 chr d--------- 2 root root 40 Nov 23 10:32 dir p--------- 1 root root 0 Nov 23 10:32 fifo ---------- 1 root root 0 Nov 23 10:32 reg s--------- 1 root root 0 Nov 23 10:32 sock ``` The above is in `/run/systemd/inaccessible`, obviously. This is now confirmed on two machines, both upgraded identically from F38->F39. Same Problem. Also upgraded from f38 (Same Problem). Another Laptop with f39 installed new, does not have this Problem. The kernel update on F39 is completely broken now. Removing kernel-core to fix dependency issues also no longer works.
```
# dnf upgrade -y
Last metadata expiration check: 0:00:48 ago on Mon 04 Dec 2023 12:21:53 PM EST.
Dependencies resolved.
============================================================================================================================================================================================================================================
Package Architecture Version Repository Size
============================================================================================================================================================================================================================================
Installing:
kernel x86_64 6.6.3-200.fc39 updates 153 k
kernel-core x86_64 6.6.3-200.fc39 updates 16 M
kernel-modules x86_64 6.6.3-200.fc39 updates 58 M
kernel-modules-core x86_64 6.6.3-200.fc39 updates 32 M
kernel-modules-extra x86_64 6.6.3-200.fc39 updates 2.6 M
kernel-modules-internal x86_64 6.6.3-200.fc39 updates 1.0 M
Upgrading:
acl x86_64 2.3.1-9.fc39 updates 71 k
kernel-headers x86_64 6.6.3-200.fc39 updates 1.6 M
kernel-tools x86_64 6.6.3-200.fc39 updates 312 k
kernel-tools-libs x86_64 6.6.3-200.fc39 updates 28 k
libacl x86_64 2.3.1-9.fc39 updates 23 k
Removing:
kernel x86_64 6.5.12-300.fc39 @updates 0
kernel-core x86_64 6.5.9-200.fc38 @@System 65 M
kernel-core x86_64 6.5.10-200.fc38 @@System 65 M
kernel-core x86_64 6.5.12-300.fc39 @@System 65 M
kernel-modules x86_64 6.5.12-300.fc39 @updates 56 M
kernel-modules-core x86_64 6.5.12-300.fc39 @updates 30 M
kernel-modules-extra x86_64 6.5.12-300.fc39 @updates 2.4 M
kernel-modules-internal x86_64 6.5.12-300.fc39 @updates 804 k
Transaction Summary
============================================================================================================================================================================================================================================
Install 6 Packages
Upgrade 5 Packages
Remove 8 Packages
Total size: 111 M
Downloading Packages:
[SKIPPED] kernel-6.6.3-200.fc39.x86_64.rpm: Already downloaded
[SKIPPED] kernel-core-6.6.3-200.fc39.x86_64.rpm: Already downloaded
[SKIPPED] kernel-modules-6.6.3-200.fc39.x86_64.rpm: Already downloaded
[SKIPPED] kernel-modules-core-6.6.3-200.fc39.x86_64.rpm: Already downloaded
[SKIPPED] kernel-modules-extra-6.6.3-200.fc39.x86_64.rpm: Already downloaded
[SKIPPED] kernel-modules-internal-6.6.3-200.fc39.x86_64.rpm: Already downloaded
[SKIPPED] acl-2.3.1-9.fc39.x86_64.rpm: Already downloaded
[SKIPPED] kernel-headers-6.6.3-200.fc39.x86_64.rpm: Already downloaded
[SKIPPED] kernel-tools-6.6.3-200.fc39.x86_64.rpm: Already downloaded
[SKIPPED] kernel-tools-libs-6.6.3-200.fc39.x86_64.rpm: Already downloaded
[SKIPPED] libacl-2.3.1-9.fc39.x86_64.rpm: Already downloaded
Running transaction check
Error: transaction check vs depsolve:
kernel-modules-core-uname-r = 6.5.12-300.fc39.x86_64 is needed by (installed) kernel-core-6.5.12-300.fc39.x86_64
kernel-modules-core-uname-r = 6.5.12-300.fc39.x86_64 is needed by (installed) kernel-core-6.5.12-300.fc39.x86_64
kernel-modules-core-uname-r = 6.5.12-300.fc39.x86_64 is needed by (installed) kernel-core-6.5.12-300.fc39.x86_64
To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'.
You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue.
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
[root@fw1 admin]# dnf remove kernel\*6\*.fc39\*
Dependencies resolved.
============================================================================================================================================================================================================================================
Package Architecture Version Repository Size
============================================================================================================================================================================================================================================
Removing:
kernel x86_64 6.5.12-300.fc39 @updates 0
kernel x86_64 6.6.2-201.fc39 @updates 0
kernel-core x86_64 6.5.12-300.fc39 @@System 65 M
kernel-core x86_64 6.6.2-201.fc39 @updates 66 M
kernel-headers x86_64 6.6.2-201.fc39 @updates 6.1 M
kernel-modules x86_64 6.5.12-300.fc39 @updates 56 M
kernel-modules x86_64 6.6.2-201.fc39 @updates 56 M
kernel-modules-core x86_64 6.5.12-300.fc39 @updates 30 M
kernel-modules-core x86_64 6.6.2-201.fc39 @updates 31 M
kernel-modules-extra x86_64 6.5.12-300.fc39 @updates 2.4 M
kernel-modules-extra x86_64 6.6.2-201.fc39 @updates 2.4 M
kernel-modules-internal x86_64 6.5.12-300.fc39 @updates 804 k
kernel-modules-internal x86_64 6.6.2-201.fc39 @updates 832 k
kernel-tools x86_64 6.6.2-201.fc39 @updates 884 k
kernel-tools-libs x86_64 6.6.2-201.fc39 @updates 36 k
Removing dependent packages:
gcc x86_64 13.2.1-4.fc39 @updates 95 M
Removing unused dependencies:
binutils x86_64 2.40-13.fc39 @fedora 25 M
binutils-gold x86_64 2.40-13.fc39 @fedora 2.2 M
cpp x86_64 13.2.1-4.fc39 @updates 31 M
glibc-devel x86_64 2.38-11.fc39 @updates 35 k
glibc-headers-x86 noarch 2.38-11.fc39 @updates 2.1 M
libxcrypt-devel x86_64 4.4.36-2.fc39 @fedora 30 k
Transaction Summary
============================================================================================================================================================================================================================================
Remove 22 Packages
Freed space: 472 M
Is this ok [y/N]: N
Operation aborted.
[root@fw1 admin]# dnf remove kernel\*6\*.fc39\* --exclude \*headers\*
Dependencies resolved.
============================================================================================================================================================================================================================================
Package Architecture Version Repository Size
============================================================================================================================================================================================================================================
Removing:
kernel x86_64 6.5.12-300.fc39 @updates 0
kernel x86_64 6.6.2-201.fc39 @updates 0
kernel-core x86_64 6.5.12-300.fc39 @@System 65 M
kernel-core x86_64 6.6.2-201.fc39 @updates 66 M
kernel-modules x86_64 6.5.12-300.fc39 @updates 56 M
kernel-modules x86_64 6.6.2-201.fc39 @updates 56 M
kernel-modules-core x86_64 6.5.12-300.fc39 @updates 30 M
kernel-modules-core x86_64 6.6.2-201.fc39 @updates 31 M
kernel-modules-extra x86_64 6.5.12-300.fc39 @updates 2.4 M
kernel-modules-extra x86_64 6.6.2-201.fc39 @updates 2.4 M
kernel-modules-internal x86_64 6.5.12-300.fc39 @updates 804 k
kernel-modules-internal x86_64 6.6.2-201.fc39 @updates 832 k
kernel-tools x86_64 6.6.2-201.fc39 @updates 884 k
kernel-tools-libs x86_64 6.6.2-201.fc39 @updates 36 k
Transaction Summary
============================================================================================================================================================================================================================================
Remove 14 Packages
Freed space: 311 M
Is this ok [y/N]: y
Running transaction check
Error: transaction check vs depsolve:
kernel-modules-core-uname-r = 6.5.12-300.fc39.x86_64 is needed by (installed) kernel-core-6.5.12-300.fc39.x86_64
kernel-modules-core-uname-r = 6.5.12-300.fc39.x86_64 is needed by (installed) kernel-core-6.5.12-300.fc39.x86_64
kernel-modules-core-uname-r = 6.5.12-300.fc39.x86_64 is needed by (installed) kernel-core-6.5.12-300.fc39.x86_64
To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'.
You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue.
[root@fw1 admin]# uname -a
Linux fw1.fl.home.ivanovy.net 6.5.11-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Nov 8 21:42:06 UTC 2023 x86_64 GNU/Linux
[root@fw1 admin]# ^C reinstall
[root@fw1 admin]# dnf reinstall kernel-core\*
Last metadata expiration check: 0:02:29 ago on Mon 04 Dec 2023 12:21:53 PM EST.
Installed package kernel-core-6.5.9-200.fc38.x86_64 (from @System) not available.
Installed package kernel-core-6.5.10-200.fc38.x86_64 (from @System) not available.
Installed package kernel-core-6.5.11-200.fc38.x86_64 (from updates) not available.
Installed package kernel-core-6.5.12-300.fc39.x86_64 (from @System) not available.
Installed package kernel-core-6.5.12-300.fc39.x86_64 (from @System) not available.
Installed package kernel-core-6.5.12-300.fc39.x86_64 (from @System) not available.
Installed package kernel-core-6.5.12-300.fc39.x86_64 (from @System) not available.
Installed package kernel-core-6.6.2-201.fc39.x86_64 (from updates) not available.
Error: No packages marked for reinstall.
[root@fw1 admin]# rpm -Va --nofiles --nodigest
Unsatisfied dependencies for kernel-core-6.5.9-200.fc38.x86_64:
kernel-modules-core-uname-r = 6.5.9-200.fc38.x86_64 is needed by (installed) kernel-core-6.5.9-200.fc38.x86_64
Unsatisfied dependencies for kernel-core-6.5.10-200.fc38.x86_64:
kernel-modules-core-uname-r = 6.5.10-200.fc38.x86_64 is needed by (installed) kernel-core-6.5.10-200.fc38.x86_64
[root@fw1 admin]# dnf remove kernel-core-6.5.9\* kernel-core-6.5.10\*
Dependencies resolved.
============================================================================================================================================================================================================================================
Package Architecture Version Repository Size
============================================================================================================================================================================================================================================
Removing:
kernel-core x86_64 6.5.9-200.fc38 @@System 65 M
kernel-core x86_64 6.5.10-200.fc38 @@System 65 M
Transaction Summary
============================================================================================================================================================================================================================================
Remove 2 Packages
Freed space: 131 M
Is this ok [y/N]: y
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: kernel-core-6.5.10-200.fc38.x86_64 1/2
/run/systemd/inaccessible/blk: Failed to create blkid probe: Permission denied
error: %preun(kernel-core-6.5.10-200.fc38.x86_64) scriptlet failed, exit status 1
Error in PREUN scriptlet in rpm package kernel-core
Running scriptlet: kernel-core-6.5.9-200.fc38.x86_64 2/2
error: kernel-core-6.5.10-200.fc38.x86_64: erase failed
/run/systemd/inaccessible/blk: Failed to create blkid probe: Permission denied
error: %preun(kernel-core-6.5.9-200.fc38.x86_64) scriptlet failed, exit status 1
Error in PREUN scriptlet in rpm package kernel-core
Verifying : kernel-core-6.5.9-200.fc38.x86_64 1/2
Verifying : kernel-core-6.5.10-200.fc38.x86_64 2/2
Failed:
kernel-core-6.5.9-200.fc38.x86_64 kernel-core-6.5.10-200.fc38.x86_64
Error: Transaction failed
```
The kernel install scripts are owned by systemd-udev. Reassigning to systemd Adding Reddit thread for reference as there is more debug output in the comments there: https://www.reddit.com/r/Fedora/comments/182egi8/kernel6512300fc39x86_64_install_failed_with/ Hopefully fixed by https://github.com/systemd/systemd/pull/30321, though I have not tested it yet. Just for record keeping: the affected systems seem to be btrfs RAID mirrors on /boot partition. (In reply to Marcel Hetzendorfer from comment #5) > Also upgraded from f38 (Same Problem). > > Another Laptop with f39 installed new, does not have this Problem. @mh7596 could you confirm you have a mirror BTRFS on /boot as well? (In reply to Arcadiy Ivanov from comment #11) > (In reply to Marcel Hetzendorfer from comment #5) > > Also upgraded from f38 (Same Problem). > > > > Another Laptop with f39 installed new, does not have this Problem. > > @mh7596 could you confirm you have a mirror BTRFS on /boot as well? Yes i had. as a test i moved the /boot in a small new partition and now no more permission denied, kernel update works The PR has been merged into main. This now needs a backport. This has been backported to 254.9 - https://github.com/systemd/systemd-stable/commits/v254.9 https://github.com/systemd/systemd-stable/commit/724b0a109e474bc25032f8bad33a3a9d1beafa1c https://github.com/systemd/systemd-stable/commit/dde9fd85e3823c87508cfb6b0ef1063c8e0b728e Systemd needs an upgrade in F39. *** Bug 2263106 has been marked as a duplicate of this bug. *** FEDORA-2024-31190af00b (systemd-254.9-1.fc39) has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2024-31190af00b FEDORA-2024-31190af00b (systemd-254.9-1.fc39) has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report. |
Upgraded from F38 to F39 via `dnf system-upgrade download --releasever=39`. After `dnf system-upgrade reboot` noticed the kernel was still from F38. Ran: ``` # dnf reinstall kernel-core\* Last metadata expiration check: 0:08:46 ago on Thu 23 Nov 2023 12:34:50 PM EST. Dependencies resolved. ====================================================================================================================================================================================== Package Architecture Version Repository Size ====================================================================================================================================================================================== Reinstalling: kernel-core x86_64 6.5.12-300.fc39 updates 16 M replacing kernel-core.x86_64 6.5.12-300.fc39 Removing: kernel-core x86_64 6.5.9-200.fc38 @@System 65 M Transaction Summary ====================================================================================================================================================================================== Remove 1 Package Total download size: 16 M Is this ok [y/N]: y Downloading Packages: kernel-core-6.5.12-300.fc39.x86_64.rpm 13 MB/s | 16 MB 00:01 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 9.8 MB/s | 16 MB 00:01 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Reinstalling : kernel-core-6.5.12-300.fc39.x86_64 1/5 Running scriptlet: kernel-core-6.5.12-300.fc39.x86_64 1/5 Running scriptlet: kernel-core-6.5.9-200.fc38.x86_64 2/5 /run/systemd/inaccessible/blk: Failed to create blkid probe: Permission denied error: %preun(kernel-core-6.5.9-200.fc38.x86_64) scriptlet failed, exit status 1 Error in PREUN scriptlet in rpm package kernel-core Running scriptlet: kernel-core-6.5.12-300.fc39.x86_64 3/5 error: kernel-core-6.5.9-200.fc38.x86_64: erase failed /run/systemd/inaccessible/blk: Failed to create blkid probe: Permission denied error: %preun(kernel-core-6.5.12-300.fc39.x86_64) scriptlet failed, exit status 1 Error in PREUN scriptlet in rpm package kernel-core Running scriptlet: kernel-core-6.5.12-300.fc39.x86_64 4/5 error: kernel-core-6.5.12-300.fc39.x86_64: erase failed /run/systemd/inaccessible/blk: Failed to create blkid probe: Permission denied error: %preun(kernel-core-6.5.12-300.fc39.x86_64) scriptlet failed, exit status 1 Error in PREUN scriptlet in rpm package kernel-core Running scriptlet: kernel-core-6.5.12-300.fc39.x86_64 5/5 error: kernel-core-6.5.12-300.fc39.x86_64: erase failed /run/systemd/inaccessible/blk: Failed to create blkid probe: Permission denied error: %preun(kernel-core-6.5.12-300.fc39.x86_64) scriptlet failed, exit status 1 Error in PREUN scriptlet in rpm package kernel-core error: kernel-core-6.5.12-300.fc39.x86_64: erase failed /run/systemd/inaccessible/blk: Failed to create blkid probe: Permission denied warning: %posttrans(kernel-core-6.5.12-300.fc39.x86_64) scriptlet failed, exit status 1 Error in POSTTRANS scriptlet in rpm package kernel-core TransactionItem not found for key: kernel-core-6.5.12-300.fc39.x86_64 Verifying : kernel-core-6.5.12-300.fc39.x86_64 1/4 Verifying : kernel-core-6.5.12-300.fc39.x86_64 2/4 Verifying : kernel-core-6.5.12-300.fc39.x86_64 3/4 Verifying : kernel-core-6.5.9-200.fc38.x86_64 4/4 Reinstalled: kernel-core-6.5.12-300.fc39.x86_64 Failed: kernel-core-6.5.9-200.fc38.x86_64 kernel-core-6.5.12-300.fc39.x86_64 kernel-core-6.5.12-300.fc39.x86_64 Error: Transaction failed ``` Reproducible: Always