Bug 2248838 - Failed to start polkit.service - Authorization Manager
Summary: Failed to start polkit.service - Authorization Manager
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: polkit
Version: 39
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
Assignee: Jan Rybar
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-11-09 08:56 UTC by Ralf Corsepius
Modified: 2023-12-14 11:12 UTC (History)
6 users (show)

Fixed In Version: polkit-123-1.fc39.1
Clone Of:
Environment:
Last Closed: 2023-11-23 01:24:35 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
journalctl -r -b > f38.log (29.81 KB, application/x-xz)
2023-11-09 17:16 UTC, Ralf Corsepius
no flags Details
journalctl -r -b -u polkit > f38.polkit.log (589 bytes, text/plain)
2023-11-09 17:17 UTC, Ralf Corsepius
no flags Details
journalctl -r -b > f39.0.log (31.26 KB, application/x-xz)
2023-11-09 17:18 UTC, Ralf Corsepius
no flags Details
journalctl -r -b -u polkit > f39.0.polkit.log (793 bytes, text/plain)
2023-11-09 17:19 UTC, Ralf Corsepius
no flags Details
journalctl -r -b > f39.1.log (29.38 KB, application/x-xz)
2023-11-09 17:20 UTC, Ralf Corsepius
no flags Details
journalctl -r -b -u polkit > f39.1.polkit.log (264 bytes, text/plain)
2023-11-09 17:21 UTC, Ralf Corsepius
no flags Details
journalctl -r -b > f39.2.log (31.73 KB, application/x-xz)
2023-11-09 17:22 UTC, Ralf Corsepius
no flags Details
journalctl -r -b -u polkit > f39.2.polkit.log (589 bytes, text/plain)
2023-11-09 17:24 UTC, Ralf Corsepius
no flags Details

Description Ralf Corsepius 2023-11-09 08:56:05 UTC
Description of problem:
After upgrading from f38 to f39, polkit stopped working:

# journalctl -r -b | grep polkit.service
Nov 09 09:48:57 lewis systemd[1]: Starting polkit.service - Authorization Manager...
Nov 09 09:48:57 lewis systemd[1]: Failed to start polkit.service - Authorization Manager.
Nov 09 09:48:57 lewis systemd[1]: polkit.service: Failed with result 'timeout'.
Nov 09 09:48:11 lewis systemd[1]: polkit.service: start operation timed out. Terminating.
Nov 09 09:46:41 lewis systemd[1]: Starting polkit.service - Authorization Manager...


Version-Release number of selected component (if applicable):
polkit-123-1.fc39.x86_64

How reproducible:
Deterministic, on 2 different machines.


Actual results:
- The error messages above.
- X11 fails to start.

Expected results:
Function.

Additional info:
No idea about the cause. I never touched nor changes polkit, but this error to me renders F39 completely unusable and gives sufficent reasons to downgrade to f38.

Comment 1 Ralf Corsepius 2023-11-09 09:21:33 UTC
FWIW: Installing
polkit-libs-122-3.fc38.1.x86_64
polkit-122-3.fc38.1.x86_64

on fc39 seems to fix this issue for me.

Comment 2 Jan Rybar 2023-11-09 09:24:33 UTC
Hello, please try the newest polkit and upgraded selinux-policy.  
polkit-123 introduced new utilization of systemd's security sandboxing and changes to selinux-policy had to be made. Those changes should have been present on F39 since August. Investigating right now.

Comment 3 Jan Rybar 2023-11-09 09:33:41 UTC
Can you please provide journal info?
$ sudo journalctl -S today -u polkit.service

and SElinux info:
$ sudo ausearch -ts recent

I presume it is going to be long, so please attach it as files.

Thank you.

Comment 4 Ralf Corsepius 2023-11-09 09:48:36 UTC
(In reply to Jan Rybar from comment #2)
> Hello, please try the newest polkit and upgraded selinux-policy.

Which packages do you want me to try?

- polkit-123-1.fc39 is the newest, I can find. This is the version, which exposes the issues for me.

- I have selinux-policy-39.1-1.fc39 installed. There is a newer version for f40 in koji, but *-39.1-1 is the newest for fc39

Comment 5 Ralf Corsepius 2023-11-09 09:58:44 UTC
Another observation.

As I wrote above, for testing, I temporarily installed polkit-122-3.fc38.1.fc38. When (re-) upgrading to polkit-libs-123-1.fc39, this happened: 

# dnf update --refresh
...
(1/2): polkit-libs-123-1.fc39.x86_64.rpm                                                                  542 kB/s |  64 kB     00:00    
(2/2): polkit-123-1.fc39.x86_64.rpm                                                                       1.2 MB/s | 153 kB     00:00    
...
[Long wait]
...
Job for polkit.service failed because a timeout was exceeded.
See "systemctl status polkit.service" and "journalctl -xeu polkit.service" for details.
...
Upgraded:
  polkit-123-1.fc39.x86_64                                          polkit-libs-123-1.fc39.x86_64                                         
..

Comment 6 Ralf Corsepius 2023-11-09 10:20:15 UTC
Now, things are going to behave bizarre ;)

After the dnf update from #5, the error is gone!

To summarize:
1. After upgrading from f38 to f39, the error occurred.

2. Then downgrading to polkit-122-3.fc38.1 and rebooting caused to error to vanish.

3. Then "dnf update"-ing to polkit-123-1.fc39 exposed the "hanger"

4. After another reboot, the error seems to have gone.


From this, my wild guess is, something might go wrong wrt. polkit during the f38->f39 dist-upgrade, but doesn't go wrong (despite the hanger) when normally "dnf update"-ing from *-122->*-123 on fc39.

Comment 7 Ralf Corsepius 2023-11-09 12:06:04 UTC
Another observation:

I upgraded another system from f38 to f39, intentionally not fooling around with different versions of rpms.

This time, all errors went away upon the 3rd or 4th reboot.

I interpret this, as something is fishy inside of systemd's and polkit's interaction. May-be systemd's services execution order, may-be rpm's installation scripts.

Comment 8 Jan Rybar 2023-11-09 12:14:50 UTC
Without data from journal and audit I can only guess wildly, but I bet this is caused by a weird combination of outdated selinux-policy package at the time of installing newest polkit package.  
I just installed fresh f38, updated it and then upgraded to f38 and no issue like this has emerged.  
Did you update the system before dist-upgrading it?

Comment 9 Jan Rybar 2023-11-09 12:17:46 UTC
(In reply to Jan Rybar from comment #8)
> Without data from journal and audit I can only guess wildly, but I bet this
> is caused by a weird combination of outdated selinux-policy package at the
> time of installing newest polkit package.  
> I just installed fresh f38, updated it and then upgraded to f38 and no issue

... upgraded to f39 and no issue...

> like this has emerged.  
> Did you update the system before dist-upgrading it?

Comment 10 Ralf Corsepius 2023-11-09 15:52:54 UTC
(In reply to Jan Rybar from comment #9)
> Without data from journal and audit
I have some, but due to my experiments, no clean ones, yet.

I'll post them, when I have some.

> > Did you update the system before dist-upgrading it?
Of course, I did.

Particularities about my installations is me heavily using yp, autofs, nfs, lightdm+xfce and using some custom SELinux rules.

Comment 11 Ralf Corsepius 2023-11-09 17:16:07 UTC
Created attachment 1998136 [details]
journalctl -r -b > f38.log

last time booting f38 before upgrade

Comment 12 Ralf Corsepius 2023-11-09 17:17:12 UTC
Created attachment 1998137 [details]
journalctl -r -b -u polkit > f38.polkit.log

Comment 13 Ralf Corsepius 2023-11-09 17:18:40 UTC
Created attachment 1998138 [details]
journalctl -r -b > f39.0.log

Log from 1. time booting f39.

Comment 14 Ralf Corsepius 2023-11-09 17:19:27 UTC
Created attachment 1998139 [details]
journalctl -r -b -u polkit > f39.0.polkit.log

Comment 15 Ralf Corsepius 2023-11-09 17:20:34 UTC
Created attachment 1998140 [details]
journalctl -r -b > f39.1.log

Log from 2. time booting fc39

Comment 16 Ralf Corsepius 2023-11-09 17:21:20 UTC
Created attachment 1998141 [details]
journalctl -r -b -u polkit > f39.1.polkit.log

Comment 17 Ralf Corsepius 2023-11-09 17:22:32 UTC
Created attachment 1998142 [details]
journalctl -r -b > f39.2.log

Log from 3. time booting f39

Comment 18 Ralf Corsepius 2023-11-09 17:24:28 UTC
Created attachment 1998143 [details]
journalctl -r -b -u polkit > f39.2.polkit.log

Comment 19 Ralf Corsepius 2023-11-09 17:32:39 UTC
I've add 4 log having been generated from 4 consecutive boots.

f38.log, the last boot.log before upgrading.
 
f39.0.log, boot.log from the 1st boot after upgrading
f39.1.log, boot.log from the 2nd boot after upgrading
f39.2.log, boot.log from the 3rd boot after upgrading.

The 1st and 2nd boot failed, the 3rd one succeeded.

Comment 20 Jeffrey Walton 2023-11-15 21:09:54 UTC
(In reply to Ralf Corsepius from comment #10)
> (In reply to Jan Rybar from comment #9)
> > Without data from journal and audit
> I have some, but due to my experiments, no clean ones, yet.
> 
> I'll post them, when I have some.
> 
> > > Did you update the system before dist-upgrading it?
> Of course, I did.
> 
> Particularities about my installations is me heavily using yp, autofs, nfs,
> lightdm+xfce and using some custom SELinux rules.

I'm almost afraid to suggest it... Relabel with the latest SELinux policy command is `sudo fixfiles -B onboot`.

But with a lot of customizations, I would be concerned about doing more harm than good.

Comment 21 Ralf Corsepius 2023-11-16 13:21:24 UTC
(In reply to Ralf Corsepius from comment #1)
> FWIW: Installing
> polkit-libs-122-3.fc38.1.x86_64
> polkit-122-3.fc38.1.x86_64
> 
> on fc39 seems to fix this issue for me.

I found *123 works for me, if I replace /usr/lib/systemd/system/polkit.service with *-122's /usr/lib/systemd/system/polkit.service.

As the only difference between *122's and *123's /usr/lib/systemd/system/polkit.service is the "[Service]" section, I started to experiment with settings in *-122's /usr/lib/systemd/system/polkit.service.

Result:  Commenting out 
"IPAddressDeny=any"
fixes my bootup problems.

Obviously, "IPAddressDeny=any" breaks yp/nis.

Comment 22 Jan Rybar 2023-11-20 10:48:22 UTC
Yeah, this little line seems to make a lot of mess, so it's been decided to remove it in upstream.  
https://gitlab.freedesktop.org/polkit/polkit/-/commit/597d3e0d2643c96cbb1c8282066f0b0bc8534b5c

It's probably worth sending a backport to an update.  
Not that it's a common issue though.  

Thank you for your investigation!  
Stay tuned.

Comment 23 Fedora Update System 2023-11-20 14:56:36 UTC
FEDORA-2023-cf201a2c64 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-cf201a2c64

Comment 24 Fedora Update System 2023-11-21 02:31:19 UTC
FEDORA-2023-cf201a2c64 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-cf201a2c64`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-cf201a2c64

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 25 Fedora Update System 2023-11-23 01:24:35 UTC
FEDORA-2023-cf201a2c64 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 26 Bug 32 2023-12-14 09:47:40 UTC
This issue affects Fedora 38 too, when you fresh install using the network installer with the latest packages. GNOME desktop is crippled as a result.

Comment 27 Bug 32 2023-12-14 11:12:26 UTC
 dbus-broker-launch[899]: Invalid user-name in /usr/share/dubs-1/system.d/org.freedesktop.PolicyKit1.conf +16: user="polkitd"


Note You need to log in before you can comment on or make changes to this bug.