Bug 1614751

Summary:	systemctl daemon-reload is called after "systemctl try-restart" on package upgrades breaking every package where service file is changed
Product:	[Fedora] Fedora	Reporter:	Miroslav Lichvar <mlichvar>
Component:	systemd	Assignee:	systemd-maint
Status:	CLOSED NEXTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	34	CC:	dominik, lnykryn, msekleta, nicolas.mailhot, paul, pmatilai, pzhukov, s, systemd-maint, terje.rosten, tom, zbyszek
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	systemd-248~rc1-2.fc34	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-02-23 18:23:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Miroslav Lichvar 2018-08-10 10:43:06 UTC

Description of problem:
I'm not sure if this is a bug in the systemd triggers, rpm, or something else.

In the chronyd unit file the PIDFile has changed recently, but this change is not respected on package upgrades, which results in a timeout and systemd killing the daemon.

It seems the problem is that the systemd trigger which runs "systemctl daemon-reload" is called after chrony's %postun, which calls "systemctl try-restart", so systemd doesn't see the new PIDFile when the service is restarted.

Version-Release number of selected component (if applicable):
systemd-239-3.fc29.x86_64

How reproducible:
always

Steps to Reproduce:
1. install chrony-3.3-1.fc29
2. start chronyd
3. update to chrony-3.3-4.fc29

Actual results:
slow rpm transaction, timeout, and chronyd is killed

Expected results:
no timeout, chronyd stays running

Additional information:

From an rpm log:

D: %triggerun(systemd-239-3.fc29.x86_64): execv(/bin/sh) pid 7867
D: Plugin: calling hook scriptlet_fork_post in selinux plugin
D: setexecfilecon: (/bin/sh) 
+ test -d /run/systemd/system
+ mkdir -p /var/lib/rpm-state/systemd
+ touch /var/lib/rpm-state/systemd/needs-reload
D: %triggerun(systemd-239-3.fc29.x86_64): waitpid(7867) rc 7867 status 0

...

D: %postun(chrony-3.3-1.fc29.x86_64): execv(/bin/sh) pid 7878
D: Plugin: calling hook scriptlet_fork_post in selinux plugin
D: setexecfilecon: (/bin/sh) 
+ '[' 1 -ge 1 ']'
+ systemctl try-restart chronyd.service
+ :
D: %postun(chrony-3.3-1.fc29.x86_64): waitpid(7878) rc 7878 status 0

...

D: %triggerin(systemd-239-3.fc29.x86_64): execv(/bin/sh) pid 7917
D: Plugin: calling hook scriptlet_fork_post in selinux plugin
D: setexecfilecon: (/bin/sh) 
+ test -d /run/systemd/system
+ /usr/bin/systemctl daemon-reload
D: %triggerin(systemd-239-3.fc29.x86_64): waitpid(7917) rc 7917 status 0

Comment 1 Jan Kurik 2018-08-14 08:37:24 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 29 development cycle.
Changing version to '29'.

Comment 2 Zbigniew Jędrzejewski-Szmek 2019-08-14 14:55:25 UTC

*** Bug 1715737 has been marked as a duplicate of this bug. ***

Comment 3 Ben Cotton 2019-10-31 18:57:19 UTC

This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 4 Miroslav Lichvar 2019-11-04 09:36:01 UTC

I can still reproduce this on Fedora 31, even when I rebuild the new package to use the current systemd macros.

Comment 5 Terje Røsten 2020-05-10 10:50:20 UTC

Just repeating the analysis from  1715737:

 Zbigniew Jędrzejewski-Szmek 2019-05-31 08:34:00 UTC

We recently changes systemd macros to not silence warnings (they used to be redirected to /dev/null, now they are not).
Unfortunately this shows that there is a problem, and the warnings are valid:
'systemctl try-restart' is called during package's %postun, and 'systemctl daemon-reload' is
called later, during %transfiletriggerin. It seems the whole scheme is broken ;(,
and the recent change simply uncovered the error.

So the whole move of to file triggers for "systemctl daemon-reload" is non-functional, this seems like
serious bug to me.

Comment 6 Ben Cotton 2020-11-03 15:01:28 UTC

This message is a reminder that Fedora 31 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '31'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 31 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 7 Zbigniew Jędrzejewski-Szmek 2021-02-05 17:39:03 UTC

https://github.com/systemd/systemd/pull/18481

Comment 8 Ben Cotton 2021-02-09 15:06:28 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.

Comment 9 Zbigniew Jędrzejewski-Szmek 2021-02-23 18:23:13 UTC

systemd now supports marking services for restart and restarting them at the end of the transaction.
From systemd-248-rc1 NEWS entry:

  Units acquired a new property "Markers" that takes a list of zero,
  one or two of the following strings: "needs-reload" and
  "needs-restart". These markers may be set via "systemctl
  set-property". Once a marker is set, "systemctl reload-or-restart
  --marked" may be invoked to execute the operation the units are
  marked for. This is useful for package managers that want to mark
  units for restart/reload while updating, but effect the actual
  operations at a later step at once.

Systemd rpm macros and transfiletriggers have been adjusted to make use of this.

Unfortunately I missed the mass rebuild with this, so macros in individual packages
have not been rebuilt. I'll open a fesco ticket to ask for a target rebuild to update
the macros.

I'll close this here, since $subject is fixed. We need to fix individual packages,
but I don't need to track this in bugzilla.