This bug was initially created as a copy of Bug #1719364

I am copying this bug because: 
It's still broken in RHEL-9.


Description of problem:

When a OnCalendar timer is used and RandomizedDelaySec is not zero (case of insights-client.timer for example), the timer may never run, typically when systemd regularly reloads its configuration.


Version-Release number of selected component (if applicable):

systemd-239-13.el8_0.3


How reproducible:

Always


Steps to Reproduce:
1. Create a dummy service

# cat << EOF > /etc/systemd/system/my.service
[Unit]
Description=My one-shot service triggered by a timer
[Service]
Type=oneshot
ExecStart=/bin/echo "I'm running"
EOF

2. Create a timer triggered on calendar with random delay (here, every 10 minutes + 5 minutes delay)

# cat << EOF > /etc/systemd/system/my.timer
[Timer]
OnCalendar=*-*-* *:5,15,25,35,45,55:00
RandomizedDelaySec=300
EOF

3. Set debug level and reload systemd

# systemd-analyze set-log-level debug
# systemctl daemon-reload

4. Follow the journal and start the timer

# journalctl --follow -u my.service -u my.timer &
# systemctl start my.timer

5. Just before timer elapses (but after expected time, issue systemctl daemon-reload)

Journal:

Jun 11 17:08:24 vm-rhel8 systemd[1]: my.timer: Adding 3min 35.811652s random time.
Jun 11 17:08:24 vm-rhel8 systemd[1]: my.timer: Realtime timer elapses at Tue 2019-06-11 17:18:35 CEST.

--> issue "systemctl daemon-reload" between 17:15 and 17:18
In the example below, it was done at 17:16:

Jun 11 17:16:01 vm-rhel8 systemd[1]: my.timer: Adding 4min 19.759514s random time.
Jun 11 17:16:01 vm-rhel8 systemd[1]: my.timer: Realtime timer elapses at Tue 2019-06-11 17:29:19 CEST.

--> service will not run before 17:29

...

Additional info:

This is a huge issue for timer elapsing only once a day, such as insights-client.timer which runs Daily + 4 hours:
as soon as a daemon-reload occurs between midnight and 4 o'clock, the timer won't elapse until next day.