RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1719366 - OnCalendar timers with RandomizedDelaySec != 0 may never run
Summary: OnCalendar timers with RandomizedDelaySec != 0 may never run
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: systemd
Version: 7.6
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: systemd-maint
QA Contact: Frantisek Sumsal
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-11 15:25 UTC by Renaud Métrich
Modified: 2020-11-11 21:46 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1719364
Environment:
Last Closed: 2020-11-11 21:46:31 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 4276781 0 Customize or extend None Units defining OnCalendar timers with RandomizedDelaySec != 0 may never run 2019-07-10 09:47:11 UTC

Description Renaud Métrich 2019-06-11 15:25:21 UTC 
+++ This bug was initially created as a clone of Bug #1719364 +++

Description of problem:

When a OnCalendar timer is used and RandomizedDelaySec is not zero (case of insights-client.timer for example), the timer may never run, typically when systemd regularly reloads its configuration.


Version-Release number of selected component (if applicable):

systemd-219


How reproducible:

Always


Steps to Reproduce:
1. Create a dummy service

# cat << EOF > /etc/systemd/system/my.service
[Unit]
Description=My one-shot service triggered by a timer
[Service]
Type=oneshot
ExecStart=/bin/echo "I'm running"
EOF

2. Create a timer triggered on calendar with random delay (here, every 10 minutes + 5 minutes delay)

# cat << EOF > /etc/systemd/system/my.timer
[Timer]
OnCalendar=*-*-* *:5,15,25,35,45,55:00
RandomizedDelaySec=300
EOF

3. Set debug level and reload systemd

# systemd-analyze set-log-level debug
# systemctl daemon-reload

4. Follow the journal and start the timer

# journalctl --follow -u my.service -u my.timer &
# systemctl start my.timer

5. Just before timer elapses (but after expected time, issue systemctl daemon-reload)

Journal:

Jun 11 17:08:24 vm-rhel8 systemd[1]: my.timer: Adding 3min 35.811652s random time.
Jun 11 17:08:24 vm-rhel8 systemd[1]: my.timer: Realtime timer elapses at Tue 2019-06-11 17:18:35 CEST.

--> issue "systemctl daemon-reload" between 17:15 and 17:18
In the example below, it was done at 17:16:

Jun 11 17:16:01 vm-rhel8 systemd[1]: my.timer: Adding 4min 19.759514s random time.
Jun 11 17:16:01 vm-rhel8 systemd[1]: my.timer: Realtime timer elapses at Tue 2019-06-11 17:29:19 CEST.

--> service will not run before 17:29

...

Additional info:

This is a huge issue for timer elapsing only once a day, such as insights-client.timer which runs Daily + 4 hours:
as soon as a daemon-reload occurs between midnight and 4 o'clock, the timer won't elapse until next day.
Comment 2 David Tardon 2020-04-28 06:31:34 UTC 
Doing a lot of reloads is not a typical scenario, I assume...
Comment 3 RHEL Program Management 2020-04-28 06:31:42 UTC 
Development Management has reviewed and declined this request. You may appeal this decision by using your Red Hat support channels, who will make certain  the issue receives the proper prioritization with product and development management.

https://www.redhat.com/support/process/production/#howto
Comment 4 Renaud Métrich 2020-04-28 07:05:09 UTC 
I disagree with the decision, this bug is hit regularly by our customers having insights-client installed.

The insights-client.timer unit is configured with this:

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
[Timer]
OnCalendar=daily
RandomizedDelaySec=14400
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

"daily" means every midnight 00:00:00 from systemd.time(7)
14400 delay is 4 hours

With this setup, it is enough that *one* reload happens some time after midnight for insights-client to not run that day, something that customers notice through their monitoring tools.
Usually automatic administration (ansible, puppet) is done during the night during these hours, this kind of administration tasks definitely causes spurious reloads leading to the issue.

Depending on the administration task performed, we may take a very long time, the spurious reloads can lead to not executing insights-client timer at all, as shown in the example below:

1. Timestamp 00:00 (midnight): say systemd configured insights-client to run at 2 o'clock (middle of the interval)
2. Timestamp 01:30: admin task provokes a reload implicitly (e.g. because a "systemctl enable foo" is executed, *even* when foo was already enabled ... systemctl *will* reload in any case)
  -> systemd reconfigures insights-client to run NEXT DAY (because we are past midnight, so next midnight, being the reference for timer computation will be the day after)

  -> insights-client not executing today

Since everyday the admin task will reload implicitly, insights-clients will *never* run.

Hence, please reconsider.
Comment 6 Chris Williams 2020-11-11 21:46:31 UTC 
Red Hat Enterprise Linux 7 shipped it's final minor release on September 29th, 2020. 7.9 was the last minor releases scheduled for RHEL 7.
From intial triage it does not appear the remaining Bugzillas meet the inclusion criteria for Maintenance Phase 2 and will now be closed. 

From the RHEL life cycle page:
https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_2_Phase
"During Maintenance Support 2 Phase for Red Hat Enterprise Linux version 7,Red Hat defined Critical and Important impact Security Advisories (RHSAs) and selected (at Red Hat discretion) Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available."

If this BZ was closed in error and meets the above criteria please re-open it flag for 7.9.z, provide suitable business and technical justifications, and follow the process for Accelerated Fixes:
https://source.redhat.com/groups/public/pnt-cxno/pnt_customer_experience_and_operations_wiki/support_delivery_accelerated_fix_release_handbook  

Feature Requests can re-opened and moved to RHEL 8 if the desired functionality is not already present in the product. 

Please reach out to the applicable Product Experience Engineer[0] if you have any questions or concerns.  

[0] https://bugzilla.redhat.com/page.cgi?id=agile_component_mapping.html&product=Red+Hat+Enterprise+Linux+7
Note You need to log in before you can comment on or make changes to this bug.