Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2230187

Summary: RHEL 4 Edge /usr modify timestamp not updated
Product: Red Hat Enterprise Linux 9 Reporter: ckrell
Component: rpm-ostreeAssignee: RHCOS SST <rhcos-sst>
Status: CLOSED MIGRATED QA Contact: RHCOS SST QE <rhcos-sst-qe>
Severity: medium Docs Contact:
Priority: medium    
Version: 9.2CC: travier
Target Milestone: rcKeywords: MigratedToJIRA
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-13 10:48:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description ckrell 2023-08-08 20:39:07 UTC
Description of problem:

On edge device deployment and after updates using rpm-ostree, the modify timestamp for /usr stays at 1969-12-31 18:00:00.000000000 -0600 while the other stat times update appropriately. 

This impacts the functionality of systemd as systemd units relying on ConditionNeedsUpdate may not trigger correctly as this relies on an update of /usr mtime.

Version-Release number of selected component (if applicable):
rpm-ostree

How reproducible:
Everytime -

Steps to Reproduce:
1. Install a RHEL 4 Edge System
2. `stat /usr`
3. Update RHEL 4 Edge System using rpm-ostree
4. `stat /usr`

Actual results:
~~~
# stat /usr
  File: /usr
  Size: 155       Blocks: 0          IO Block: 4096   directory
Device: fd00h/64768dInode: 201333897   Links: 12
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:usr_t:s0
Access: 2023-08-07 12:58:26.783227201 -0500
Modify: 1969-12-31 18:00:00.000000000 -0600
Change: 2023-08-07 12:58:27.959240721 -0500
Birth: 2023-08-07 12:58:26.783227201 -0500
~~~
Expected results:
The expectation is that the mtime is updated accordingly similar to a normal RHEL system update. 

Additional info:


Verifying on my own RHEL 4 Edge test system:

~~~
$ stat /usr
  File: /usr
  Size: 155       	Blocks: 0          IO Block: 4096   directory
Device: fc03h/64515d	Inode: 19570154    Links: 12
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:usr_t:s0
Access: 2023-01-20 12:43:08.558000000 -0500
Modify: 1969-12-31 19:00:00.000000000 -0500
Change: 2023-01-20 12:43:09.356000000 -0500
Birth: 2023-01-20 12:43:08.558000000 -0500
~~~

vs a standard RHEL 9 minimal install system:

~~~
$ stat /usr
  File: /usr
  Size: 155       	Blocks: 0          IO Block: 4096   directory
Device: fd00h/64768d	Inode: 16813426    Links: 13
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:usr_t:s0
Access: 2023-08-08 22:06:10.371504686 +0200
Modify: 2023-06-20 22:13:53.619673667 +0200
Change: 2023-06-20 22:13:53.619673667 +0200
 Birth: 2022-07-25 21:36:31.310000000 +0200
~~~

From man 5 systemd.unit:
~~~
ConditionNeedsUpdate=

Takes one of /var/ or /etc/ as argument, possibly prefixed with a "!"  (to invert the condition). This condition may be used to conditionalize units on whether the specified directory requires an update because /usr/'s modification time is newer than the stamp file .updated in the specified directory. This is useful to implement offline updates of the vendor operating system resources in /usr/ that require updating of /etc/ or /var/ on the next following boot. Units making use of this condition should order themselves before systemd-update-done.service(8), to make sure they run before the stamp file's modification time gets reset indicating a completed update.

If the systemd.condition-needs-update= option is specified on the kernel command line (taking a boolean), it will override the result of this condition check, taking precedence over any file modification time checks. If the kernel command line option is used, systemd-update-done.service will not have immediate effect on any following ConditionNeedsUpdate= checks, until the system is rebooted where the kernel command line option is not specified anymore.

Note that to make this scheme effective, the timestamp of /usr/ should be explicitly updated after its contents are modified. The kernel will automatically update modification timestamp on a directory only when immediate children of a directory are modified; an modification of nested files will not automatically result in mtime of /usr/ being updated.

Also note that if the update method includes a call to execute appropriate post-update steps itself, it should not touch the timestamp of /usr/. In a typical distribution packaging scheme, packages will do any required update steps as part of the installation or upgrade, to make package contents immediately usable.  ConditionNeedsUpdate= should be used with other update mechanisms where such an immediate update does not happen.
~~~

From this man page seems that there needs to be an explicit call to update the mtime of /usr after the contents are modified since normally only direct child updates update the directory mtime (I'm thinking with the method RHEL4Edge/rpm-ostree updates systems even this doesn't happen but I'm not an expert on that.)

I also checked a RHEL system (not edge) built with Image Builder and didn't see the same behavior.

Comment 1 Timothée Ravier 2023-08-10 08:49:29 UTC
This is part of the design of ostree (https://ostreedev.github.io/ostree/repo/#content-objects) so will not be easy to change.

A quick grep on my system turn out:
```
$ grep -Rni ConditionNeedsUpdate= /usr/lib/systemd
/usr/lib/systemd/system/ldconfig.service:14:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/systemd-hwdb-update.service:17:ConditionNeedsUpdate=/etc
/usr/lib/systemd/system/systemd-journal-catalog-update.service:17:ConditionNeedsUpdate=/var
/usr/lib/systemd/system/systemd-sysusers.service:20:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/systemd-update-done.service:17:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/systemd-update-done.service:18:ConditionNeedsUpdate=|/var
/usr/lib/systemd/system/sysinit.target.wants/ldconfig.service:14:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-hwdb-update.service:17:ConditionNeedsUpdate=/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-journal-catalog-update.service:17:ConditionNeedsUpdate=/var
/usr/lib/systemd/system/sysinit.target.wants/systemd-sysusers.service:20:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-update-done.service:17:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-update-done.service:18:ConditionNeedsUpdate=|/var
```

Comment 2 Timothée Ravier 2023-08-10 15:33:15 UTC
Started a discussion upstream: https://github.com/coreos/fedora-coreos-tracker/issues/1538

Comment 3 RHEL Program Management 2023-09-13 10:46:49 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 4 RHEL Program Management 2023-09-13 10:48:40 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.