Bug 2230187

Summary: RHEL 4 Edge /usr modify timestamp not updated
Product: Red Hat Enterprise Linux 9 Reporter: ckrell
Component: rpm-ostreeAssignee: RHCOS SST <rhcos-sst>
Status: NEW --- QA Contact: RHCOS SST QE <rhcos-sst-qe>
Severity: medium Docs Contact:
Priority: medium    
Version: 9.2CC: travier
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description ckrell 2023-08-08 20:39:07 UTC
Description of problem:

On edge device deployment and after updates using rpm-ostree, the modify timestamp for /usr stays at 1969-12-31 18:00:00.000000000 -0600 while the other stat times update appropriately. 

This impacts the functionality of systemd as systemd units relying on ConditionNeedsUpdate may not trigger correctly as this relies on an update of /usr mtime.

Version-Release number of selected component (if applicable):
rpm-ostree

How reproducible:
Everytime -

Steps to Reproduce:
1. Install a RHEL 4 Edge System
2. `stat /usr`
3. Update RHEL 4 Edge System using rpm-ostree
4. `stat /usr`

Actual results:
~~~
# stat /usr
  File: /usr
  Size: 155       Blocks: 0          IO Block: 4096   directory
Device: fd00h/64768dInode: 201333897   Links: 12
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:usr_t:s0
Access: 2023-08-07 12:58:26.783227201 -0500
Modify: 1969-12-31 18:00:00.000000000 -0600
Change: 2023-08-07 12:58:27.959240721 -0500
Birth: 2023-08-07 12:58:26.783227201 -0500
~~~
Expected results:
The expectation is that the mtime is updated accordingly similar to a normal RHEL system update. 

Additional info:


Verifying on my own RHEL 4 Edge test system:

~~~
$ stat /usr
  File: /usr
  Size: 155       	Blocks: 0          IO Block: 4096   directory
Device: fc03h/64515d	Inode: 19570154    Links: 12
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:usr_t:s0
Access: 2023-01-20 12:43:08.558000000 -0500
Modify: 1969-12-31 19:00:00.000000000 -0500
Change: 2023-01-20 12:43:09.356000000 -0500
Birth: 2023-01-20 12:43:08.558000000 -0500
~~~

vs a standard RHEL 9 minimal install system:

~~~
$ stat /usr
  File: /usr
  Size: 155       	Blocks: 0          IO Block: 4096   directory
Device: fd00h/64768d	Inode: 16813426    Links: 13
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:usr_t:s0
Access: 2023-08-08 22:06:10.371504686 +0200
Modify: 2023-06-20 22:13:53.619673667 +0200
Change: 2023-06-20 22:13:53.619673667 +0200
 Birth: 2022-07-25 21:36:31.310000000 +0200
~~~

From man 5 systemd.unit:
~~~
ConditionNeedsUpdate=

Takes one of /var/ or /etc/ as argument, possibly prefixed with a "!"  (to invert the condition). This condition may be used to conditionalize units on whether the specified directory requires an update because /usr/'s modification time is newer than the stamp file .updated in the specified directory. This is useful to implement offline updates of the vendor operating system resources in /usr/ that require updating of /etc/ or /var/ on the next following boot. Units making use of this condition should order themselves before systemd-update-done.service(8), to make sure they run before the stamp file's modification time gets reset indicating a completed update.

If the systemd.condition-needs-update= option is specified on the kernel command line (taking a boolean), it will override the result of this condition check, taking precedence over any file modification time checks. If the kernel command line option is used, systemd-update-done.service will not have immediate effect on any following ConditionNeedsUpdate= checks, until the system is rebooted where the kernel command line option is not specified anymore.

Note that to make this scheme effective, the timestamp of /usr/ should be explicitly updated after its contents are modified. The kernel will automatically update modification timestamp on a directory only when immediate children of a directory are modified; an modification of nested files will not automatically result in mtime of /usr/ being updated.

Also note that if the update method includes a call to execute appropriate post-update steps itself, it should not touch the timestamp of /usr/. In a typical distribution packaging scheme, packages will do any required update steps as part of the installation or upgrade, to make package contents immediately usable.  ConditionNeedsUpdate= should be used with other update mechanisms where such an immediate update does not happen.
~~~

From this man page seems that there needs to be an explicit call to update the mtime of /usr after the contents are modified since normally only direct child updates update the directory mtime (I'm thinking with the method RHEL4Edge/rpm-ostree updates systems even this doesn't happen but I'm not an expert on that.)

I also checked a RHEL system (not edge) built with Image Builder and didn't see the same behavior.

Comment 1 Timothée Ravier 2023-08-10 08:49:29 UTC
This is part of the design of ostree (https://ostreedev.github.io/ostree/repo/#content-objects) so will not be easy to change.

A quick grep on my system turn out:
```
$ grep -Rni ConditionNeedsUpdate= /usr/lib/systemd
/usr/lib/systemd/system/ldconfig.service:14:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/systemd-hwdb-update.service:17:ConditionNeedsUpdate=/etc
/usr/lib/systemd/system/systemd-journal-catalog-update.service:17:ConditionNeedsUpdate=/var
/usr/lib/systemd/system/systemd-sysusers.service:20:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/systemd-update-done.service:17:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/systemd-update-done.service:18:ConditionNeedsUpdate=|/var
/usr/lib/systemd/system/sysinit.target.wants/ldconfig.service:14:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-hwdb-update.service:17:ConditionNeedsUpdate=/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-journal-catalog-update.service:17:ConditionNeedsUpdate=/var
/usr/lib/systemd/system/sysinit.target.wants/systemd-sysusers.service:20:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-update-done.service:17:ConditionNeedsUpdate=|/etc
/usr/lib/systemd/system/sysinit.target.wants/systemd-update-done.service:18:ConditionNeedsUpdate=|/var
```

Comment 2 Timothée Ravier 2023-08-10 15:33:15 UTC
Started a discussion upstream: https://github.com/coreos/fedora-coreos-tracker/issues/1538