Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1848453

Summary: Enable ostree-remount.service needed for RHEL Edge
Product: Red Hat Enterprise Linux 8 Reporter: Christian Kellner <ckellner>
Component: redhat-releaseAssignee: Djordje Todorovic <dtodorov>
Status: CLOSED ERRATA QA Contact: Release Test Team <release-test-team-automation>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.3CC: bbreard, lisas, lmiksik, lueberni, mezhang, perobins, pkotvan, releng-maint-list, sbueno, xiaofwan
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-04 01:47:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1815140    

Description Christian Kellner 2020-06-18 11:44:37 UTC 
RHEL Edge is based on OSTree, which needs special setup in the initramfs compared to traditional systems.  One such ingredient is ostree-remount.service which takes care of mounting /sysroot and /var as read-write. If this is not done, lots of services will fail to work properly.

On Fedora this is being enabled by default and run. Preset: https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/90-default.preset#_219
Log:
● ostree-remount.service - OSTree Remount OS/ Bind Mounts
     Loaded: loaded (/usr/lib/systemd/system/ostree-remount.service; enabled; vendor preset: enabled)
     Active: active (exited) since Thu 2020-06-18 13:03:00 CEST; 2min 35s ago
       Docs: man:ostree(1)
    Process: 648 ExecStart=/usr/lib/ostree/ostree-remount (code=exited, status=0/SUCCESS)
   Main PID: 648 (code=exited, status=0/SUCCESS)
        CPU: 3ms


But on RHEL the preset is missing from redhat-release: http://pkgs.devel.redhat.com/cgit/rpms/redhat-release/tree/90-default.preset?h=rhel-8.3.0

Log:
● ostree-remount.service - OSTree Remount OS/ Bind Mounts
   Loaded: loaded (/usr/lib/systemd/system/ostree-remount.service; disabled; ve>
   Active: inactive (dead)
     Docs: man:ostree(1)

As a result /var is read-only (and so is /sysroot) and lots of things fail:

un 18 04:21:41 localhost.localdomain systemd-tmpfiles[708]: Failed to create directory or subvolume "/var/empty/sshd": No such file or directory
Jun 18 04:21:41 localhost.localdomain systemd-tmpfiles[708]: Failed to create directory or subvolume "/var/empty/sshd": No such file or directory
Jun 18 04:21:41 localhost.localdomain systemd-tmpfiles[708]: /var/lib/portables does not exist and cannot be created as the file system is read-only.
Jun 18 04:21:41 localhost.localdomain systemd-tmpfiles[708]: Unable to fix SELinux security context of /var/usrlocal/bin: Read-only file system
Jun 18 04:21:41 localhost.localdomain systemd-tmpfiles[708]: Unable to fix SELinux security context of /var/usrlocal/man: Read-only file system
Jun 18 04:21:41 localhost.localdomain systemd-tmpfiles[708]: /var/lib/games does not exist and cannot be created as the file system is read-only.

Jun 18 04:21:43 localhost.localdomain systemd[1]: Started Login Service.
Jun 18 04:21:44 localhost.localdomain firewalld[718]: ERROR: Failed to open log file '/var/log/firewalld': [Errno 30] Read-only file system: '/var/log/f
irewalld'
Jun 18 04:21:46 localhost.localdomain systemd[1]: sshd-keygen: Succeeded.
Jun 18 04:21:46 localhost.localdomain systemd[1]: Started OpenSSH rsa Server Key Generation.
Jun 18 04:21:46 localhost.localdomain systemd[1]: Reached target sshd-keygen.target.
Jun 18 04:21:48 localhost.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
Jun 18 04:21:48 localhost.localdomain systemd[1]: Reached target Network (Pre).
Jun 18 04:21:48 localhost.localdomain systemd[1]: Starting Network Manager...
Jun 18 04:21:48 localhost.localdomain NetworkManager[761]: Cannot create '/var/lib/NetworkManager': Read-only file system
Jun 18 04:21:48 localhost.localdomain systemd[1]: NetworkManager.service: Main process exited, code=exited, status=1/FAILURE
Comment 1 Christian Kellner 2020-06-18 11:46:55 UTC 
Discovered by Xiaofeng Wang.
Comment 2 Christian Kellner 2020-06-18 12:45:08 UTC 
I filed https://src.osci.redhat.com/rpms/redhat-release/pull-request/20 for this.
Comment 4 Lars Karlitski 2020-06-24 17:59:29 UTC 
Answering on behalf of Xiaofeng — hope that's ok.

He will verify this fix as part of the "Image-builder with ostree" test plan, which is meant to exactly just that:

    https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=~xiaofwan&title=Image-builder+with+ostree

Peter, is that enough for you to accept the fix that Christian linked to?

It's important for RHEL for Edge: no services work without this patch, because `/var` is currently mounted read-only.
Comment 5 Peter Kotvan 2020-06-25 06:34:12 UTC 
(In reply to Lars Karlitski from comment #4)
> Answering on behalf of Xiaofeng — hope that's ok.
> 
> He will verify this fix as part of the "Image-builder with ostree" test
> plan, which is meant to exactly just that:
> 
>    
> https://docs.engineering.redhat.com/pages/viewpage.
> action?spaceKey=~xiaofwan&title=Image-builder+with+ostree
> 
> Peter, is that enough for you to accept the fix that Christian linked to?
> 
> It's important for RHEL for Edge: no services work without this patch,
> because `/var` is currently mounted read-only.

Hi Lars, Xiaofeng,

thanks for the explanation. RTT will verify the fix as well since we have a test to verify redhat-release contents (default services included).

Since RTT is a qe contact for redhat-release component, please reach out to release test team (release-test-team) to ask for qe_ack+ in the future.

Thanks.
Comment 11 Lisa S 2020-08-03 14:57:48 UTC 
Ticket to track work:
https://projects.engineering.redhat.com/browse/RHELBLD-2270
Comment 15 Xiaofeng Wang 2020-08-17 06:34:03 UTC 
Tested on RHEL 8.3 nightly compose and build. Issue got fixed on redhat-release-8.3-0.4.el8.x86_64. Thanks @Christian Kellner! 

[admin@ostree-guest ~]$ sudo systemctl status ostree-remount.service
● ostree-remount.service - OSTree Remount OS/ Bind Mounts
   Loaded: loaded (/usr/lib/systemd/system/ostree-remount.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2020-08-17 06:28:16 UTC; 1min 36s ago
     Docs: man:ostree(1)
  Process: 646 ExecStart=/usr/lib/ostree/ostree-remount (code=exited, status=0/SUCCESS)
 Main PID: 646 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 23500)
   Memory: 0B
   CGroup: /system.slice/ostree-remount.service

Aug 17 06:28:15 localhost.localdomain systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Aug 17 06:28:16 localhost.localdomain ostree-remount[646]: Remounted rw: /var
Aug 17 06:28:16 localhost.localdomain systemd[1]: Started OSTree Remount OS/ Bind Mounts.
[admin@ostree-guest ~]$ rpm -qa |grep redhat-release
redhat-release-8.3-0.4.el8.x86_64
redhat-release-eula-8.3-0.4.el8.x86_64
Comment 18 errata-xmlrpc 2020-11-04 01:47:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (redhat-release bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4495