Bug 1848453

Summary:	Enable ostree-remount.service needed for RHEL Edge
Product:	Red Hat Enterprise Linux 8	Reporter:	Christian Kellner <ckellner>
Component:	redhat-release	Assignee:	Djordje Todorovic <dtodorov>
Status:	CLOSED ERRATA	QA Contact:	Release Test Team <release-test-team-automation>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.3	CC:	bbreard, lisas, lmiksik, lueberni, mezhang, perobins, pkotvan, releng-maint-list, sbueno, xiaofwan
Target Milestone:	rc
Target Release:	8.0
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-11-04 01:47:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1815140

Description Christian Kellner 2020-06-18 11:44:37 UTC

RHEL Edge is based on OSTree, which needs special setup in the initramfs compared to traditional systems.  One such ingredient is ostree-remount.service which takes care of mounting /sysroot and /var as read-write. If this is not done, lots of services will fail to work properly.

On Fedora this is being enabled by default and run. Preset: https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/90-default.preset#_219
Log:
● ostree-remount.service - OSTree Remount OS/ Bind Mounts
     Loaded: loaded (/usr/lib/systemd/system/ostree-remount.service; enabled; vendor preset: enabled)
     Active: active (exited) since Thu 2020-06-18 13:03:00 CEST; 2min 35s ago
       Docs: man:ostree(1)
    Process: 648 ExecStart=/usr/lib/ostree/ostree-remount (code=exited, status=0/SUCCESS)
   Main PID: 648 (code=exited, status=0/SUCCESS)
        CPU: 3ms


But on RHEL the preset is missing from redhat-release: http://pkgs.devel.redhat.com/cgit/rpms/redhat-release/tree/90-default.preset?h=rhel-8.3.0

Log:
● ostree-remount.service - OSTree Remount OS/ Bind Mounts
   Loaded: loaded (/usr/lib/systemd/system/ostree-remount.service; disabled; ve>
   Active: inactive (dead)
     Docs: man:ostree(1)

As a result /var is read-only (and so is /sysroot) and lots of things fail:

un 18 04:21:41 localhost.localdomain systemd-tmpfiles[708]: Failed to create directory or subvolume "/var/empty/sshd": No such file or directory
Jun 18 04:21:41 localhost.localdomain systemd-tmpfiles[708]: Failed to create directory or subvolume "/var/empty/sshd": No such file or directory
Jun 18 04:21:41 localhost.localdomain systemd-tmpfiles[708]: /var/lib/portables does not exist and cannot be created as the file system is read-only.
Jun 18 04:21:41 localhost.localdomain systemd-tmpfiles[708]: Unable to fix SELinux security context of /var/usrlocal/bin: Read-only file system
Jun 18 04:21:41 localhost.localdomain systemd-tmpfiles[708]: Unable to fix SELinux security context of /var/usrlocal/man: Read-only file system
Jun 18 04:21:41 localhost.localdomain systemd-tmpfiles[708]: /var/lib/games does not exist and cannot be created as the file system is read-only.

Jun 18 04:21:43 localhost.localdomain systemd[1]: Started Login Service.
Jun 18 04:21:44 localhost.localdomain firewalld[718]: ERROR: Failed to open log file '/var/log/firewalld': [Errno 30] Read-only file system: '/var/log/f
irewalld'
Jun 18 04:21:46 localhost.localdomain systemd[1]: sshd-keygen: Succeeded.
Jun 18 04:21:46 localhost.localdomain systemd[1]: Started OpenSSH rsa Server Key Generation.
Jun 18 04:21:46 localhost.localdomain systemd[1]: Reached target sshd-keygen.target.
Jun 18 04:21:48 localhost.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
Jun 18 04:21:48 localhost.localdomain systemd[1]: Reached target Network (Pre).
Jun 18 04:21:48 localhost.localdomain systemd[1]: Starting Network Manager...
Jun 18 04:21:48 localhost.localdomain NetworkManager[761]: Cannot create '/var/lib/NetworkManager': Read-only file system
Jun 18 04:21:48 localhost.localdomain systemd[1]: NetworkManager.service: Main process exited, code=exited, status=1/FAILURE

Comment 1 Christian Kellner 2020-06-18 11:46:55 UTC

Discovered by Xiaofeng Wang.

Comment 2 Christian Kellner 2020-06-18 12:45:08 UTC

I filed https://src.osci.redhat.com/rpms/redhat-release/pull-request/20 for this.

Comment 4 Lars Karlitski 2020-06-24 17:59:29 UTC

Answering on behalf of Xiaofeng — hope that's ok.

He will verify this fix as part of the "Image-builder with ostree" test plan, which is meant to exactly just that:

    https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=~xiaofwan&title=Image-builder+with+ostree

Peter, is that enough for you to accept the fix that Christian linked to?

It's important for RHEL for Edge: no services work without this patch, because `/var` is currently mounted read-only.

Comment 5 Peter Kotvan 2020-06-25 06:34:12 UTC

(In reply to Lars Karlitski from comment #4)
> Answering on behalf of Xiaofeng — hope that's ok.
> 
> He will verify this fix as part of the "Image-builder with ostree" test
> plan, which is meant to exactly just that:
> 
>    
> https://docs.engineering.redhat.com/pages/viewpage.
> action?spaceKey=~xiaofwan&title=Image-builder+with+ostree
> 
> Peter, is that enough for you to accept the fix that Christian linked to?
> 
> It's important for RHEL for Edge: no services work without this patch,
> because `/var` is currently mounted read-only.

Hi Lars, Xiaofeng,

thanks for the explanation. RTT will verify the fix as well since we have a test to verify redhat-release contents (default services included).

Since RTT is a qe contact for redhat-release component, please reach out to release test team (release-test-team) to ask for qe_ack+ in the future.

Thanks.

Comment 11 Lisa S 2020-08-03 14:57:48 UTC

Ticket to track work:
https://projects.engineering.redhat.com/browse/RHELBLD-2270

Comment 15 Xiaofeng Wang 2020-08-17 06:34:03 UTC

Tested on RHEL 8.3 nightly compose and build. Issue got fixed on redhat-release-8.3-0.4.el8.x86_64. Thanks @Christian Kellner! 

[admin@ostree-guest ~]$ sudo systemctl status ostree-remount.service
● ostree-remount.service - OSTree Remount OS/ Bind Mounts
   Loaded: loaded (/usr/lib/systemd/system/ostree-remount.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2020-08-17 06:28:16 UTC; 1min 36s ago
     Docs: man:ostree(1)
  Process: 646 ExecStart=/usr/lib/ostree/ostree-remount (code=exited, status=0/SUCCESS)
 Main PID: 646 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 23500)
   Memory: 0B
   CGroup: /system.slice/ostree-remount.service

Aug 17 06:28:15 localhost.localdomain systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Aug 17 06:28:16 localhost.localdomain ostree-remount[646]: Remounted rw: /var
Aug 17 06:28:16 localhost.localdomain systemd[1]: Started OSTree Remount OS/ Bind Mounts.
[admin@ostree-guest ~]$ rpm -qa |grep redhat-release
redhat-release-8.3-0.4.el8.x86_64
redhat-release-eula-8.3-0.4.el8.x86_64

Comment 18 errata-xmlrpc 2020-11-04 01:47:12 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (redhat-release bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4495