Bug 1503066

Summary: systemd read-only container fails to start systemd-journald.socket
Product: Red Hat Enterprise Linux 7 Reporter: Jan Pazdziora <jpazdziora>
Component: runcAssignee: Jindrich Novy <jnovy>
Status: CLOSED CURRENTRELEASE QA Contact: atomic-bugs <atomic-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.4CC: amurdaca, dwalsh, jpazdziora, lsm5, mpatel
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-06-03 13:59:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jan Pazdziora 2017-10-17 10:12:06 UTC
Description of problem:

Attempt to run the equivalent of reproducer of bug 1390191 (to check behaviour on RHEL in bug 1478002) shows failing systemd-journald.socket with RHEL 7 containers.

Version-Release number of selected component (if applicable):

On the host:

docker-1.12.6-55.gitc4618fb.el7.x86_64
oci-systemd-hook-0.1.12-1.git1e84754.el7.x86_64
selinux-policy-3.13.1-166.el7_4.5.noarch
container-selinux-2.21-2.gitba103ac.el7.noarch

In the container:

registry.access.redhat.com/rhel7 7.4 549b1c5d7a44 2 weeks ago 195.9 MB

How reproducible:

Deterministic.

Steps to Reproduce:
1. docker run --read-only=true --tmpfs /var --name systemd-ro -e container=docker --rm -ti registry.access.redhat.com/rhel7:7.4 /usr/sbin/init

Actual results:

# docker run --read-only=true --tmpfs /var --name systemd-ro -e container=docker --rm -ti registry.access.redhat.com/rhel7:7.4 /usr/sbin/init
Unable to find image 'registry.access.redhat.com/rhel7:7.4' locally
Trying to pull repository registry.access.redhat.com/rhel7 ... 
7.4: Pulling from registry.access.redhat.com/rhel7

26e5ed6899db: Already exists 
66dbe984a319: Already exists 
Digest: sha256:82c6d9163b4c101ae41470dca5ca5fbe09c546b77f2c0478e031c73d8e270fee
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Red Hat Enterprise Linux Server 7.4 (Maipo)!

Set hostname to <b37df108bc4a>.
Cannot add dependency job for unit sys-fs-fuse-connections.mount, ignoring: Unit is masked.
Cannot add dependency job for unit systemd-logind.service, ignoring: Unit is masked.
Cannot add dependency job for unit getty.target, ignoring: Unit is masked.
[  OK  ] Created slice Root Slice.
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Created slice System Slice.
[  OK  ] Reached target Swap.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
systemd-journald.socket failed to listen on sockets: Read-only file system
[FAILED] Failed to listen on Journal Socket.
See 'systemctl status systemd-journald.socket' for details.
[DEPEND] Dependency failed for Journal Service.
[DEPEND] Dependency failed for Flush Journal to Persistent Storage.
Job systemd-journal-flush.service/start failed with result 'dependency'.
Job systemd-journald.service/start failed with result 'dependency'.
Unit systemd-journald.socket entered failed state.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
         Starting Update is Completed...
[  OK  ] Reached target Paths.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slices.
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Update is Completed.
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
[  OK  ] Started D-Bus System Message Bus.
         Starting D-Bus System Message Bus...
         Starting Permit User Sessions...
         Starting Cleanup of Temporary Directories...
[  OK  ] Started Permit User Sessions.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Cleanup of Temporary Directories.
[  OK  ] Started Update UTMP about System Runlevel Changes.
Startup finished in 41ms.

Expected results:

# docker run --read-only=true --tmpfs /var --name systemd-ro -e container=docker --rm -ti registry.access.redhat.com/rhel7:7.4 /usr/sbin/init
Unable to find image 'registry.access.redhat.com/rhel7:7.4' locally
Trying to pull repository registry.access.redhat.com/rhel7 ... 
7.4: Pulling from registry.access.redhat.com/rhel7

26e5ed6899db: Already exists 
66dbe984a319: Already exists 
Digest: sha256:82c6d9163b4c101ae41470dca5ca5fbe09c546b77f2c0478e031c73d8e270fee
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Red Hat Enterprise Linux Server 7.4 (Maipo)!

Set hostname to <b37df108bc4a>.
Cannot add dependency job for unit sys-fs-fuse-connections.mount, ignoring: Unit is masked.
Cannot add dependency job for unit systemd-logind.service, ignoring: Unit is masked.
Cannot add dependency job for unit getty.target, ignoring: Unit is masked.
[  OK  ] Created slice Root Slice.
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Created slice System Slice.
[  OK  ] Reached target Swap.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
         Starting Update is Completed...
[  OK  ] Reached target Paths.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slices.
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Update is Completed.
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
[  OK  ] Started D-Bus System Message Bus.
         Starting D-Bus System Message Bus...
         Starting Permit User Sessions...
         Starting Cleanup of Temporary Directories...
[  OK  ] Started Permit User Sessions.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Cleanup of Temporary Directories.
[  OK  ] Started Update UTMP about System Runlevel Changes.
Startup finished in 41ms.

Additional info:

With fedora:24 image on the same RHEL 7 host, only the bug 1390191 issue is shown:

# docker run --read-only=true --tmpfs /var --name systemd-ro -e container=docker --rm -ti fedora:24 /usr/sbin/initUnable to find image 'fedora:24' locally
Trying to pull repository registry.access.redhat.com/fedora ... 
Trying to pull repository docker.io/library/fedora ... 
24: Pulling from docker.io/library/fedora
d489011951f5: Pull complete 
Digest: sha256:0c1580c63e623ecfa0ef2d4a548d73a655e8072725bcca01bc6f2e446914a7bc
systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Fedora 24 (Twenty Four)!

Set hostname to <3e73b48c3f89>.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket.
[  OK  ] Listening on Process Core Dump Socket.
[  OK  ] Reached target Swap.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Local File Systems.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Created slice System Slice.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Slices.
         Starting Journal Service...
         Starting Update is Completed...
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Update is Completed.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[FAILED] Failed to start Create Volatile Files and Directories.
See 'systemctl status systemd-tmpfiles-setup.service' for details.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Started dnf makecache timer.
[  OK  ] Reached target Basic System.
         Starting Permit User Sessions...
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Reached target Timers.
[  OK  ] Started Permit User Sessions.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.

Comment 2 Jan Pazdziora 2017-10-17 10:26:13 UTC
What seems to be missing in the read-only container is /dev/log. The /dev seems to be mounted read-only:

tmpfs on /dev type tmpfs (ro,relatime,context="system_u:object_r:svirt_sandbox_file_t:s0:c31,c804",mode=755)

Comment 3 Daniel Walsh 2017-10-17 13:14:31 UTC
Well /dev can be readonly but /dev/log should not.  But maybe this is being created.  Which means that /dev/ should really not be readonly.  This might be an issue in runc.

Comment 4 Daniel Walsh 2017-10-17 13:16:06 UTC
Antonio can you check if runc is setting /dev read-only if the read-only flag is passed?

Comment 5 Jan Pazdziora 2017-10-17 13:45:28 UTC
(In reply to Daniel Walsh from comment #3)
> Well /dev can be readonly but /dev/log should not.  But maybe this is being
> created.  Which means that /dev/ should really not be readonly.

Right, /dev/log does not exist in /dev and systemd-journald.socket attempts to create it. I'm not sure if we have a way to have /dev/log exist (maybe as symlink to /var?) in the read-only /dev filesystem ...

Comment 6 Daniel Walsh 2017-10-17 14:19:38 UTC
No, we need to fix runc or docker to not mount /dev as readonly that is not part of the image and should be read/write

Comment 7 Jan Pazdziora 2017-10-27 09:23:00 UTC
The situation when running fedora:24 is strange though -- the systemd upon startup says

         Starting Update is Completed...
[  OK  ] Listening on Journal Socket (/dev/log).
         Starting Journal Service...
[  OK  ] Started Load/Save Random Seed.

yet there is no /dev/log in the container

# docker exec systemd-ro ls -la /dev/log
ls: cannot access '/dev/log': No such file or directory

and of course the /dev is mounted read-only there as well.

# docker exec systemd-ro mount | grep '/dev '
tmpfs on /dev type tmpfs (ro,relatime,context="system_u:object_r:svirt_sandbox_file_t:s0:c14,c449",mode=755)

I would test with newer fedora:* images but due to bug 1373780, the systemd does not show status upon boot at all, so it's hard to see what is actually going on there.

Comment 8 Daniel Walsh 2020-06-03 13:59:04 UTC
I believe this works on RHEL8.  Reopen if I am mistaken.