RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1503066 - systemd read-only container fails to start systemd-journald.socket
Summary: systemd read-only container fails to start systemd-journald.socket
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: runc
Version: 7.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Jindrich Novy
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-17 10:12 UTC by Jan Pazdziora
Modified: 2020-06-03 13:59 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-03 13:59:04 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Jan Pazdziora 2017-10-17 10:12:06 UTC
Description of problem:

Attempt to run the equivalent of reproducer of bug 1390191 (to check behaviour on RHEL in bug 1478002) shows failing systemd-journald.socket with RHEL 7 containers.

Version-Release number of selected component (if applicable):

On the host:

docker-1.12.6-55.gitc4618fb.el7.x86_64
oci-systemd-hook-0.1.12-1.git1e84754.el7.x86_64
selinux-policy-3.13.1-166.el7_4.5.noarch
container-selinux-2.21-2.gitba103ac.el7.noarch

In the container:

registry.access.redhat.com/rhel7 7.4 549b1c5d7a44 2 weeks ago 195.9 MB

How reproducible:

Deterministic.

Steps to Reproduce:
1. docker run --read-only=true --tmpfs /var --name systemd-ro -e container=docker --rm -ti registry.access.redhat.com/rhel7:7.4 /usr/sbin/init

Actual results:

# docker run --read-only=true --tmpfs /var --name systemd-ro -e container=docker --rm -ti registry.access.redhat.com/rhel7:7.4 /usr/sbin/init
Unable to find image 'registry.access.redhat.com/rhel7:7.4' locally
Trying to pull repository registry.access.redhat.com/rhel7 ... 
7.4: Pulling from registry.access.redhat.com/rhel7

26e5ed6899db: Already exists 
66dbe984a319: Already exists 
Digest: sha256:82c6d9163b4c101ae41470dca5ca5fbe09c546b77f2c0478e031c73d8e270fee
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Red Hat Enterprise Linux Server 7.4 (Maipo)!

Set hostname to <b37df108bc4a>.
Cannot add dependency job for unit sys-fs-fuse-connections.mount, ignoring: Unit is masked.
Cannot add dependency job for unit systemd-logind.service, ignoring: Unit is masked.
Cannot add dependency job for unit getty.target, ignoring: Unit is masked.
[  OK  ] Created slice Root Slice.
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Created slice System Slice.
[  OK  ] Reached target Swap.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
systemd-journald.socket failed to listen on sockets: Read-only file system
[FAILED] Failed to listen on Journal Socket.
See 'systemctl status systemd-journald.socket' for details.
[DEPEND] Dependency failed for Journal Service.
[DEPEND] Dependency failed for Flush Journal to Persistent Storage.
Job systemd-journal-flush.service/start failed with result 'dependency'.
Job systemd-journald.service/start failed with result 'dependency'.
Unit systemd-journald.socket entered failed state.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
         Starting Update is Completed...
[  OK  ] Reached target Paths.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slices.
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Update is Completed.
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
[  OK  ] Started D-Bus System Message Bus.
         Starting D-Bus System Message Bus...
         Starting Permit User Sessions...
         Starting Cleanup of Temporary Directories...
[  OK  ] Started Permit User Sessions.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Cleanup of Temporary Directories.
[  OK  ] Started Update UTMP about System Runlevel Changes.
Startup finished in 41ms.

Expected results:

# docker run --read-only=true --tmpfs /var --name systemd-ro -e container=docker --rm -ti registry.access.redhat.com/rhel7:7.4 /usr/sbin/init
Unable to find image 'registry.access.redhat.com/rhel7:7.4' locally
Trying to pull repository registry.access.redhat.com/rhel7 ... 
7.4: Pulling from registry.access.redhat.com/rhel7

26e5ed6899db: Already exists 
66dbe984a319: Already exists 
Digest: sha256:82c6d9163b4c101ae41470dca5ca5fbe09c546b77f2c0478e031c73d8e270fee
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Red Hat Enterprise Linux Server 7.4 (Maipo)!

Set hostname to <b37df108bc4a>.
Cannot add dependency job for unit sys-fs-fuse-connections.mount, ignoring: Unit is masked.
Cannot add dependency job for unit systemd-logind.service, ignoring: Unit is masked.
Cannot add dependency job for unit getty.target, ignoring: Unit is masked.
[  OK  ] Created slice Root Slice.
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Created slice System Slice.
[  OK  ] Reached target Swap.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
         Starting Update is Completed...
[  OK  ] Reached target Paths.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slices.
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Update is Completed.
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
[  OK  ] Started D-Bus System Message Bus.
         Starting D-Bus System Message Bus...
         Starting Permit User Sessions...
         Starting Cleanup of Temporary Directories...
[  OK  ] Started Permit User Sessions.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Cleanup of Temporary Directories.
[  OK  ] Started Update UTMP about System Runlevel Changes.
Startup finished in 41ms.

Additional info:

With fedora:24 image on the same RHEL 7 host, only the bug 1390191 issue is shown:

# docker run --read-only=true --tmpfs /var --name systemd-ro -e container=docker --rm -ti fedora:24 /usr/sbin/initUnable to find image 'fedora:24' locally
Trying to pull repository registry.access.redhat.com/fedora ... 
Trying to pull repository docker.io/library/fedora ... 
24: Pulling from docker.io/library/fedora
d489011951f5: Pull complete 
Digest: sha256:0c1580c63e623ecfa0ef2d4a548d73a655e8072725bcca01bc6f2e446914a7bc
systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Fedora 24 (Twenty Four)!

Set hostname to <3e73b48c3f89>.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket.
[  OK  ] Listening on Process Core Dump Socket.
[  OK  ] Reached target Swap.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Local File Systems.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Created slice System Slice.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Slices.
         Starting Journal Service...
         Starting Update is Completed...
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Update is Completed.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[FAILED] Failed to start Create Volatile Files and Directories.
See 'systemctl status systemd-tmpfiles-setup.service' for details.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Started dnf makecache timer.
[  OK  ] Reached target Basic System.
         Starting Permit User Sessions...
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Reached target Timers.
[  OK  ] Started Permit User Sessions.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.

Comment 2 Jan Pazdziora 2017-10-17 10:26:13 UTC
What seems to be missing in the read-only container is /dev/log. The /dev seems to be mounted read-only:

tmpfs on /dev type tmpfs (ro,relatime,context="system_u:object_r:svirt_sandbox_file_t:s0:c31,c804",mode=755)

Comment 3 Daniel Walsh 2017-10-17 13:14:31 UTC
Well /dev can be readonly but /dev/log should not.  But maybe this is being created.  Which means that /dev/ should really not be readonly.  This might be an issue in runc.

Comment 4 Daniel Walsh 2017-10-17 13:16:06 UTC
Antonio can you check if runc is setting /dev read-only if the read-only flag is passed?

Comment 5 Jan Pazdziora 2017-10-17 13:45:28 UTC
(In reply to Daniel Walsh from comment #3)
> Well /dev can be readonly but /dev/log should not.  But maybe this is being
> created.  Which means that /dev/ should really not be readonly.

Right, /dev/log does not exist in /dev and systemd-journald.socket attempts to create it. I'm not sure if we have a way to have /dev/log exist (maybe as symlink to /var?) in the read-only /dev filesystem ...

Comment 6 Daniel Walsh 2017-10-17 14:19:38 UTC
No, we need to fix runc or docker to not mount /dev as readonly that is not part of the image and should be read/write

Comment 7 Jan Pazdziora 2017-10-27 09:23:00 UTC
The situation when running fedora:24 is strange though -- the systemd upon startup says

         Starting Update is Completed...
[  OK  ] Listening on Journal Socket (/dev/log).
         Starting Journal Service...
[  OK  ] Started Load/Save Random Seed.

yet there is no /dev/log in the container

# docker exec systemd-ro ls -la /dev/log
ls: cannot access '/dev/log': No such file or directory

and of course the /dev is mounted read-only there as well.

# docker exec systemd-ro mount | grep '/dev '
tmpfs on /dev type tmpfs (ro,relatime,context="system_u:object_r:svirt_sandbox_file_t:s0:c14,c449",mode=755)

I would test with newer fedora:* images but due to bug 1373780, the systemd does not show status upon boot at all, so it's hard to see what is actually going on there.

Comment 8 Daniel Walsh 2020-06-03 13:59:04 UTC
I believe this works on RHEL8.  Reopen if I am mistaken.


Note You need to log in before you can comment on or make changes to this bug.