Bug 1478002 - systemd read-only container produces errors
systemd read-only container produces errors
Status: ON_QA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: util-linux (Show other bugs)
7.5
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Karel Zak
Radka Skvarilova
:
Depends On: 1390191
Blocks: 1465901
  Show dependency treegraph
 
Reported: 2017-08-03 08:38 EDT by Daniel Walsh
Modified: 2017-10-27 05:59 EDT (History)
20 users (show)

See Also:
Fixed In Version: util-linux-2.23.2-44.el7
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1390191
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Daniel Walsh 2017-08-03 08:38:52 EDT
+++ This bug was initially created as a clone of Bug #1390191 +++

Description of problem:

Running systemd-based container on host with oci-systemd-hook, there are errors shown on console and in journal.

Version-Release number of selected component (if applicable):

docker-1.10.3-54.gite03ddb8.fc24.x86_64
oci-systemd-hook-0.1.4-1.fc24.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. $ docker run --read-only=true --name systemd-ro -e container=docker --rm -ti fedora:24 /usr/sbin/init
2. Check the docker run output and also run in another terminal $ docker exec -ti systemd-ro journalctl -l

Actual results:

systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Fedora 24 (Twenty Four)!

Set hostname to <aed69bad6512>.
[  OK  ] Created slice System Slice.
[  OK  ] Listening on Process Core Dump Socket.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Listening on Journal Socket.
         Starting Load/Save Random Seed...
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
         Starting Journal Service...
[  OK  ] Reached target Slices.
[  OK  ] Reached target Local File Systems.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Swap.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[FAILED] Failed to start Create Volatile Files and Directories.
See 'systemctl status systemd-tmpfiles-setup.service' for details.
         Starting Update UTMP about System Boot/Shutdown...
[FAILED] Failed to start Update UTMP about System Boot/Shutdown.
See 'systemctl status systemd-update-utmp.service' for details.
[DEPEND] Dependency failed for Update UTMP about System Runlevel Changes.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Started dnf makecache timer.
[  OK  ] Reached target Basic System.
         Starting Permit User Sessions...
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Started Permit User Sessions.
[  OK  ] Reached target Multi-User System.

Oct 31 12:53:46 aed69bad6512 systemd[1]: Starting Create Volatile Files and Directories...
Oct 31 12:53:46 aed69bad6512 systemd-tmpfiles[23]: symlink(../proc/self/mounts, /etc/mtab) failed: Read-only file system
Oct 31 12:53:46 aed69bad6512 systemd-tmpfiles[23]: Setting default ACL "u::rwx,g::r-x,g:adm:r-x,g:wheel:r-x,m::r-x,o::r-x" on /var/log/journal failed: Read-only file system
Oct 31 12:53:46 aed69bad6512 systemd-tmpfiles[23]: Setting access ACL "u::rwx,g::r-x,g:adm:r-x,g:wheel:r-x,m::r-x,o::r-x" on /var/log/journal failed: Read-only file system
Oct 31 12:53:46 aed69bad6512 systemd-tmpfiles[23]: Cannot set file attribute for '/var/log/journal', value=0x00800000, mask=0x00800000: Operation not supported
Oct 31 12:53:46 aed69bad6512 systemd-tmpfiles[23]: Cannot set file attribute for '/var/log/journal/aed69bad65129148b1cf5728eaf69368', value=0x00800000, mask=0x00800000: Operation not supported
Oct 31 12:53:46 aed69bad6512 systemd-tmpfiles[23]: rm(/var/lib/rpm/__db.001): Read-only file system
Oct 31 12:53:46 aed69bad6512 systemd-tmpfiles[23]: rm(/var/lib/rpm/__db.002): Read-only file system
Oct 31 12:53:46 aed69bad6512 systemd-tmpfiles[23]: rm(/var/lib/rpm/__db.003): Read-only file system
Oct 31 12:53:46 aed69bad6512 systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Oct 31 12:53:46 aed69bad6512 systemd[1]: Failed to start Create Volatile Files and Directories.
Oct 31 12:53:46 aed69bad6512 systemd[1]: systemd-tmpfiles-setup.service: Unit entered failed state.
Oct 31 12:53:46 aed69bad6512 systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
Oct 31 12:53:46 aed69bad6512 systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Oct 31 12:53:46 aed69bad6512 systemd-update-utmp[24]: Failed to write utmp record: Read-only file system
Oct 31 12:53:46 aed69bad6512 systemd[1]: systemd-update-utmp.service: Main process exited, code=exited, status=1/FAILURE
Oct 31 12:53:46 aed69bad6512 systemd[1]: Failed to start Update UTMP about System Boot/Shutdown.
Oct 31 12:53:46 aed69bad6512 systemd[1]: Dependency failed for Update UTMP about System Runlevel Changes.
Oct 31 12:53:46 aed69bad6512 systemd[1]: systemd-update-utmp-runlevel.service: Job systemd-update-utmp-runlevel.service/start failed with result 'dependency'.
Oct 31 12:53:46 aed69bad6512 systemd[1]: systemd-update-utmp.service: Unit entered failed state.
Oct 31 12:53:46 aed69bad6512 systemd[1]: systemd-update-utmp.service: Failed with result 'exit-code'.
Oct 31 12:53:46 aed69bad6512 systemd[1]: Reached target System Initialization.

Oct 31 12:53:46 aed69bad6512 systemd[1]: systemd-journald.service: Couldn't add fd to fd store: Operation not permitted

Expected results:

No errors, no failed services / targets.

Additional info:

--- Additional comment from Daniel Walsh on 2016-10-31 09:11:01 EDT ---

I think you should volume mount in /var to eliminate a lot of these. 

The /etc/mtab one will be difficult to fix. 

mkdir /var/systemd-ro
# docker run --read-only=true --name systemd-ro -e container=docker -v /var/systemd-ro:/var:Z --rm -ti fedora:24 /usr/sbin/init
Unable to find image 'fedora:24' locally
Trying to pull repository atomic-registry.usersys.redhat.com/fedora ... 
Pulling repository atomic-registry.usersys.redhat.com/fedora
Trying to pull repository docker.io/library/fedora ... 
sha256:64a02df6aac27d1200c2572fe4b9949f1970d05f74d367ce4af994ba5dc3669e: Pulling from docker.io/library/fedora
Digest: sha256:64a02df6aac27d1200c2572fe4b9949f1970d05f74d367ce4af994ba5dc3669e
Status: Image is up to date for docker.io/fedora:24
systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Fedora 24 (Twenty Four)!

Set hostname to <758086f85759>.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Created slice System Slice.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Reached target Local File Systems.
[  OK  ] Reached target Swap.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Listening on Journal Socket.
         Starting Load/Save Random Seed...
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Slices.
         Starting Update is Completed...
[  OK  ] Listening on Process Core Dump Socket.
         Starting Journal Service...
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Update is Completed.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[FAILED] Failed to start Create Volatile Files and Directories.
See 'systemctl status systemd-tmpfiles-setup.service' for details.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Started dnf makecache timer.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
[  OK  ] Started D-Bus System Message Bus.
         Starting Permit User Sessions...
[  OK  ] Started Permit User Sessions.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.

--- Additional comment from Daniel Walsh on 2016-10-31 09:12:19 EDT ---

The only failure I see is the creating of the /etc/mtab -> ../proc/self/mount

--- Additional comment from Jan Pazdziora on 2016-10-31 09:16:05 EDT ---

The

Oct 31 13:14:22 a13a6fc1d422 systemd[1]: systemd-journald.service: Couldn't add fd to fd store: Operation not permitted

also stays in the journal, with --tmpfs /var.

--- Additional comment from Daniel Walsh on 2016-10-31 09:20:26 EDT ---

I guess the container image could remove the L+ from 

/lib/systemd-tmpfiles.d/etc.conf

grep mtab /lib/tmpfiles.d/etc.conf 
L+ /etc/mtab - - - - ../proc/self/mounts

I am not sure what the Journlad message means?

--- Additional comment from Daniel Walsh on 2017-03-12 08:10:11 EDT ---

Franticek could you see if this works with the latest rhel7-init, fedora-init and centos-init containers?

--- Additional comment from Fedora End Of Life on 2017-07-25 19:41:41 EDT ---

This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

--- Additional comment from Jan Pazdziora on 2017-07-28 08:40:31 EDT ---

I don't see neither of rhel7-init, fedora-init, or centos-init available:

$ docker run -ti --rm fedora-init bash -c 'set | grep container'
Unable to find image 'fedora-init:latest' locally
Trying to pull repository registry.access.redhat.com/fedora-init ... 
Trying to pull repository docker.io/library/fedora-init ... 
/usr/bin/docker-current: unauthorized: authentication required.
See '/usr/bin/docker-current run --help'.

$ docker run -ti --rm rhel-init bash -c 'set | grep container'
Unable to find image 'rhel-init:latest' locally
Trying to pull repository registry.access.redhat.com/rhel-init ... 
Trying to pull repository docker.io/library/rhel-init ... 
/usr/bin/docker-current: unauthorized: authentication required.
See '/usr/bin/docker-current run --help'.

$ docker run -ti --rm centos-init bash -c 'set | grep container'
Unable to find image 'centos-init:latest' locally
Trying to pull repository registry.access.redhat.com/centos-init ... 
Trying to pull repository docker.io/library/centos-init ... 
/usr/bin/docker-current: unauthorized: authentication required.
See '/usr/bin/docker-current run --help'.

--- Additional comment from Jan Pazdziora on 2017-07-28 08:41:05 EDT ---

The same with

$ docker run -ti --rm registry.fedoraproject.org/fedora-init bash -c 'set | grep container'
Unable to find image 'registry.fedoraproject.org/fedora-init:latest' locally
Trying to pull repository registry.fedoraproject.org/fedora-init ... 
/usr/bin/docker-current: manifest unknown: manifest unknown.
See '/usr/bin/docker-current run --help'.

--- Additional comment from Jan Pazdziora on 2017-07-28 08:46:49 EDT ---

The issue is still present with fedora:25

$ docker run --read-only=true --name systemd-ro -e container=docker --rm -ti registry.fedoraproject.org/fedora:25 /usr/sbin/init

It does not write anything to the output but in other terminal

$ docker exec systemd-ro systemctl | grep failed
● systemd-tmpfiles-setup.service                         loaded failed failed    Create Volatile Files and Directories
● systemd-update-utmp.service                            loaded failed failed    Update UTMP about System Boot/Shutdown

and the same result with registry.fedoraproject.org/fedora:rawhide.

This is with

docker-1.12.6-6.gitae7d637.fc25.x86_64
oci-systemd-hook-0.1.7-1.git1788cf2.fc25.x86_64

Running the container with --tmpfs /var fixes the systemd-update-utmp.service issue:

$ docker exec systemd-ro systemctl status systemd-update-utmp.service
● systemd-update-utmp.service - Update UTMP about System Boot/Shutdown
   Loaded: loaded (/usr/lib/systemd/system/systemd-update-utmp.service; static; vendor preset: disabled)
   Active: active (exited) since Fri 2017-07-28 12:45:26 UTC; 12s ago
     Docs: man:systemd-update-utmp.service(8)
           man:utmp(5)
  Process: 24 ExecStart=/usr/lib/systemd/systemd-update-utmp reboot (code=exited, status=0/SUCCESS)
 Main PID: 24 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/docker-574a515cc8cef21682cf750db3149a01e69b4babaf31da34a19cbc2d852e5e57.scope/system.slice/systemd-update-utmp.service

Jul 28 12:45:25 574a515cc8ce systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Jul 28 12:45:26 574a515cc8ce systemd[1]: Started Update UTMP about System Boot/Shutdown.

But systemd-tmpfiles-setup.service is still failing.

--- Additional comment from Daniel Walsh on 2017-07-28 08:57:56 EDT ---

I think the utmp service is not something we should fix, I think running with a --read-only container means the user should have to add /var as a mount point for systemd.  

 systemd-tmpfiles-setup.service Should probably be fixed.  

Is this the only one that fails with -v /PATH:/var:Z

Oct 31 12:53:46 aed69bad6512 systemd-tmpfiles[23]: symlink(../proc/self/mounts, /etc/mtab) failed: Read-only file system

--- Additional comment from Jan Pazdziora on 2017-07-28 09:25:56 EDT ---

Note that the /etc/mtab symlink is there

$ docker run --rm -ti registry.fedoraproject.org/fedora:25 ls -la /etc/mtab
lrwxrwxrwx. 1 root root 12 Jul 28 13:23 /etc/mtab -> /proc/mounts

it just points directly to /proc/mounts, instead of ../proc/self/mounts which would then point to /proc/self/mounts via

$ docker run --rm -ti registry.fedoraproject.org/fedora:25 ls -la /proc/mounts
lrwxrwxrwx. 1 root root 11 Jul 28 13:24 /proc/mounts -> self/mounts

So either changing util-linux to create the symlink to match what tmpfiles.d does, or changing it during base image build, would likely work.

--- Additional comment from Jan Pazdziora on 2017-07-28 09:27:34 EDT ---

Karel, what is your opinion about changing /etc/mtab symlink in util-linux to point to ../proc/self/mounts instead of to /proc/mounts, to minimize writes to /etc during startup?

--- Additional comment from Daniel Walsh on 2017-07-28 10:41:03 EDT ---

I agree that util-linux should change.  If we change it in the image, then will rpm -Vf /etc/mtab 
Complain?

--- Additional comment from Jan Pazdziora on 2017-07-28 10:49:24 EDT ---

It won't, the %files entry is defined as

%ghost %verify(not md5 size mtime) %config(noreplace,missingok) /etc/mtab

--- Additional comment from Daniel Walsh on 2017-07-28 10:50:30 EDT ---

Ok Franticek for now, lets fix the -init images to have /etc/mtab point at /proc/self/mounts

--- Additional comment from Karel Zak on 2017-08-01 05:17:35 EDT ---

If I good understand, the issuse is

  ln -sf /proc/mounts %{buildroot}/etc/mtab

in the util-linux.spec, because it does not match with systemd tmpfiles.d where is ../proc/self/mounts, right? I don't see a problem to update util-linux.spec file.

--- Additional comment from Daniel Walsh on 2017-08-01 08:29:01 EDT ---

Yes everytime a RHEL/Fedora system boots, systemd-tmpfiles is changing the default. so lets just make the rpm use the default and then we can handle readonly /etc partitions, without complaining.

--- Additional comment from Karel Zak on 2017-08-02 09:36:39 EDT ---

The mtab problem should be fixed by util-linux-2.30.1-3.fc27.

--- Additional comment from Daniel Walsh on 2017-08-02 09:54:19 EDT ---

Thanks, any chance of getting this into RHEL/Centos in the future.

--- Additional comment from Karel Zak on 2017-08-03 03:17:44 EDT ---

You can clone this BZ for RHEL7. We'll have rhel7.5 update (very probably), so add this trivial issue should not be a problem.

--- Additional comment from Daniel Walsh on 2017-08-03 08:38:05 EDT ---

Fixed in util-linux-2.30.1-3
Comment 3 Jan Pazdziora 2017-10-17 06:20:28 EDT
Hmm, it looks like this change was not needed in RHEL 7 after all, see output in bug 1503066.
Comment 4 Karel Zak 2017-10-17 08:14:54 EDT
(In reply to Jan Pazdziora from comment #3)
> Hmm, it looks like this change was not needed in RHEL 7 after all, see
> output in bug 1503066.

IMHO util-linux is completely irrelevant in this context. The util-linux change only makes spec file more robust to not fail on read-only system, all the change is:

-ln -sf /proc/mounts %{buildroot}/etc/mtab
+ln -sf ../proc/self/mounts %{buildroot}/etc/mtab
...
-       ln -fs /proc/mounts /etc/mtab
+       ln -sf ../proc/self/mounts /etc/mtab || :

I don't think it introduces any regression. The core of the functionality is systemd...

Note You need to log in before you can comment on or make changes to this bug.