Bug 1390191

I think you should volume mount in /var to eliminate a lot of these. 

The /etc/mtab one will be difficult to fix. 

mkdir /var/systemd-ro
# docker run --read-only=true --name systemd-ro -e container=docker -v /var/systemd-ro:/var:Z --rm -ti fedora:24 /usr/sbin/init
Unable to find image 'fedora:24' locally
Trying to pull repository atomic-registry.usersys.redhat.com/fedora ... 
Pulling repository atomic-registry.usersys.redhat.com/fedora
Trying to pull repository docker.io/library/fedora ... 
sha256:64a02df6aac27d1200c2572fe4b9949f1970d05f74d367ce4af994ba5dc3669e: Pulling from docker.io/library/fedora
Digest: sha256:64a02df6aac27d1200c2572fe4b9949f1970d05f74d367ce4af994ba5dc3669e
Status: Image is up to date for docker.io/fedora:24
systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Fedora 24 (Twenty Four)!

Set hostname to <758086f85759>.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Created slice System Slice.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Reached target Local File Systems.
[  OK  ] Reached target Swap.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Listening on Journal Socket.
         Starting Load/Save Random Seed...
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Slices.
         Starting Update is Completed...
[  OK  ] Listening on Process Core Dump Socket.
         Starting Journal Service...
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Update is Completed.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[FAILED] Failed to start Create Volatile Files and Directories.
See 'systemctl status systemd-tmpfiles-setup.service' for details.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Started dnf makecache timer.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
[  OK  ] Started D-Bus System Message Bus.
         Starting Permit User Sessions...
[  OK  ] Started Permit User Sessions.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.

The only failure I see is the creating of the /etc/mtab -> ../proc/self/mount

The

Oct 31 13:14:22 a13a6fc1d422 systemd[1]: systemd-journald.service: Couldn't add fd to fd store: Operation not permitted

also stays in the journal, with --tmpfs /var.

I guess the container image could remove the L+ from 

/lib/systemd-tmpfiles.d/etc.conf

grep mtab /lib/tmpfiles.d/etc.conf 
L+ /etc/mtab - - - - ../proc/self/mounts

I am not sure what the Journlad message means?

Franticek could you see if this works with the latest rhel7-init, fedora-init and centos-init containers?

This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

I don't see neither of rhel7-init, fedora-init, or centos-init available:

$ docker run -ti --rm fedora-init bash -c 'set | grep container'
Unable to find image 'fedora-init:latest' locally
Trying to pull repository registry.access.redhat.com/fedora-init ... 
Trying to pull repository docker.io/library/fedora-init ... 
/usr/bin/docker-current: unauthorized: authentication required.
See '/usr/bin/docker-current run --help'.

$ docker run -ti --rm rhel-init bash -c 'set | grep container'
Unable to find image 'rhel-init:latest' locally
Trying to pull repository registry.access.redhat.com/rhel-init ... 
Trying to pull repository docker.io/library/rhel-init ... 
/usr/bin/docker-current: unauthorized: authentication required.
See '/usr/bin/docker-current run --help'.

$ docker run -ti --rm centos-init bash -c 'set | grep container'
Unable to find image 'centos-init:latest' locally
Trying to pull repository registry.access.redhat.com/centos-init ... 
Trying to pull repository docker.io/library/centos-init ... 
/usr/bin/docker-current: unauthorized: authentication required.
See '/usr/bin/docker-current run --help'.

The same with

$ docker run -ti --rm registry.fedoraproject.org/fedora-init bash -c 'set | grep container'
Unable to find image 'registry.fedoraproject.org/fedora-init:latest' locally
Trying to pull repository registry.fedoraproject.org/fedora-init ... 
/usr/bin/docker-current: manifest unknown: manifest unknown.
See '/usr/bin/docker-current run --help'.

The issue is still present with fedora:25

$ docker run --read-only=true --name systemd-ro -e container=docker --rm -ti registry.fedoraproject.org/fedora:25 /usr/sbin/init

It does not write anything to the output but in other terminal

$ docker exec systemd-ro systemctl | grep failed
● systemd-tmpfiles-setup.service                         loaded failed failed    Create Volatile Files and Directories
● systemd-update-utmp.service                            loaded failed failed    Update UTMP about System Boot/Shutdown

and the same result with registry.fedoraproject.org/fedora:rawhide.

This is with

docker-1.12.6-6.gitae7d637.fc25.x86_64
oci-systemd-hook-0.1.7-1.git1788cf2.fc25.x86_64

Running the container with --tmpfs /var fixes the systemd-update-utmp.service issue:

$ docker exec systemd-ro systemctl status systemd-update-utmp.service
● systemd-update-utmp.service - Update UTMP about System Boot/Shutdown
   Loaded: loaded (/usr/lib/systemd/system/systemd-update-utmp.service; static; vendor preset: disabled)
   Active: active (exited) since Fri 2017-07-28 12:45:26 UTC; 12s ago
     Docs: man:systemd-update-utmp.service(8)
           man:utmp(5)
  Process: 24 ExecStart=/usr/lib/systemd/systemd-update-utmp reboot (code=exited, status=0/SUCCESS)
 Main PID: 24 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/docker-574a515cc8cef21682cf750db3149a01e69b4babaf31da34a19cbc2d852e5e57.scope/system.slice/systemd-update-utmp.service

Jul 28 12:45:25 574a515cc8ce systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Jul 28 12:45:26 574a515cc8ce systemd[1]: Started Update UTMP about System Boot/Shutdown.

But systemd-tmpfiles-setup.service is still failing.

I think the utmp service is not something we should fix, I think running with a --read-only container means the user should have to add /var as a mount point for systemd.  

 systemd-tmpfiles-setup.service Should probably be fixed.  

Is this the only one that fails with -v /PATH:/var:Z

Oct 31 12:53:46 aed69bad6512 systemd-tmpfiles[23]: symlink(../proc/self/mounts, /etc/mtab) failed: Read-only file system

Note that the /etc/mtab symlink is there

$ docker run --rm -ti registry.fedoraproject.org/fedora:25 ls -la /etc/mtab
lrwxrwxrwx. 1 root root 12 Jul 28 13:23 /etc/mtab -> /proc/mounts

it just points directly to /proc/mounts, instead of ../proc/self/mounts which would then point to /proc/self/mounts via

$ docker run --rm -ti registry.fedoraproject.org/fedora:25 ls -la /proc/mounts
lrwxrwxrwx. 1 root root 11 Jul 28 13:24 /proc/mounts -> self/mounts

So either changing util-linux to create the symlink to match what tmpfiles.d does, or changing it during base image build, would likely work.

Karel, what is your opinion about changing /etc/mtab symlink in util-linux to point to ../proc/self/mounts instead of to /proc/mounts, to minimize writes to /etc during startup?

I agree that util-linux should change.  If we change it in the image, then will rpm -Vf /etc/mtab 
Complain?

It won't, the %files entry is defined as

%ghost %verify(not md5 size mtime) %config(noreplace,missingok) /etc/mtab

Ok Franticek for now, lets fix the -init images to have /etc/mtab point at /proc/self/mounts

If I good understand, the issuse is

  ln -sf /proc/mounts %{buildroot}/etc/mtab

in the util-linux.spec, because it does not match with systemd tmpfiles.d where is ../proc/self/mounts, right? I don't see a problem to update util-linux.spec file.

Yes everytime a RHEL/Fedora system boots, systemd-tmpfiles is changing the default. so lets just make the rpm use the default and then we can handle readonly /etc partitions, without complaining.

The mtab problem should be fixed by util-linux-2.30.1-3.fc27.

Thanks, any chance of getting this into RHEL/Centos in the future.

You can clone this BZ for RHEL7. We'll have rhel7.5 update (very probably), so add this trivial issue should not be a problem.

Fixed in util-linux-2.30.1-3

https://bugzilla.redhat.com/show_bug.cgi?id=1478002

Unfortunately, this broke rawhide composes. ;( 

https://koji.fedoraproject.org/koji/taskinfo?taskID=21019385

https://kojipkgs.fedoraproject.org//work/tasks/9385/21019385/root.log

...
DEBUG util.py:439:  2017-08-03 18:01:49,563: Installing util-linux.x86_64 (710/1541)
DEBUG util.py:439:  2017-08-03 18:01:49,563: 
DEBUG util.py:439:  2017-08-03 18:01:49,563: The installation was stopped due to incomplete spokes detected while running in non-interactive cmdline mode. Since there cannot be any questions in cmdline mode, edit your kickstart file and retry installation.
DEBUG util.py:439:  2017-08-03 18:01:49,563: The exact error message is:
DEBUG util.py:439:  2017-08-03 18:01:49,563: 
DEBUG util.py:439:  2017-08-03 18:01:49,564: Non interactive installation failed: DNF error: Non-fatal POSTIN scriptlet failure in rpm package util-linux.
DEBUG util.py:439:  2017-08-03 18:01:49,564: 

I'm going to untag it from rawhide so we can get a compose...

Kevin does the compose look for links to non existent files?  If you have a tools that is looking for links that don't connect this could be an issue.  Since there might not be a /proc/self/mountinfo inside of chroot image.

The problem is that an rpm scriptlet exited non 0. 

I suspect: 

       ln -sf ../proc/self/mounts %{buildroot}/etc/mtab

is buildroot usable at install time there?
Shouldnt that just be /etc/mtab?

if you do 'ln -sf foo bar/baz/doesntexist' it indeed errors. ;) 

%ln -sf foo bar/baz/doesntexist
ln: failed to create symbolic link 'bar/baz/doesntexist': No such file or directory
% echo $?
1

Yes, copy & past bug. Sorry.

Fixed by util-linux-2.30.1-4.fc27 (I hope;-))

Still seems to be happening with -4. ;( 

https://koji.fedoraproject.org/koji/taskinfo?taskID=21048244

https://kojipkgs.fedoraproject.org//work/tasks/8244/21048244/root.log

...
DEBUG util.py:439:   util-linux                      x86_64 2.30.1-4.fc27               build 2.4 M
...
DEBUG util.py:439:  2017-08-04 15:02:47,867: Installing shadow-utils.x86_64 (708/1529)
DEBUG util.py:439:  2017-08-04 15:02:47,868: Installing libutempter.x86_64 (709/1529)
DEBUG util.py:439:  2017-08-04 15:02:47,868: Installing util-linux.x86_64 (710/1529)
DEBUG util.py:439:  2017-08-04 15:02:47,868: 
DEBUG util.py:439:  2017-08-04 15:02:47,868: The installation was stopped due to incomplete spokes detected while running in non-interactive cmdline mode. Since there cannot be any questions in cmdline mode, edit your kickstart file and retry installation.
DEBUG util.py:439:  2017-08-04 15:02:47,868: The exact error message is:
DEBUG util.py:439:  2017-08-04 15:02:47,868: 
DEBUG util.py:439:  2017-08-04 15:02:47,868: Non interactive installation failed: DNF error: Non-fatal POSTIN scriptlet failure in rpm package util-linux.
DEBUG util.py:439:  2017-08-04 15:02:47,868:

But why? The diff between versions -2 and -4:

--- a/util-linux.spec
+++ b/util-linux.spec
@@ -2,7 +2,7 @@
 Summary: A collection of basic system utilities
 Name: util-linux
 Version: 2.30.1
-Release: 2%{?dist}
+Release: 4%{?dist}
 License: GPLv2 and GPLv2+ and LGPLv2+ and BSD with advertising and Public Domain
 Group: System Environment/Base
 URL: http://en.wikipedia.org/wiki/Util-linux
@@ -394,7 +394,7 @@ chmod 644 misc-utils/getopt-*.{bash,tcsh}
 rm -f ${RPM_BUILD_ROOT}%{_datadir}/doc/util-linux/getopt/*
 rmdir ${RPM_BUILD_ROOT}%{_datadir}/doc/util-linux/getopt
 
-ln -sf /proc/mounts %{buildroot}/etc/mtab
+ln -sf ../proc/self/mounts %{buildroot}/etc/mtab
 
 # remove static libs
 rm -f $RPM_BUILD_ROOT%{_libdir}/lib{uuid,blkid,mount,smartcols,fdisk}.a
@@ -435,7 +435,7 @@ if [ -x /usr/sbin/selinuxenabled ] && /usr/sbin/selinuxenabled; then
        fi
 fi
 if [ ! -L /etc/mtab ]; then
-       ln -fs /proc/mounts /etc/mtab
+       ln -sf ../proc/self/mounts /etc/mtab
 fi
 
 %post -n libblkid
@@ -930,6 +930,12 @@ exit 0
 %{_libdir}/python*/site-packages/libmount/*
 
 %changelog
+* Fri Aug  4 2017 Karel Zak <kzak> - 2.30.1-4
+- fix post install script
+
+* Wed Aug  2 2017 Karel Zak <kzak> - 2.30.1-3
+- fix #1390191 - systemd read-only container produces errors
+

The POSTIN script is "ln -sf ../proc/self/mounts /etc/mtab" now, and it works as expected if I try it locally (f25, update by rpm -U).

Maybe use absolute path for ln(1), but...

I don't see it either. ;( 

Perhaps you could test in a mock chroot?

"fedpkg mockbuild" works as expected

And if I modify mock setting to use local -4 builds then 

 $ mock -r fedora-rawhide-x86_64 --init

works as expected and

  ...
  Installing  : util-linux-2.30.1-4.fc27.x86_64                             
  ...

returns no errors, and verification:

   mock -r fedora-rawhide-x86_64 --shell
   ...
   <mock-chroot> sh-4.4# rpm -q util-linux
   util-linux-2.30.1-4.fc27.x86_64

Kevin, maybe something is wrong with  @livemedia-build group. How I can reproduce this locally? It seems the group does not exist in the default rawhide.

 Warning: Group 'livemedia-build' does not exist.
 Error: Nothing to do.

Well, I see the logs (comment #28) again, and the issue is installation triggered by livemedia-creator kickstart with in already successfully installed mock root.

DEBUG util.py:522:  Executing command: ['/sbin/livemedia-creator', '--ks', '/tmp/koji-image-f27-build-21048244.ks', '--logfile', '/tmp/lmc-logs/livemedia-out.log', '--no-virt', '--resultdir', '/tmp/lmc', '--project', 'Fedora-Workstation-Live', '--make-iso', '--volid', 'Fedora-WS-Live-Rawhide-20170804.', '--iso-only', '--iso-name', 'Fedora-Workstation-Live-x86_64-Rawhide-20170804.n.0.iso', '--releasever', 'Rawhide', '--title', 'Fedora-Workstation-Live', '--macboot']

Yeah, this is livemedia-creator in mock calling lorax to do the installs, perhaps thats the part that matters. ;( 

I'm not sure how best to test this on the side. I guess a repo with your package and running livemedia-creator in mock ( https://rhinstaller.github.io/lorax/livemedia-creator.html#using-mock-and-no-virt-to-create-images )

There is also the shotgun approach of just adding || : to everywhere so it never "fails".

Ah yes, "|| :" ... I'll try this poor man hack in util-linux-2.30.1-5.fc27 ;-) We'll see.

Did we get this change to some Fedora 27 package?

f27 branch:

* Fri Sep 22 2017 Karel Zak <kzak> - 2.30.2-1
- upgrade to v2.30.2
  http://ftp.kernel.org/pub/linux/utils/util-linux/v2.30/v2.30.2-ReleaseNotes
 
* Mon Aug 14 2017 Karel Zak <kzak> - 2.30.1-5
- make ln-s usage more robust
 
* Fri Aug  4 2017 Karel Zak <kzak> - 2.30.1-4
- fix post install script

My point is, this bugzilla should likely have been CLOSED resolved long time ago, shouldn't it? ;-)

Thanks. ;-)