Bug 1626071

I know we patched runc to fix this one - might not have a build with the patch in 7.5?

Dan - you wrote the patch, can you verify?

Well, I am not sure why which runc patches are you referring too, but I tried the latest upstream runc and failed with similar errors.

# /usr/bin/runc -v
runc version 1.0.0-rc5+dev
commit: 70ca035aa6ecfc496e13365fdef20383408501ba
spec: 1.0.0

# /usr/bin/podman run --name init -d brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhel7:7.5 init
container create failed: container_linux.go:336: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"tmpfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/f4807fca11cef5fc2d95599f59b7c8ee9c804ef83408b95d8155b4ee82f39375/merged\\\" at \\\"/tmp/runctop097532344/runctmpdir613014455\\\" caused \\\"tmpcopyup: failed to copy /var/lib/containers/storage/overlay/f4807fca11cef5fc2d95599f59b7c8ee9c804ef83408b95d8155b4ee82f39375/merged/sys/fs/cgroup/systemd to /tmp/runctop097532344/runctmpdir613014455: read /var/lib/containers/storage/overlay/f4807fca11cef5fc2d95599f59b7c8ee9c804ef83408b95d8155b4ee82f39375/merged/sys/fs/cgroup/systemd/cgroup.event_control: invalid argument\\\"\""
: internal libpod error

I wonder if this has something to do with sharing on /tmp

Could you try 

mount --make-rslave /tmp

On the host and then see if podman works?

# mount --make-rslave /tmp
mount: /tmp is not mountpoint or bad option

       In some cases useful info is found in syslog - try
       dmesg | tail or so.


# mount -t tmpfs none /tmp/

# mount --make-rslave /tmp

# findmnt -o propagation /tmp/
PROPAGATION
private

# podman run -d rhel7 init
container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/containers/storage/overlay-containers/b3d9bff4246b26f3c6707763b9e3daeaef62479eba6843108c8edce6796cc5e6/userdata/tmpfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/47b58ca79b3120718aaa64f1e2689cec1574e1fbf1ff5b7e074133a164bfb8d5/merged\\\" at \\\"/tmp/runctmpdir727174249\\\" caused \\\"tmpcopyup: failed to copy /var/lib/containers/storage/overlay/47b58ca79b3120718aaa64f1e2689cec1574e1fbf1ff5b7e074133a164bfb8d5/merged/sys/fs/cgroup/systemd to /tmp/runctmpdir727174249: read /var/lib/containers/storage/overlay/47b58ca79b3120718aaa64f1e2689cec1574e1fbf1ff5b7e074133a164bfb8d5/merged/sys/fs/cgroup/systemd/cgroup.event_control: invalid argument\\\"\""
: internal libpod error

Also tried runc-1.0.0-52.dev.git70ca035.el7_5. No luck.

There are 2 problems.

One is that there is file not readable.

# ls -l /sys/fs/cgroup/systemd/cgroup.event_control 
--w--w--w-. 1 root root 0 Sep 13 09:45 /sys/fs/cgroup/systemd/cgroup.event_control

so it need a patch in runc to skip it during tmpcopyup().

# diff -u vendor/github.com/mrunalp/fileutils/fileutils.go.orig vendor/github.com/mrunalp/fileutils/fileutils.go
--- vendor/github.com/mrunalp/fileutils/fileutils.go.orig	2018-09-13 10:32:28.435461780 -0400
+++ vendor/github.com/mrunalp/fileutils/fileutils.go	2018-09-13 11:02:46.372461780 -0400
@@ -58,6 +58,12 @@
 		}
 		defer sf.Close()
 
+		data := make([]byte, 100)
+		_, err = sf.Read(data)
+		if err != nil {
+			return nil
+		}
+
 		df, err := os.Create(dest)
 		if err != nil {
 			return err

Once applied the above patch, for some reasons /sys/fs/cgroup/systemd/ still have not been copy completely caused systemd not running properly in container.

# podman exec systemd find /sys/fs/cgroup/systemd
/sys/fs/cgroup/systemd
/sys/fs/cgroup/systemd/machine.slice
/sys/fs/cgroup/systemd/machine.slice/libpod-0649cfce13dd0f4f40e4d7a9afd596b8adad34ecffb13eebfb165ffd1e4421e2.scope
/sys/fs/cgroup/systemd/tasks
/sys/fs/cgroup/systemd/notify_on_release
/sys/fs/cgroup/systemd/cgroup.procs
/sys/fs/cgroup/systemd/cgroup.clone_children

If I manually bind mount the path then, it works fine.

# podman run --name systemd -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd:rw -d rhel7 init

# podman exec systemd ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  43092  3244 ?        Ss   15:08   0:00 init
root        15  0.0  0.1  39080  1912 ?        Ss   15:08   0:00 /usr/lib/systemd/systemd-journald
dbus        23  0.0  0.1  58064  2084 ?        Ss   15:08   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root        39  0.0  0.0  51708  1716 ?        Rs   15:15   0:00 ps aux

Lets push this back to RHEL7.6 to fix.  I don't think we should block release for RHEL7.5.5

The thing is if we can get this runc-1.0.0-52.dev.git70ca035.el7_5 version in RHEL-7.5.4, as least we have a workaround to run systemd container in podman.

# podman run --name systemd -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd:rw -d rhel7 init

Otherwise, it would fail with the current runc (runc-1.0.0-37.rc5.dev.gitad0f525.el7).

# podman run --name systemd -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd:rw -d rhel7 init
container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/containers/storage/overlay-containers/1d1e8aa92e0045b1f45822cd150c2a3768f65b335444ef85b6a2d8173704853e/userdata/tmpfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/22e3d2f65e17a46d7d95724c97a34ec834ff0ed74b6cbbaad4ed91f244c69029/merged\\\" at \\\"/tmp/runctmpdir188124887\\\" caused \\\"tmpcopyup: failed to move mount /tmp/runctmpdir188124887 to /var/lib/containers/storage/overlay/22e3d2f65e17a46d7d95724c97a34ec834ff0ed74b6cbbaad4ed91f244c69029/merged/tmp: invalid argument\\\"\""
: internal libpod error

runc-1.0.0-52.dev.git70ca035.el7_5 is built.

Coming from the github bug https://github.com/containers/libpod/issues/1470#issuecomment-421387171.

Any chance we could have a build of runc-1.0.0-52.dev.git70ca035.el7_5 at https://cbs.centos.org/koji/packageinfo?packageID=3160?

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2798