RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1626071 - unable to run systemd container
Summary: unable to run systemd container
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: runc
Version: 7.5
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Frantisek Kluknavsky
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1627891
TreeView+ depends on / blocked
 
Reported: 2018-09-06 14:06 UTC by Qian Cai
Modified: 2020-07-20 08:32 UTC (History)
9 users (show)

Fixed In Version: runc-1.0.0-52.dev.git70ca035.el7_5
Doc Type: Known Issue
Doc Text:
Clone Of:
: 1627891 (view as bug list)
Environment:
Last Closed: 2018-09-26 07:48:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:2798 0 None None None 2018-09-26 07:48:19 UTC

Description Qian Cai 2018-09-06 14:06:35 UTC
Description of problem:
# podman run --name init -d brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhel7:7.5 init
container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/containers/storage/overlay-containers/1eb0619f939e5b4a9513c2f5eef99243827b1b4267117cd8db7945eff30de840/userdata/tmpfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/867e156851315d71fe16e064249997106e5dec9c132ca8a0adac1249623f0abf/merged\\\" at \\\"/tmp/runctmpdir635713766\\\" caused \\\"tmpcopyup: failed to move mount /tmp/runctmpdir635713766 to /var/lib/containers/storage/overlay/867e156851315d71fe16e064249997106e5dec9c132ca8a0adac1249623f0abf/merged/run: invalid argument\\\"\""
: internal libpod error

Reverted this commit fixes the issue.
commit 27ca091c08674b1a34058c9639d60455023ffa30
Date:   Tue Aug 21 07:31:00 2018 -0400

    Add proper support for systemd inside of podman

Version-Release number of selected component (if applicable):
podman-0.8.5-2.gitdc5a711.el7.ppc64le

How reproducible:
always

Comment 1 Matthew Heon 2018-09-06 14:21:39 UTC
I know we patched runc to fix this one - might not have a build with the patch in 7.5?

Dan - you wrote the patch, can you verify?

Comment 2 Qian Cai 2018-09-06 15:10:39 UTC
Well, I am not sure why which runc patches are you referring too, but I tried the latest upstream runc and failed with similar errors.

# /usr/bin/runc -v
runc version 1.0.0-rc5+dev
commit: 70ca035aa6ecfc496e13365fdef20383408501ba
spec: 1.0.0

# /usr/bin/podman run --name init -d brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhel7:7.5 init
container create failed: container_linux.go:336: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"tmpfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/f4807fca11cef5fc2d95599f59b7c8ee9c804ef83408b95d8155b4ee82f39375/merged\\\" at \\\"/tmp/runctop097532344/runctmpdir613014455\\\" caused \\\"tmpcopyup: failed to copy /var/lib/containers/storage/overlay/f4807fca11cef5fc2d95599f59b7c8ee9c804ef83408b95d8155b4ee82f39375/merged/sys/fs/cgroup/systemd to /tmp/runctop097532344/runctmpdir613014455: read /var/lib/containers/storage/overlay/f4807fca11cef5fc2d95599f59b7c8ee9c804ef83408b95d8155b4ee82f39375/merged/sys/fs/cgroup/systemd/cgroup.event_control: invalid argument\\\"\""
: internal libpod error

Comment 4 Daniel Walsh 2018-09-13 13:09:51 UTC
I wonder if this has something to do with sharing on /tmp

Could you try 

mount --make-rslave /tmp

On the host and then see if podman works?

Comment 5 Qian Cai 2018-09-13 13:52:47 UTC
# mount --make-rslave /tmp
mount: /tmp is not mountpoint or bad option

       In some cases useful info is found in syslog - try
       dmesg | tail or so.


# mount -t tmpfs none /tmp/

# mount --make-rslave /tmp

# findmnt -o propagation /tmp/
PROPAGATION
private

# podman run -d rhel7 init
container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/containers/storage/overlay-containers/b3d9bff4246b26f3c6707763b9e3daeaef62479eba6843108c8edce6796cc5e6/userdata/tmpfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/47b58ca79b3120718aaa64f1e2689cec1574e1fbf1ff5b7e074133a164bfb8d5/merged\\\" at \\\"/tmp/runctmpdir727174249\\\" caused \\\"tmpcopyup: failed to copy /var/lib/containers/storage/overlay/47b58ca79b3120718aaa64f1e2689cec1574e1fbf1ff5b7e074133a164bfb8d5/merged/sys/fs/cgroup/systemd to /tmp/runctmpdir727174249: read /var/lib/containers/storage/overlay/47b58ca79b3120718aaa64f1e2689cec1574e1fbf1ff5b7e074133a164bfb8d5/merged/sys/fs/cgroup/systemd/cgroup.event_control: invalid argument\\\"\""
: internal libpod error

Also tried runc-1.0.0-52.dev.git70ca035.el7_5. No luck.

Comment 6 Qian Cai 2018-09-13 15:16:04 UTC
There are 2 problems.

One is that there is file not readable.

# ls -l /sys/fs/cgroup/systemd/cgroup.event_control 
--w--w--w-. 1 root root 0 Sep 13 09:45 /sys/fs/cgroup/systemd/cgroup.event_control

so it need a patch in runc to skip it during tmpcopyup().

# diff -u vendor/github.com/mrunalp/fileutils/fileutils.go.orig vendor/github.com/mrunalp/fileutils/fileutils.go
--- vendor/github.com/mrunalp/fileutils/fileutils.go.orig	2018-09-13 10:32:28.435461780 -0400
+++ vendor/github.com/mrunalp/fileutils/fileutils.go	2018-09-13 11:02:46.372461780 -0400
@@ -58,6 +58,12 @@
 		}
 		defer sf.Close()
 
+		data := make([]byte, 100)
+		_, err = sf.Read(data)
+		if err != nil {
+			return nil
+		}
+
 		df, err := os.Create(dest)
 		if err != nil {
 			return err

Once applied the above patch, for some reasons /sys/fs/cgroup/systemd/ still have not been copy completely caused systemd not running properly in container.

# podman exec systemd find /sys/fs/cgroup/systemd
/sys/fs/cgroup/systemd
/sys/fs/cgroup/systemd/machine.slice
/sys/fs/cgroup/systemd/machine.slice/libpod-0649cfce13dd0f4f40e4d7a9afd596b8adad34ecffb13eebfb165ffd1e4421e2.scope
/sys/fs/cgroup/systemd/tasks
/sys/fs/cgroup/systemd/notify_on_release
/sys/fs/cgroup/systemd/cgroup.procs
/sys/fs/cgroup/systemd/cgroup.clone_children

If I manually bind mount the path then, it works fine.

# podman run --name systemd -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd:rw -d rhel7 init

# podman exec systemd ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  43092  3244 ?        Ss   15:08   0:00 init
root        15  0.0  0.1  39080  1912 ?        Ss   15:08   0:00 /usr/lib/systemd/systemd-journald
dbus        23  0.0  0.1  58064  2084 ?        Ss   15:08   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root        39  0.0  0.0  51708  1716 ?        Rs   15:15   0:00 ps aux

Comment 7 Daniel Walsh 2018-09-13 15:38:16 UTC
Lets push this back to RHEL7.6 to fix.  I don't think we should block release for RHEL7.5.5

Comment 10 Qian Cai 2018-09-13 15:48:15 UTC
The thing is if we can get this runc-1.0.0-52.dev.git70ca035.el7_5 version in RHEL-7.5.4, as least we have a workaround to run systemd container in podman.

# podman run --name systemd -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd:rw -d rhel7 init

Otherwise, it would fail with the current runc (runc-1.0.0-37.rc5.dev.gitad0f525.el7).

# podman run --name systemd -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd:rw -d rhel7 init
container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/containers/storage/overlay-containers/1d1e8aa92e0045b1f45822cd150c2a3768f65b335444ef85b6a2d8173704853e/userdata/tmpfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/22e3d2f65e17a46d7d95724c97a34ec834ff0ed74b6cbbaad4ed91f244c69029/merged\\\" at \\\"/tmp/runctmpdir188124887\\\" caused \\\"tmpcopyup: failed to move mount /tmp/runctmpdir188124887 to /var/lib/containers/storage/overlay/22e3d2f65e17a46d7d95724c97a34ec834ff0ed74b6cbbaad4ed91f244c69029/merged/tmp: invalid argument\\\"\""
: internal libpod error

Comment 12 Daniel Walsh 2018-09-13 17:40:04 UTC
runc-1.0.0-52.dev.git70ca035.el7_5 is built.

Comment 15 space88man 2018-09-15 01:56:52 UTC
Coming from the github bug https://github.com/containers/libpod/issues/1470#issuecomment-421387171.

Any chance we could have a build of runc-1.0.0-52.dev.git70ca035.el7_5 at https://cbs.centos.org/koji/packageinfo?packageID=3160?

Comment 17 errata-xmlrpc 2018-09-26 07:48:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2798


Note You need to log in before you can comment on or make changes to this bug.