Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1627891

Summary: unable to run systemd container
Product: Red Hat Enterprise Linux 7 Reporter: Qian Cai <qcai>
Component: podmanAssignee: Brent Baude <bbaude>
Status: CLOSED ERRATA QA Contact: Martin Jenner <mjenner>
Severity: high Docs Contact:
Priority: high    
Version: 7.6CC: dornelas, dwalsh, fkluknav, jligon, lsm5, mheon, mjenner, mpatel, pthomas, umohnani, ypu
Target Milestone: rcKeywords: Extras, Regression
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: podman-0.10.1.2-1.git2b4f8d1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1626071
: 1640211 (view as bug list) Environment:
Last Closed: 2018-10-30 11:54:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1626071    
Bug Blocks:    

Description Qian Cai 2018-09-11 19:14:48 UTC
Same thing for RHEL-7.6, i.e.,

podman-0.8.5-2.gitdc5a711.el7
runc-1.0.0-37.rc5.dev.gitad0f525.el7

# podman run -d --name httpd httpd-systemd
container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/containers/storage/overlay-containers/6ae5896c73c291b3f370c22589cafc8f389d8cab09e1a80ab67df60e26ebbcce/userdata/tmpfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/5b7fc8601658187a5548513dff37ac1c9db389e6464b9634352969f04528171d/merged\\\" at \\\"/tmp/runctmpdir173256754\\\" caused \\\"tmpcopyup: failed to move mount /tmp/runctmpdir173256754 to /var/lib/containers/storage/overlay/5b7fc8601658187a5548513dff37ac1c9db389e6464b9634352969f04528171d/merged/run: invalid argument\\\"\""
: internal libpod error

+++ This bug was initially created as a clone of Bug #1626071 +++

Description of problem:
# podman run --name init -d brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhel7:7.5 init
container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/containers/storage/overlay-containers/1eb0619f939e5b4a9513c2f5eef99243827b1b4267117cd8db7945eff30de840/userdata/tmpfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/867e156851315d71fe16e064249997106e5dec9c132ca8a0adac1249623f0abf/merged\\\" at \\\"/tmp/runctmpdir635713766\\\" caused \\\"tmpcopyup: failed to move mount /tmp/runctmpdir635713766 to /var/lib/containers/storage/overlay/867e156851315d71fe16e064249997106e5dec9c132ca8a0adac1249623f0abf/merged/run: invalid argument\\\"\""
: internal libpod error

Reverted this commit fixes the issue.
commit 27ca091c08674b1a34058c9639d60455023ffa30
Date:   Tue Aug 21 07:31:00 2018 -0400

    Add proper support for systemd inside of podman

Version-Release number of selected component (if applicable):
podman-0.8.5-2.gitdc5a711.el7.ppc64le

How reproducible:
always

--- Additional comment from Matthew Heon on 2018-09-06 10:21:39 EDT ---

I know we patched runc to fix this one - might not have a build with the patch in 7.5?

Dan - you wrote the patch, can you verify?

--- Additional comment from Qian Cai on 2018-09-06 11:10:39 EDT ---

Well, I am not sure why which runc patches are you referring too, but I tried the latest upstream runc and failed with similar errors.

# /usr/bin/runc -v
runc version 1.0.0-rc5+dev
commit: 70ca035aa6ecfc496e13365fdef20383408501ba
spec: 1.0.0

# /usr/bin/podman run --name init -d brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhel7:7.5 init
container create failed: container_linux.go:336: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"tmpfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/f4807fca11cef5fc2d95599f59b7c8ee9c804ef83408b95d8155b4ee82f39375/merged\\\" at \\\"/tmp/runctop097532344/runctmpdir613014455\\\" caused \\\"tmpcopyup: failed to copy /var/lib/containers/storage/overlay/f4807fca11cef5fc2d95599f59b7c8ee9c804ef83408b95d8155b4ee82f39375/merged/sys/fs/cgroup/systemd to /tmp/runctop097532344/runctmpdir613014455: read /var/lib/containers/storage/overlay/f4807fca11cef5fc2d95599f59b7c8ee9c804ef83408b95d8155b4ee82f39375/merged/sys/fs/cgroup/systemd/cgroup.event_control: invalid argument\\\"\""
: internal libpod error

Comment 1 Daniel Walsh 2018-09-13 12:17:50 UTC
We need a newer version of runc for rhel7.

Comment 2 Daniel Walsh 2018-09-13 12:20:37 UTC
rpm -q runc?

We are looking for 
runc-1.0.0-51.dev.gitfdd8055.fc28.x86_64
or later.

Comment 3 Daniel Walsh 2018-09-13 12:21:36 UTC
Franticek it looks like  a newer version of runc was built for rhel7.6, has this been released?

Comment 4 Qian Cai 2018-09-13 17:53:46 UTC
There are 2 problems.

One is that there is file not readable. 

# ls -l /sys/fs/cgroup/systemd/cgroup.event_control 
--w--w--w-. 1 root root 0 Sep 13 09:45 /sys/fs/cgroup/systemd/cgroup.event_control

Hence,

https://github.com/opencontainers/runc/blob/master/libcontainer/rootfs_linux.go

mountToRootfs() -> fileutils.CopyDirectory(dest, tmpDir) -> CopyFile() -> io.Copy () will error out.

tmpcopyup: failed to copy /var/lib/containers/storage/overlay/f4807fca11cef5fc2d95599f59b7c8ee9c804ef83408b95d8155b4ee82f39375/merged/sys/fs/cgroup/systemd to /tmp/runctop097532344/runctmpdir613014455: read /var/lib/containers/storage/overlay/f4807fca11cef5fc2d95599f59b7c8ee9c804ef83408b95d8155b4ee82f39375/merged/sys/fs/cgroup/systemd/cgroup.event_control: invalid argument

so it need a patch in runc latest upstream to skip it during tmpcopyup().

# diff -u vendor/github.com/mrunalp/fileutils/fileutils.go.orig vendor/github.com/mrunalp/fileutils/fileutils.go
--- vendor/github.com/mrunalp/fileutils/fileutils.go.orig	2018-09-13 10:32:28.435461780 -0400
+++ vendor/github.com/mrunalp/fileutils/fileutils.go	2018-09-13 11:02:46.372461780 -0400
@@ -58,6 +58,12 @@
 		}
 		defer sf.Close()
 
+		data := make([]byte, 100)
+		_, err = sf.Read(data)
+		if err != nil {
+			return nil
+		}
+
 		df, err := os.Create(dest)
 		if err != nil {
 			return err

Once applied the above patch, for some reasons /sys/fs/cgroup/systemd/ still have not been copy completely caused systemd not running properly in container.

# podman exec systemd find /sys/fs/cgroup/systemd
/sys/fs/cgroup/systemd
/sys/fs/cgroup/systemd/machine.slice
/sys/fs/cgroup/systemd/machine.slice/libpod-0649cfce13dd0f4f40e4d7a9afd596b8adad34ecffb13eebfb165ffd1e4421e2.scope
/sys/fs/cgroup/systemd/tasks
/sys/fs/cgroup/systemd/notify_on_release
/sys/fs/cgroup/systemd/cgroup.procs
/sys/fs/cgroup/systemd/cgroup.clone_children

If I manually bind mount the path then, it works fine.

# podman run --name systemd -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd:rw -d rhel7 init

# podman exec systemd ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  43092  3244 ?        Ss   15:08   0:00 init
root        15  0.0  0.1  39080  1912 ?        Ss   15:08   0:00 /usr/lib/systemd/systemd-journald
dbus        23  0.0  0.1  58064  2084 ?        Ss   15:08   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root        39  0.0  0.0  51708  1716 ?        Rs   15:15   0:00 ps aux

Comment 5 Qian Cai 2018-09-13 19:37:02 UTC
For 2nd problem.

podman will setup /sys/fs/cgroup/systemd as a tmpfs.

        {
            "destination": "/sys/fs/cgroup/systemd",
            "options": [
                "rw",
                "private",
                "noexec",
                "nosuid",
                "nodev",
                "tmpcopyup",
                "size=65536k"
            ],
            "source": "tmpfs",
            "type": "tmpfs"
        }

where end up with systemd did not spawn additional processes.

ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  42684  1912 ?        Ss   18:37   0:00 /usr/sbin/init
root        14  0.0  0.0  51740  1728 ?        Rs   18:38   0:00 ps aux

However, docker works fine with oci-systemd-hook. In oci-systemd-hook,

https://github.com/projectatomic/oci-systemd-hook/blob/master/src/systemdhook.c

/sys/fs/cgroup/systemd is actually bind mounted while /sys/fs/cgroup/ is a tmpfs,

	if (bind_mount(id, CGROUP_SYSTEMD, systemd_path, true)) {
		pr_perror("%s: Failed to bind mount %s on %s", id, CGROUP_SYSTEMD, systemd_path);
		return -1;
	}

	named_path = get_process_cgroup_subsystem_path(id, pid, "name=systemd");

	if (bind_mount(id, named_path, systemd_named_path, false)) {
		pr_perror("%s: Failed to bind mount %s on %s", id, CGROUP_SYSTEMD, systemd_named_path);
		return -1;
	}

	/* Mount tmpfs at new cgroup directory */
	if (mount("tmpfs", cgroup_path, "tmpfs", MS_NODEV|MS_NOSUID, options) == -1) {
		pr_perror("%s: Failed to mount tmpfs at %s", id, cgroup_path);
		return -1;
	}

Is this an oversight in podman?

Comment 6 Daniel Walsh 2018-09-14 13:10:15 UTC
I was able to get systemd to work without that, but I guess you expect the cgroup directory on the host to be bind mounted into the container.

you expect 

/sys/fs/cgroup/systemd/machine.slice/libpod-UUID.scope/ to be mounted on /sys/fs/cgroup/systemd?

Comment 7 Qian Cai 2018-09-14 14:25:54 UTC
Are you testing on a regular RHEL-7.6?

No, I expect that podman works fine with systemd container without -v /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd:rw

The problem is that during tmpcopyup of /sys/fs/cgroup/systemd, there is a lot of files there with RHEL-7.6 kernel not readable. Hence, tmpcopyup will fail on RHEL-7.6. It also looks like without those files been available in containers, systemd in container would refused to start properly.
 

# cp -r /sys/fs/cgroup/systemd/ /tmp/
cp: error reading ‘/sys/fs/cgroup/systemd/myRuntime/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/myRuntime/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/machine.slice/libpod-09d66bf25488f1b3baab18e1fecf9788766e6894e40122a73d5a1e7405eb7b5a.scope/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/machine.slice/libpod-09d66bf25488f1b3baab18e1fecf9788766e6894e40122a73d5a1e7405eb7b5a.scope/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/machine.slice/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/machine.slice/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/user.slice/user-0.slice/session-26.scope/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/user.slice/user-0.slice/session-26.scope/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/user.slice/user-0.slice/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/user.slice/user-0.slice/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/user.slice/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/user.slice/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/var-lib-docker-overlay2.mount/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/var-lib-docker-overlay2.mount/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/var-lib-docker-containers.mount/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/var-lib-docker-containers.mount/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/docker.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/docker.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/rhel-push-plugin.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/rhel-push-plugin.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/dev-mapper-rhel_ibm\\x2d\\x2dp8\\x2d\\x2dstorage\\x2d\\x2d02\\x2d\\x2dguest\\x2d\\x2d15\\x2dswap.swap/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/dev-mapper-rhel_ibm\\x2d\\x2dp8\\x2d\\x2dstorage\\x2d\\x2d02\\x2d\\x2dguest\\x2d\\x2d15\\x2dswap.swap/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/dev-rhel_ibm\\x2dp8\\x2dstorage\\x2d02\\x2dguest\\x2d15-swap.swap/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/dev-rhel_ibm\\x2dp8\\x2dstorage\\x2d02\\x2dguest\\x2d15-swap.swap/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/dev-dm\\x2d1.swap/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/dev-dm\\x2d1.swap/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/dev-disk-by\\x2did-dm\\x2duuid\\x2dLVM\\x2d7qds57lHcYamhcf3Gi9WaAdjkQJdgZOw5hlaX0uai5YOQTCGrb2lcClg0ajDjThy.swap/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/dev-disk-by\\x2did-dm\\x2duuid\\x2dLVM\\x2d7qds57lHcYamhcf3Gi9WaAdjkQJdgZOw5hlaX0uai5YOQTCGrb2lcClg0ajDjThy.swap/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/dev-disk-by\\x2did-dm\\x2dname\\x2drhel_ibm\\x2d\\x2dp8\\x2d\\x2dstorage\\x2d\\x2d02\\x2d\\x2dguest\\x2d\\x2d15\\x2dswap.swap/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/dev-disk-by\\x2did-dm\\x2dname\\x2drhel_ibm\\x2d\\x2dp8\\x2d\\x2dstorage\\x2d\\x2d02\\x2d\\x2dguest\\x2d\\x2d15\\x2dswap.swap/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/dev-disk-by\\x2duuid-f7699125\\x2d6050\\x2d4b41\\x2d9968\\x2dc08798cd239d.swap/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/dev-disk-by\\x2duuid-f7699125\\x2d6050\\x2d4b41\\x2d9968\\x2dc08798cd239d.swap/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/root-rootfs.mount/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/root-rootfs.mount/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/run-netns.mount/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/run-netns.mount/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/kdump.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/kdump.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/chrony-wait.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/chrony-wait.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/rhnsd.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/rhnsd.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/crond.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/crond.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/network.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/network.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/rsyslog.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/rsyslog.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/sshd.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/sshd.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/tuned.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/tuned.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/postfix.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/postfix.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/rhsmcertd.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/rhsmcertd.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/NetworkManager-wait-online.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/NetworkManager-wait-online.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/sshd-keygen.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/sshd-keygen.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/-.mount/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/-.mount/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/sys-kernel-config.mount/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/sys-kernel-config.mount/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-vconsole-setup.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-vconsole-setup.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/rhel-dmesg.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/rhel-dmesg.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-user-sessions.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-user-sessions.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-update-utmp.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-update-utmp.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-logind.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-logind.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/NetworkManager.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/NetworkManager.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/dbus.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/dbus.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/rtas_errd.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/rtas_errd.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/chronyd.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/chronyd.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/polkit.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/polkit.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/irqbalance.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/irqbalance.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-tmpfiles-setup.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-tmpfiles-setup.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/auditd.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/auditd.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/rhel-import-state.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/rhel-import-state.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-update-done.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-update-done.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-journal-catalog-update.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-journal-catalog-update.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/home.mount/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/home.mount/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/lvm2-monitor.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/lvm2-monitor.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/boot.mount/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/boot.mount/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-udev-trigger.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-udev-trigger.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/system-lvm2\\x2dpvscan.slice/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/system-lvm2\\x2dpvscan.slice/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-hwdb-update.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-hwdb-update.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/rhel-readonly.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/rhel-readonly.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/rhel-domainname.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/rhel-domainname.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-journal-flush.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-journal-flush.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/lvm2-lvmetad.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/lvm2-lvmetad.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-tmpfiles-setup-dev.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-tmpfiles-setup-dev.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-random-seed.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-random-seed.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-udevd.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-udevd.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-remount-fs.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-remount-fs.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-sysctl.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-sysctl.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/kmod-static-nodes.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/kmod-static-nodes.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/system-getty.slice/getty/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/system-getty.slice/getty/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/system-getty.slice/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/system-getty.slice/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/dev-mqueue.mount/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/dev-mqueue.mount/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/sys-kernel-debug.mount/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/sys-kernel-debug.mount/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/system-serial\\x2dgetty.slice/serial-getty/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/system-serial\\x2dgetty.slice/serial-getty/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/system-serial\\x2dgetty.slice/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/system-serial\\x2dgetty.slice/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/system-selinux\\x2dpolicy\\x2dmigrate\\x2dlocal\\x2dchanges.slice/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/system-selinux\\x2dpolicy\\x2dmigrate\\x2dlocal\\x2dchanges.slice/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/systemd-journald.service/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/systemd-journald.service/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/system.slice/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/system.slice/cgroup.event_control’: Invalid argument
cp: error reading ‘/sys/fs/cgroup/systemd/cgroup.event_control’: Invalid argument
cp: failed to extend ‘/tmp/systemd/cgroup.event_control’: Invalid argument

Comment 8 Qian Cai 2018-09-19 12:52:46 UTC
Maybe the easiest thing to do is to revert this podman commit,

commit 27ca091c08674b1a34058c9639d60455023ffa30
Date:   Tue Aug 21 07:31:00 2018 -0400

    Add proper support for systemd inside of podman

and let podman depends on oci-systemd-hook until some fixed tmpcopyup() in github.com/mrunalp/fileutils/fileutils.go properly.

Comment 11 Qian Cai 2018-10-17 14:08:04 UTC
This is still broken using podman-0.10.1.1-1.git4bea3e9.el7.

# podman run -d --name systemd brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhel7:7.6 /sbin/init

RRO[0001] container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:58: mounting \\\"/sys/fs/cgroup/systemd/libpod_parent/libpod-f442f1027f01a78a699c1352b8d06465d67c57386042484c2d46b4dc4dc3395d\\\" to rootfs \\\"/var/lib/containers/storage/overlay/cb3bb916810a2fa01e2f2e36b460f9f01995e212b4f80df44009a8f07695e596/merged\\\" at \\\"/sys/fs/cgroup/systemd\\\" caused \\\"stat /sys/fs/cgroup/systemd/libpod_parent/libpod-f442f1027f01a78a699c1352b8d06465d67c57386042484c2d46b4dc4dc3395d: no such file or directory\\\"\""
: internal libpod error

Comment 12 Qian Cai 2018-10-17 14:25:49 UTC
I believe it has something to with this bind mount in the generated config.json.

        {
            "destination": "/sys/fs/cgroup/systemd",
            "options": [
                "bind",
                "private"
            ],
            "source": "/sys/fs/cgroup/systemd/libpod_parent/libpod-f442f1027f01a78a699c1352b8d06465d67c57386042484c2d46b4dc4dc3395d",
            "type": "bind"
        }

In order to generate that source path, it needs to pass "--cgroup-manager cgroupfs" to podman.

Comment 13 Daniel Walsh 2018-10-17 16:42:28 UTC
Qian can you try this against the podman in master?

Comment 15 Qian Cai 2018-10-17 18:04:06 UTC
The new build works fine on s390x so far. Running on other arches...

Comment 17 Joy Pu 2018-10-19 10:40:19 UTC
Test in x86_64 and seems it works with following command:
# podman run --name systemd -d rhel7 init
b76098d0b616662d3d39809b149182e09853c872b355a8fb9d9e66cfd2cb8fa9

So set this to verified.

Comment 19 errata-xmlrpc 2018-10-30 11:54:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3370