This is not a systemd bug, but application bug.

See https://access.redhat.com/solutions/4420581

In a nutshell, the service unit is executing some program that doesn't run in the expected cgroup (system.slice -> <service>.service), but probably in a user session, due to using "sudo" or "su"

I am aware of the KB and the reason why it fails, but I can't do much about it since that's the way the mongodb service is packaged.

Where should I file a bug report? I have tried in Foreman (where I initially found the issue), then libpod (since I use podman to start the container as a systemd container) and was then asked to file it here.

Another thing I noted was that running on a RHEL server is fails when using the CentOS container that is given above, but when I changed to ubi7-init and used `yum-config-manager --enable rhel-server-rhscl-7-rpms`, all other commands equal, it works.

It is the same version of the RPM file for mongodb and systemd inside the container. Not sure really what's changed but it seems that it only happens in the CentOS SCL and no the Red Hat SCL.

Where should I file a bug report to get this fixed?

Thanks!

I guess you should file against centos-scl then.

The problem looks to be on podman but not with systemd. 

<snip>

# podman start mongob
Error: unable to start container "mongob": container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"rootfs_linux.go:58: mounting \\\"/sys/fs/cgroup/systemd/machine.slice/libpod-d02aaf40cc0142ec5d2a7dba2e29aa756bc919eca9eb20a89bb69bd477e4e707.scope\\\" to rootfs \\\"/var/lib/containers/storage/overlay/9858159dbbcb30fdab3ba11ee8ecaf6ae1d2397dc5e68b8f3859ceb5611c70b2/merged\\\" at \\\"/sys/fs/cgroup/systemd\\\" caused \\\"stat /sys/fs/cgroup/systemd/machine.slice/libpod-d02aaf40cc0142ec5d2a7dba2e29aa756bc919eca9eb20a89bb69bd477e4e707.scope: no such file or directory\\\"\""
: OCI runtime error

<snip>

I checked that with docker-1.13.1-103.git7f2769b.el7.x86_64, I am able to run mongodb on centos container. 

# systemctl status rh-mongodb34-mongod
● rh-mongodb34-mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/rh-mongodb34-mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-10-08 13:21:27 UTC; 24min ago
  Process: 68 ExecStart=/opt/rh/rh-mongodb34/root/usr/libexec/mongodb-scl-helper enable $RH_MONGODB34_SCLS_ENABLED -- /opt/rh/rh-mongodb34/root/usr/bin/mongod $OPTIONS run (code=exited, status=0/SUCCESS)
 Main PID: 74 (mongod)
   CGroup: /system.slice/docker-c91cdf7b18086164b1e055c80b432a2923bbfed4e4a8ff4aa36efd9dcb745c6f.scope/system.slice/rh-mongodb34-mongod.service
           └─74 /opt/rh/rh-mongodb34/root/usr/bin/mongod -f /etc/opt/rh/rh-mongodb34/mongod.conf run

Oct 08 13:21:27 c91cdf7b1808 systemd[1]: Starting High-performance, schema-free document-oriented database...
Oct 08 13:21:27 c91cdf7b1808 mongodb-scl-helper[68]: about to fork child process, waiting until server is ready for connections.
Oct 08 13:21:27 c91cdf7b1808 mongodb-scl-helper[68]: forked process: 74
Oct 08 13:21:27 c91cdf7b1808 systemd[1]: Started High-performance, schema-free document-oriented database.


$ docker version 
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-103.git7f2769b.el7.x86_64
 Go version:      go1.10.8
 Git commit:      7f2769b/1.13.1
 Built:           Fri Aug  2 10:19:53 2019
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-103.git7f2769b.el7.x86_64
 Go version:      go1.10.8
 Git commit:      7f2769b/1.13.1
 Built:           Fri Aug  2 10:19:53 2019
 OS/Arch:         linux/amd64
 Experimental:    false


So filling a bug with podman will address the issue.

As per the description I did file a bug with podman but they redirected me to bugzilla and speculated that the issue is with systemd. However, I suspect that it is with podman, not properly taking https://access.redhat.com/solutions/4420581 into account when using `--systemd`

Also, I am able to start the container with podman, not getting the OCI error that Ramesh is seeing. My issue is that the service inside the container cannot start, due to systemd not trusting the PID file. My guess is that there is some sort of mismatch between the cgroups on host vs container, but I don't know enough about systemd to know exactly what's going on here.

Perhaps someone could explain to me exactly how systemd determines whether or not to trust the PID file?

I will put some quality time to investigate the podman issue tomorrow and update result.

I did go ahead and update podman to a 1.6.x revision on the same RHEL 7 host where I reproduced the problem and found that the newer revision was no longer susceptible to this problem.

# podman --version
podman version 1.6.4

# podman run --systemd=true --name test -d --privileged test
1e5976482e56ff8e53943ff4ed8840747c9dc9751d4ead102f8808fa78ddfac5

# podman exec -it test systemctl status rh-mongodb34-mongod
● rh-mongodb34-mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/rh-mongodb34-mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2020-02-15 20:28:07 UTC; 957ms ago
  Process: 73 ExecStart=/opt/rh/rh-mongodb34/root/usr/libexec/mongodb-scl-helper enable $RH_MONGODB34_SCLS_ENABLED -- /opt/rh/rh-mongodb34/root/usr/bin/mongod $OPTIONS run (code=exited, status=0/SUCCESS)
 Main PID: 85 (mongod)
   CGroup: /machine.slice/libpod-1e5976482e56ff8e53943ff4ed8840747c9dc9751d4ead102f8808fa78ddfac5.scope/system.slice/rh-mongodb34-mongod.service
           └─85 /opt/rh/rh-mongodb34/root/usr/bin/mongod -f /etc/opt/rh/rh-mongodb34/mongod.conf run

Feb 15 20:28:04 1e5976482e56 systemd[1]: Starting High-performance, schema-free document-oriented database...
Feb 15 20:28:06 1e5976482e56 mongodb-scl-helper[73]: about to fork child process, waiting until server is ready for connections.
Feb 15 20:28:06 1e5976482e56 mongodb-scl-helper[73]: forked process: 85
Feb 15 20:28:07 1e5976482e56 systemd[1]: Started High-performance, schema-free document-oriented database.

With that being the case, I'm fairly well convinced that the problem is actually in the podman stack. To that end, I am moving this bug to the applicable component to track the ongoing efforts to update to the 1.6.x podman revision in a future release.

Setting to post and assigning to to Jindrich on the off chance there's any packaging concerns.  I don't believe there are as this appears to be fixed as of podman 1.6.4-7.el7_8

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1227