Description of problem: Docker containers fail to start because ici-register-machine fails. Version-Release number of selected component (if applicable): docker-1.10.3-3.gitd93ee51.fc25.x86_64 How reproducible: 100% Steps to Reproduce: 1. docker run --rm busybox date 2. 3. Actual results: Container fails to start Expected results: Container starts Additional info: from the journal: Mar 23 11:05:21 hack2015 docker[7326]: time="2016-03-23T11:05:21.655385103-04:00" level=debug msg="Calling POST /v1.22/containers/7b687c0c73f3824e1bc0f8145621b54dd46768f0d24bb b02de1995895e12fa3c/start" Mar 23 11:05:21 hack2015 docker[7326]: time="2016-03-23T11:05:21.655413547-04:00" level=debug msg="POST /v1.22/containers/7b687c0c73f3824e1bc0f8145621b54dd46768f0d24bbb02de199 5895e12fa3c/start" ... [snip] ... Mar 23 11:05:21 hack2015 oci-register-machine[7740]: 2016/03/23 11:05:21 Register machine: prestart %!d(string=7b687c0c73f3824e1bc0f8145621b54dd46768f0d24bbb02de1995895e12fa3c) %!s(int=7713) /var/lib/docker/devicemapper/mnt/6f9a36c0ab1a6dfeb991b51dbfc2f4636d31cfc71c346ce930ea6c42b396ab2a/rootfs Mar 23 11:05:21 hack2015 oci-register-machine[7740]: 2016/03/23 11:05:21 Register machine failed: Failed to determine unit of process 7713 : No such device or address Mar 23 11:05:21 hack2015 docker[7326]: time="2016-03-23T11:05:21.976319905-04:00" level=warning msg="signal: killed" Mar 23 11:05:21 hack2015 oci-systemd-hook[7746]: systemdhook <debug>: Skipping as container command is date, not init or systemd Mar 23 11:05:21 hack2015 oci-register-machine[7747]: 2016/03/23 11:05:21 Register machine: poststop %!d(string=7b687c0c73f3824e1bc0f8145621b54dd46768f0d24bbb02de1995895e12fa3c) %!s(int=0) /var/lib/docker/devicemapper/mnt/6f9a36c0ab1a6dfeb991b51dbfc2f4636d31cfc71c346ce930ea6c42b396ab2a/rootfs Mar 23 11:05:21 hack2015 oci-register-machine[7747]: 2016/03/23 11:05:21 TerminateMachine failed: No machine '7b687c0c73f3824e1bc0f8145621b54dd46768f0d24bbb02de1995895e12fa3c'
This looks like a new feature of systemd?
systemd guys, is there some new requirement for registering a machine, that is just showing up in rawhide? Andy could you attach the output of cat /proc/self/cgroup From inside of the container.
I can't start any containers, so I can't provide that info. Do you want me to disable the oci-register-machine hook and then get the cgroup info? If so, here it is: $ sudo docker run --rm busybox cat /proc/self/cgroup 11:hugetlb:/docker/5b93dd25de4f0330f34ac997784f0610983187028ad3240922093f5d76306054 10:devices:/docker/5b93dd25de4f0330f34ac997784f0610983187028ad3240922093f5d76306054 9:cpu,cpuacct:/docker/5b93dd25de4f0330f34ac997784f0610983187028ad3240922093f5d76306054 8:freezer:/docker/5b93dd25de4f0330f34ac997784f0610983187028ad3240922093f5d76306054 7:blkio:/docker/5b93dd25de4f0330f34ac997784f0610983187028ad3240922093f5d76306054 6:pids:/system.slice/docker.service 5:memory:/docker/5b93dd25de4f0330f34ac997784f0610983187028ad3240922093f5d76306054 4:perf_event:/docker/5b93dd25de4f0330f34ac997784f0610983187028ad3240922093f5d76306054 3:net_cls,net_prio:/docker/5b93dd25de4f0330f34ac997784f0610983187028ad3240922093f5d76306054 2:cpuset:/docker/5b93dd25de4f0330f34ac997784f0610983187028ad3240922093f5d76306054 1:name=systemd:/docker/5b93dd25de4f0330f34ac997784f0610983187028ad3240922093f5d76306054 Note, I am running systemd-229-7.fc25.x86_64.
At the first glance this looks like a race. I think docker creates scope for the container but doesn't wait for systemd start job to complete and proceeds immediately with registration of the machine. This may fail because machined tries to determine cgroup of the container scope, however cgroup was not realized and processes were not migrated there yet. Are you sure you wait for completion of scope start job?
Andy are you running with the standard docker unit file? Is your docker daemon running with --exec-opt native.cgroupdriver=systemd Michal We are registering the machine after the process is created, so I figure it is registered with the cgroup. But Mrunal would no for sure. Is this a new check in systemd-229-7? We have not seen this problem before.
I'm using the standard unit file from the rpm. The only change I made to /etc/sysconfig/docker was to enable debug logging (-D). It does not appear to be running with any --exec-opt settings. The output from ps is /usr/bin/docker daemon --selinux-enabled --log-driver=journald -D
Could you add that option to /etc/sysconfig/docker and see if that fixes the issue. docker-1.10 is no longer using systemd by default for setting up cgroups. We are supposed to have changed the default back in our package.
Setting the cgroupdriver to systemd fixed the problem.
Lokesh could you make sure this flag is in /etc/sysconfig/docker
*** Bug 1321618 has been marked as a duplicate of this bug. ***
see: http://koji.fedoraproject.org/koji/taskinfo?taskID=13496951 and http://pkgs.fedoraproject.org/cgit/rpms/docker.git/commit/?id=e4b2e3cdf939b8deb074306bf5aa4c180962bdad
I'm having the same problem in Fedora 24. Any plans to backport the fix?
Yes we need to make this the default in F24 and all docker-1.10 packages.
(In reply to Andreu Botella from comment #12) > I'm having the same problem in Fedora 24. Any plans to backport the fix? Yes, I've added this to f24 as well, the update can be found here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-976f49409c . Please add karma so we can move it to stable quicker.
Lokesh that fixes the problem, but you should have put the cgroup option into /etc/sysconfig/docker rather then in the unit file, so people could easily change it.