Bug 1444662
| Summary: | native.cgroupdriver=cgroupfs causes Docker hook to fail | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Nandhakumar Venkatachalam <v.nandha> |
| Component: | oci-register-machine | Assignee: | Jindrich Novy <jnovy> |
| Status: | CLOSED WONTFIX | QA Contact: | atomic-bugs <atomic-bugs> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.5 | CC: | amurdaca, christian.carle, dornelas, dwalsh, jamills, jhunsaker, lsm5, monotek23, schoudha |
| Target Milestone: | rc | Keywords: | Extras |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-08-01 02:30:00 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1186913, 1513780, 1561841, 1591075, 1611206, 1636853, 1647729, 1668273, 1688343 | ||
Yes it is fine to remove or disable this plugin. *** Bug 1546720 has been marked as a duplicate of this bug. *** I can reproduce this on the latest packages
# atomic host status
State: idle; auto updates disabled
Deployments:
● ostree://rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
Version: 7.5.3 (2018-08-15 11:18:39)
Commit: 03d524a16c8d76897f097565ca7452c1a5e2541f8c2beab145adf622499c7c64
ostree://rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
Version: 7.5.2 (2018-06-21 19:45:26)
Commit: 7eae04224d894f6f0b57bf3c77f78c749d64813bd1543290f4b0276c81082617
# rpm -q docker oci-register-machine
docker-1.13.1-74.git6e3bb8e.el7.x86_64
oci-register-machine-0-6.git2b44233.el7.x86_64
# systemctl cat docker.service
# /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target rhel-push-plugin.socket registries.service
Wants=docker-storage-setup.service
Requires=rhel-push-plugin.socket registries.service
Requires=docker-cleanup.timer
[Service]
Type=notify
NotifyAccess=main
EnvironmentFile=-/run/containers/registries.conf
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd-current \
--add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
--default-runtime=docker-runc \
--authorization-plugin=rhel-push-plugin \
--exec-opt native.cgroupdriver=systemd \
--userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
--init-path=/usr/libexec/docker/docker-init-current \
--seccomp-profile=/etc/docker/seccomp.json \
$OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$ADD_REGISTRY \
$BLOCK_REGISTRY \
$INSECURE_REGISTRY \
$REGISTRIES
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
KillMode=process
[Install]
WantedBy=multi-user.target
# /usr/lib/systemd/system/docker.service.d/flannel.conf
[Service]
EnvironmentFile=-/run/flannel/docker
# /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStart=
ExecStart=-/usr/bin/dockerd-current \
--add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
--default-runtime=docker-runc \
--authorization-plugin=rhel-push-plugin \
--exec-opt native.cgroupdriver=cgroupfs \
--userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
--init-path=/usr/libexec/docker/docker-init-current \
--seccomp-profile=/etc/docker/seccomp.json \
$OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$ADD_REGISTRY \
$BLOCK_REGISTRY \
$INSECURE_REGISTRY \
$REGISTRIES
# ps axwww | grep dockerd-current
12174 ? Ssl 0:00 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=cgroupfs --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --storage-driver overlay2 --add-registry registry.access.redhat.com
# docker info | grep cgroup
WARNING: You're not using the default seccomp profile
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Cgroup Driver: cgroupfs
# cat /etc/oci-register-machine.conf
# Disable oci-register-machine by setting the disabled field to true
##disabled : true
disabled : false
# docker run hello-world
Unable to find image 'hello-world:latest' locally
Trying to pull repository registry.access.redhat.com/hello-world ...
Trying to pull repository docker.io/library/hello-world ...
latest: Pulling from docker.io/library/hello-world
9db2ca6ccae0: Pull complete
Digest: sha256:4b8ff392a12ed9ea17784bd3c9a8b1fa3299cac44aca35a85c90c5e3c7afacdc
Status: Downloaded newer image for docker.io/hello-world:latest
container_linux.go:247: starting container process caused "process_linux.go:339: running prestart hook 1 caused \"error running hook: exit status 1, stdout: , stderr: \""
/usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:339: running prestart hook 1 caused \"error running hook: exit status 1, stdout: , stderr: \"".
This issue is only reproducible when the oci-register-machine hook is manually enabled. This hook is disabled by default, as it does not currently serve a purpose in Red Hat Enterprise Linux 7. The package/hook has been completely removed in Red Hat Enterprise Linux 8. The permanent fix for this issue is to disable the hook. To disable this hook from running please edit /etc/oci-register-machine.conf and change the setting back to 'disabled : true'. |
Description of problem: Version-Release number of selected component (if applicable): How reproducible: # docker run hello-world container_linux.go:247: starting container process caused "process_linux.go:334: running prestart hook 1 caused \"error running hook: exit status 1, stdout: , stderr: \"" /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:334: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"\n". # Steps to Reproduce: 1. On rhel 7.1, install following docker packages # uname -r 3.10.0-514.6.1.el7 # rpm -qa |grep docker docker-client-1.12.6-16.el7.x86_64 docker-1.12.6-16.el7.x86_64 docker-common-1.12.6-16.el7.x86_64 docker-rhel-push-plugin-1.12.6-16.el7.x86_64 # rpm -qa |grep systemd systemd-sysv-219-30.el7_3.8.x86_64 systemd-219-30.el7_3.8.x86_64 systemd-libs-219-30.el7_3.8.x86_64 oci-systemd-hook-0.1.7-2.git2788078.el7.x86_64 # With docker storage setup, my process arg looks like this # ps aux |grep docker root 2239 3.8 0.0 5013464 84172 ? Ssl 18:12 6:30 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --log-level=warn --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.use_deferred_deletion=true --iptables=false --ip-masq=false --storage-opt dm.use_deferred_removal=true --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.basesize=100G --storage-opt dm.thinpooldev=/dev/mapper/sys-docker--pool root 2259 0.0 0.0 1725232 21892 ? Ssl 18:12 0:05 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc root 30152 0.0 0.0 112644 972 pts/0 S+ 21:00 0:00 grep --color=auto docker Note: there is an option native.cgroupdriver=cgroupfs Actual results: [root@kubenode15 vnandha]# docker run hello-world container_linux.go:247: starting container process caused "process_linux.go:334: running prestart hook 1 caused \"error running hook: exit status 1, stdout: , stderr: \"" /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:334: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"\n". [root@kubenode15 vnandha]# Expected results: [root@kubenode15 vnandha]# docker run hello-world Unable to find image 'hello-world:latest' locally Trying to pull repository docker.io/library/hello-world ... sha256:c5515758d4c5e1e838e9cd307f6c6a0d620b5e07e6f927b07d05f6d12a1ac8d7: Pulling from docker.io/library/hello-world 78445dd45222: Pull complete Digest: sha256:c5515758d4c5e1e838e9cd307f6c6a0d620b5e07e6f927b07d05f6d12a1ac8d7 Status: Downloaded newer image for docker.io/hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://cloud.docker.com/ For more examples and ideas, visit: https://docs.docker.com/engine/userguide/ [root@kubenode15 vnandha]# docker run -it ubuntu bash Unable to find image 'ubuntu:latest' locally Trying to pull repository docker.io/library/ubuntu ... sha256:c2bbf50d276508d73dd865cda7b4ee9b5243f2648647d21e3a471dd3cc4209a0: Pulling from docker.io/library/ubuntu c62795f78da9: Pull complete d4fceeeb758e: Pull complete 5c9125a401ae: Pull complete 0062f774e994: Pull complete 6b33fd031fac: Pull complete Digest: sha256:c2bbf50d276508d73dd865cda7b4ee9b5243f2648647d21e3a471dd3cc4209a0 Status: Downloaded newer image for docker.io/ubuntu:latest root@aee01ae444a3:/# exit Additional info: If i set native.cgroupdriver=systemd things works fine.