| Summary: | during 'atomic install' - "Error response from daemon: invalid header field value..." | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Micah Abbott <miabbott> |
| Component: | docker | Assignee: | Daniel Walsh <dwalsh> |
| Status: | CLOSED EOL | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 25 | CC: | adimania, admiller, amurdaca, dwalsh, ichavero, jcajka, jchaloup, lsm5, marianne, miminar, mnguyen, nalin, riek, vbatts |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-12 10:40:59 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Looks like a hook failed to run and it didn't even provide a useful error message. So, first, could you try running with selinux Permissive? (it selinux isn't the problem here re-enable it) Secondly, could you ls -la /usr/libexec/oci/hooks.d/ directory and paste the output here? Lastly, remove (backup) every hook under the aforementioned directory (one by one) to see if some of them are causing this failure. (In reply to Antonio Murdaca from comment #1) > Looks like a hook failed to run and it didn't even provide a useful error > message. > > So, first, could you try running with selinux Permissive? > > (it selinux isn't the problem here re-enable it) Setting SELinux to Permissive helped: # setenforce 0 # getenforce Permissive # atomic --debug install docker.io/cockpit/ws Copying blob sha256:2bf01635e2a0f7ed3800c8cb3effc5ff46adc6b9b86f0e80743c956371efe553 69.45 MB / 69.50 MB [=========================================================] Copying blob sha256:fc813b2010153b217b1db393382860b65f749528c3e2a956fd3196734d488762 0 B / 13.38 KB [--------------------------------------------------------------] Copying blob sha256:07ea723a4787f1b33d848471e51a5927330c56bacdc5f12e5eac2ff9dfd7ed54 101.39 MB / 101.42 MB [=======================================================] Copying blob sha256:a94d88741a195fda31dbd8859c2192a301862eb6db15151d1cc4c579dfa685ce 0 B / 184 B [-----------------------------------------------------------------] Copying config sha256:eee19b785c06d101981d2dba694f3215dd025399e8157a4fa915e33526985d13 0 B / 5.54 KB [---------------------------------------------------------------] Writing manifest to image destination Storing signatures 5.54 KB / 5.54 KB [===========================================================]/usr/bin/docker run -ti --rm --privileged -v /:/host docker.io/cockpit/ws /container/atomic-install + sed -e /pam_selinux/d -e /pam_sepermit/d /etc/pam.d/cockpit + mkdir -p /host/etc/cockpit/ws-certs.d + chmod 755 /host/etc/cockpit/ws-certs.d + chown root:root /host/etc/cockpit/ws-certs.d + mkdir -p /host/var/lib/cockpit + chmod 775 /host/var/lib/cockpit + chown root:wheel /host/var/lib/cockpit + /bin/mount --bind /host/etc/cockpit /etc/cockpit + /usr/sbin/remotectl certificate --ensure Re-ran the 'atomic install' with SELinux enforcing and captured the following denial in the journal: Dec 13 21:34:00 fedora25ah-testing-dev audit[1607]: AVC avc: denied { search } for pid=1607 comm="systemd-machine" name="2062" dev="proc" ino=30873 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=dir permissive=0 Dec 13 21:34:00 fedora25ah-testing-dev oci-register-machine[2112]: 2016/12/13 21:34:00 Register machine failed: Failed to determine unit of process 2062 : Permission denied > Secondly, could you ls -la /usr/libexec/oci/hooks.d/ directory and paste the > output here? # ls -la /usr/libexec/oci/hooks.d/ total 2916 drwxr-xr-x. 2 root root 58 Jan 1 1970 . drwxr-xr-x. 3 root root 21 Jan 1 1970 .. -rwxr-xr-x. 3 root root 2960288 Jan 1 1970 oci-register-machine -rwxr-xr-x. 2 root root 23768 Jan 1 1970 oci-systemd-hook As a bonus: # rpm -q oci-register-machine oci-systemd-hook oci-register-machine-0-2.7.gitbb20b00.fc25.x86_64 oci-systemd-hook-0.1.4-4.git15c2f48.fc25.x86_64 > Lastly, remove (backup) every hook under the aforementioned directory (one > by one) to see if some of them are causing this failure. I can't modify the filesystem under /usr because this is an Atomic Host. That might require a new BZ... Dan, looks like a Selinux issue Looks like your docker daemon is running as unconfined_service_t? ps -eZ | grep docker dnf reinstall container-selinux (In reply to Daniel Walsh from comment #4) > Looks like your docker daemon is running as unconfined_service_t? > > ps -eZ | grep docker # ps -eZ | grep docker system_u:system_r:unconfined_service_t:s0 807 ? 00:00:00 docker-containe system_u:system_r:unconfined_service_t:s0 1000 ? 00:00:00 dockerd-current > dnf reinstall container-selinux This is not an option, since the host is F25 Atomic Host (testing). This is the version installed on 25.13 (91e5b2966f): # rpm -q container-selinux container-selinux-1.12.3-15.git0423d89.fc25.x86_64 Even upgrading to the next ostree commit (25.14 558575c9cd), which brings in container-selinux-1.12.4-2.git1b5971a.fc25.x86_64, I'm still seeing the docker process running as unconfined_service_t: # atomic host status State: idle Deployments: ● fedora-atomic:fedora-atomic/25/x86_64/testing/docker-host Version: 25.14 (2016-12-14 01:38:41) Commit: 558575c9cd2d0b91466fab034ed98b00c1ee987c5b9134796c0d9ce224034901 OSName: fedora-atomic fedora-atomic:fedora-atomic/25/x86_64/testing/docker-host Version: 25.13 (2016-12-13 05:09:43) Commit: 91e5b2966f8cc0b56cecc2957452b49f9ed2b5ebf9b759a3c5dedf1e2afbfed9 OSName: fedora-atomic # rpm -q container-selinux container-selinux-1.12.4-2.git1b5971a.fc25.x86_64 # ps -eZ | grep docker system_u:system_r:unconfined_service_t:s0 801 ? 00:00:00 docker-containe system_u:system_r:unconfined_service_t:s0 1002 ? 00:00:00 dockerd-current restorecon -R -v /usr/bin/docker* # restorecon -R -v /usr/bin/docker* # systemctl restart docker # ps -eZ | grep docker system_u:system_r:unconfined_service_t:s0 801 ? 00:00:00 docker-containe system_u:system_r:unconfined_service_t:s0 1511 ? 00:00:00 dockerd-current # ls -lZ /usr/bin/docker* -rwxr-xr-x. 3 root root system_u:object_r:bin_t:s0 532 Jan 1 1970 /usr/bin/docker -rwxr-xr-x. 2 root root system_u:object_r:bin_t:s0 11055464 Jan 1 1970 /usr/bin/docker-current -rwxr-xr-x. 2 root root system_u:object_r:bin_t:s0 29968064 Jan 1 1970 /usr/bin/dockerd-current -rwxr-xr-x. 3 root root system_u:object_r:bin_t:s0 28874 Jan 1 1970 /usr/bin/docker-storage-setup So this looks like this package is not setting up proper labeling. See if there is an update in updates-testing to fix this. chcon -t contianer_runtime_exec_t /usr/bin/docker* Should fix the issue until the package is updated. I'm at the latest from the Atomic Host testing branch (fedora-atomic/25/x86_64/testing/docker-host), so the most recent docker/container-linux package in bodhi hasn't landed in there: https://bodhi.fedoraproject.org/updates/FEDORA-2016-2a18b9e056 I'll retest once a newer version lands in that branch. Could you pull the new package that is building now, and make sure it fixes your issue. Using the latest docker build from koji: https://koji.fedoraproject.org/koji/buildinfo?buildID=825385 It looks like 'dockerd-current' is labeled as 'container_runtime_t', but 'docker-containerd-current' is labled as 'unconfined_service_t'. # ostree admin unlock --hotfix Copying /etc changes: 26 modified, 0 removed, 70 added Transaction complete; bootconfig swap: yes deployment count change: 0 Freed objects: 189.4 MB Hotfix mode enabled. A writable overlayfs is now mounted on /usr for this booted deployment. A non-hotfixed clone has been created as the non-default rollback target. # rpm -Uhv docker-1.12.4-6.git1b5971a.fc25.x86_64.rpm docker-common-1.12.4-6.git1b5971a.fc25.x86_64.rpm container-selinux-1.12.4-6.git1b5971a.fc25.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:container-selinux-2:1.12.4-6.git1################################# [ 17%] 2:docker-common-2:1.12.4-6.git1b597################################# [ 33%] 3:docker-2:1.12.4-6.git1b5971a.fc25################################# [ 50%] Cleaning up / removing... 4:docker-2:1.12.4-2.git1b5971a.fc25################################# [ 67%] 5:container-selinux-2:1.12.4-2.git1################################# [ 83%] 6:docker-common-2:1.12.4-2.git1b597################################# [100%] # restorecon -R -v /usr/bin/docker* # ls -lZ /usr/bin/docker* -rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 532 Dec 14 14:21 /usr/bin/docker -rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 11055456 Dec 14 14:25 /usr/bin/docker-current -rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 29968000 Dec 14 14:25 /usr/bin/dockerd-current -rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 28874 Dec 14 14:25 /usr/bin/docker-storage-setup # restorecon -R -v /usr/libexec/docker/* # ls -lZ /usr/libexec/docker/* -rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 7009928 Dec 14 14:25 /usr/libexec/docker/docker-containerd-current -rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 1701888 Dec 14 14:25 /usr/libexec/docker/docker-containerd-shim-current -rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 6498672 Dec 14 14:25 /usr/libexec/docker/docker-ctr-current -rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 1683576 Dec 14 14:25 /usr/libexec/docker/docker-proxy-current -rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 5041576 Dec 14 14:25 /usr/libexec/docker/docker-runc-current # systemctl restart docker # ps -eZ | grep docker system_u:system_r:unconfined_service_t:s0 801 ? 00:00:00 docker-containe system_u:system_r:container_runtime_t:s0 2725 ? 00:00:00 dockerd-current restart containerd also. Oops, that was the trick. Thanks Dan. Everything looks fixed now. You can mark this fixed in 'docker-1.12.4-6.git1b5971a' # systemctl restart docker-containerd # ps -eZ | grep docker system_u:system_r:container_runtime_t:s0 2962 ? 00:00:00 docker-containe system_u:system_r:container_runtime_t:s0 3043 ? 00:00:00 dockerd-current Fixed in docker-1.12.4-6.git1b5971a This message is a reminder that Fedora 25 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 25. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '25'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 25 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Fedora 25 changed to end-of-life (EOL) status on 2017-12-12. Fedora 25 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. |
Running the F25 Testing Atomic Host, I tried to 'atomic install docker.io/cockpit/ws'. The operation failed with the following error: # atomic --debug install docker.io/cockpit/ws:latest Copying blob sha256:2bf01635e2a0f7ed3800c8cb3effc5ff46adc6b9b86f0e80743c956371efe553 69.30 MB / 69.50 MB [=========================================================] Copying blob sha256:fc813b2010153b217b1db393382860b65f749528c3e2a956fd3196734d488762 0 B / 13.38 KB [--------------------------------------------------------------] Copying blob sha256:07ea723a4787f1b33d848471e51a5927330c56bacdc5f12e5eac2ff9dfd7ed54 101.36 MB / 101.42 MB [=======================================================] Copying blob sha256:a94d88741a195fda31dbd8859c2192a301862eb6db15151d1cc4c579dfa685ce 0 B / 184 B [-----------------------------------------------------------------] Copying config sha256:eee19b785c06d101981d2dba694f3215dd025399e8157a4fa915e33526985d13 0 B / 5.54 KB [---------------------------------------------------------------] Writing manifest to image destination Storing signatures 5.54 KB / 5.54 KB [===========================================================]/usr/bin/docker run -ti --rm --privileged -v /:/host docker.io/cockpit/ws:latest /container/atomic-install /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:334: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"\n". Traceback (most recent call last): File "/usr/bin/atomic", line 186, in <module> sys.exit(_func()) File "/usr/lib/python3.5/site-packages/Atomic/install.py", line 100, in install return util.check_call(cmd) File "/usr/lib/python3.5/site-packages/Atomic/util.py", line 138, in check_call return subprocess.check_call(cmd, env=env, stdin=stdin, stderr=stderr, stdout=stdout, close_fds=True) File "/usr/lib64/python3.5/subprocess.py", line 581, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/usr/bin/docker', 'run', '-ti', '--rm', '--privileged', '-v', '/:/host', 'docker.io/cockpit/ws:latest', '/container/atomic-install']' returned non-zero exit status 125 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/bin/atomic", line 196, in <module> sys.exit(e.returncode) SystemExit: 125 Some system info/package info: # atomic host status State: idle Deployments: ● fedora-atomic:fedora-atomic/25/x86_64/testing/docker-host Version: 25.13 (2016-12-13 05:09:43) Commit: 91e5b2966f8cc0b56cecc2957452b49f9ed2b5ebf9b759a3c5dedf1e2afbfed9 OSName: fedora-atomic fedora-atomic:fedora-atomic/25/x86_64/docker-host Version: 25.9 (2016-12-07 05:09:55) Commit: 7c0b62b1c6226c0826383e899b4f33992445f6a349018685b52e1c10e0d9160e OSName: fedora-atomic # rpm -q docker skopeo atomic docker-1.12.3-15.git0423d89.fc25.x86_64 skopeo-0.1.17-1.dev.git2b3af4a.fc25.x86_64 atomic-1.13.8-1.git8046db2.fc25.x86_64