Description of problem: oc rsh <<pod_name>> worked but commands hung for this pod. Eventually, receive error message rsh'ing into pod Version-Release number of selected component (if applicable): 3.6 How reproducible: 100% for this pod, other pod tested worked Steps to Reproduce: 1. oc rsh <<pod_name>> Actual results: rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:83: executing setns process caused \\\"exit status 16\\\"\"\n" command terminated with exit code 126 Expected results: Command prompt Additional info: CNS-backed pod running Coolstore MSA (gogs-postgresql) From node /var/log/messages: level=error msg="Error running exec in container: rpc error: code = 13 desc = invalid header field value \"oci runtime error: exec failed: container_linux.go:240: creating new parent process caused \\\"container_linux.go:1248: running lstat on namespace path \\\\\\\"/proc/94175/ns/ipc\\\\\\\" caused \\\\\\\"lstat /proc/94175/ns/ipc: no such file or directory\\\\\\\"\\\"\\n\"" W1107 12:09:22.963147 22572 prober.go:98] No ref for container "docker://1e61d150672b7737252f51834d71891c26fae394d63bf57fd60be8d9d74fa249" (gogs-postgresql-6-0dd1x_ci-ocuser(65ea0067-c3dc-11e7-92cf-001a4a160152):postgresql) prober.go:106] Readiness probe for "gogs-postgresql-6-0dd1x_ci-ocuser(65ea0067-c3dc-11e7-92cf-001a4a160152):postgresql" failed (failure): rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:240: creating new parent process caused \"container_linux.go:1248: running lstat on namespace path \\\"/proc/94175/ns/ipc\\\" caused \\\"lstat /proc/94175/ns/ipc: no such file or directory\\\"\"\n"
Possible related upstream issue: https://github.com/opencontainers/runc/issues/1476
Patch backported to: - projectatomic/runc @ docker-1.12.6 - projectatomic/runc @ docker-1.13.1-rhel Lokesh, pls make sure to rebuid docker with latest commits from the above branches.
Is there is step I can follow to verify the bug? I don't know how to reproduce the bug. thanks
which docker version fix the issue? I follow the step in https://github.com/opencontainers/runc/issues/1476 still have the same failure [root@ip-172-18-10-184 netns]# docker run -tid --name test000 ubuntu 916e495f4d2ad6459000655d8c19b14adf8228530c12185946897f87e6fe2f2d [root@ip-172-18-10-184 netns]# docker run -tid --name test111 --ipc container:test000 ubuntu bee80c5c44d86611f9283240cb29627ab39936e31d1fc408c21624c8f3de1e03 [root@ip-172-18-10-184 netns]# docker stop test000 test000 [root@ip-172-18-10-184 netns]# docker exec -ti test111 bash rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:240: creating new parent process caused \"container_linux.go:1248: running lstat on namespace path \\\"/proc/40504/ns/ipc\\\" caused \\\"lstat /proc/40504/ns/ipc: no such file or directory\\\"\"\n" [root@ip-172-18-10-184 netns]# [root@ip-172-18-10-184 netns]# docker version Client: Version: 1.12.6 API version: 1.24 Package version: docker-1.12.6-68.gitec8512b.el7.x86_64 Go version: go1.8.3 Git commit: ec8512b/1.12.6 Built: Thu Nov 16 15:19:17 2017 OS/Arch: linux/amd64 Server: Version: 1.12.6 API version: 1.24 Package version: docker-1.12.6-68.gitec8512b.el7.x86_64 Go version: go1.8.3 Git commit: ec8512b/1.12.6 Built: Thu Nov 16 15:19:17 2017 OS/Arch: linux/amd64
This is fixed in latest RPMs built from dist-git docker for 7.4 afaict. Package version: docker-1.12.6-71.git3e8e77d.el7.x86_64 ^^ that's an old version from December actually, please test the latest package. Assigning this to Lokesh, code wise, we have the patches in runc
Verify on docker-1.13.1-53.git774336d.el7.x86_64 # docker version Client: Version: 1.13.1 API version: 1.26 Package version: <unknown> Go version: go1.8.3 Git commit: 774336d/1.13.1 Built: Tue Feb 20 13:46:34 2018 OS/Arch: linux/amd64 Server: Version: 1.13.1 API version: 1.26 (minimum version 1.12) Package version: <unknown> Go version: go1.8.3 Git commit: 774336d/1.13.1 Built: Tue Feb 20 13:46:34 2018 OS/Arch: linux/amd64 Experimental: false # docker run -tid --name test000 ubuntu Unable to find image 'ubuntu:latest' locally Trying to pull repository docker.io/library/ubuntu ... latest: Pulling from docker.io/library/ubuntu 1be7f2b886e8: Pull complete 6fbc4a21b806: Pull complete c71a6f8e1378: Pull complete 4be3072e5a37: Pull complete 06c6d2f59700: Pull complete Digest: sha256:e27e9d7f7f28d67aa9e2d7540bdc2b33254b452ee8e60f388875e5b7d9b2b696 Status: Downloaded newer image for docker.io/ubuntu:latest b37edd6140b6e768fca995f8b6a57440935c66b875196926567a3338a7792669 # docker run -tid --name test111 --ipc container:test000 ubuntu 2e55e3a29218035c60e53e74d5f4101e832202317e209f01473d78adfefb621a # docker stop test000 test000 # docker exec -ti test111 bash root@2e55e3a29218:/#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0489
*** Bug 1599756 has been marked as a duplicate of this bug. ***