Bug 1280291 - Docker daemon panics when using socat in container
Docker daemon panics when using socat in container
Status: CLOSED EOL
Product: Fedora
Classification: Fedora
Component: docker (Show other bugs)
23
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Antonio Murdaca
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-11 07:18 EST by Michal Fojtik
Modified: 2016-12-20 10:39 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-12-20 10:39:57 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Michal Fojtik 2015-11-11 07:18:49 EST
Description of problem:

This is a little bit crazy scenario, but I managed to panic host docker daemon
by running socat in container :-)

First some background. I'm experimenting with the 'tianon/wine:64' image to run
tests for source-to-image project. Since this project also publish windows
binary, we should test it.
So the idea was to run the test in container with wine and use 'socat' to relay
/var/run/docker.sock to tcp://127.0.0.1:50333.

I have this script: https://gist.github.com/mfojtik/f89b13e49c740920365c that is also the reproducer. If you run this script, the docker will crash with following panic in logs:

Nov 11 13:09:59 pi docker[7472]: time="2015-11-11T13:09:59.486666851+01:00" level=info msg="GET /images/centos/ruby-22-centos7:latest/json"
Nov 11 13:09:59 pi docker[7472]: 2015/11/11 13:09:59 http: panic serving @: runtime error: invalid memory address or nil pointer dereference
Nov 11 13:09:59 pi docker[7472]: goroutine 322 [running]:
Nov 11 13:09:59 pi docker[7472]: net/http.(*conn).serve.func1(0xc8208e8f20, 0x7fe52829a2d8, 0xc820250588)
Nov 11 13:09:59 pi docker[7472]: /usr/lib/golang/src/net/http/server.go:1287 +0xb5 fp=0xc820d794c8 sp=0xc820d793f8
Nov 11 13:09:59 pi docker[7472]: runtime.call32(0x0, 0x1287be8, 0xc82025a530, 0x1800000018)
Nov 11 13:09:59 pi docker[7472]: /usr/lib/golang/src/runtime/asm_amd64.s:437 +0x3e fp=0xc820d794f0 sp=0xc820d794c8
Nov 11 13:09:59 pi docker[7472]: runtime.gopanic(0xf0f880, 0xc82000e070)
Nov 11 13:09:59 pi docker[7472]: /usr/lib/golang/src/runtime/panic.go:423 +0x4e9 fp=0xc820d79570 sp=0xc820d794f0
Nov 11 13:09:59 pi docker[7472]: runtime.panicmem()
Nov 11 13:09:59 pi docker[7472]: /usr/lib/golang/src/runtime/panic.go:42 +0x49 fp=0xc820d79598 sp=0xc820d79570
Nov 11 13:09:59 pi docker[7472]: runtime.sigpanic()
Nov 11 13:09:59 pi docker[7472]: /usr/lib/golang/src/runtime/sigpanic_unix.go:24 +0x2ba fp=0xc820d795e8 sp=0xc820d79598
Nov 11 13:09:59 pi docker[7472]: github.com/docker/docker/api/server.getpwuid(0xffffffff, 0x0, 0x0, 0x0, 0x0)
Nov 11 13:09:59 pi docker[7472]: /builddir/build/BUILD/docker-28c300fafb58c380d78381e08e1be35dfed5d4f9/_build/src/github.com/docker/docker/api/server/credentials_linux.go:75 +0x1f6 fp=0xc820d79678 sp=0xc820d795e8
Nov 11 13:09:59 pi docker[7472]: github.com/docker/docker/api/server.(*Server).LogAction(0xc820268400, 0x7fe5262595a0, 0xc820b946e0, 0xc820ba22a0, 0x0, 0x0)
Nov 11 13:09:59 pi docker[7472]: /builddir/build/BUILD/docker-28c300fafb58c380d78381e08e1be35dfed5d4f9/_build/src/github.com/docker/docker/api/server/credentials_linux.go:172 +0xaac fp=0xc820d79850 sp=0xc820d79678
Nov 11 13:09:59 pi docker[7472]: github.com/docker/docker/api/server.(*Server).makeHttpHandler.func1(0x7fe5262595a0, 0xc820b946e0, 0xc820ba22a0)
Nov 11 13:09:59 pi docker[7472]: /builddir/build/BUILD/docker-28c300fafb58c380d78381e08e1be35dfed5d4f9/_build/src/github.com/docker/docker/api/server/server.go:1680 +0x8aa fp=0xc820d79aa8 sp=0xc820d79850
Nov 11 13:09:59 pi docker[7472]: net/http.HandlerFunc.ServeHTTP(0xc82026ca80, 0x7fe5262595a0, 0xc820b946e0, 0xc820ba22a0)
Nov 11 13:09:59 pi docker[7472]: /usr/lib/golang/src/net/http/server.go:1422 +0x3a fp=0xc820d79ac8 sp=0xc820d79aa8
Nov 11 13:09:59 pi docker[7472]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc820272820, 0x7fe5262595a0, 0xc820b946e0, 0xc820ba22a0)
Nov 11 13:09:59 pi docker[7472]: /builddir/build/BUILD/docker-28c300fafb58c380d78381e08e1be35dfed5d4f9/vendor/src/github.com/gorilla/mux/mux.go:98 +0x29e fp=0xc820d79be0 sp=0xc820d79ac8
Nov 11 13:09:59 pi docker[7472]: net/http.serverHandler.ServeHTTP(0xc8203f0060, 0x7fe5262595a0, 0xc820b946e0, 0xc820ba22a0)
Nov 11 13:09:59 pi docker[7472]: /usr/lib/golang/src/net/http/server.go:1862 +0x19e fp=0xc820d79c40 sp=0xc820d79be0
Nov 11 13:09:59 pi docker[7472]: net/http.(*conn).serve(0xc8208e8f20)
Nov 11 13:09:59 pi docker[7472]: /usr/lib/golang/src/net/http/server.go:1361 +0xbee fp=0xc820d79f98 sp=0xc820d79c40
Nov 11 13:09:59 pi docker[7472]: runtime.goexit()
Nov 11 13:09:59 pi docker[7472]: /usr/lib/golang/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820d79fa0 sp=0xc820d79f98
Nov 11 13:09:59 pi docker[7472]: created by net/http.(*Server).Serve
Nov 11 13:09:59 pi docker[7472]: /usr/lib/golang/src/net/http/server.go:1910 +0x3f6



Version-Release number of selected component (if applicable):

Client:
 Version:      1.8.2-fc23
 API version:  1.20
 Package Version: docker-1.8.2-10.git28c300f.fc23.x86_64
 Go version:   go1.5.1
 Git commit:   cc2d489-dirty
 Built:        Tue Nov  3 06:41:23 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2-fc23
 API version:  1.20
 Package Version: 
 Go version:   go1.5.1
 Git commit:   cc2d489-dirty
 Built:        Tue Nov  3 06:41:23 UTC 2015
 OS/Arch:      linux/amd64



How reproducible:

$ git clone https://github.com/openshift/source-to-image
$ cd source-to-image && make release
$ copy the script from above to ./hack folder
$ ./hack/test-windows.sh

Wait for it to crash.


Actual results:

See panic.


Expected results:

It should not panic :-)
Comment 1 Michal Fojtik 2015-11-11 07:25:57 EST
BTW.

The "github.com/docker/docker/api/server/credentials_linux.go:75" does not even exists in upstream docker 1.8.2.
Comment 2 Pavel Odvody 2015-11-11 07:31:22 EST
Caused by a RH patch:
https://github.com/rhatdan/docker/pull/109
Comment 3 Nalin Dahyabhai 2015-11-11 16:09:36 EST
Michal, any chance I can get you to try again with http://koji.fedoraproject.org/koji/taskinfo?taskID=11796355?  The crash looks like the one described at https://bugzilla.redhat.com/show_bug.cgi?id=1275593#c15, and I think https://github.com/rhatdan/docker/pull/151 would fix it.

When I run your reproducer here, after I get a hexdump of the centos/ruby-22-centos7 image's metadata, s2i.exe seems to be stuck while reading from a pipe.  Am I missing a step in trying to reproduce the problem?
Comment 4 Michal Fojtik 2015-11-12 06:21:15 EST
I see this error in logs:


Nov 12 12:19:53 pi docker[22178]: time="2015-11-12T12:19:53.387069054+01:00" level=info msg="GET /images/centos/ruby-22-centos7:latest/json"
Nov 12 12:19:53 pi Docker[22178]: {Action=json, LoginUID=4294967295, PID=29784, }
Nov 12 12:19:53 pi docker[22178]: time="2015-11-12T12:19:53.387327326+01:00" level=error msg="Failed to get pwuid struct for UID 4294967295: strconv.ParseInt: parsing \"%!u(int=4294967295)\": invalid syntax"
Comment 5 Michal Fojtik 2015-11-12 07:08:32 EST
(In reply to Nalin Dahyabhai from comment #3)
> Michal, any chance I can get you to try again with
> http://koji.fedoraproject.org/koji/taskinfo?taskID=11796355?  The crash
> looks like the one described at
> https://bugzilla.redhat.com/show_bug.cgi?id=1275593#c15, and I think
> https://github.com/rhatdan/docker/pull/151 would fix it.
> 
> When I run your reproducer here, after I get a hexdump of the
> centos/ruby-22-centos7 image's metadata, s2i.exe seems to be stuck while
> reading from a pipe.  Am I missing a step in trying to reproduce the problem?

It is stuck because the Docker daemon crashed :-) So it will never get the message back.
With the latest RPM, the Docker daemon wont crash, but the request will error out with the error above. I think that is a parsing bug.
Comment 10 Daniel Walsh 2016-02-23 08:46:22 EST
Fixed in docker-1.10.
Comment 11 Fedora End Of Life 2016-11-24 08:17:45 EST
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 12 Fedora End Of Life 2016-12-20 10:39:57 EST
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.