Created attachment 1015215 [details] go-caller patch Description of problem: docker using gcc-go crashes Version-Release number of selected component (if applicable): gcc-5.0.1-0.1.fc22 How reproducible: Always(reproduced on x86_64, ppc64le, s390x, but expecting ppc64 too) Steps to Reproduce: 1.install docker from https://repos.fedorapeople.org/repos/jcajka/docker-gccgo/ (ppc64(le), s390x) or https://copr.fedoraproject.org/coprs/jcajka/docker-gccgo/ (x86_64) 2.systemctl start docker 3.mkdir /root/dir 4.chcon -Rt svirt_sandbox_file_t /root/dir/ (just in case) 5.1 for x86: docker run fedora -itv /root/dir/:/root/dir/ /bin/bash 5.2 for ppc64(le),s390x(replace arch in docker.io/jcajka/fedora22-...): docker run docker.io/jcajka/fedora22-ppc64 -itv /root/dir/:/root/dir/ /bin/bash Actual results on ppc64le/x86: docker run docker.io/jcajka/fedora22-ppc64le -itv /root/dir/:/root/dir/ /bin/bash panic: runtime error: invalid memory address or nil pointer dereference on ppc64le: [signal 0xb code=0x1 addr=0x8] on x86_64: [signal 0xb code=0x1 addr=0x0] goroutine 16 [running]: github_com_docker_libcontainer_stacktrace.NewFrame /builddir/build/BUILD/docker-1dcc59a0b40f7d29ab2e0b8fdeb0adf537dbcbaf/vendor/src/github.com/docker/libcontainer/stacktrace/frame.go:12 github_com_docker_libcontainer_stacktrace.Capture /builddir/build/BUILD/docker-1dcc59a0b40f7d29ab2e0b8fdeb0adf537dbcbaf/vendor/src/github.com/docker/libcontainer/stacktrace/capture.go:18 libcontainer.newSystemError /builddir/build/BUILD/docker-1dcc59a0b40f7d29ab2e0b8fdeb0adf537dbcbaf/vendor/src/github.com/docker/libcontainer/generic_error.go:48 libcontainer.$nested4 /builddir/build/BUILD/docker-1dcc59a0b40f7d29ab2e0b8fdeb0adf537dbcbaf/vendor/src/github.com/docker/libcontainer/factory_linux.go:212 github_com_docker_libcontainer.StartInitialization.pN43_github_com_docker_libcontainer.LinuxFactory /builddir/build/BUILD/docker-1dcc59a0b40f7d29ab2e0b8fdeb0adf537dbcbaf/vendor/src/github.com/docker/libcontainer/factory_linux.go:224 native.initializer /builddir/build/BUILD/docker-1dcc59a0b40f7d29ab2e0b8fdeb0adf537dbcbaf/_build/src/github.com/docker/docker/daemon/execdriver/native/init.go:35 github_com_docker_docker_pkg_reexec.Init /builddir/build/BUILD/docker-1dcc59a0b40f7d29ab2e0b8fdeb0adf537dbcbaf/_build/src/github.com/docker/docker/pkg/reexec/reexec.go:26 main.main /builddir/build/BUILD/docker-1dcc59a0b40f7d29ab2e0b8fdeb0adf537dbcbaf/docker/docker.go:29 goroutine 18 [finalizer wait]: created by runtime_createfing ../../../libgo/runtime/mgc0.c:2572 goroutine 19 [syscall]: goroutine in C code; stack unavailable created by os_signal..import ../../../libgo/go/os/signal/signal_unix.go:25 on s390x(from log, timestamp and hostname ommited): . . . kernel: SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses mountpoint labeling kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses mountpoint labeling kernel: User process fault: interruption code 003b ilc:2in systemd-coredump[2aacf268000+1a000] kernel: failing address: 0000000000000000 TEID: 0000000000000400 kernel: Fault in primary space mode while using user ASCE. kernel: AS:000000004f8e01c7 R3:0000000000000024 kernel: CPU: 0 PID: 34376 Comm: systemd-coredum Not tainted 3.19.0-0.rc5.git2.1.fc22.s390x #1 kernel: task: 0000000060c557c0 ti: 00000000037cc000 task.ti: 00000000037cc000 kernel: User PSW : 0705000180000000 000002aacf274ce6 kernel: R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:1 AS:0 CC:0 PM:0 EA:3 User GPRS: ffffffff00000000 0000000000000000 0000000000000000 0000000000000001 kernel: 000002aacf274c6c 000003fffd5249f8 000002aacf27d928 000003ffffa7cd28 kernel: 000003ffffa7cd30 ffffffff00000000 000002aafe081250 000003ffffa7cc78 kernel: 000002aacf282ac8 000002aacf27d8f8 000002aacf274c6c 000003ffffa7cc50 kernel: User Code: 000002aacf274cd8: e320b0a80004 lg %r2,168(%r11) 000002aacf274cde: a7980000 lhi %r9,0 #000002aacf274ce2: 41121000 la %r1,0(%r2,%r1) >000002aacf274ce6: 92001000 mvi 0(%r1),0 000002aacf274cea: e310b0a00004 lg %r1,160(%r11) 000002aacf274cf0: e320b0b00004 lg %r2,176(%r11) 000002aacf274cf6: e32010000024 stg %r2,0(%r1) 000002aacf274cfc: a7190000 lghi %r1,0 kernel: Last Breaking-Event-Address: kernel: [<000002aacf274c74>] 0x2aacf274c74 kernel: docker0: port 1(veth260f764) entered disabled state kernel: device veth260f764 left promiscuous mode kernel: docker0: port 1(veth260f764) entered disabled state . . . Expected results: docker run docker.io/jcajka/fedora22-s390x -itv /root/dir/:/root/dir/ /bin/bash FATA[0000] Error response from daemon: Cannot start container fb6ed2f3efd57fa07ddf5c4d8576408d0f4cb95fe645a335063f7bed014a0654: [8] System error: exec: "-itv": executable file not found in $PATH Additional info: Crash seems to be triggered by creating new Frame using invalid pc/filename/line in libcontainer(seen this crash from time to time, but haven't got reliable reproducer, until now). As libgo's runtime.Caller(capture.go:14) returns seemingly invalid data(0,"",0) later used in runtime.FuncForPC/parseFunctionName(frame.go). This could be workaround by extending/fixing check in capture.go:15, but I think that runtime.Caller shouldn't return ok state when returned data seems invalid. Patch in attachment prevents this crash.
I will submit a gccgo bug on this.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65798 has been opened for this.
A fix was committed for this gcc bugzilla. It is not exactly as was suggested -- please verify this resolves the problem and if not we need to understand what was on the stack at the point runtime.Caller was called to understand why it returned invalid results.
Created attachment 1018046 [details] patch Hm..., it doesn't fix it. I have messed it up a bit..., problem might not be the runtime.Caller...( as the rejected function name part was working around the problem). I have prepare small program triggering the crash outside docker. package main import "runtime" import "fmt" func main() { for i := 1;i <= 5;i++ { pc, fname, line,ok := runtime.Caller(i) fmt.Printf("%x",pc) fmt.Print(" ",fname," ",line," ",ok,"\n") fn := runtime.FuncForPC(pc) fmt.Print(fn.Name(),"\n\n") } } Crashes with gcc-go: 7fefdc22b2c6 ../../../libgo/runtime/proc.c 550 true runtime_main 7fefdc228f8b ../../../libgo/runtime/proc.c 235 true kickoff 7fefdb20ffff 0 true panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x0] goroutine 16 [running]: main.main /root/test.go:12 goroutine 18 [finalizer wait]: created by runtime_createfing ../../../libgo/runtime/mgc0.c:2572 but works fine with golang(built with --ldflags "-linkmode external"): ./test 4125e3 /usr/lib/golang/src/runtime/proc.go 63 true runtime.main 4377e1 /usr/lib/golang/src/runtime/asm_amd64.s 2232 true runtime.goexit 0 0 false 0 0 false 0 0 false There is difference(probably expected?, this confused me...), in number of callers. But also libgo seems to behave differently than golang when accessing nil Func's Name, as ommiting/replacing the fn.Name() with just fn yields with gcc-go: 7feb8a48e2c6 ../../../libgo/runtime/proc.c 550 true &{{}} 7feb8a48bf8b ../../../libgo/runtime/proc.c 235 true &{{}} 7feb89472fff 0 true <nil> 0 0 false <nil> 0 0 false <nil> (Also tried just fn := runtime.FuncForPC(0) fmt.Print(fn.Name(),"\n\n") which works with golang, but fails with gcc-go) I have checked golang's code and indeed it does checking for nil. (symtab.go) func cfuncname(f *_func) *byte { if f == nil || f.nameoff == 0 { return nil } datap := findmoduledatap(f.entry) // inefficient return (*byte)(unsafe.Pointer(&datap.pclntable[f.nameoff])) } but libgo doesn't: (caller.c) String runtime_funcname_go (Func *f) { return f->name; } changing it: @@ -231,7 +231,13 @@ String runtime_funcname_go (Func *f) String runtime_funcname_go (Func *f) { - return f->name; + String str; + if (!f) + { + runtime_memclr (&str, sizeof str); + return str; + } + else return f->name; } seems to fix the crash(tested on x86, will do ppc). Also it seems that Entry() is not protected in both golang and libgo. I would expect same behavior as in Name(). Will open golang ticket. I hope now it is finally correct fix(or at least at right spot)... Patch in attachment.
Golang upstream ticket: https://github.com/golang/go/issues/10696
Opened gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66016
gcc-5.1.1-3.fc22,gcc-python-plugin-0.14-2.fc22 has been submitted as an update for Fedora 22. https://admin.fedoraproject.org/updates/gcc-5.1.1-3.fc22,gcc-python-plugin-0.14-2.fc22
Package gcc-5.1.1-3.fc22, gcc-python-plugin-0.14-2.fc22: * should fix your issue, * was pushed to the Fedora 22 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing gcc-5.1.1-3.fc22 gcc-python-plugin-0.14-2.fc22' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-9967/gcc-5.1.1-3.fc22,gcc-python-plugin-0.14-2.fc22 then log in and leave karma (feedback).
gcc-5.1.1-3.fc22, gcc-python-plugin-0.14-2.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.