Description of problem: After export http_proxy=<proxy server>:3128 and export https_proxy=<proxy server>:3128, run: $ oc rsh <pod name> $ oc exec <pod name> -- ls /etc/hosts $ oc port-forward hello-openshift 5006:8080 Both commands fail with panic: panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x68 pc=0xe65294] ... (see more below) Only reproduced in Online. Origin (v1.3.0-alpha.0-528-g703ad65) and OSE (openshift v3.2.0.44) don't reproduce. Version-Release number of selected component (if applicable): dev-preview-stg: openshift v3.2.0.44 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 and dev-preview-int How reproducible: Always Steps to Reproduce: 1. Login to Online via CLI, create project 2. Create app: $ oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-hello-world.git $ oc create -f origin/examples/hello-openshift/hello-pod.json 3. After pod deployed, set proxy env: $ export http_proxy=<proxy server>:3128 $ export https_proxy=<proxy server>:3128 4. Run: 1> oc get pod NAME READY STATUS RESTARTS AGE ruby-hello-world-1-build 0/1 Completed 0 25m ruby-hello-world-1-qo8a0 1/1 Running 0 17m hello-openshift 1/1 Running 0 4m 2> oc rsh ruby-hello-world-1-qo8a0 3> oc exec ruby-hello-world-1-qo8a0 -- ls /etc/hosts 4> oc port-forward hello-openshift 5006:8080 5. Unset the proxy env: $ unset http_proxy https_proxy Then repeat step 4. Actual results: 4. 2>, 3> and 4>: both commands throw out panic messages: panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x68 pc=0xe65294] goroutine 1 [running]: k8s.io/kubernetes/pkg/util/httpstream/spdy.(*SpdyRoundTripper).dial(0xc208732fc0, 0xc208434d00, 0x0, 0x0, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/k8s.io/kubernetes/pkg/util/httpstream/spdy/roundtripper.go:128 +0x524 k8s.io/kubernetes/pkg/util/httpstream/spdy.(*SpdyRoundTripper).RoundTrip(0xc208732fc0, 0xc208434d00, 0xd, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/k8s.io/kubernetes/pkg/util/httpstream/spdy/roundtripper.go:212 +0x107 k8s.io/kubernetes/pkg/client/transport.(*bearerAuthRoundTripper).RoundTrip(0xc20872dd80, 0xc208434c30, 0xa, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/k8s.io/kubernetes/pkg/client/transport/round_trippers.go:151 +0x242 k8s.io/kubernetes/pkg/client/transport.(*userAgentRoundTripper).RoundTrip(0xc20872dda0, 0xc208434b60, 0xc2086acff0, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/k8s.io/kubernetes/pkg/client/transport/round_trippers.go:89 +0x134 net/http.send(0xc208434a90, 0x7ff119b90f98, 0xc20872dda0, 0xe2, 0x0, 0x0) /usr/lib/golang/src/net/http/client.go:219 +0x4fc net/http.(*Client).send(0xc208732ff0, 0xc208434a90, 0xe2, 0x0, 0x0) /usr/lib/golang/src/net/http/client.go:142 +0x15b net/http.(*Client).doFollowingRedirects(0xc208732ff0, 0xc208434a90, 0x226d890, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/http/client.go:367 +0xb25 net/http.(*Client).Do(0xc208732ff0, 0xc208434a90, 0x19, 0x0, 0x0) /usr/lib/golang/src/net/http/client.go:177 +0x192 k8s.io/kubernetes/pkg/client/unversioned/remotecommand.(*streamExecutor).Dial(0xc2086a36c0, 0xc208607960, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/k8s.io/kubernetes/pkg/client/unversioned/remotecommand/remotecommand.go:117 +0x38c k8s.io/kubernetes/pkg/client/unversioned/remotecommand.(*streamExecutor).Stream(0xc2086a36c0, 0x7ff119b7e418, 0xc20802e000, 0x7ff119b7e390, 0xc20802e008, 0x7ff119b7e390, 0xc20802e010, 0x1, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/k8s.io/kubernetes/pkg/client/unversioned/remotecommand/remotecommand.go:151 +0xaa k8s.io/kubernetes/pkg/kubectl/cmd.(*DefaultRemoteExecutor).Execute(0x2cb0300, 0x1cd5c20, 0x4, 0xc208266070, 0xc208281540, 0x7ff119b7e418, 0xc20802e000, 0x7ff119b7e390, 0xc20802e008, 0x7ff119b7e390, ...) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/k8s.io/kubernetes/pkg/kubectl/cmd/exec.go:91 +0x111 k8s.io/kubernetes/pkg/kubectl/cmd.(*ExecOptions).Run(0xc20840e780, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/k8s.io/kubernetes/pkg/kubectl/cmd/exec.go:241 +0xb2f github.com/openshift/origin/pkg/cmd/cli/cmd.(*RshOptions).Run(0xc2083def60, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_build/src/github.com/openshift/origin/pkg/cmd/cli/cmd/rsh.go:135 +0x3e github.com/openshift/origin/pkg/cmd/cli/cmd.func·030(0xc20844ce00, 0xc2083f6e90, 0x1, 0x1) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_build/src/github.com/openshift/origin/pkg/cmd/cli/cmd/rsh.go:72 +0xc9 github.com/spf13/cobra.(*Command).execute(0xc20844ce00, 0xc2083f6e20, 0x1, 0x1, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/github.com/spf13/cobra/command.go:572 +0x82f github.com/spf13/cobra.(*Command).ExecuteC(0xc20825b400, 0xc20844ce00, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/github.com/spf13/cobra/command.go:662 +0x4db github.com/spf13/cobra.(*Command).Execute(0xc20825b400, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/github.com/spf13/cobra/command.go:618 +0x3a main.main() /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_build/src/github.com/openshift/origin/cmd/oc/oc.go:27 +0x175 goroutine 5 [chan receive]: github.com/golang/glog.(*loggingT).flushDaemon(0x2ca0900) /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/github.com/golang/glog/glog.go:879 +0x78 created by github.com/golang/glog.init·1 /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/github.com/golang/glog/glog.go:410 +0x2a7 goroutine 17 [syscall, locked to thread]: runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:2232 +0x1 goroutine 13 [syscall]: os/signal.loop() /usr/lib/golang/src/os/signal/signal_unix.go:21 +0x1f created by os/signal.init·1 /usr/lib/golang/src/os/signal/signal_unix.go:27 +0x35 goroutine 32 [select]: net/http.(*persistConn).writeLoop(0xc20804e000) /usr/lib/golang/src/net/http/transport.go:945 +0x41d created by net/http.(*Transport).dialConn /usr/lib/golang/src/net/http/transport.go:661 +0xcbc goroutine 31 [IO wait]: net.(*pollDesc).Wait(0xc2083d60d0, 0x72, 0x0, 0x0) /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2083d60d0, 0x0, 0x0) /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2083d6070, 0xc2086fa000, 0x1000, 0x1000, 0x0, 0x7ff119b7e468, 0xc2086e6aa8) /usr/lib/golang/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20802f0b0, 0xc2086fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/net.go:121 +0xdc crypto/tls.(*block).readFromUntil(0xc2084d8150, 0x7ff119b92300, 0xc20802f0b0, 0x5, 0x0, 0x0) /usr/lib/golang/src/crypto/tls/conn.go:454 +0xe6 crypto/tls.(*Conn).readRecord(0xc2084ec000, 0x17, 0x0, 0x0) /usr/lib/golang/src/crypto/tls/conn.go:539 +0x2da crypto/tls.(*Conn).Read(0xc2084ec000, 0xc2084e8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/crypto/tls/conn.go:904 +0x166 net/http.noteEOFReader.Read(0x7ff119b96d60, 0xc2084ec000, 0xc20804e058, 0xc2084e8000, 0x1000, 0x1000, 0x180f0e0, 0x0, 0x0) /usr/lib/golang/src/net/http/transport.go:1270 +0x6e net/http.(*noteEOFReader).Read(0xc208694b80, 0xc2084e8000, 0x1000, 0x1000, 0xc208012000, 0x0, 0x0) <autogenerated>:125 +0xd4 bufio.(*Reader).fill(0xc2084db260) /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).Peek(0xc2084db260, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/bufio/bufio.go:132 +0xf0 net/http.(*persistConn).readLoop(0xc20804e000) /usr/lib/golang/src/net/http/transport.go:842 +0xa4 created by net/http.(*Transport).dialConn /usr/lib/golang/src/net/http/transport.go:660 +0xc9f goroutine 35 [chan receive]: github.com/docker/docker/pkg/term.func·001() /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/github.com/docker/docker/pkg/term/term.go:114 +0x64 created by github.com/docker/docker/pkg/term.handleInterrupt /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/github.com/docker/docker/pkg/term/term.go:117 +0x1d5 goroutine 36 [chan receive]: k8s.io/kubernetes/pkg/kubectl/cmd.func·031() /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/k8s.io/kubernetes/pkg/kubectl/cmd/exec.go:210 +0x51 created by k8s.io/kubernetes/pkg/kubectl/cmd.(*ExecOptions).Run /builddir/build/BUILD/atomic-openshift-git-0.a4463d9/_thirdpartyhacks/src/k8s.io/kubernetes/pkg/kubectl/cmd/exec.go:213 +0x7a4 5. Both oc rsh and oc exec run successfully. Expected results: 4. Both oc rsh and oc exec should can run successfully behind proxy. Additional info:
Fixed in Origin in https://github.com/openshift/origin/pull/8896. Backport is in progress.
(In reply to Xingxing Xia from comment #0) > Description of problem: > After export http_proxy=<proxy server>:3128 and export https_proxy=<proxy ...... > Only reproduced in Online. Origin (v1.3.0-alpha.0-528-g703ad65) and OSE > (openshift v3.2.0.44) don't reproduce. Saw the PR 233 is merged into v3.2.0.45. But as said above, the bug only happened in Online. OSE doesn't has the problem, v3.2.0.44 and today's latest puddle v3.2.0.45 both work. So the result is not sufficient to have the bug VERIFIED until Online with >=v3.2.0.45 works. Or, do you have suggestion on this?
Found approach. Against Online dev-preview-stg, tried comparison test with oc v3.2.0.44 and oc v3.2.0.45 (both with auth and no-auth proxy): For oc v3.2.0.44, bug reproduces. For oc v3.2.0.45, the commands could work now, bug is fixed.
@Fabiano, But still curious about what makes Online reproduced but OSE didn't. Thank you!
This could only be reproduced when the API server has a SSL certificate from a trusted Certificate Authority, which doesn't require the CLI to skip the certificate authority check. We have it in online, but in OSE installations we usually use self-signed certs. Can you confirm that was the case?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1221