Description of problem: The following command line causes oc rsync to crash: oc --loglevel=3 rsync --strategy=tar logging-ci-test-runner:/tmp/artifacts/ /tmp/artifacts The loglevel doesn't matter - I've tried without it and with several other log levels. Here are the last few log lines and the stacktrace: I0411 00:41:40.627385 9812 sibling.go:16] Found parent command: oc I0411 00:41:40.627429 9812 sibling.go:22] Setting root command to: oc I0411 00:41:40.627438 9812 sibling.go:27] The sibling command is: oc rsh I0411 00:41:40.635882 9812 copy_tar.go:119] Copying files with tar I0411 00:41:40.636149 9812 copy_tar.go:147] Creating local tar file /tmp/rsync348440340 from remote path /tmp/artifacts/ I0411 00:41:40.636176 9812 copy_tar.go:203] Tarring /tmp/artifacts/ remotely I0411 00:41:40.636189 9812 copy_tar.go:227] Remote tar command: tar -C /tmp/artifacts/ -c . I0411 00:41:40.636200 9812 exec_remote.go:30] Remote executor running command: tar -C /tmp/artifacts/ -c . panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0x21b3b0f] goroutine 1 [running]: github.com/openshift/origin/pkg/oc/cli/rsync.(*remoteExecutor).Execute(0xc4218968a0, 0xc421900200, 0x5, 0x8, 0x0, 0x0, 0x2cc52e0, 0xc4218635e8, 0x2cc0000, 0xc42025d3b0, ...) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/exec_remote.go:45 +0x13f github.com/openshift/origin/pkg/oc/cli/rsync.tarRemote(0x2cc0d40, 0xc4218968a0, 0xc4218bbd27, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/copy_tar.go:228 +0x917 github.com/openshift/origin/pkg/oc/cli/rsync.(*tarStrategy).Copy(0xc4217b8630, 0xc421906240, 0xc4219062a0, 0x2cc52e0, 0xc4200f4008, 0x2cc52e0, 0xc4200f4010, 0x0, 0x0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/copy_tar.go:149 +0xba9 github.com/openshift/origin/pkg/oc/cli/rsync.(*RsyncOptions).RunRsync(0xc421376400, 0x0, 0x0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/rsync.go:260 +0x7a github.com/openshift/origin/pkg/oc/cli/rsync.NewCmdRsync.func1(0xc42137a500, 0xc42148b300, 0x2, 0x4) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/rsync.go:123 +0xb9 github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc42137a500, 0xc42148b2c0, 0x4, 0x4, 0xc42137a500, 0xc42148b2c0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:760 +0x2c1 github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc42126cc80, 0xc420f2fdf0, 0x13ff8a9, 0xc4202b48c0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:846 +0x30a github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(0xc42126cc80, 0x2, 0xc42126cc80) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:794 +0x2b main.main() /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/cmd/oc/oc.go:68 +0x554 Version-Release number of selected component (if applicable): oc version Client Version: version.Info{Major:"4", Minor:"0+", GitVersion:"v4.0.0-alpha.0+3e176c9-1989-dirty", GitCommit:"3e176c9", GitTreeState:"", BuildDate:"2019-04-10T22:23:49Z", GoVersion:"", Compiler:"", Platform:""} Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+3e176c9", GitCommit:"3e176c9", GitTreeState:"clean", BuildDate:"2019-04-10T22:26:36Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} How reproducible: Crashes every time in logging CI - this is blocking logging CI Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Adam you may want to involve Luis since he made recent changes: https://github.com/openshift/origin/commit/b85af71f7db31e0840f36424719fabb463981000
see also: https://github.com/openshift/origin/issues/22535
This is blocking logging CI The test full-integ-aws launches a test pod running in the remote cluster - the only way we can get the test artifacts from the test pod is using oc rsync - so now we cannot get detailed information when a test fails, so it is very difficult to debug test failures.
PR opened: https://github.com/openshift/origin/pull/22552
Update: I tried --type=rsync and got a different crash: I0411 19:30:24.512179 9663 copy_rsync.go:72] Copying files with rsync I0411 19:30:24.512324 9663 exec_local.go:19] Local executor running command: rsync --blocking-io --archive --no-owner --no-group --omit-dir-times --numeric-ids -v -e oc rsh --loglevel=3 logging-ci-test-runner:/tmp/artifacts/ /tmp/artifacts receiving incremental file list ./ ... all of the files were copied ... sent 3,303 bytes received 14,495,885 bytes 4,142,625.14 bytes/sec total size is 14,476,708 speedup is 1.00 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0x21b7fde] goroutine 1 [running]: github.com/openshift/origin/pkg/oc/cli/rsync.podAPIChecker.CheckPod(0x0, 0x0, 0xc421524c40, 0x11, 0xc4216278c0, 0x16, 0x2774b60, 0xc42023f7c0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/util.go:136 +0x2e github.com/openshift/origin/pkg/oc/cli/rsync.(*rsyncStrategy).Copy(0xc420cb5380, 0xc42023f5e0, 0xc42023f640, 0x2cc52e0, 0xc42000e018, 0x2cc52e0, 0xc42000e020, 0x0, 0x2a756c0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/copy_rsync.go:79 +0x392 github.com/openshift/origin/pkg/oc/cli/rsync.(*RsyncOptions).RunRsync(0xc42138c400, 0x0, 0x0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/rsync.go:260 +0x7a github.com/openshift/origin/pkg/oc/cli/rsync.NewCmdRsync.func1(0xc42138ea00, 0xc42148f300, 0x2, 0x4) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/rsync.go:123 +0xb9 github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc42138ea00, 0xc42148f2c0, 0x4, 0x4, 0xc42138ea00, 0xc42148f2c0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:760 +0x2c1 github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4206c3180, 0xc4211cddf0, 0x13ff8a9, 0xc4202848c0) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:846 +0x30a github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4206c3180, 0x2, 0xc4206c3180) /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:794 +0x2b main.main() /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/cmd/oc/oc.go:68 +0x554 So at least we have a workaround for now, so lowering the priority
OC is owned by master and Luis already has a PR moving over to us.
The PR merged, moving to to QA.
Rich, could you help check again if latest oc (https://mirror.openshift.com/pub/openshift-v4/clients/oc/4.1/linux/) works for you? I first wanted to reproduce it against my env's some pods with old oc that does not include above fix PR, but I cann't reproduce the crash thus not sure of verifying it. Thanks!
(In reply to Xingxing Xia from comment #8) > I first wanted to reproduce it against my env's some pods with old > oc that does not include above fix PR, but I cann't reproduce the crash thus > not sure of verifying it. Thanks! I didn't reproduce it because my old oc does not include https://github.com/openshift/origin/commit/b85af71f7db31e0840f36424719fabb463981000 either. Now, by below command, get the one that can reproduce it with --strategy=tar and --strategy=rsync-daemon (--strategy=rsync worked, though): $ git tag --sort=taggerdate --contains b85af71 --no-contains e5b0115 openshift-4.1.0-201904100032 ... [xxia@fedora29 4.1.0-201904100032-2d04c98]$ ./oc rsh mysql-1-9j4vk mkdir /tmp/artifacts # Pod is created by `oc new-app centos/mysql-57-centos7 --name=mysql --env=MYSQL_USER=user1 --env=MYSQL_PASSWORD=password1 --env=MYSQL_DATABASE=mydb` [xxia@fedora29 4.1.0-201904100032-2d04c98]$ ./oc rsh mysql-1-9j4vk touch /tmp/artifacts/abc; mkdir local [xxia@fedora29 4.1.0-201904100032-2d04c98]$ ./oc rsync --strategy=tar mysql-1-9j4vk:/tmp/artifacts/ local panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0x2266fdf] ... [xxia@fedora29 4.1.0-201904100032-2d04c98]$ ./oc rsync --strategy=rsync-daemon mysql-1-9j4vk:/tmp/artifacts/ local panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0x2266fdf] ... Now try latest oc openshift-clients-4.1.0-201904190232.git.0.8f1385d.el7.x86_64 with --strategy=tar and --strategy=rsync-daemon and --strategy=rsync, all work.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758