Bug 1699048 - oc rsync crashes using --type=tar and --type=rsync
Summary: oc rsync crashes using --type=tar and --type=rsync
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 4.1.0
Assignee: Luis Sanchez
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-11 15:53 UTC by Rich Megginson
Modified: 2019-06-04 10:47 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:47:22 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:47:30 UTC
Origin (Github) 22535 None None None 2019-04-11 15:53:35 UTC

Description Rich Megginson 2019-04-11 15:53:35 UTC
Description of problem:
The following command line causes oc rsync to crash:

oc --loglevel=3 rsync --strategy=tar logging-ci-test-runner:/tmp/artifacts/ /tmp/artifacts

The loglevel doesn't matter - I've tried without it and with several other log levels.

Here are the last few log lines and the stacktrace:

I0411 00:41:40.627385    9812 sibling.go:16] Found parent command: oc
I0411 00:41:40.627429    9812 sibling.go:22] Setting root command to: oc
I0411 00:41:40.627438    9812 sibling.go:27] The sibling command is: oc rsh
I0411 00:41:40.635882    9812 copy_tar.go:119] Copying files with tar
I0411 00:41:40.636149    9812 copy_tar.go:147] Creating local tar file /tmp/rsync348440340 from remote path /tmp/artifacts/
I0411 00:41:40.636176    9812 copy_tar.go:203] Tarring /tmp/artifacts/ remotely
I0411 00:41:40.636189    9812 copy_tar.go:227] Remote tar command: tar -C /tmp/artifacts/ -c .
I0411 00:41:40.636200    9812 exec_remote.go:30] Remote executor running command: tar -C /tmp/artifacts/ -c .
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0x21b3b0f]

goroutine 1 [running]:
github.com/openshift/origin/pkg/oc/cli/rsync.(*remoteExecutor).Execute(0xc4218968a0, 0xc421900200, 0x5, 0x8, 0x0, 0x0, 0x2cc52e0, 0xc4218635e8, 0x2cc0000, 0xc42025d3b0, ...)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/exec_remote.go:45 +0x13f
github.com/openshift/origin/pkg/oc/cli/rsync.tarRemote(0x2cc0d40, 0xc4218968a0, 0xc4218bbd27, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/copy_tar.go:228 +0x917
github.com/openshift/origin/pkg/oc/cli/rsync.(*tarStrategy).Copy(0xc4217b8630, 0xc421906240, 0xc4219062a0, 0x2cc52e0, 0xc4200f4008, 0x2cc52e0, 0xc4200f4010, 0x0, 0x0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/copy_tar.go:149 +0xba9
github.com/openshift/origin/pkg/oc/cli/rsync.(*RsyncOptions).RunRsync(0xc421376400, 0x0, 0x0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/rsync.go:260 +0x7a
github.com/openshift/origin/pkg/oc/cli/rsync.NewCmdRsync.func1(0xc42137a500, 0xc42148b300, 0x2, 0x4)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/rsync.go:123 +0xb9
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc42137a500, 0xc42148b2c0, 0x4, 0x4, 0xc42137a500, 0xc42148b2c0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:760 +0x2c1
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc42126cc80, 0xc420f2fdf0, 0x13ff8a9, 0xc4202b48c0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:846 +0x30a
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(0xc42126cc80, 0x2, 0xc42126cc80)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:794 +0x2b
main.main()
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/cmd/oc/oc.go:68 +0x554


Version-Release number of selected component (if applicable):
oc version
Client Version: version.Info{Major:"4", Minor:"0+", GitVersion:"v4.0.0-alpha.0+3e176c9-1989-dirty", GitCommit:"3e176c9", GitTreeState:"", BuildDate:"2019-04-10T22:23:49Z", GoVersion:"", Compiler:"", Platform:""}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+3e176c9", GitCommit:"3e176c9", GitTreeState:"clean", BuildDate:"2019-04-10T22:26:36Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

How reproducible:
Crashes every time in logging CI - this is blocking logging CI



Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Ben Parees 2019-04-11 15:56:59 UTC
Adam you may want to involve Luis since he made recent changes:
https://github.com/openshift/origin/commit/b85af71f7db31e0840f36424719fabb463981000

Comment 2 Ben Parees 2019-04-11 15:57:34 UTC
see also: https://github.com/openshift/origin/issues/22535

Comment 3 Rich Megginson 2019-04-11 15:59:55 UTC
This is blocking logging CI
The test full-integ-aws launches a test pod running in the remote cluster - the only way we can get the test artifacts from the test pod is using oc rsync - so now we cannot get detailed information when a test fails, so it is very difficult to debug test failures.

Comment 4 Luis Sanchez 2019-04-11 19:34:34 UTC
PR opened: https://github.com/openshift/origin/pull/22552

Comment 5 Rich Megginson 2019-04-11 19:38:19 UTC
Update: I tried --type=rsync and got a different crash:

I0411 19:30:24.512179    9663 copy_rsync.go:72] Copying files with rsync
I0411 19:30:24.512324    9663 exec_local.go:19] Local executor running command: rsync --blocking-io --archive --no-owner --no-group --omit-dir-times --numeric-ids -v -e oc rsh --loglevel=3 logging-ci-test-runner:/tmp/artifacts/ /tmp/artifacts
receiving incremental file list
./
... all of the files were copied ...
sent 3,303 bytes  received 14,495,885 bytes  4,142,625.14 bytes/sec
total size is 14,476,708  speedup is 1.00
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0x21b7fde]

goroutine 1 [running]:
github.com/openshift/origin/pkg/oc/cli/rsync.podAPIChecker.CheckPod(0x0, 0x0, 0xc421524c40, 0x11, 0xc4216278c0, 0x16, 0x2774b60, 0xc42023f7c0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/util.go:136 +0x2e
github.com/openshift/origin/pkg/oc/cli/rsync.(*rsyncStrategy).Copy(0xc420cb5380, 0xc42023f5e0, 0xc42023f640, 0x2cc52e0, 0xc42000e018, 0x2cc52e0, 0xc42000e020, 0x0, 0x2a756c0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/copy_rsync.go:79 +0x392
github.com/openshift/origin/pkg/oc/cli/rsync.(*RsyncOptions).RunRsync(0xc42138c400, 0x0, 0x0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/rsync.go:260 +0x7a
github.com/openshift/origin/pkg/oc/cli/rsync.NewCmdRsync.func1(0xc42138ea00, 0xc42148f300, 0x2, 0x4)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/oc/cli/rsync/rsync.go:123 +0xb9
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc42138ea00, 0xc42148f2c0, 0x4, 0x4, 0xc42138ea00, 0xc42148f2c0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:760 +0x2c1
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4206c3180, 0xc4211cddf0, 0x13ff8a9, 0xc4202848c0)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:846 +0x30a
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4206c3180, 0x2, 0xc4206c3180)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:794 +0x2b
main.main()
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/cmd/oc/oc.go:68 +0x554

So at least we have a workaround for now, so lowering the priority

Comment 6 Maciej Szulik 2019-04-12 10:20:06 UTC
OC is owned by master and Luis already has a PR moving over to us.

Comment 7 Michal Fojtik 2019-04-16 09:05:17 UTC
The PR merged, moving to to QA.

Comment 8 Xingxing Xia 2019-04-18 11:04:42 UTC
Rich, could you help check again if latest oc (https://mirror.openshift.com/pub/openshift-v4/clients/oc/4.1/linux/) works for you? I first wanted to reproduce it against my env's some pods with old oc that does not include above fix PR, but I cann't reproduce the crash thus not sure of verifying it. Thanks!

Comment 9 Xingxing Xia 2019-04-19 09:31:02 UTC
(In reply to Xingxing Xia from comment #8)
> I first wanted to reproduce it against my env's some pods with old
> oc that does not include above fix PR, but I cann't reproduce the crash thus
> not sure of verifying it. Thanks!

I didn't reproduce it because my old oc does not include https://github.com/openshift/origin/commit/b85af71f7db31e0840f36424719fabb463981000 either.

Now, by below command, get the one that can reproduce it with --strategy=tar and --strategy=rsync-daemon (--strategy=rsync worked, though):
$ git tag --sort=taggerdate --contains b85af71 --no-contains e5b0115
openshift-4.1.0-201904100032
...

[xxia@fedora29 4.1.0-201904100032-2d04c98]$ ./oc rsh mysql-1-9j4vk mkdir /tmp/artifacts # Pod is created by `oc new-app centos/mysql-57-centos7 --name=mysql --env=MYSQL_USER=user1 --env=MYSQL_PASSWORD=password1 --env=MYSQL_DATABASE=mydb`
[xxia@fedora29 4.1.0-201904100032-2d04c98]$ ./oc rsh mysql-1-9j4vk touch /tmp/artifacts/abc; mkdir local
[xxia@fedora29 4.1.0-201904100032-2d04c98]$ ./oc rsync --strategy=tar mysql-1-9j4vk:/tmp/artifacts/ local
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0x2266fdf]
...

[xxia@fedora29 4.1.0-201904100032-2d04c98]$ ./oc rsync --strategy=rsync-daemon mysql-1-9j4vk:/tmp/artifacts/ local
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0x2266fdf]
...

Now try latest oc openshift-clients-4.1.0-201904190232.git.0.8f1385d.el7.x86_64 with --strategy=tar and --strategy=rsync-daemon and --strategy=rsync, all work.

Comment 11 errata-xmlrpc 2019-06-04 10:47:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.