Bug 1409878

Summary: oc get build, fails when sorting by startTimestamp
Product: OpenShift Container Platform Reporter: rcerven
Component: ocAssignee: Fabiano Franz <ffranz>
Status: CLOSED ERRATA QA Contact: Xingxing Xia <xxia>
Severity: medium Docs Contact:
Priority: high    
Version: 3.3.1CC: aos-bugs, avagarwa, haowang, jokerman, mmccomas, smunilla, xiaocwan, xxia
Target Milestone: ---Keywords: Rebase
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: '--sort-by' in 'oc get' fails when any object in the list contains an empty value in the field used to sort. Consequence: panic. Fix: correctly handle empty fields in '--sort-by'. Result: Output of 'oc get' is printed correctly and empty fields are considered in sorting.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-10 05:17:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
oc output none

Description rcerven 2017-01-03 17:05:52 UTC
Created attachment 1236929 [details]
oc output

Description of problem:
oc get build, fails when sorting by startTimestamp
oc get builds --sort-by={.status.startTimestamp}

it will fail if there exists any builds which haven't started yet

Version-Release number of selected component (if applicable):
3.3.1.3

How reproducible:
every time, if there exists any builds which haven't started yet

Steps to Reproduce:
1.oc get builds --sort-by={.status.startTimestamp}
2.
3.

Actual results:
in attachment

Expected results:
not fail, and treat empty startTimestamp as valid

Additional info:

Comment 1 Xingxing Xia 2017-01-04 03:09:09 UTC
FYI, I didn't get panic as above attachment.

Paste following 2 lines commands together to execute them quickly:
versions/v3.3.1.3/oc start-build ruby-sample-build
versions/v3.3.1.3/oc get builds --sort-by={.status.startTimestamp}

Then I got from shell screen:
$ versions/v3.3.1.3/oc start-build ruby-sample-build
build "ruby-sample-build-2" started

$ versions/v3.3.1.3/oc get builds --sort-by={.status.startTimestamp}
proto: no encoder for TypeMeta unversioned.TypeMeta [GetProperties]
proto: tag has too few fields: "-"
proto: no coders for struct *reflect.rtype
proto: no encoder for sec int64 [GetProperties]
proto: no encoder for nsec int32 [GetProperties]
proto: no encoder for loc *time.Location [GetProperties]
proto: no encoder for Time time.Time [GetProperties]
proto: no encoder for i resource.int64Amount [GetProperties]
proto: no encoder for d resource.infDecAmount [GetProperties]
proto: no encoder for s string [GetProperties]
proto: no encoder for Format resource.Format [GetProperties]
F0104 10:59:28.254133   25064 sorting_printer.go:207] Field {.status.startTimestamp} in TypeMeta:<kind:"Build" apiVersion:"v1" > metadata:<name:"ruby-sample-build-2" generateName:"" namespace:"xxia-proj" selfLink:"/oapi/v1/namespaces/xxia-proj/builds/ruby-sample-build-2" uid:"cd115c0a-d229-11e6-a084-fa163ef56398" resourceVersion:"316673" generation:0 creationTimestamp:<2017-01-04T10:59:25+08:00> labels:<key:"app" value:"ruby-helloworld-sample" > labels:<key:"buildconfig" value:"ruby-sample-build" > labels:<key:"name" value:"ruby-sample-build" > labels:<key:"openshift.io/build-config.name" value:"ruby-sample-build" > labels:<key:"openshift.io/build.start-policy" value:"Serial" > labels:<key:"template" value:"application-template-stibuild" > annotations:<key:"openshift.io/build-config.name" value:"ruby-sample-build" > annotations:<key:"openshift.io/build.number" value:"2" > > spec:<commonSpec:<serviceAccount:"builder" source:<type:"Git" git:<uri:"https://github.com/openshift/ruby-hello-world.git" ref:"" > contextDir:"" > strategy:<type:"Source" sourceStrategy:<from:<kind:"DockerImage" namespace:"" name:"centos/ruby-22-centos7@sha256:aace2b3ef701be868ef3a1feff7f1d766caa9e80026ba55dbf2d2e9c8f37bf7e" uid:"" apiVersion:"" resourceVersion:"" fieldPath:"" > env:<name:"EXAMPLE" value:"sample-app" > scripts:"" forcePull:false > > output:<to:<kind:"ImageStreamTag" namespace:"" name:"origin-ruby-sample:latest" uid:"" apiVersion:"" resourceVersion:"" fieldPath:"" > pushSecret:<name:"builder-dockercfg-h7488" > > resources:<> postCommit:<args:"bundle" args:"exec" args:"rake" args:"test" script:"" > > triggeredBy:<message:"Manually triggered" > > status:<phase:"New" cancelled:false reason:"" message:"" duration:0 outputDockerImageReference:"" config:<kind:"BuildConfig" namespace:"xxia-proj" name:"ruby-sample-build" uid:"" apiVersion:"" resourceVersion:"" fieldPath:"" > >  is an unsortable type: ptr, err: unsortable type: invalid



$ versions/v3.3.1.3/oc get builds --sort-by={.status.startTimestamp}
NAME                  TYPE      FROM          STATUS     STARTED              DURATION
ruby-sample-build-1   Source    Git@e79d887   Complete   About a minute ago   1m13s
ruby-sample-build-2   Source    Git@e79d887   Running    2 seconds ago        2s

$ versions/v3.3.1.3/oc version
oc v3.3.1.3
kubernetes v1.3.0+52492b4
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://<master>:8443
openshift v3.3.1.8
kubernetes v1.3.0+52492b4

Comment 2 rcerven 2017-01-04 10:02:20 UTC
there has to be build which never started, and failed/canceled

to achieve to obtain such a build (without starttimestamp attribute)

you can try: submit build and asap cancel it, so it doesn't have change to even start

or you can submit too many builds, so some will be waiting in queue and then you have chance to cancel these, and you will get build without starttimestamp

Comment 3 Xingxing Xia 2017-01-04 10:12:14 UTC
Good info in comment 2.
Reproduced the panic now:
Given the bc's spec.policy is Serial (by default), start several builds:

$ versions/v3.3.1.3/oc start-build ruby-sample-build
build "ruby-sample-build-11" started
$ versions/v3.3.1.3/oc start-build ruby-sample-build
build "ruby-sample-build-12" started
$ versions/v3.3.1.3/oc start-build ruby-sample-build
build "ruby-sample-build-13" started
$ versions/v3.3.1.3/oc start-build ruby-sample-build
build "ruby-sample-build-14" started

$ versions/v3.3.1.3/oc get builds --sort-by={.status.startTimestamp}
panic: reflect: call of reflect.Value.Interface on zero Value

goroutine 1 [running]:
panic(0x23c9a60, 0xc820826620)
	/usr/lib/golang/src/runtime/panic.go:481 +0x3e6
reflect.valueInterface(0x0, 0x0, 0x0, 0x2825c01, 0x0, 0x0)
	/usr/lib/golang/src/reflect/value.go:913 +0x7f
reflect.Value.Interface(0x0, 0x0, 0x0, 0x0, 0x0)
...skipped, similar to above attachment....

$ versions/v3.3.1.3/oc get build
NAME                   TYPE      FROM          STATUS     STARTED              DURATION
...
ruby-sample-build-11   Source    Git@e79d887   Complete   25 seconds ago       23s
ruby-sample-build-12   Source    Git           Pending    2 seconds ago        1s
ruby-sample-build-13   Source    Git           New                             
ruby-sample-build-14   Source    Git           New

Comment 4 Xingxing Xia 2017-01-04 10:21:25 UTC
What's worse is, if one build with status "New" is canceled, then it always gets the panic even after all other builds become complete/running:

$ versions/v3.3.1.3/oc start-build ruby-sample-build
build "ruby-sample-build-15" started
[star@pc_f25 oc]$ versions/v3.3.1.3/oc start-build ruby-sample-build
build "ruby-sample-build-16" started
[star@pc_f25 oc]$ versions/v3.3.1.3/oc start-build ruby-sample-build
build "ruby-sample-build-17" started
[star@pc_f25 oc]$ oc cancel-build ruby-sample-build-17
build "ruby-sample-build-17" cancelled

$ versions/v3.3.1.3/oc get build
NAME                   TYPE      FROM          STATUS      STARTED          DURATION
...
ruby-sample-build-15   Source    Git@e79d887   Complete    36 seconds ago   24s
ruby-sample-build-16   Source    Git@e79d887   Running     9 seconds ago    9s
ruby-sample-build-17   Source    Git           Cancelled

$ versions/v3.3.1.3/oc get builds --sort-by={.status.startTimestamp}
panic: reflect: call of reflect.Value.Interface on zero Value

goroutine 1 [running]:
panic(0x23c9a60, 0xc82082afa0)
...

Comment 5 Xingxing Xia 2017-01-11 01:20:25 UTC
*** Bug 1411964 has been marked as a duplicate of this bug. ***

Comment 6 Fabiano Franz 2017-01-11 20:43:13 UTC
Can you confirm that in *Origin* master the panic does not happen anymore? I'm now seeing something like 

Failed to get i values for &runtime.Unstructured
(...)
using {.status.startTimestamp} (&errors.errorString{s:"startTimestamp is not found"})

Comment 7 Xingxing Xia 2017-01-12 05:22:48 UTC
Yes, Origin master (devenv-rhel7_5660 openshift/v1.5.0-alpha.1+fc73b80-204) does not reproduce the panic, it reproduces info like yours.

Comment 8 Fabiano Franz 2017-01-26 20:19:22 UTC
This is being fixed upstream in https://github.com/kubernetes/kubernetes/pull/40541.

I'm adjusting the tags here and we'll wait for the fix to land in the next Kube rebase.

Comment 9 Fabiano Franz 2017-05-08 20:27:37 UTC
Fixed in master by the latest Kubernetes rebase.

Comment 11 XiaochuanWang 2017-06-05 07:07:18 UTC
This is not reproduced and verified on oc v3.6.94

# oc get builds --sort-by={.status.startTimestamp}
NAME                  TYPE      FROM          STATUS                       STARTED              DURATION
ruby-sample-build-5   Source    Git           New                                               
ruby-sample-build-3   Source    Git           New                                               
ruby-sample-build-1   Source    Git@022d87e   Running                      About a minute ago   
ruby-sample-build-2   Source    Git           Cancelled (CancelledBuild)   23 seconds ago       
ruby-sample-build-4   Source    Git           Cancelled (CancelledBuild)   8 seconds ago

Comment 13 errata-xmlrpc 2017-08-10 05:17:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716