Red Hat Bugzilla – Bug 1608277
When using priority, oc will panic due to index out of range
Last modified: 2018-10-11 03:22:29 EDT
Description of problem: When pod is preempting node, `oc get pods` will got panic. Version-Release number of selected component (if applicable): # oc version oc v3.11.0-0.9.0 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://qe-wjiang-master-etcd-1:8443 openshift v3.11.0-0.9.0 kubernetes v1.11.0+d4cacc0 How reproducible: always Steps to Reproduce: 1. Create a priority for testing echo 'apiVersion: scheduling.k8s.io/v1beta1 kind: PriorityClass metadata: name: low-priority value: -1 globalDefault: false description: "This priority class should be used for XYZ service pods only."' | oc create -f - 2. make pods to consume all resources oc run h --image=aosqe/sleep --requests="cpu=500m,memory=500Mi" --replicas=8 3. make pods in step 2 as low-priority, both priorityClassName and priority should be present in podtemplate 4. Create another dc (since default priority is 0), will preempt nodes from step 2 pods oc run h1 --image=openshift/hello-openshift:latest --requests="cpu=1000m,memory=1000Mi" 5. oc scale dc/h1 --replicas=3 && oc get pods Actual results: # oc scale dc/h1 --replicas=3 && oc get pods --loglevel=8 deploymentconfig.apps.openshift.io/h1 scaled I0725 04:58:14.713156 14551 loader.go:359] Config loaded from file /root/.kube/config I0725 04:58:14.714762 14551 loader.go:359] Config loaded from file /root/.kube/config I0725 04:58:14.722219 14551 loader.go:359] Config loaded from file /root/.kube/config I0725 04:58:14.732702 14551 loader.go:359] Config loaded from file /root/.kube/config I0725 04:58:14.733137 14551 round_trippers.go:383] GET https://qe-wjiang-master-etcd-1:8443/api/v1/namespaces/default/pods?limit=500 I0725 04:58:14.733161 14551 round_trippers.go:390] Request Headers: I0725 04:58:14.733172 14551 round_trippers.go:393] Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json I0725 04:58:14.733182 14551 round_trippers.go:393] User-Agent: oc/v1.11.0+d4cacc0 (linux/amd64) kubernetes/d4cacc0 I0725 04:58:14.745903 14551 round_trippers.go:408] Response Status: 200 OK in 12 milliseconds I0725 04:58:14.745975 14551 round_trippers.go:411] Response Headers: I0725 04:58:14.745988 14551 round_trippers.go:414] Cache-Control: no-store I0725 04:58:14.746000 14551 round_trippers.go:414] Content-Type: application/json I0725 04:58:14.746011 14551 round_trippers.go:414] Date: Wed, 25 Jul 2018 08:58:14 GMT I0725 04:58:14.746278 14551 request.go:897] Response Body: {"kind":"Table","apiVersion":"meta.k8s.io/v1beta1","metadata":{"selfLink":"/api/v1/namespaces/default/pods","resourceVersion":"52384"},"columnDefinitions":[{"name":"Name","type":"string","format":"name","description":"Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names","priority":0},{"name":"Ready","type":"string","format":"","description":"The aggregate readiness state of this pod for accepting traffic.","priority":0},{"name":"Status","type":"string","format":"","description":"The aggregate status of the containers in this pod.","priority":0},{"name":"Restarts","type":"integer","format":"","description":"The number of times the containers in this pod have been restarted.","priority":0},{"name":"Age","type":"string [truncated 17871 chars] I0725 04:58:14.749050 14551 get.go:439] no kind is registered for the type v1beta1.Table in scheme "k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29" panic: runtime error: index out of range goroutine 1 [running]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/printers.PrintTable(0xc420d17780, 0x3084260, 0xc42011d800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/atomic-openshift-git-0.0bffd7b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/printers/humanreadable.go:374 +0x5d6 github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/printers.(*HumanReadablePrinter).PrintObj(0xc420a68300, 0x3093380, 0xc420d17780, 0x3084260, 0xc42011d800, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.0bffd7b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/printers/humanreadable.go:294 +0x7ca github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/printers.(ResourcePrinter).PrintObj-fm(0x3093380, 0xc420d17780, 0x3084260, 0xc42011d800, 0x1, 0x1) /builddir/build/BUILD/atomic-openshift-git-0.0bffd7b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:252 +0x57 github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/printers.ResourcePrinterFunc.PrintObj(0xc420555620, 0x3093380, 0xc420d17780, 0x3084260, 0xc42011d800, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.0bffd7b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/printers/interface.go:38 +0x4e github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/get.(*GetOptions).Run(0xc42050c7e0, 0x30ecee0, 0xc42130a060, 0xc42134e500, 0xc42062b900, 0x1, 0x2, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.0bffd7b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:440 +0xae9 github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/get.NewCmdGet.func1(0xc42134e500, 0xc42062b900, 0x1, 0x2) /builddir/build/BUILD/atomic-openshift-git-0.0bffd7b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:159 +0x115 github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc42134e500, 0xc42062b8a0, 0x2, 0x2, 0xc42134e500, 0xc42062b8a0) /builddir/build/BUILD/atomic-openshift-git-0.0bffd7b/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:760 +0x2c1 github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4212f8f00, 0xc4212f8f00, 0x12a4669, 0xc42015c150) /builddir/build/BUILD/atomic-openshift-git-0.0bffd7b/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:846 +0x30a github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4212f8f00, 0x2, 0xc4212f8f00) /builddir/build/BUILD/atomic-openshift-git-0.0bffd7b/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:794 +0x2b main.main() /builddir/build/BUILD/atomic-openshift-git-0.0bffd7b/_output/local/go/src/github.com/openshift/origin/cmd/oc/oc.go:40 +0x300 Expected results: oc should not panic during preempting Additional info:
Will reproduce and get back once I have some findings.
https://github.com/openshift/origin/pull/20437
(In reply to Avesh Agarwal from comment #2) > https://github.com/openshift/origin/pull/20437 Above PR is merged. Moving it to modified.
Checked with # oc version oc v3.11.0-0.11.0 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://ip-172-18-4-116.ec2.internal:8443 openshift v3.11.0-0.11.0 kubernetes v1.11.0+d4cacc0 And the bug is fixed
Checked again with # oc version oc v3.11.0-0.22.0 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://qe-wjiang-master-etcd-1:8443 openshift v3.11.0-0.22.0 kubernetes v1.11.0+d4cacc0 And the issue has been fixed
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2652