Bug 1737127

Summary: oc idle service cause runtime error: invalid memory address or nil pointer dereference
Product: OpenShift Container Platform Reporter: Weibin Liang <weliang>
Component: NetworkingAssignee: Dan Williams <dcbw>
Status: CLOSED ERRATA QA Contact: Weibin Liang <weliang>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.2.0CC: aos-bugs, cdc, dcbw, piqin
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 06:34:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Weibin Liang 2019-08-02 17:59:44 UTC
Description of problem:
oc idle service cause panic: runtime error: invalid memory address or nil pointer dereference


Version-Release number of selected component (if applicable):
4.1.0-0.ci-2019-08-01-113006

How reproducible:
Every time

Steps to Reproduce:
1. Create 10 service
2. oc idle service-1

Actual results:
[root@dhcp-41-193 ~]# oc get svc
NAME              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
blue-service-1    ClusterIP   172.30.207.243   <none>        8080/TCP   6m12s
blue-service-10   ClusterIP   172.30.17.202    <none>        8080/TCP   6m1s
blue-service-2    ClusterIP   172.30.28.82     <none>        8080/TCP   6m11s
blue-service-3    ClusterIP   172.30.94.86     <none>        8080/TCP   6m10s
blue-service-4    ClusterIP   172.30.120.213   <none>        8080/TCP   6m9s
blue-service-5    ClusterIP   172.30.150.106   <none>        8080/TCP   6m8s
blue-service-6    ClusterIP   172.30.228.234   <none>        8080/TCP   6m7s
blue-service-7    ClusterIP   172.30.116.43    <none>        8080/TCP   6m5s
blue-service-8    ClusterIP   172.30.114.171   <none>        8080/TCP   6m4s
blue-service-9    ClusterIP   172.30.112.9     <none>        8080/TCP   6m3s
[root@dhcp-41-193 ~]# 

[root@dhcp-41-193 ~]# oc idle  blue-service-1
WARNING: idling when network policies are in place may cause connections to bypass network policy entirely
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4643fa]

goroutine 1 [running]:
github.com/openshift/origin/vendor/github.com/openshift/oc/pkg/cli/idle.normalizedNSOwnerRef(...)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/oc/pkg/cli/idle/idle.go:373
github.com/openshift/origin/vendor/github.com/openshift/oc/pkg/cli/idle.findScalableResourcesForEndpoints(0xc000d6e3c0, 0xc0003d7b60, 0xc0003d7b80, 0x7f4237449e10, 0xc000d6e3c0, 0xc000405901)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/oc/pkg/cli/idle/idle.go:408 +0x361
github.com/openshift/origin/vendor/github.com/openshift/oc/pkg/cli/idle.(*IdleOptions).calculateIdlableAnnotationsByService.func3(0xc000e12150, 0x0, 0x0, 0x0, 0x0)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/oc/pkg/cli/idle/idle.go:315 +0xcf
github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource.DecoratedVisitor.Visit.func1(0xc000e12150, 0x0, 0x0, 0x18, 0xc0003d7ae0)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:330 +0xd9
github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource.ContinueOnErrorVisitor.Visit.func1(0xc000e12150, 0x0, 0x0, 0x20, 0x2470920)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:354 +0x114
github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource.FlattenListVisitor.Visit.func1(0xc000e12150, 0x0, 0x0, 0x2e2f380, 0xc0012b1cc0)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:392 +0x7ff
github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource.EagerVisitorList.Visit.func1(0xc000e12150, 0x0, 0x0, 0xc001009801, 0xc0003d7c00)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:213 +0x114
github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource.(*Info).Visit(0xc000e12150, 0xc0003d7c00, 0x38, 0x38)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:95 +0x38
github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource.EagerVisitorList.Visit(0xc000405a30, 0x1, 0x1, 0xc0010098c0, 0xc0003d7b01, 0xc0010098c0)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:208 +0xc5
github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource.FlattenListVisitor.Visit(0x2e035e0, 0xc0003d7ae0, 0x2e2f240, 0xc000202d20, 0xc0013f7ec0, 0xc0003d7bc0, 0x8, 0xc0013a0b38)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:387 +0x86
github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource.ContinueOnErrorVisitor.Visit(0x2e03620, 0xc0012a94d0, 0xc001009880, 0x1, 0xc001009880)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:349 +0xc3
github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource.DecoratedVisitor.Visit(0x2e035a0, 0xc000405a60, 0xc0003d7b20, 0x3, 0x4, 0xc0012a9620, 0x30, 0x2701280)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:321 +0x86
github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource.(*Result).Visit(0xc0013ff000, 0xc0012a9620, 0x279db01, 0xc0012a9620)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/cli-runtime/pkg/resource/result.go:99 +0x62
github.com/openshift/origin/vendor/github.com/openshift/oc/pkg/cli/idle.(*IdleOptions).calculateIdlableAnnotationsByService(0xc001088480, 0xc0013a1038, 0x4d88bc0, 0xc0011128a0, 0x1, 0x1)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/oc/pkg/cli/idle/idle.go:301 +0x159
github.com/openshift/origin/vendor/github.com/openshift/oc/pkg/cli/idle.(*IdleOptions).RunIdle(0xc001088480, 0x0, 0x2a53598)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/oc/pkg/cli/idle/idle.go:605 +0x451
github.com/openshift/origin/vendor/github.com/openshift/oc/pkg/cli/idle.NewCmdIdle.func1(0xc0010e1180, 0xc0011128a0, 0x1, 0x1)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/oc/pkg/cli/idle/idle.go:99 +0xa0
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc0010e1180, 0xc001112870, 0x1, 0x1, 0xc0010e1180, 0xc001112870)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:830 +0x2ae
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000e8a500, 0x2, 0xc000e8a500, 0x2)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:914 +0x2fc
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:864
main.main()
	/builddir/build/BUILD/openshift-git-0.739670a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/oc/cmd/oc/oc.go:99 +0x823


Expected results:
Should not cause panic

Additional info:

Comment 1 Dan Williams 2019-08-05 15:45:26 UTC
	for _, pod := range podRefs {
		controllerRef := metav1.GetControllerOf(pod)
		ref := normalizedNSOwnerRef(pod.Namespace, controllerRef)
		if controllerRef == nil {
			return nil, fmt.Errorf("unable to find controller for pod %s/%s: no creator reference listed", pod.Namespace, pod.Name)
		}
		immediateControllerRefs[ref] = struct{}{}
	}

GetControllerOf() can return nil, and normalizedNSOwnerRef() doesn't appear to handle that case since it dereferences the controllerRef pointer.  Why it's nil, I have no idea.

Comment 2 Dan Williams 2019-08-05 15:50:15 UTC
diff --git a/staging/src/github.com/openshift/oc/pkg/cli/idle/idle.go b/staging/src/github.com/openshift/oc/pkg/cli/idle/idle.go
index 0dd48a390ce1f..04c79f89a5f6b 100644
--- a/staging/src/github.com/openshift/oc/pkg/cli/idle/idle.go
+++ b/staging/src/github.com/openshift/oc/pkg/cli/idle/idle.go
@@ -405,10 +405,10 @@ func findScalableResourcesForEndpoints(endpoints *corev1.Endpoints, getPod func(
        immediateControllerRefs := make(map[namespacedOwnerReference]struct{})
        for _, pod := range podRefs {
                controllerRef := metav1.GetControllerOf(pod)
-               ref := normalizedNSOwnerRef(pod.Namespace, controllerRef)
                if controllerRef == nil {
                        return nil, fmt.Errorf("unable to find controller for pod %s/%s: no creator reference listed", pod.Namespace, pod.Name)
                }
+               ref := normalizedNSOwnerRef(pod.Namespace, controllerRef)
                immediateControllerRefs[ref] = struct{}{}
        }

Comment 3 Casey Callendrello 2019-08-05 15:53:33 UTC
Ricky's on PTO for a while. Assigning to you, dan.

Comment 4 Dan Williams 2019-08-05 21:16:29 UTC
https://github.com/openshift/oc/pull/43

Comment 6 Weibin Liang 2019-08-20 13:22:21 UTC
Tested and verified in v4.2.0-0.nightly-2019-08-20-043744, no panic messages found.

[root@dhcp-41-193 FILE]# oc get svc
NAME     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
svc-1    ClusterIP   172.30.246.172   <none>        8080/TCP   12s
svc-10   ClusterIP   172.30.205.89    <none>        8080/TCP   6s
svc-2    ClusterIP   172.30.247.104   <none>        8080/TCP   10s
svc-3    ClusterIP   172.30.212.216   <none>        8080/TCP   9s
svc-4    ClusterIP   172.30.173.88    <none>        8080/TCP   9s
svc-5    ClusterIP   172.30.66.225    <none>        8080/TCP   9s
svc-6    ClusterIP   172.30.207.22    <none>        8080/TCP   8s
svc-7    ClusterIP   172.30.107.245   <none>        8080/TCP   8s
svc-8    ClusterIP   172.30.21.177    <none>        8080/TCP   7s
svc-9    ClusterIP   172.30.146.3     <none>        8080/TCP   7s
[root@dhcp-41-193 FILE]# oc idle svc-1
WARNING: idling when network policies are in place may cause connections to bypass network policy entirely
error: unable to mark the service "p1/svc-1" as idled.
Make sure that the service is not already marked as idled and that it is associated with resources that can be scaled.
See 'oc idle -h' for help and examples.
[root@dhcp-41-193 FILE]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.2.0-0.nightly-2019-08-20-043744   True        False         8m16s   Cluster version is 4.2.0-0.nightly-2019-08-20-043744

Comment 7 errata-xmlrpc 2019-10-16 06:34:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922