Bug 1922454

Summary: CRI-O doesn't enable pprof by default
Product: OpenShift Container Platform Reporter: Mrunal Patel <mpatel>
Component: NodeAssignee: Mrunal Patel <mpatel>
Node sub component: CRI-O QA Contact: Sunil Choudhary <schoudha>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: low CC: aos-bugs, rphillips
Version: 4.7   
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:57:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Mrunal Patel 2021-01-29 19:00:19 UTC
Description of problem:
CRI-O does not enable pprof by default making it hard to debug issues in production.


Version-Release number of selected component (if applicable):




Expected results:
CRI-O enables pprof by default over the default unix socket.

Additional info:

Comment 2 Sunil Choudhary 2021-02-03 08:42:19 UTC
Verified on 4.7.0-0.nightly-2021-02-02-223803

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-02-02-223803   True        False         76m     Cluster version is 4.7.0-0.nightly-2021-02-02-223803

$ oc get nodes -o wide
NAME                                                        STATUS   ROLES    AGE   VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
sunilc030247-zpzb4-master-0.c.openshift-qe.internal         Ready    master   97m   v1.20.0+8f6e1f4   10.0.0.5      <none>        Red Hat Enterprise Linux CoreOS 47.83.202102021844-0 (Ootpa)   4.18.0-240.10.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.gitce4f759.el8.44
sunilc030247-zpzb4-master-1.c.openshift-qe.internal         Ready    master   97m   v1.20.0+8f6e1f4   10.0.0.3      <none>        Red Hat Enterprise Linux CoreOS 47.83.202102021844-0 (Ootpa)   4.18.0-240.10.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.gitce4f759.el8.44
sunilc030247-zpzb4-master-2.c.openshift-qe.internal         Ready    master   97m   v1.20.0+8f6e1f4   10.0.0.4      <none>        Red Hat Enterprise Linux CoreOS 47.83.202102021844-0 (Ootpa)   4.18.0-240.10.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.gitce4f759.el8.44
sunilc030247-zpzb4-worker-a-bh8fh.c.openshift-qe.internal   Ready    worker   91m   v1.20.0+8f6e1f4   10.0.32.3     <none>        Red Hat Enterprise Linux CoreOS 47.83.202102021844-0 (Ootpa)   4.18.0-240.10.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.gitce4f759.el8.44
sunilc030247-zpzb4-worker-b-ccrdg.c.openshift-qe.internal   Ready    worker   88m   v1.20.0+8f6e1f4   10.0.32.4     <none>        Red Hat Enterprise Linux CoreOS 47.83.202102021844-0 (Ootpa)   4.18.0-240.10.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.gitce4f759.el8.44
sunilc030247-zpzb4-worker-c-hp54w.c.openshift-qe.internal   Ready    worker   91m   v1.20.0+8f6e1f4   10.0.32.2     <none>        Red Hat Enterprise Linux CoreOS 47.83.202102021844-0 (Ootpa)   4.18.0-240.10.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.gitce4f759.el8.44


$ oc debug node/sunilc030247-zpzb4-worker-a-bh8fh.c.openshift-qe.internal
Starting pod/sunilc030247-zpzb4-worker-a-bh8fhcopenshift-qeinternal-debug ...
...


sh-4.4# curl --unix-socket /var/run/crio/crio.sock http://localhost/debug/pprof/goroutine?debug=1
goroutine profile: total 49
5 @ 0x5586ce5e53c6 0x5586ce5b130f 0x5586ce5b0f4b 0x5586cf54859a 0x5586ce61bba1
#	0x5586cf548599	github.com/cri-o/cri-o/vendor/github.com/docker/spdystream.(*Connection).Serve.func1+0x39	/builddir/build/BUILD/cri-o-ce4f7593ab387a0e27b6ba6873a358b9e81bd7c6/_output/src/github.com/cri-o/cri-o/vendor/github.com/docker/spdystream/connection.go:306

5 @ 0x5586ce5e53c6 0x5586ce617d1c 0x5586ce617cf2 0x5586ce63287f 0x5586cf5469a6 0x5586cf543671 0x5586cf54862d 0x5586ce61bba1
#	0x5586ce617cf1	sync.runtime_notifyListWait+0xd1								/usr/lib/golang/src/runtime/sema.go:513
#	0x5586ce63287e	sync.(*Cond).Wait+0x9e										/usr/lib/golang/src/sync/cond.go:56
#	0x5586cf5469a5	github.com/cri-o/cri-o/vendor/github.com/docker/spdystream.(*PriorityFrameQueue).Pop+0x85	/builddir/build/BUILD/cri-o-ce4f7593ab387a0e27b6ba6873a358b9e81bd7c6/_output/src/github.com/cri-o/cri-o/vendor/github.com/docker/spdystream/priority.go:86
#	0x5586cf543670	github.com/cri-o/cri-o/vendor/github.com/docker/spdystream.(*Connection).frameHandler+0x30	/builddir/build/BUILD/cri-o-ce4f7593ab387a0e27b6ba6873a358b9e81bd7c6/_output/src/github.com/cri-o/cri-o/vendor/github.com/docker/spdystream/connection.go:398
#	0x5586cf54862c	github.com/cri-o/cri-o/vendor/github.com/docker/spdystream.(*Connection).Serve.func2+0x6c	/builddir/build/BUILD/cri-o-ce4f7593ab387a0e27b6ba6873a358b9e81bd7c6/_output/src/github.com/cri-o/cri-o/vendor/github.com/docker/spdystream/connection.go:315

3 @ 0x5586ce5e53c6 0x5586ce5dda5c 0x5586ce615ec5 0x5586ce68a2a7 0x5586ce68b305 0x5586ce68b2e3 0x5586ce7ed151 0x5586ce802390 0x5586cfcf5f18 0x5586cfcf7391 0x5586ce6f3ae4 0x5586ce661629 0x5586ceb24849 0x5586ceb247fc 0x5586ceb250c5 0x5586cf06af5a 0x5586cf0bf1bd 0x5586cf0d1645 0x5586ce61bba1
#	0x5586ce615ec4	internal/poll.runtime_pollWait+0x64										/usr/lib/golang/src/runtime/netpoll.go:222
#	0x5586ce68a2a6	internal/poll.(*pollDesc).wait+0x46										/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87
#	0x5586ce68b304	internal/poll.(*pollDesc).waitRead+0x1a4									/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92
#	0x5586ce68b2e2	internal/poll.(*FD).Read+0x182											/usr/lib/golang/src/internal/poll/fd_unix.go:159
#	0x5586ce7ed150	net.(*netFD).Read+0x50												/usr/lib/golang/src/net/fd_posix.go:55
#	0x5586ce80238f	net.(*conn).Read+0x8f												/usr/lib/golang/src/net/net.go:182

Comment 5 errata-xmlrpc 2021-02-24 15:57:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633