2033733 – [4.9.z] vsphere-problem-detector: session leak

Bug 2033733 - [4.9.z] vsphere-problem-detector: session leak

Summary: [4.9.z] vsphere-problem-detector: session leak

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Storage
Sub Component:
Version:	4.10
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.9.z
Assignee:	Robert Bost
QA Contact:	Wei Duan
Docs Contact:
URL:
Whiteboard:
Depends On:	2011903
Blocks:	2036967
TreeView+	depends on / blocked

Reported:	2021-12-17 17:44 UTC by Robert Bost
Modified:	2022-01-12 10:06 UTC (History)
CC List:	1 user (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-01-10 08:50:46 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift vsphere-problem-detector pull 67	0	None	open	Bug 2033733: Deferred logout after checks are run	2021-12-17 18:04:53 UTC
Red Hat Product Errata	RHBA-2022:0029	0	None	None	None	2022-01-10 08:51:15 UTC

Description Robert Bost 2021-12-17 17:44:20 UTC

This bug was initially created as a copy of Bug #2011903

I am copying this bug because: Backport to 4.9.z



Description of problem:

vSphere Problem Detector may be leaking sessions in vSphere due to client not being logged out between syncs. 

Version-Release number of selected component (if applicable): 4.10 and master

How reproducible: Always

Steps to Reproduce:
1. Monitor sessions using `govc session.ls` while vsphere-problem-detector is running

Actual results: 
Sessions count increases from 1 to 14 in about 17 minutes.

In the output below, I was running build from https://github.com/openshift/vsphere-problem-detector/pull/58

$ while true; do date; govc session.ls | grep vsphere-prob; sleep 5m ; done
Thu Oct  7 10:20:26 AM MDT 2021
523e179e-4ead-fd7b-b51a-4f51ad4789a0  VSPHERE.LOCAL\rbost    2021-10-07 16:20  12s     x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
Thu Oct  7 10:25:26 AM MDT 2021
5237d10f-eb55-be7b-08f3-199397fec29d  VSPHERE.LOCAL\rbost    2021-10-07 16:23  2m8s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
523e179e-4ead-fd7b-b51a-4f51ad4789a0  VSPHERE.LOCAL\rbost    2021-10-07 16:20  5m12s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
528a51b6-142d-9e56-afbd-2e19b384b975  VSPHERE.LOCAL\rbost    2021-10-07 16:24  1m7s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
528f6e0d-da37-ffe0-247a-11f9bff5a2de  VSPHERE.LOCAL\rbost    2021-10-07 16:22  3m10s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52f4a6c1-c603-bdf5-1922-0f3129bc0196  VSPHERE.LOCAL\rbost    2021-10-07 16:21  4m11s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52facf01-0f12-d658-077a-2485d25b602e  VSPHERE.LOCAL\rbost    2021-10-07 16:25  7s      x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
Thu Oct  7 10:30:26 AM MDT 2021
5212af06-f896-be51-8088-4c881459226f  VSPHERE.LOCAL\rbost    2021-10-07 16:26  4m5s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
522071e9-0ab3-d843-e02e-d44924644ea2  VSPHERE.LOCAL\rbost    2021-10-07 16:28  2m3s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
5237d10f-eb55-be7b-08f3-199397fec29d  VSPHERE.LOCAL\rbost    2021-10-07 16:23  7m9s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
523e179e-4ead-fd7b-b51a-4f51ad4789a0  VSPHERE.LOCAL\rbost    2021-10-07 16:20  10m13s  x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
528a51b6-142d-9e56-afbd-2e19b384b975  VSPHERE.LOCAL\rbost    2021-10-07 16:24  6m8s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
528f6e0d-da37-ffe0-247a-11f9bff5a2de  VSPHERE.LOCAL\rbost    2021-10-07 16:22  8m10s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52b62109-7618-33f1-12ab-0eccf6c46c77  VSPHERE.LOCAL\rbost    2021-10-07 16:27  3m4s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52d08b6b-9be2-b4de-c7bc-4aae330242ab  VSPHERE.LOCAL\rbost    2021-10-07 16:29  1m2s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52df60e9-7dd4-666f-711c-99d872e0a85a  VSPHERE.LOCAL\rbost    2021-10-07 16:30  2s      x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52f4a6c1-c603-bdf5-1922-0f3129bc0196  VSPHERE.LOCAL\rbost    2021-10-07 16:21  9m11s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52facf01-0f12-d658-077a-2485d25b602e  VSPHERE.LOCAL\rbost    2021-10-07 16:25  5m7s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
Thu Oct  7 10:35:27 AM MDT 2021
5211905e-4c06-e20e-edc2-6d7debaf3af8  VSPHERE.LOCAL\rbost    2021-10-07 16:31  4m0s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
5212af06-f896-be51-8088-4c881459226f  VSPHERE.LOCAL\rbost    2021-10-07 16:26  9m5s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
522071e9-0ab3-d843-e02e-d44924644ea2  VSPHERE.LOCAL\rbost    2021-10-07 16:28  7m3s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
5234bcde-f49b-f53f-edcf-fcdd0db91de5  VSPHERE.LOCAL\rbost    2021-10-07 16:32  2m59s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
5237d10f-eb55-be7b-08f3-199397fec29d  VSPHERE.LOCAL\rbost    2021-10-07 16:23  12m9s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
523e179e-4ead-fd7b-b51a-4f51ad4789a0  VSPHERE.LOCAL\rbost    2021-10-07 16:20  15m13s  x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
528a51b6-142d-9e56-afbd-2e19b384b975  VSPHERE.LOCAL\rbost    2021-10-07 16:24  11m8s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
528f6e0d-da37-ffe0-247a-11f9bff5a2de  VSPHERE.LOCAL\rbost    2021-10-07 16:22  13m10s  x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52b62109-7618-33f1-12ab-0eccf6c46c77  VSPHERE.LOCAL\rbost    2021-10-07 16:27  8m4s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52d08b6b-9be2-b4de-c7bc-4aae330242ab  VSPHERE.LOCAL\rbost    2021-10-07 16:29  6m2s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52db6e7f-d860-dc5a-109c-e2a936be556e  VSPHERE.LOCAL\rbost    2021-10-07 16:34  56s     x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52df60e9-7dd4-666f-711c-99d872e0a85a  VSPHERE.LOCAL\rbost    2021-10-07 16:30  5m1s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52ebd12c-0084-4ee6-e6b5-339337f7a3d1  VSPHERE.LOCAL\rbost    2021-10-07 16:33  1m57s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52f4a6c1-c603-bdf5-1922-0f3129bc0196  VSPHERE.LOCAL\rbost    2021-10-07 16:21  14m11s  x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52facf01-0f12-d658-077a-2485d25b602e  VSPHERE.LOCAL\rbost    2021-10-07 16:25  10m7s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown

Comment 4 Wei Duan 2021-12-24 04:48:36 UTC

Verified on 4.9.0-0.nightly-2021-12-23-045233

1. Using following command to monitor the session and restart the vsphere-problem-detector-operator pod, we see new session launched and releases immediately
$ while true; do date; govc session.ls | wc -l; done
$ oc -n openshift-cluster-storage-operator delete pod vsphere-problem-detector-operator-8794689bc-kqm7l (restart vsphere-problem-detector-operator pod)  

Fri 24 Dec 2021 04:36:33 AM UTC
14
Fri 24 Dec 2021 04:36:33 AM UTC
14
Fri 24 Dec 2021 04:36:33 AM UTC
14
Fri 24 Dec 2021 04:36:33 AM UTC
14
Fri 24 Dec 2021 04:36:33 AM UTC
14
Fri 24 Dec 2021 04:36:33 AM UTC
15
Fri 24 Dec 2021 04:36:33 AM UTC
15
Fri 24 Dec 2021 04:36:33 AM UTC
15
Fri 24 Dec 2021 04:36:33 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14


The session number raise from 14 to 15 and back to 14 immediately. The related session is:
5210a470-3389-d449-8bf4-35f292fda301  VSPHERE.LOCAL\Administrator                                        2021-12-24 04:41  1s      10.8.33.214  Go-http-client/1.1         

Change status to "Verified".

Comment 7 errata-xmlrpc 2022-01-10 08:50:46 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.13 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0029

Note You need to log in before you can comment on or make changes to this bug.