Bug 2033733

Summary: [4.9.z] vsphere-problem-detector: session leak
Product: OpenShift Container Platform Reporter: Robert Bost <rbost>
Component: StorageAssignee: Robert Bost <rbost>
Storage sub component: Operators QA Contact: Wei Duan <wduan>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: aos-bugs
Version: 4.10   
Target Milestone: ---   
Target Release: 4.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-01-10 08:50:46 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2011903    
Bug Blocks: 2036967    

Description Robert Bost 2021-12-17 17:44:20 UTC
This bug was initially created as a copy of Bug #2011903

I am copying this bug because: Backport to 4.9.z



Description of problem:

vSphere Problem Detector may be leaking sessions in vSphere due to client not being logged out between syncs. 

Version-Release number of selected component (if applicable): 4.10 and master

How reproducible: Always

Steps to Reproduce:
1. Monitor sessions using `govc session.ls` while vsphere-problem-detector is running

Actual results: 
Sessions count increases from 1 to 14 in about 17 minutes.

In the output below, I was running build from https://github.com/openshift/vsphere-problem-detector/pull/58

$ while true; do date; govc session.ls | grep vsphere-prob; sleep 5m ; done
Thu Oct  7 10:20:26 AM MDT 2021
523e179e-4ead-fd7b-b51a-4f51ad4789a0  VSPHERE.LOCAL\rbost    2021-10-07 16:20  12s     x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
Thu Oct  7 10:25:26 AM MDT 2021
5237d10f-eb55-be7b-08f3-199397fec29d  VSPHERE.LOCAL\rbost    2021-10-07 16:23  2m8s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
523e179e-4ead-fd7b-b51a-4f51ad4789a0  VSPHERE.LOCAL\rbost    2021-10-07 16:20  5m12s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
528a51b6-142d-9e56-afbd-2e19b384b975  VSPHERE.LOCAL\rbost    2021-10-07 16:24  1m7s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
528f6e0d-da37-ffe0-247a-11f9bff5a2de  VSPHERE.LOCAL\rbost    2021-10-07 16:22  3m10s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52f4a6c1-c603-bdf5-1922-0f3129bc0196  VSPHERE.LOCAL\rbost    2021-10-07 16:21  4m11s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52facf01-0f12-d658-077a-2485d25b602e  VSPHERE.LOCAL\rbost    2021-10-07 16:25  7s      x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
Thu Oct  7 10:30:26 AM MDT 2021
5212af06-f896-be51-8088-4c881459226f  VSPHERE.LOCAL\rbost    2021-10-07 16:26  4m5s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
522071e9-0ab3-d843-e02e-d44924644ea2  VSPHERE.LOCAL\rbost    2021-10-07 16:28  2m3s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
5237d10f-eb55-be7b-08f3-199397fec29d  VSPHERE.LOCAL\rbost    2021-10-07 16:23  7m9s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
523e179e-4ead-fd7b-b51a-4f51ad4789a0  VSPHERE.LOCAL\rbost    2021-10-07 16:20  10m13s  x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
528a51b6-142d-9e56-afbd-2e19b384b975  VSPHERE.LOCAL\rbost    2021-10-07 16:24  6m8s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
528f6e0d-da37-ffe0-247a-11f9bff5a2de  VSPHERE.LOCAL\rbost    2021-10-07 16:22  8m10s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52b62109-7618-33f1-12ab-0eccf6c46c77  VSPHERE.LOCAL\rbost    2021-10-07 16:27  3m4s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52d08b6b-9be2-b4de-c7bc-4aae330242ab  VSPHERE.LOCAL\rbost    2021-10-07 16:29  1m2s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52df60e9-7dd4-666f-711c-99d872e0a85a  VSPHERE.LOCAL\rbost    2021-10-07 16:30  2s      x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52f4a6c1-c603-bdf5-1922-0f3129bc0196  VSPHERE.LOCAL\rbost    2021-10-07 16:21  9m11s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52facf01-0f12-d658-077a-2485d25b602e  VSPHERE.LOCAL\rbost    2021-10-07 16:25  5m7s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
Thu Oct  7 10:35:27 AM MDT 2021
5211905e-4c06-e20e-edc2-6d7debaf3af8  VSPHERE.LOCAL\rbost    2021-10-07 16:31  4m0s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
5212af06-f896-be51-8088-4c881459226f  VSPHERE.LOCAL\rbost    2021-10-07 16:26  9m5s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
522071e9-0ab3-d843-e02e-d44924644ea2  VSPHERE.LOCAL\rbost    2021-10-07 16:28  7m3s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
5234bcde-f49b-f53f-edcf-fcdd0db91de5  VSPHERE.LOCAL\rbost    2021-10-07 16:32  2m59s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
5237d10f-eb55-be7b-08f3-199397fec29d  VSPHERE.LOCAL\rbost    2021-10-07 16:23  12m9s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
523e179e-4ead-fd7b-b51a-4f51ad4789a0  VSPHERE.LOCAL\rbost    2021-10-07 16:20  15m13s  x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
528a51b6-142d-9e56-afbd-2e19b384b975  VSPHERE.LOCAL\rbost    2021-10-07 16:24  11m8s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
528f6e0d-da37-ffe0-247a-11f9bff5a2de  VSPHERE.LOCAL\rbost    2021-10-07 16:22  13m10s  x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52b62109-7618-33f1-12ab-0eccf6c46c77  VSPHERE.LOCAL\rbost    2021-10-07 16:27  8m4s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52d08b6b-9be2-b4de-c7bc-4aae330242ab  VSPHERE.LOCAL\rbost    2021-10-07 16:29  6m2s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52db6e7f-d860-dc5a-109c-e2a936be556e  VSPHERE.LOCAL\rbost    2021-10-07 16:34  56s     x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52df60e9-7dd4-666f-711c-99d872e0a85a  VSPHERE.LOCAL\rbost    2021-10-07 16:30  5m1s    x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52ebd12c-0084-4ee6-e6b5-339337f7a3d1  VSPHERE.LOCAL\rbost    2021-10-07 16:33  1m57s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52f4a6c1-c603-bdf5-1922-0f3129bc0196  VSPHERE.LOCAL\rbost    2021-10-07 16:21  14m11s  x.x.x.x  vsphere-problem-detector/v0.0.0-unknown                     
52facf01-0f12-d658-077a-2485d25b602e  VSPHERE.LOCAL\rbost    2021-10-07 16:25  10m7s   x.x.x.x  vsphere-problem-detector/v0.0.0-unknown

Comment 4 Wei Duan 2021-12-24 04:48:36 UTC
Verified on 4.9.0-0.nightly-2021-12-23-045233

1. Using following command to monitor the session and restart the vsphere-problem-detector-operator pod, we see new session launched and releases immediately
$ while true; do date; govc session.ls | wc -l; done
$ oc -n openshift-cluster-storage-operator delete pod vsphere-problem-detector-operator-8794689bc-kqm7l (restart vsphere-problem-detector-operator pod)  

Fri 24 Dec 2021 04:36:33 AM UTC
14
Fri 24 Dec 2021 04:36:33 AM UTC
14
Fri 24 Dec 2021 04:36:33 AM UTC
14
Fri 24 Dec 2021 04:36:33 AM UTC
14
Fri 24 Dec 2021 04:36:33 AM UTC
14
Fri 24 Dec 2021 04:36:33 AM UTC
15
Fri 24 Dec 2021 04:36:33 AM UTC
15
Fri 24 Dec 2021 04:36:33 AM UTC
15
Fri 24 Dec 2021 04:36:33 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:34 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
15
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14
Fri 24 Dec 2021 04:36:35 AM UTC
14


The session number raise from 14 to 15 and back to 14 immediately. The related session is:
5210a470-3389-d449-8bf4-35f292fda301  VSPHERE.LOCAL\Administrator                                        2021-12-24 04:41  1s      10.8.33.214  Go-http-client/1.1         

Change status to "Verified".

Comment 7 errata-xmlrpc 2022-01-10 08:50:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.13 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0029