Bug 1813200

Summary: gssproxy memory leak with rpc.gssd
Product: Red Hat Enterprise Linux 8 Reporter: Yongcheng Yang <yoyang>
Component: gssproxyAssignee: Robbie Harwood <rharwood>
Status: CLOSED ERRATA QA Contact: anuja <amore>
Severity: high Docs Contact:
Priority: high    
Version: 8.2CC: dpal, fdvorak, fs-qe, ndehadra, plambri, rharwood, steved, swhiteho, xzhou
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.3   
Hardware: Unspecified   
OS: Linux   
URL: https://github.com/gssapi/gssproxy/pull/6
Whiteboard:
Fixed In Version: gssproxy-0.8.0-19.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1809277 Environment:
Last Closed: 2021-05-18 14:42:07 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1894575    
Attachments:
Description Flags
valgrind output with gssproxy-0.8.3-3.fc33
none
Entire valgrid log with gssproxy disabled
none
Entire valgrid log with gssproxy enabled using an Window AD
none
Entire valgrid log with gssproxy enabled with gss_release_oid() added
none
Entire valgrid log with gssproxy disabled with gss_release_oid() added none

Comment 23 Robbie Harwood 2020-07-31 20:33:33 UTC
gssproxy-0.8.3-3.fc33 should contain the fixes.  If you would like a RHEL build to advance test with, I'm happy to provide one.

Comment 24 Yongcheng Yang 2020-08-03 08:18:02 UTC
Created attachment 1703242 [details]
valgrind output with gssproxy-0.8.3-3.fc33

Comment 25 Yongcheng Yang 2020-08-03 08:26:01 UTC
I have checked the reproducer on fedora33 (Fedora-Rawhide-20200802.n.0 from beaker).

Please find the above attached output.

Comment 27 Steve Dickson 2020-08-04 21:39:32 UTC
(In reply to Yongcheng Yang from comment #24)
> Created attachment 1703242 [details]
> valgrind output with gssproxy-0.8.3-3.fc33

What was the valgrind command line did you use?

Comment 28 Yongcheng Yang 2020-08-05 00:55:26 UTC
(In reply to Steve Dickson from comment #27)
> (In reply to Yongcheng Yang from comment #24)
> > Created attachment 1703242 [details]
> > valgrind output with gssproxy-0.8.3-3.fc33
> 
> What was the valgrind command line did you use?

valgrind --log-file=valgrind.out.xxx --trace-children=yes --leak-check=full --track-origins=yes -v --show-reachable=yes rpc.gssd

Comment 30 Steve Dickson 2020-08-06 11:12:21 UTC
(In reply to Yongcheng Yang from comment #28)
> (In reply to Steve Dickson from comment #27)
> > (In reply to Yongcheng Yang from comment #24)
> > > Created attachment 1703242 [details]
> > > valgrind output with gssproxy-0.8.3-3.fc33
> > 
> > What was the valgrind command line did you use?
> 
> valgrind --log-file=valgrind.out.xxx --trace-children=yes --leak-check=full
> --track-origins=yes -v --show-reachable=yes rpc.gssd

I ran the above valgrind command with rpc.gssd not using 
gssproxy (/etc/nfs.conf: use-gss-proxy=0) over night and
got the following results:

==115828== LEAK SUMMARY:
==115828==    definitely lost: 0 bytes in 0 blocks
==115828==    indirectly lost: 0 bytes in 0 blocks
==115828==      possibly lost: 0 bytes in 0 blocks
==115828==    still reachable: 12,942 bytes in 44 blocks
==115828==         suppressed: 88 bytes in 1 blocks
==115828== 
==115828== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

So the interaction between rpc.gssd and gssproxy is causing the memory leaks.

Yongcheng, would you mind verifying this case in your testing?

Comment 31 Yongcheng Yang 2020-08-06 13:27:15 UTC
(In reply to Steve Dickson from comment #30)
...
> Yongcheng, would you mind verifying this case in your testing?

Yes, have just checked that NO memleak found with "use-gss-proxy=0" set under "[gssd]" in /etc/nfs.conf.
I.e. https://beaker.engineering.redhat.com/jobs/4467749

But why the nfs krb5 mounting can success even when rpc.gssd not using gssproxy. (I thought it's mis-configured before..)

Btw, seemingly the usage of "use-gss-proxy" is not mentioned in the man page rpc.gssd(8).

Comment 33 Steve Dickson 2020-08-07 11:01:32 UTC
Created attachment 1710785 [details]
Entire valgrid log with gssproxy disabled

Comment 35 Steve Dickson 2020-08-07 11:10:49 UTC
(In reply to Yongcheng Yang from comment #31)
> (In reply to Steve Dickson from comment #30)
> ...
> > Yongcheng, would you mind verifying this case in your testing?
> 
> Yes, have just checked that NO memleak found with "use-gss-proxy=0" set
> under "[gssd]" in /etc/nfs.conf.
> I.e. https://beaker.engineering.redhat.com/jobs/4467749
Thanks!

> 
> But why the nfs krb5 mounting can success even when rpc.gssd not using
> gssproxy. (I thought it's mis-configured before..)
It is only required on the server since it replaced rpc.svcgssd(8)
which was a bit buggy... It is optional on the client side.

> 
> Btw, seemingly the usage of "use-gss-proxy" is not mentioned in the man page
> rpc.gssd(8).
This is probably a doc bug...

Comment 41 Steve Dickson 2020-08-15 15:06:13 UTC
Created attachment 1711485 [details]
Entire valgrid log with gssproxy enabled using an Window AD

Just for grins... I ran the set up in comment 39 using 
a Window's AD as the KDC... Doing a quick scan it appears
the errors are similar which is not too surprising. 

Just trying to look at this from all angels

Comment 46 Steve Dickson 2020-08-24 18:47:32 UTC
Created attachment 1712420 [details]
Entire valgrid log with gssproxy enabled with gss_release_oid() added

Comment 47 Steve Dickson 2020-08-24 18:49:00 UTC
Created attachment 1712422 [details]
Entire valgrid log with gssproxy disabled with gss_release_oid() added

Comment 48 Robbie Harwood 2020-08-26 18:38:59 UTC
Thanks, that first one is the log I was looking for.  With that information, the correct fix is on the gssproxy side and you don't need to do anything for it: https://github.com/gssapi/gssproxy/pull/10

Comment 69 anuja 2020-12-02 09:09:49 UTC
Verified with sanity run as per comment #68 and comment #67
Verfied bug using dev compose:
gssproxy-0.8.0-19.el8.x86_64
ipa-server-4.8.7-13.module+el8.3.0+8376+0bba7131.x86_64

Adding report using bash::ipa-client-automount:

2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho] ====================================== Final Report =======================================
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]             Test Date: Wed Dec  2 03:52:40 EST 2020         
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]                Total : [57]       
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]                Passed: [57]        
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]                Failed: [0]        
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]            Unfinished: [0]  
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]                Abort : [0]       
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]                Crash : [0]     
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]  +-----------------------------[RPMs & OS: [RedHat - x86_64]-----------------------------+
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho] |       ipa-client-4.8.7-13.module+el8.3.0+8376+0bba7131.x86_64
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho] |       ipa-client-common-4.8.7-13.module+el8.3.0+8376+0bba7131.noarch
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho] |       sssd-ipa-2.4.0-2.el8.x86_64
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho] ------------------------------------------------------------------------------------------
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]  +-----------------------------------------------------------------------------------------+
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]      Test:[/ipa-server/rhel80/ipa-client-automount/root]: [ Pass(57/57): 100% ] 
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]  +-----------------------------------------------------------------------------------------+
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]  +----------------------------------------------------------------------+
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]                     Fail / unfinished / ABORT [ Fail(0/57): 0% ]
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho]  +----------------------------------------------------------------------+
2020-12-02T08:52:41+0000 [ci-vm-10-0-155-38.ho] =========================== end of report [/tmp/rhts.report.23479.txt]===============================
 
Based on this marking Verified:Tested SanityOnly

Comment 73 anuja 2020-12-17 07:28:43 UTC
Verified with sanity run as per comment #68 and comment #67

Adding report using bash::ipa-client-automount:

2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho] ====================================== Final Report =======================================
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho]             Test Date: Thu Dec 17 07:17:45 UTC 2020         
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho]                Total : [57]       
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho]                Passed: [57]        
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho]                Failed: [0]        
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho]            Unfinished: [0]  
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho]                Abort : [0]       
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho]                Crash : [0]     
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho]  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho]  +-----------------------------[RPMs & OS: [RedHat - x86_64]-----------------------------+
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho] |       ipa-client-4.9.0-0.5.rc3.module+el8.4.0+9124+ced20601.x86_64
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho] |       ipa-client-common-4.9.0-0.5.rc3.module+el8.4.0+9124+ced20601.noarch
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho] |       sssd-ipa-2.4.0-3.el8.x86_64
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho] ------------------------------------------------------------------------------------------
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho]  +-----------------------------------------------------------------------------------------+
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho]      Test:[/ipa-server/rhel80/ipa-client-automount/root]: [ Pass(57/57): 100% ] 
2020-12-17T07:17:46+0000 [ci-vm-10-0-107-23.ho]  +-----------------------------------------------------------------------------------------+

Based on this marking verified.

Comment 75 errata-xmlrpc 2021-05-18 14:42:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (gssproxy bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1592