Bug 1932477

Description of problem:
On both RHEL7.8 and RHEL8.2 systems with perl-GSSAPI rpm installed, when I try to run the example client/server code (/usr/share/doc/perl-GSSAPI/examples/gss-server.pl and gss-client.pl), the server portion dies after the client attempts to make a connection (segfault on RHEL8, a long messy invalid pointer error in RHEL7)


Version-Release number of selected component (if applicable):
RHEL8.2: 
perl-GSSAPI-0.28-23.el8.x86_64
krb5-libs-1.17-18.el8.x86_64

RHEL7.8: 
perl-GSSAPI-0.28-9.el7.x86_64
krb5-libs-1.15.1-46.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
On host XXX.YYY.umd.edu
1. Have existing keytab /etc/krb5.keytab.hpccdb with hpccdb/XXX.YYY.umd.edu keys for the given hostname, e.g. 
   1    1 hpccdb/XXX.YYY.umd.edu (aes256-cts-hmac-sha1-96) 
   2    1 hpccdb/XXX.YYY.umd.edu (des3-cbc-sha1) 
   3    1 hpccdb/XXX.YYY.umd.edu (arcfour-hmac)
2. Start up two terminals on XXX.YYY.umd.edu
3. On first terminal, become root and run
perl /usr/share/doc/perl-GSSAPI-0.28/examples/gss-server.pl --port=2601 --keytabfile=/etc/krb5.keytab.hpccdb
4) On second terminal, as normal user with valid kerberos tickets, run:
perl /usr/share/doc/perl-GSSAPI/examples/gss-client.pl --prodid=hpccdb --hostname=$HOSTNAME --port=2601

(the paths to gss-server.pl/gss-client.pl are slightly different on RHEL7.8)

Actual results:

The server process starts up and waits for a client request, but dies with segfault or other error (depending on OS) when the first client request is received (it does appear able to verify the correct authenticated client (payerle), but aborts for some reason)

(The text in [%%% ... %%%] is commentary, not in actual output.  Mainly
to indicate timing between actions in two windows).

On RHEL8:
Server window:
XX.YYY.umd.edu# perl /usr/share/doc/perl-GSSAPI/examples/gss-server.pl --port=2601 --keytabfile=/etc/krb5.keytab.hpccdb
usr/share/doc/perl-GSSAPI/examples/gss-server.pl: -name not specified, using hostname result [XXX.YYY.umd.edu]
/usr/share/doc/perl-GSSAPI/examples/gss-server.pl: using [XXX.YYY.umd.edu:2601]
SERVER set environment variable KRB5_KTNAME to FILE:/etc/krb5.keytab.hpccdb
Listening on port 2601 ...

SERVER::waiting for request ...
[%%% sits until client request sent %%%]
SERVER::accepted connection from client ...
SERVER::received token (length is 626):
SERVER::authenticated client name is payerle
Segmentation fault (core dumped)

Client Window:
XXX.YYY.umd.edu> perl /usr/share/doc/perl-GSSAPI/examples/gss-client.pl --prodid=hpccdb --hostname=$HOSTNAME --port=2601
/usr/share/doc/perl-GSSAPI/examples/gss-client.pl: using [hpccdb.umd.edu:2601]
CLIENT::principal [hpccdb.umd.edu] means going to communicate with server name [hpccdb.umd.edu]
CLIENT::gss_init_sec_context success
CLIENT::going to identify client to server
CLIENT::have token to send ...
CLIENT::GSS token length is 626
CLIENT::sent token to server
Argument "" isn't numeric in null operation at /usr/share/doc/perl-GSSAPI/examples/gss-client.pl line 153.
	(in cleanup) oid has no value at /usr/share/doc/perl-GSSAPI/examples/gss-client.pl line 153.

RHEL7.8:
Server Window:
ZZZ.umd.edu# perl /usr/share/doc/perl-GSSAPI-0.28/examples/gss-server.pl --port=2601 --keytabfile=/etc/krb5.keytab.hpccdb
/usr/share/doc/perl-GSSAPI-0.28/examples/gss-server.pl: -name not specified, using hostname result [ZZZ.umd.edu]
/usr/share/doc/perl-GSSAPI-0.28/examples/gss-server.pl: using [ZZZ.umd.edu:2601]
SERVER set environment variable KRB5_KTNAME to FILE:/etc/krb5.keytab.hpccdb
Listening on port 2601 ...

SERVER::waiting for request ...
[%%% sits until client request sent %%%]
SERVER::accepted connection from client ...
SERVER::received token (length is 617):
SERVER::authenticated client name is payerle
*** Error in `perl': munmap_chunk(): invalid pointer: 0x00007f6cb22122be ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f3e4)[0x7f6cb940b3e4]
/lib64/libgssapi_krb5.so.2(+0xd5a2)[0x7f6cb21df5a2]
/usr/lib64/perl5/vendor_perl/auto/GSSAPI/GSSAPI.so(+0xc13c)[0x7f6cb242b13c]
/usr/lib64/perl5/CORE/libperl.so(Perl_pp_entersub+0x58f)[0x7f6cba7a642f]
/usr/lib64/perl5/CORE/libperl.so(Perl_call_sv+0x69d)[0x7f6cba73640d]
/usr/lib64/perl5/CORE/libperl.so(+0xc5185)[0x7f6cba7af185]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x4b0)[0x7f6cba7afaa0]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x4a)[0x7f6cba7b010a]
/usr/lib64/perl5/CORE/libperl.so(Perl_leave_scope+0x1125)[0x7f6cba7d49e5]
/usr/lib64/perl5/CORE/libperl.so(Perl_pp_leave+0xce)[0x7f6cba7df0de]
/usr/lib64/perl5/CORE/libperl.so(Perl_runops_standard+0x16)[0x7f6cba79eba6]
/usr/lib64/perl5/CORE/libperl.so(perl_run+0x355)[0x7f6cba73b995]
perl[0x400ce9]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f6cb93ae555]
perl[0x400d21]
======= Memory map: ========
00400000-00402000 r-xp 00000000 fd:00 67182007                           /usr/bin/perl
00601000-00602000 r--p 00001000 fd:00 67182007                           /usr/bin/perl
00602000-00603000 rw-p 00002000 fd:00 67182007                           /usr/bin/perl
01fb9000-022d4000 rw-p 00000000 00:00 0                                  [heap]
7f6cb09be000-7f6cb09d3000 r-xp 00000000 fd:00 101713175                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f6cb09d3000-7f6cb0bd2000 ---p 00015000 fd:00 101713175                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f6cb0bd2000-7f6cb0bd3000 r--p 00014000 fd:00 101713175                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f6cb0bd3000-7f6cb0bd4000 rw-p 00015000 fd:00 101713175                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f6cb0bd4000-7f6cb0bf2000 r-xp 00000000 fd:00 101713593                  /usr/lib64/libgssrpc.so.4.2
7f6cb0bf2000-7f6cb0df2000 ---p 0001e000 fd:00 101713593                  /usr/lib64/libgssrpc.so.4.2
7f6cb0df2000-7f6cb0df3000 r--p 0001e000 fd:00 101713593                  /usr/lib64/libgssrpc.so.4.2
7f6cb0df3000-7f6cb0df4000 rw-p 0001f000 fd:00 101713593                  /usr/lib64/libgssrpc.so.4.2
7f6cb0df4000-7f6cb0e0f000 r-xp 00000000 fd:00 67220109                   /usr/lib64/gssproxy/proxymech.so
7f6cb0e0f000-7f6cb100f000 ---p 0001b000 fd:00 67220109                   /usr/lib64/gssproxy/proxymech.so
7f6cb100f000-7f6cb1010000 r--p 0001b000 fd:00 67220109                   /usr/lib64/gssproxy/proxymech.so
7f6cb1010000-7f6cb1011000 rw-p 0001c000 fd:00 67220109                   /usr/lib64/gssproxy/proxymech.so
7f6cb1011000-7f6cb1014000 r-xp 00000000 fd:00 115910                     /usr/lib64/perl5/auto/MIME/Base64/Base64.so
7f6cb1014000-7f6cb1213000 ---p 00003000 fd:00 115910                     /usr/lib64/perl5/auto/MIME/Base64/Base64.so
7f6cb1213000-7f6cb1214000 r--p 00002000 fd:00 115910                     /usr/lib64/perl5/auto/MIME/Base64/Base64.so
7f6cb1214000-7f6cb1215000 rw-p 00003000 fd:00 115910                     /usr/lib64/perl5/auto/MIME/Base64/Base64.so
7f6cb1215000-7f6cb1275000 r-xp 00000000 fd:00 100668866                  /usr/lib64/libpcre.so.1.2.0
7f6cb1275000-7f6cb1475000 ---p 00060000 fd:00 100668866                  /usr/lib64/libpcre.so.1.2.0
7f6cb1475000-7f6cb1476000 r--p 00060000 fd:00 100668866                  /usr/lib64/libpcre.so.1.2.0
7f6cb1476000-7f6cb1477000 rw-p 00061000 fd:00 100668866                  /usr/lib64/libpcre.so.1.2.0
7f6cb1477000-7f6cb149b000 r-xp 00000000 fd:00 100664542                  /usr/lib64/libselinux.so.1
7f6cb149b000-7f6cb169a000 ---p 00024000 fd:00 100664542                  /usr/lib64/libselinux.so.1
7f6cb169a000-7f6cb169b000 r--p 00023000 fd:00 100664542                  /usr/lib64/libselinux.so.1
7f6cb169b000-7f6cb169c000 rw-p 00024000 fd:00 100664542                  /usr/lib64/libselinux.so.1
7f6cb169c000-7f6cb169e000 rw-p 00000000 00:00 0 
7f6cb169e000-7f6cb16a1000 r-xp 00000000 fd:00 100694568                  /usr/lib64/libkeyutils.so.1.5
7f6cb16a1000-7f6cb18a0000 ---p 00003000 fd:00 100694568                  /usr/lib64/libkeyutils.so.1.5
7f6cb18a0000-7f6cb18a1000 r--p 00002000 fd:00 100694568                  /usr/lib64/libkeyutils.so.1.5
7f6cb18a1000-7f6cb18a2000 rw-p 00003000 fd:00 100694568                  /usr/lib64/libkeyutils.so.1.5
7f6cb18a2000-7f6cb18b0000 r-xp 00000000 fd:00 101713709                  /usr/lib64/libkrb5support.so.0.1
7f6cb18b0000-7f6cb1ab0000 ---p 0000e000 fd:00 101713709                  /usr/lib64/libkrb5support.so.0.1
7f6cb1ab0000-7f6cb1ab1000 r--p 0000e000 fd:00 101713709                  /usr/lib64/libkrb5support.so.0.1
7f6cb1ab1000-7f6cb1ab2000 rw-p 0000f000 fd:00 101713709                  /usr/lib64/libkrb5support.so.0.1
7f6cb1ab2000-7f6cb1ab5000 r-xp 00000000 fd:00 101685022                  /usr/lib64/libcom_err.so.2.1
7f6cb1ab5000-7f6cb1cb4000 ---p 00003000 fd:00 101685022                  /usr/lib64/libcom_err.so.2.1
7f6cb1cb4000-7f6cb1cb5000 r--p 00002000 fd:00 101685022                  /usr/lib64/libcom_err.so.2.1
7f6cb1cb5000-7f6cb1cb6000 rw-p 00003000 fd:00 101685022                  /usr/lib64/libcom_err.so.2.1
7f6cb1cb6000-7f6cb1ce7000 r-xp 00000000 fd:00 101713695                  /usr/lib64/libk5crypto.so.3.1
7f6cb1ce7000-7f6cb1ee6000 ---p 00031000 fd:00 101713695                  /usr/lib64/libk5crypto.so.3.1
7f6cb1ee6000-7f6cb1ee8000 r--p 00030000 fd:00 101713695                  /usr/lib64/libk5crypto.so.3.1
7f6cb1ee8000-7f6cb1ee9000 rw-p 00032000 fd:00 101713695                  /usr/lib64/libk5crypto.so.3.1
7f6cb1ee9000-7f6cb1fc2000 r-xp 00000000 fd:00 100845584                  /usr/lib64/libkrb5.so.3.3
7f6cb1fc2000-7f6cb21c1000 ---p 000d9000 fd:00 100845584                  /usr/lib64/libkrb5.so.3.3
7f6cb21c1000-7f6cb21cf000 r--p 000d8000 fd:00 100845584                  /usr/lib64/libkrb5.so.3.3
7f6cb21cf000-7f6cb21d2000 rw-p 000e6000 fd:00 100845584                  /usr/lib64/libkrb5.so.3.3
7f6cb21d2000-7f6cb221c000 r-xp 00000000 fd:00 100694874                  /usr/lib64/libgssapi_krb5.so.2.2
7f6cb221c000-7f6cb241c000 ---p 0004a000 fd:00 100694874                  /usr/lib64/libgssapi_krb5.so.2.2
7f6cb241c000-7f6cb241d000 r--p 0004a000 fd:00 100694874                  /usr/lib64/libgssapi_krb5.so.2.2
7f6cb241d000-7f6cb241f000 rw-p 0004b000 fd:00 100694874                  /usr/lib64/libgssapi_krb5.so.2.2
7f6cb241f000-7f6cb2433000 r-xp 00000000 fd:00 67937994                   /usr/lib64/perl5/vendor_perl/auto/GSSAPI/GSSAPI.soAborted

Client window:
ZZZ.umd.edu$ perl /usr/share/doc/perl-GSSAPI-0.28/examples/gss-client.pl -prodid=hpccdb --hostname=$HOSTNAME --port=2601
/usr/share/doc/perl-GSSAPI-0.28/examples/gss-client.pl: using [hpccdb.edu:2601]
CLIENT::principal [hpccdb.edu] means going to communicate with server name [hpccdb.edu]
CLIENT::gss_init_sec_context success
CLIENT::going to identify client to server
CLIENT::have token to send ...
CLIENT::GSS token length is 617
CLIENT::sent token to server
Argument "" isn't numeric in null operation at /usr/share/doc/perl-GSSAPI-0.28/examples/gss-client.pl line 153.
	(in cleanup) oid has no value at /usr/share/doc/perl-GSSAPI-0.28/examples/gss-client.pl line 153.


Expected results:

The server successfully processes a client request and waits for another request until is terminated with control-c, etc.

(the output below from an old RHEL6 system with locally built perl 5.16.3  and GSSAPI v0.28 from CPAN.  The port number used below is different from the examples above because port 2601 is used for the real hpccdb application which
I want to run on XXX.YYY and ZZZ hosts above.  I confirmed that 2601 is not in use on XXX.YYY and ZZZ, and can reproduce same errors using port 3601 on XXX.YYY and ZZZ)

Server window:
AAA.BBB.umd.edu# /usr/local/perl/5.16.3/bin/perl gss-server.pl --hostname=AAA.BBB.umd.edu -port 3601 --keytabfile=/etc/krb5.keytab.hpccdb
server.pl: using [AAA.BBB.umd.edu:3601]
SERVER set environment variable KRB5_KTNAME to FILE:/etc/krb5.keytab.hpccdb
Listening on port 3601 ...

SERVER::waiting for request ...
[%%% sits here until first client request sent %%%]
SERVER::accepted connection from client ...
SERVER::received token (length is 664):
SERVER::authenticated client name is payerle

SERVER::waiting for request ...
[%%% sits here until second client request sent %%%]
SERVER::accepted connection from client ...
SERVER::received token (length is 664):
SERVER::authenticated client name is payerle

SERVER::waiting for request ...
[%%% sits here until I control-C out of server %%%]

Client window:
AAA.BBB.umd.edu: /usr/local/perl/5.16.3/bin/perl gss-client.pl --prodid=hpccdb --hostname=$HOSTNAME --port=3601
client.pl: using [hpccdb.umd.edu:3601]
CLIENT::principal [hpccdb.umd.edu] means going to communicate with server name [hpccdb.umd.edu]
CLIENT::gss_init_sec_context success
CLIENT::going to identify client to server
CLIENT::have token to send ...
CLIENT::GSS token length is 663
CLIENT::sent token to server
Argument "" isn't numeric in null operation at client.pl line 153.
	(in cleanup) oid has no value at client.pl line 153.


Additional info:

Our KDCs are heimdal based, and in the successful RHEL6 example the GSSAPI perl module was linked against heimdal libraries.  

On the RHEL7 and RHEL8 systems, we have both MIT and heimdal kerberos libraries installed; MIT via RedHat RPMs and heimdal via EPEL repo.  The perl GSSAPI module is from rhel8_appstream (RHEL8) or rhel7_server (RHEL7) repos, and presumably link against MIT kerberos.  I have confirmed via strace that the server and client actually are loading the MIT kibraries.

The keytab files were generated with heimdal utilities (since our KDC is heimdal), but I have confirmed that both heimdal and MIT kinit and ktutils can read the keytabs (both the original keytabs generated from heimdal utilities, and keytab generated by doing a read_kt and write_kt in MIT ktutil).

Also, I am successfully able to use GSSAPI for ssh-ing into the systems, both from using MIT and heimdal versions of kinit, which suggests the issue is in the perl module not libgssapi.so, etc.