The 1st leak looks like a bug in pkcs11 engine - EVP_PKEY_METH should be freed on unloading the engine. Isn't it missed? 

The 2nd looks like a missing SSL_SESSION_free on an application/module level

No, all engine-registered methods are to be freed by openssl itself. Could you please check that the engine is properly shut down?

Could you please explain how do you deal with the engine? I have an impression that you initialize and free it many times.

I asked a maintainer of radiator to contribute since I can't answer your questions about it without assumptions. I only operate the radius servers and did the leak hunting and reported the results so far.

What I know is that radiator is a single threaded perl daemon. So I assume that it initializes and frees every TLS Session it handles and does not kill any childs like forking daemons do after a certain amount of requests. I don't know how and if the engine itself is handled differently.

Since our servers are currently running with OpenSSL 1.1.1n built from unpatched original source for a month now I can confirm that they do not leak at all like they did on el6 before.

According to information from the radiator mailing list el9 beta is not affected as well. But I didn't (lack of el9 hosts here) verify that.

can you install debuginfo for openssl and openssl-pkcs11 to get useful backtraces?

BTDT...

valgrind:
==476564== 240,128 bytes in 938 blocks are definitely lost in loss record 5,978 of 6,001
==476564==    at 0x4C37135: malloc (vg_replace_malloc.c:381)
==476564==    by 0xA39290C: CRYPTO_zalloc (mem.c:230)
==476564==    by 0xA37EAC3: EVP_PKEY_meth_new (pmeth_lib.c:187)
==476564==    by 0x40A4BD7: UnknownInlinedFun (p11_pkey.c:508)
==476564==    by 0x40A4BD7: PKCS11_pkey_meths (p11_pkey.c:682)
==476564==    by 0xA3608E4: ENGINE_get_pkey_meth (tb_pkmeth.c:74)
==476564==    by 0xA37EEA4: int_ctx_new (pmeth_lib.c:136)
==476564==    by 0xA37A543: do_sigver_init (m_sigver.c:29)
==476564==    by 0x9FD1A41: tls_construct_server_key_exchange (statem_srvr.c:2804)
==476564==    by 0x9FC433E: write_state_machine (statem.c:843)
==476564==    by 0x9FC433E: state_machine.part.5 (statem.c:443)
==476564==    by 0x9FAFC97: SSL_do_handshake (ssl_lib.c:3681)
==476564==    by 0x9D39893: ??? (in /usr/lib64/perl5/vendor_perl/auto/Net/SSLeay/SSLeay.so)
==476564==    by 0x4F304D8: Perl_pp_entersub (in /usr/lib64/libperl.so.5.26.3)

memleax shows more or less the same, but with less details:
CallStack[921]: memory expires with 256 bytes, backtrace:
    0x00007ff61649e660  libc-2.28.so  malloc()+0
    0x00007ff6142a090d  libcrypto.so  CRYPTO_zalloc()+13  crypto/mem.c:230
    0x00007ff61428cac4  libcrypto.so  EVP_PKEY_meth_new()+36  crypto/evp/pmeth_lib.c:187
    0x00007ff617b78bd8  ??
    0x00007ff61426e8e5  libcrypto.so  ENGINE_get_pkey_meth()+53  crypto/engine/tb_pkmeth.c:74
    0x00007ff61428cea5  libcrypto.so  ?()  crypto/evp/pmeth_lib.c:136
    0x00007ff614288544  libcrypto.so  ?()  crypto/evp/m_sigver.c:29
    0x00007ff6141a9c52  libcrypto.so  ASN1_item_verify()+674  crypto/asn1/a_verify.c:157
    0x00007ff614319914  libcrypto.so  ?()  crypto/x509/x509_vfy.c:1815
    0x00007ff61431b876  libcrypto.so  ?()  crypto/x509/x509_vfy.c:233
    0x00007ff61431bf10  libcrypto.so  X509_verify_cert()+160  crypto/x509/x509_vfy.c:303
    0x00007ff614659c26  libssl.so  ?()  ssl/statem/statem_lib.c:959
    0x00007ff61465d738  libssl.so  ?()  ssl/statem/statem_srvr.c:3812
    0x00007ff61464f33f  libssl.so  ?()  ssl/statem/statem.c:843
    0x00007ff61463ac98  libssl.so  SSL_do_handshake()+88  ssl/ssl_lib.c:3681
    0x00007ff6148ed894  SSLeay.so
    0x00007ff61769a4d9  libperl.so  Perl_pp_entersub()+505
    0x00007ff617692345  libperl.so  Perl_runops_standard()+53
    0x00007ff61761200d  libperl.so  perl_run()+797
    0x000055d2ab750eda  perl
    0x00007ff61643cca3  libc-2.28.so  __libc_start_main()+243
    0x000055d2ab750f1e  perl

I have an impression that you load engine once per handshake, which is not the best behavior. Probably you should explicitly load the engine once on initialization

From the setups steps, it is not clear if you use pkcs11 keys for something or is the issue going away just by not loading the pkcs11 engine (uninstalling openssl-pkcs11)?

I can confirm that removing openssl-pkcs11 silences valgrind and memleax. The upper reports are not found running radiator without it.

I changed back to stock openssl and removed pkcs11 on one of our production radius servers to see if it's memory consumption is stable as well. Will report later...

Our production host runs stable and without memory leak as well without pkcs11 engine installed.

I also did testing with and without openssl-pkcs11 package. The test is a simple: a 'while :' shell loop that calls eapol_test to run PEAP against Radiator. When openssl-pkcs11 is installed, which seems to be the default with RHEL 8 and its derivatives, Radiator process slowly grows. When I stop Radiator and do this:

% sudo rpm -e openssl-pkcs11

and then re-run Radiator and eapol_test loop, Radiator process size appears to be stable. In short: removing the pkcs11 engine helps.

Radiator attempts to load pkcs11 engine once for the process with OpenSSL ENGINE API. Typically this engine is not needed because the private key is in a regular file. Radiator could make pkcs11 engine loading configurable for those who need it.

I checked a couple of other OSes and noticed that RHEL 7 and Ubuntu 20.04 and 22.04, for example, do not come with pkcs11 engine installed by default. Package libengine-pkcs11-openssl on Ubuntu 20.04 is version 0.4.10-1 which is close to RHEL8 that has 0.4.10-2. When I repeated the test on Ubuntu 20.04 with pkcs11 engine installed, the slow leak did not appear.

It seems the pkcs11 engine has had memory leaks, for example https://github.com/OpenSC/libp11/issues/358 , and it might be they are still present in RHEL 8 version.

Same issue as bug 2097690.

Clemens if this is the same issue as the other bug, please close one of the 2 as duplicate.

I'm closing this as duplicate of 2097690, since that has more extensive debugging and reproduction information in private comments.
Please monitor bug 2097690, and thank you very much for the report and analysis.

*** This bug has been marked as a duplicate of bug 2097690 ***