Description of problem:
When you copy a 5GB file to a Samba File Server using an encrypted SMB3 connection with AES-CCM, the used memory grows up to the file size. If you transfer a 5GB file it will use 5GB of RAM.
It doesn't happen with AES-GCM!
Samba creates a cipher handle and attches it to the SMB session to avoid reallocation during a file transfer. The memory is freed once the file is transferred.
For decryption we normally call gnutls_aead_cipher_decryptv2() which grows the memory to the file size using AES-CCM. It doesn't happen with gnutls_aead_cipher_decrypt().
Note that gnutls_aead_cipher_decryptv2() will be used in RHEL 8.3 with Samba 4.12. RHEL 8.2 used gnutls_aead_cipher_decrypt().
Someone created a MR for GnuTLS to address the issue, see https://gitlab.com/gnutls/gnutls/-/merge_requests/1277
Ping. When do we get a fix for RHEL 8.3?
Trying to add back the exception flag Mark, gave with the needed ITR/ITM fields