Bug 1844112 - GnuTLS allocates more and more memory when decrypting with AES-CCM
Summary: GnuTLS allocates more and more memory when decrypting with AES-CCM
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: gnutls
Version: 8.3
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: 8.0
Assignee: Daiki Ueno
QA Contact: Alexander Sosedkin
Depends On:
TreeView+ depends on / blocked
Reported: 2020-06-04 16:22 UTC by Andreas Schneider
Modified: 2020-10-15 18:16 UTC (History)
5 users (show)

Fixed In Version: gnutls-3.6.14-6.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed:
Type: Bug
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Gitlab gnutls/gnutls/-/merge_requests/1278 None None None 2020-08-24 11:29:58 UTC
Samba Project 14399 None None None 2020-06-04 16:22:46 UTC

Description Andreas Schneider 2020-06-04 16:22:46 UTC
Description of problem:

When you copy a 5GB file to a Samba File Server using an encrypted SMB3 connection with AES-CCM, the used memory grows up to the file size. If you transfer a 5GB file it will use 5GB of RAM.

It doesn't happen with AES-GCM!

Samba creates a cipher handle and attches it to the SMB session to avoid reallocation during a file transfer. The memory is freed once the file is transferred.

For decryption we normally call gnutls_aead_cipher_decryptv2() which grows the memory to the file size using AES-CCM. It doesn't happen with gnutls_aead_cipher_decrypt().

Comment 1 Andreas Schneider 2020-06-04 16:27:04 UTC
Note that gnutls_aead_cipher_decryptv2() will be used in RHEL 8.3 with Samba 4.12. RHEL 8.2 used gnutls_aead_cipher_decrypt().

Comment 2 Andreas Schneider 2020-06-05 13:39:42 UTC
Someone created a MR for GnuTLS to address the issue, see https://gitlab.com/gnutls/gnutls/-/merge_requests/1277

Comment 3 Andreas Schneider 2020-08-24 11:15:06 UTC
Ping. When do we get a fix for RHEL 8.3?

Comment 11 Simo Sorce 2020-08-24 16:33:03 UTC
Trying to add back the exception flag Mark, gave with the needed ITR/ITM fields

Note You need to log in before you can comment on or make changes to this bug.