Bug 1116406

Summary: botan overwrites gmp's default memory functions
Product: [Fedora] Fedora Reporter: Nikos Mavrogiannopoulos <nmavrogi>
Component: botanAssignee: Thomas Moschny <thomas.moschny>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: thomas.moschny
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: botan-1.10.9-4.fc20 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-09 09:47:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1123803    

Description Nikos Mavrogiannopoulos 2014-07-04 13:20:45 UTC
Description of problem:
In GMP_Engine::GMP_Engine() botan does call:
      mp_set_memory_functions(gmp_malloc, gmp_realloc, gmp_free);

That overwrites the global gmp memory allocation functions. On the destructor it sets them to NULL. That is very very bad when one would dynamically load a library using botan.

I realized that if I load softhsm (which uses botan) from gnutls as a PKCS #11 module, and at some point unload it, I'll have a crash on gnutls which also uses gmp. That is because for the duration that the module is loaded gnutls is using botan's allocation functions, and when the module is unloaded structures are released using the default allocation functions.

As a rule libraries shouldn't change global process allocation functions, and they shouldn't assume they are the only user of gmp.

Comment 1 Thomas Moschny 2014-08-09 07:03:42 UTC
Talked to upstream.

Besides the obvious 'well don't unload it then' :) we have to options:
- disabling the gmp engine
- just disabling the allocator override

Opinions?

Comment 2 Nikos Mavrogiannopoulos 2014-08-09 09:20:53 UTC
- Don't unload isn't an option because softhsm is a module intended to be loaded and unloaded at runtime. Otherwise botan might be a bad choice for softhsm.

- Disabling the gmp engine. I have no idea what it is. Do you mean compiling botan without gmp? That's up to you as a maintainer. If it solves the issue, I'm ok with it.

- Just disabling the allocator override: I believe that's the most reasonable option.

Comment 3 Nikos Mavrogiannopoulos 2014-08-09 09:22:39 UTC
(In reply to Nikos Mavrogiannopoulos from comment #2)
> - Don't unload isn't an option because softhsm is a module intended to be
> loaded and unloaded at runtime. Otherwise botan might be a bad choice for
> softhsm.

Moreover it doesn't solve anything in the scenario I described above. By the time botan is loaded with softhsm there may be already structures allocated by another user of gmp (gnutls in that case).

Comment 4 Thomas Moschny 2014-08-10 21:41:23 UTC
So imho that's a deficit of gmp that one can only override its allocation functions globally.

Anyway, I think I will patch Botan and disable the override.

Comment 5 Fedora Update System 2015-02-26 19:08:46 UTC
botan-1.10.9-4.el7 has been submitted as an update for Fedora EPEL 7.
https://admin.fedoraproject.org/updates/botan-1.10.9-4.el7

Comment 6 Fedora Update System 2015-02-26 19:08:55 UTC
botan-1.10.9-4.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/botan-1.10.9-4.fc20

Comment 7 Fedora Update System 2015-02-26 19:08:59 UTC
botan-1.10.9-4.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/botan-1.10.9-4.fc21

Comment 8 Fedora Update System 2015-03-09 08:29:42 UTC
botan-1.10.9-4.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 9 Fedora Update System 2015-03-09 08:32:33 UTC
botan-1.10.9-4.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 10 Fedora Update System 2015-03-15 00:58:55 UTC
botan-1.10.9-4.el7 has been pushed to the Fedora EPEL 7 stable repository.  If problems still persist, please make note of it in this bug report.