Description of problem: In GMP_Engine::GMP_Engine() botan does call: mp_set_memory_functions(gmp_malloc, gmp_realloc, gmp_free); That overwrites the global gmp memory allocation functions. On the destructor it sets them to NULL. That is very very bad when one would dynamically load a library using botan. I realized that if I load softhsm (which uses botan) from gnutls as a PKCS #11 module, and at some point unload it, I'll have a crash on gnutls which also uses gmp. That is because for the duration that the module is loaded gnutls is using botan's allocation functions, and when the module is unloaded structures are released using the default allocation functions. As a rule libraries shouldn't change global process allocation functions, and they shouldn't assume they are the only user of gmp.
Talked to upstream. Besides the obvious 'well don't unload it then' :) we have to options: - disabling the gmp engine - just disabling the allocator override Opinions?
- Don't unload isn't an option because softhsm is a module intended to be loaded and unloaded at runtime. Otherwise botan might be a bad choice for softhsm. - Disabling the gmp engine. I have no idea what it is. Do you mean compiling botan without gmp? That's up to you as a maintainer. If it solves the issue, I'm ok with it. - Just disabling the allocator override: I believe that's the most reasonable option.
(In reply to Nikos Mavrogiannopoulos from comment #2) > - Don't unload isn't an option because softhsm is a module intended to be > loaded and unloaded at runtime. Otherwise botan might be a bad choice for > softhsm. Moreover it doesn't solve anything in the scenario I described above. By the time botan is loaded with softhsm there may be already structures allocated by another user of gmp (gnutls in that case).
So imho that's a deficit of gmp that one can only override its allocation functions globally. Anyway, I think I will patch Botan and disable the override.
botan-1.10.9-4.el7 has been submitted as an update for Fedora EPEL 7. https://admin.fedoraproject.org/updates/botan-1.10.9-4.el7
botan-1.10.9-4.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/botan-1.10.9-4.fc20
botan-1.10.9-4.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/botan-1.10.9-4.fc21
botan-1.10.9-4.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
botan-1.10.9-4.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.
botan-1.10.9-4.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.