A stack-based buffer overflow was found in the way the CA application of openssl, a general purpose cryptography library with TLS implementation, performed signing of certificate requests, when overly long password has been used for CA key encryption. Attempt to sign a certificate request with such a CA key would lead to openssl's 'ca' executable crash. References: [1] http://seclists.org/bugtraq/2012/May/155
Upstream bug: http://rt.openssl.org/Ticket/Display.html?id=2826&user=guest&pass=guest
This issue is not specific to ca sub-command of the openssl utility. It is an issue in the password_callback() function that is used by other openssl sub-commands too. An easy way to trigger is using genrsa (openssl genrsa -out test.key -des3), where both stack-based (first pass phrase prompt) and heap-based (verify pass phrase prompt) overflows can be reproduced. This is not a security flaw. It only affects openssl command line tool that was never intended to be installed as setuid / setgid. Therefore, no trust boundary is crossed. A patch to address this was sent to upstream bug.