Bug 1038999

All of the above code segments use the BIGNUM struct/API in a wrong way.

For example, memory is allocated for the data portion of the BN structure using the following:

x->d = (BN_ULONG *) malloc(1);

and later other BN_ functions are called to process it, however all these functions expect the data portion of the structure to be in multiple of BN_ULONG.

malloc(1) is supposed to return a pointer to a memory area of size 1 byte. (This may not happen under glibc normally as it may return a pointer to a larger area of memory)

So when a BN_ function is called, it results in a heap OOB read/write, similar to what the reporter mentions.

Changing the code to use malloc as follows fixes the problem:

x->d = (BN_ULONG *) malloc(1 * sizeof(BN_ULONG));

Note:
As mentioned above, due to various reasons malloc(1) may not return a memory area of 1 byte, but more than that. On a typical x86_64 machine it returns ~ 24 bytes. Due to this it may not cause an OOB read/write in the above test case. But that is due to glibc's implementation of malloc()

I looked at all the issues in the fd email. The ones using malloc() are similar to the ones described in comment #2.

The one using "BN_sqr(o, z, ctx);" looks like a very minor bug, i filed an upstream bug and a github pull request. This has no security angle to it though.

http://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=3410
https://github.com/openssl/openssl/pull/140