[originally reported by Dr. David Alan Gilbert]
Description of problem:
GNUTLS has a function gnutls_hash:
int gnutls_hash(gnutls_hash_hd_t handle, const void * ptext, size_t ptext_len);
it lets you perform md5 or any of the other hashes on a chunk of data;
this is implemented by a call to the nettle libraries implementation:
md5_update(struct md5_ctx *ctx,
unsigned length, <<<<<-
const uint8_t *data);
Note that here someone unwisely used an 'unsigned' length.
Thus if you pass a large ptext_len into gnutls_hash, the length gets
truncated and thus only produces a hash on part of the data.
In my case I passed a 4GB chunk which ended up hashing 0 bytes!
Nettle fixed this upstream in:
b8bfc32f (Niels Möller 2013-04-26 10:28:57 +0200)
Use size_t rather than unsigned for all hash-related functions.
I'm not sure if gnutls has any fixes.
I hit this during development, I'm not aware of anything in particular
that would hit it.
Here's a backtrace that shows the truncation:
Breakpoint 1, gnutls_hash (handle=0x7f5a74029020, text=0x7f5a83e00000, textlen=4294967296) at crypto-api.c:503
503 return _gnutls_hash((digest_hd_st *) handle, text, textlen);
(gdb) b wrap_nettle_hash_update
Breakpoint 2, wrap_nettle_hash_update (_ctx=0x7f5a74045b10, text=0x7f5a83e00000, textsize=4294967296) at mac.c:305
Breakpoint 3, nettle_md5_update (ctx=0x7f5a74045b10, length=0, data=0x7f5a83e00000 "S\377") at md5.c:62
I guess you could fixit in the wrap_nettle_hash_update (that's part of gnutls), but I guess you would also have to go on a hunting trip for anything
else that makes the same mistake. Or update nettle.
Steps to Reproduce:
1. allocate 4GB of RAM
2. Initialise gnutls
3. gnutls_hash on the 4GB chunk using md5
The md5 result for a zero length input.
The md5 result for your actual data.
gnutls-3.3.21-2.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2016-ed3e59d122
gnutls-3.3.21-2.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-ed3e59d122
gnutls-3.3.21-2.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.