Description of problem:

GNUTLS has a function gnutls_hash:

   int gnutls_hash(gnutls_hash_hd_t handle, const void * ptext, size_t ptext_len);

it lets you perform md5 or any of the other hashes on a chunk of data;
this is implemented by a call to the nettle libraries implementation:

void
md5_update(struct md5_ctx *ctx,
           unsigned length,        <<<<<-
           const uint8_t *data);

Note that here someone unwisely used an 'unsigned' length.

Thus if you pass a large ptext_len into gnutls_hash, the length gets
truncated and thus only produces a hash on part of the data.
In my case I passed a 4GB chunk which ended up hashing 0 bytes!

Nettle fixed this upstream in:
   b8bfc32f (Niels Möller 2013-04-26 10:28:57 +0200)
   Use size_t rather than unsigned for all hash-related functions.

I'm not sure if gnutls has any fixes.

I hit this during development, I'm not aware of anything in particular
that would hit it.

Here's a backtrace that shows the truncation:

Breakpoint 1, gnutls_hash (handle=0x7f5a74029020, text=0x7f5a83e00000, textlen=4294967296) at crypto-api.c:503
503             return _gnutls_hash((digest_hd_st *) handle, text, textlen);
(gdb) b wrap_nettle_hash_update

Breakpoint 2, wrap_nettle_hash_update (_ctx=0x7f5a74045b10, text=0x7f5a83e00000, textsize=4294967296) at mac.c:305
305     {
(gdb) list

Breakpoint 3, nettle_md5_update (ctx=0x7f5a74045b10, length=0, data=0x7f5a83e00000 "S\377") at md5.c:62
62	{


I guess you could fixit in the wrap_nettle_hash_update (that's part of gnutls), but I guess you would also have to go on a hunting trip for anything
else that makes the same mistake.  Or update nettle.

Version-Release number of selected component (if applicable):
gnutls-3.3.8-12.el7_1.1.x86_64
nettle-2.7.1-4.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. allocate 4GB of RAM
2. Initialise gnutls
3. gnutls_hash on the 4GB chunk using md5

Actual results:
The md5 result for a zero length input.

Expected results:
The md5 result for your actual data.

Additional info: