[originally reported by Dr. David Alan Gilbert] Description of problem: GNUTLS has a function gnutls_hash: int gnutls_hash(gnutls_hash_hd_t handle, const void * ptext, size_t ptext_len); it lets you perform md5 or any of the other hashes on a chunk of data; this is implemented by a call to the nettle libraries implementation: void md5_update(struct md5_ctx *ctx, unsigned length, <<<<<- const uint8_t *data); Note that here someone unwisely used an 'unsigned' length. Thus if you pass a large ptext_len into gnutls_hash, the length gets truncated and thus only produces a hash on part of the data. In my case I passed a 4GB chunk which ended up hashing 0 bytes! Nettle fixed this upstream in: b8bfc32f (Niels Möller 2013-04-26 10:28:57 +0200) Use size_t rather than unsigned for all hash-related functions. I'm not sure if gnutls has any fixes. I hit this during development, I'm not aware of anything in particular that would hit it. Here's a backtrace that shows the truncation: Breakpoint 1, gnutls_hash (handle=0x7f5a74029020, text=0x7f5a83e00000, textlen=4294967296) at crypto-api.c:503 503 return _gnutls_hash((digest_hd_st *) handle, text, textlen); (gdb) b wrap_nettle_hash_update Breakpoint 2, wrap_nettle_hash_update (_ctx=0x7f5a74045b10, text=0x7f5a83e00000, textsize=4294967296) at mac.c:305 305 { (gdb) list Breakpoint 3, nettle_md5_update (ctx=0x7f5a74045b10, length=0, data=0x7f5a83e00000 "S\377") at md5.c:62 62 { I guess you could fixit in the wrap_nettle_hash_update (that's part of gnutls), but I guess you would also have to go on a hunting trip for anything else that makes the same mistake. Or update nettle. Steps to Reproduce: 1. allocate 4GB of RAM 2. Initialise gnutls 3. gnutls_hash on the 4GB chunk using md5 Actual results: The md5 result for a zero length input. Expected results: The md5 result for your actual data.
https://gitlab.com/gnutls/gnutls/commit/f081fda83e7ff7be94071daba4cad8edd1c23ac0
gnutls-3.3.21-2.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2016-ed3e59d122
gnutls-3.3.21-2.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-ed3e59d122
gnutls-3.3.21-2.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.