Bug 1064825

Created attachment 862746 [details]
Example that shows the problem

Description of problem:
asan_symbolizer reports global-buffer-overflow in asan_bug/d3des.c:376, but only when compiled with optimization

Version-Release number of selected component (if applicable):
gcc (GCC) 4.8.2 20131212 (Red Hat 4.8.2-7)

Also present in gcc (GCC) 4.8.1 20130715 (Red Hat 4.8.1-4) from rhel6 devtoolset-2.

How reproducible:
Fully reproducible

Steps to Reproduce:
1. untar attachment
2. make sure you have asan_symbolize.py from llvm/clang in path
3. run make
4. Modify Makefile to disable optimization (remove -O2)
5. run make

Actual results:
asan reports global-buffer-overflow in asan_bug/d3des.c:376 when optimization is enabled and doesn't when optimization is disabled

Expected results:
asan results should be consistent independent of optimization levels. The d3des.{h,c} files I have from realvnc, but tigervnc and several other realvnc derived projects use the same. As far as I can see the code does not overflow the buffer.

The code in d3des.c around line 376 looks like this:
    
    unsigned long keys = KnL; // via function parameter
    for( round = 0; round < 8; round++ ) {
      ...
      work ^= *keys++;
      ...
      work  = right ^ *keys++;
      ...
      work ^= *keys++;
      ...
      work  = leftt ^ *keys++;  // *** d3des.c Line 376 ***
      ...
    }
    
In the eight iteration of the loop immediately after line 376, keys will have been incremented 32 times and is thus one past the end of KnL, but while post-increment (++) has higher precedence than dereference (*) it is a post-increment, so it is not dereferenced after being past the end.

So it looks to me like the address sanitizer is reporting a false positive or the compiler is generating incorrect code.