Description of problem:

Optimiser produces incorrect code for x86 builds since gcc 6 upgrade.

I have a test case extracted from a package that is failing it's self tests on x86 since gcc 6 entered rawhide.

If you unpack the attached archive and run make in the directory it will build a shared library and a test executable that invokes it, in both 32  and 64 bit mode. Running the 64 bit version gets the correct result:

rawhide [~/rgb2hsl] % ./main64
h = 0.333333
s = 1
l = 0.25098

Running the 32 bit version wrongly reports h as zero:

rawhide [~/rgb2hsl] % ./main32
h = 0
s = 1
l = 0.25098

Reducing the optimisation level of the shared library from O2 to O1 fixes it, as does removing -fPIC from the shared library build.

The problem is the rgb_to_hsl function in rgb2hsl.cpp and having followed through the generated code everything seems to be fine up to the end of this line:

        s = l > 0.5 ? delta / (2.0 - gamma) : delta / gamma;

After that it seems to get confused compiling the rest of that if block - in particular it seems to forget it has put "max" on the FPU stack and therefore generated the body of the next statement with an off-by-one error for FPU values.

That's about as far as I've got with diagnosing it so far. Trying to follow through what the optimiser has done gets tricky...

Version-Release number of selected component (if applicable):

gcc-6.0.0-0.5.fc24.x86_64