Bug 2182688 - Tests crash on Intel Core2 Quad CPU: Illegal instruction (core dumped) ./cryptlib-test because of -march=x86-64-v3
Summary: Tests crash on Intel Core2 Quad CPU: Illegal instruction (core dumped) ./cryp...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: cryptlib
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Ralf Senderek
QA Contact: Fedora Extras Quality Assurance
URL: https://docs.fedoraproject.org/en-US/...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-29 10:56 UTC by Petr Pisar
Modified: 2023-04-05 16:14 UTC (History)
3 users (show)

Fixed In Version: cryptlib-3.4.6-15.fc39 cryptlib-3.4.6-16.fc39
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-04-05 14:12:50 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Petr Pisar 2023-03-29 10:56:15 UTC
I noticed that cryptlib fails to build on my x86_64 machine:

+ echo 'Running tests on the cryptlib library. This will take a few minutes.'
+ cp /builddir/build/BUILDROOT/cryptlib-3.4.6-15.fc39.x86_64/usr/lib64/cryptlib/c/cryptlib-test.c .
+ sed -i '41s/<cryptlib\/cryptlib.h>/\".\/cryptlib.h\"/' cryptlib-test.c
+ gcc -o cryptlib-test cryptlib-test.c -L. libcl.so.3.4.6
+ ./cryptlib-test
/var/tmp/rpm-tmp.5l5CFa: line 52:  6243 Illegal instruction     (core dumped) ./cryptlib-test
error: Bad exit status from /var/tmp/rpm-tmp.5l5CFa (%check)

I believe it is because the package commands a compiler to use x86-64-v3 instruction set (-march=x86-64-v3 GCC option):

make libcl.so.3.4.6 OBJPATH=./shared-obj/ OSNAME=`uname` CFLAGS="-c -D__UNIX__ -DNDEBUG -I. -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ggdb3 -fno-omit-frame-pointer -O1 -DDATA_LITTLEENDIAN -DFIXED_SEED=0x238e2e8239b8a3fd -fPIC -DHAS_RECURSIVE_MUTEX -DHAS_ROBUST_MUTEX -march=x86-64-v3 -mtune=generic -fPIC -Wno-pointer-sign -Wno-strict-aliasing -fwrapv -fno-delete-null-pointer-checks -fstack-protector-strong -D_FORTIFY_SOURCE=2 -DOSVERSION=6 -O3 -fomit-frame-pointer -D_REENTRANT"

This makes this package unusable on x86_64 machines older than x86-64-v3 ISA base line (year 2015) <https://en.wikipedia.org/wiki/X86_64_v3#Microarchitecture_levels>.

I can see this caused by a change in x86-64patch patch file:

-+                      CCARGS="$CCARGS -march=x86-64 -mtune=generic" ;
++                      CCARGS="$CCARGS -march=x86-64-v3 -mtune=generic" ;

committed in:

commit 055adcff65ded69bc889170e7189d75cf8f75214
Author: Ralf Senderek <fedora>
Date:   Sat Mar 5 21:34:47 2022 +0100

    add subpackage cryptlib-tools

Default Fedora flags do not include -march argument. May I know why did you decided to change increase -march from x86-64 to x86-64-v3? Can you undo it?

Comment 1 Ralf Senderek 2023-03-29 15:36:19 UTC
Hello Petr, 

I know this "Illegal instruction (core dumped)" error very well, because it haunted me for more than four months in which I was unable to build the whole package on the Fedora build farm, until finally with the help of spot, I found out that I need the compiler flag  -march=x86-64-v3 to build the package reliably. The cause is simply modern
hardware on the Fedora build farm, not a bad idea. But this issue almost drove me crazy until I had a clue how to resolve it.

I then discussed this issue with upstream (Peter Gutmann) and a decision was made not to go back too far in time with this compiler flag, so we decided on version-3.
So -march=x86-64-v3 will stay for the future, sorry, if that does ruin cryptlib on rawhide on your 8 year old hardware. This is not a bug but a decision made with upstream.

Maybe it is a solution for you to go back to Fedora-34 on this old hardware?

Comment 2 Petr Pisar 2023-03-30 06:41:59 UTC
Why should I use unsupported software on an a supported hardware? Fedora definitely supports 8 year old hardware. If it weren't, it would use --march argument in system-wide cflags.
Could you at least document this misfeature in the package description?

Comment 3 Zbigniew Jędrzejewski-Szmek 2023-04-04 12:37:41 UTC
Using -march=x86-64-v3 goes contrary to the normal packaging practice.
The guidelines say that %{optflags} must be used [1], and it is
commonly understood that changing the arch baseline is not allowed.
A few months back there was a huge discussion on fedora-devel about raising
the baseline to -v3 and it was soundly rejected. The situation in which different
packages would use different baseline levels and crash randomly depending
on the hardware details would be terrible to users.

[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/#_compiler_flags

Comment 4 Ralf Senderek 2023-04-04 17:02:55 UTC
> The situation in which different packages would use different baseline levels and crash randomly depending on the hardware details would be terrible to users.

You obviously don't understand the problem: If by NOT using -march=x86-64-v3, %{optflags} resolve to "-march=native -mtune=generic" the FEDORA build machines produce packages that do crash on every computer after installation except those who are accidentally identical to the machine that had produced the package. This actually is terrible to users, 
not the use of -march=x86-64.

Have a look at the f39 scratch build, where I disabled the patch 5 that forces -march=x86-64-v3. (https://koji.fedoraproject.org/koji/taskinfo?taskID=99523515)
The result is a package that crashes on *my* laptop, although the machine that had produced the package did run a test-script during the build process successfully. But this package is totally useless for the average user with old or new hardware.

So the patch 5 will remain.

Ralf Senderek

Comment 5 Zbigniew Jędrzejewski-Szmek 2023-04-04 17:22:55 UTC
> If by NOT using -march=x86-64-v3, %{optflags} resolve to "-march=native -mtune=generic"
> the FEDORA build machines produce packages that do crash on every computer after installation
> except those who are accidentally identical to the machine that had produced the package. This
> actually is terrible to users, not the use of -march=x86-64.

You are right, this would be bad. %{optflags} does NOT use "-march=native'.
(Use 'rpmbuild --eval %{optflags}' to check.)

> Have a look at the f39 scratch build, where I disabled the patch 5 that forces -march=x86-64-v3.
> (https://koji.fedoraproject.org/koji/taskinfo?taskID=99523515)

Right. That build is broken because it injects additional build flags that override the
distro build flags. "-march=native" is the most problematic, but I also see
"-O1", "-O3", "-D_FORTIFY_SOURCE=2", "-fomit-frame-pointer" which should be dropped too.

Comment 6 Fedora Update System 2023-04-05 13:56:40 UTC
FEDORA-2023-7b7cd4bb36 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-7b7cd4bb36

Comment 7 Ralf Senderek 2023-04-05 14:12:50 UTC
> Right. That build is broken because it injects additional build flags that override the
> distro build flags. "-march=native" is the most problematic

That makes sense.

I have made sure, that the package now builds on Rawhide using "-m64 -mtune=generic" , the  %{optflags} default.
You can see the real build for Rawhide here:

https://koji.fedoraproject.org/koji/buildinfo?buildID=2181995

Updates for other releases will follow.

Ralf.

Comment 8 Fedora Update System 2023-04-05 14:29:48 UTC
FEDORA-2023-7b7cd4bb36 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 9 Zbigniew Jędrzejewski-Szmek 2023-04-05 14:40:47 UTC
Thanks for working on this!

> gcc -c -D__UNIX__ -DNDEBUG -I. -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ggdb3 -fno-omit-frame-pointer -O1 -DDATA_LITTLEENDIAN -DFIXED_SEED=0xa1a9ce099d80fdc4 -fPIC -DHAS_RECURSIVE_MUTEX -DHAS_ROBUST_MUTEX -m64 -mtune=generic -fPIC -Wno-pointer-sign -Wno-strict-aliasing -fwrapv -fno-delete-null-pointer-checks -fstack-protector-strong -D_FORTIFY_SOURCE=2 -DOSVERSION=6 -O3 -fomit-frame-pointer -D_REENTRANT -o ./shared-obj/bn_asm.o bn/bn_asm.c

-march=native is gone, so the code should be portable, but the other flags still remain :(

-D_FORTIFY_SOURCE=2
-O1
-O3
-fomit-frame-pointer

I don't know the details here, so maybe there's a good reason to inject those to
override the distro defaults. But in general, the recommendation is not to do that.
(In particular, -O1 is injected by 'flagspatch', which looks strange.)

Comment 10 Fedora Update System 2023-04-05 16:12:58 UTC
FEDORA-2023-c55dac9ab1 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-c55dac9ab1

Comment 11 Ralf Senderek 2023-04-05 16:14:33 UTC
> -O1 is injected by 'flagspatch', which looks strange.

This is my fault, this flag should long be gone. The other flags are generated by Cryptlib code.

new Rawhide build: https://koji.fedoraproject.org/koji/taskinfo?taskID=99561391

Ralf

Comment 12 Fedora Update System 2023-04-05 16:14:49 UTC
FEDORA-2023-c55dac9ab1 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.