Bug 2306353

Summary: libasan crashes without BTI/PAC on aarch64
Product: [Fedora] Fedora Reporter: Tulio Magno Quites Machado Filho <tuliom>
Component: gccAssignee: Jakub Jelinek <jakub>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 40CC: bill.roberts, dmalcolm, dvlasenk, fweimer, jakub, jlaw, josmyers, jwakely, mcermak, mpolacek, msebor, ngompa13, nickc, nixuser, pbrobinson, robdclark, sipoyare, suraj.ghimire7, yahmad
Target Milestone: ---   
Target Release: ---   
Hardware: aarch64   
OS: Linux   
Whiteboard:
Fixed In Version: gcc-14.2.1-3.fc41 gcc-14.2.1-3.fc40 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-09-17 00:15:28 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418    

Description Tulio Magno Quites Machado Filho 2024-08-20 23:34:58 UTC
We're getting an illegal instruction while using libasan.
GDB is reporting the illegal instruction happens during a branch to a PLT stub:

Core was generated by `/usr/bin/ls --color=auto'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x000040000093e0e0 in memset () from /usr/lib64/libasan.so.8
Missing separate debuginfos, use: dnf debuginfo-install coreutils-9.4-7.fc40.aarch64
>>> disas
Dump of assembler code for function memset:
=> 0x000040000093e0e0 <+0>:     b       0x4000008a0690 <__interceptor_memset@plt>
End of assembler dump.

The branch instruction looks correct.
According to comments in IRC channel #fedora-arm, the issue does not happen when the kernel is booted with parameter arm64.nobti.

Looking at the library, it does have all the features for BTI and PAC:

$ readelf -aW /usr/lib64/libasan.so.8 | grep BTI
 0x0000000070000001 (AARCH64_BTI_PLT)    
  GNU                  0x00000010       NT_GNU_PROPERTY_TYPE_0        Properties: AArch64 feature: BTI, PAC

But when we disassemble it, most of the PLT stubs do bot have BTI/PAC instructions:

/usr/lib64/libasan.so.8:     file format elf64-littleaarch64


Disassembly of section .init:

0000000000040000 <.init>:
   40000:       d503233f        paciasp
   40004:       a9bf7bfd        stp     x29, x30, [sp, #-16]!
   40008:       910003fd        mov     x29, sp
   4000c:       94000b48        bl      42d2c <*ABS*+0x8@plt+0xcc>
   40010:       a8c17bfd        ldp     x29, x30, [sp], #16
   40014:       d50323bf        autiasp
   40018:       d65f03c0        ret

Disassembly of section .plt:

0000000000040020 <__asan_report_store4@plt-0x20>:
   40020:       d503245f        bti     c
   40024:       a9bf7bf0        stp     x16, x30, [sp, #-16]!
   40028:       f0000ad0        adrp    x16, 19b000 <__asan_cplus_demangle_builtin_types@@Base+0x460>
   4002c:       f9451a11        ldr     x17, [x16, #2608]
   40030:       9128c210        add     x16, x16, #0xa30
   40034:       d61f0220        br      x17
   40038:       d503201f        nop
   4003c:       d503201f        nop

0000000000040040 <__asan_report_store4@plt>:
   40040:       f0000ad0        adrp    x16, 19b000 <__asan_cplus_demangle_builtin_types@@Base+0x460>
   40044:       f9451e11        ldr     x17, [x16, #2616]
   40048:       9128e210        add     x16, x16, #0xa38
   4004c:       d61f0220        br      x17

...

0000000000040690 <__interceptor_memset@plt>:
   40690:       f0000ad0        adrp    x16, 19b000 <__asan_cplus_demangle_builtin_types@@Base+0x460>
   40694:       f946b211        ldr     x17, [x16, #3424]
   40698:       91358210        add     x16, x16, #0xd60
   4069c:       d61f0220        br      x17

Reproducible: Always

Steps to Reproduce:
1. dnf install libasan
2. LD_PRELOAD=/usr/lib64/libasan.so.8 ls

Actual Results:  
Illegal instruction (core dumped)


Expected Results:  
Output from ls

Comment 1 Tulio Magno Quites Machado Filho 2024-08-20 23:44:25 UTC
This issue has been reproduced on Snapdragon X1 and QEMU.

Looking at other binaries, their PLT stubs also do not have BTI/PAC instructions and they run well on the same processors.
So, there is a chance this is not related to Binutils.

Comment 2 Neal Gompa 2024-08-21 11:10:02 UTC
It is also reproduced on Apple M2+ (which support BTI/PAC).

Comment 3 Tulio Magno Quites Machado Filho 2024-08-21 13:44:00 UTC
I'm starting to believe my initial description of this bug is wrong.
B is not an instruction that sets EPSR.B.

The previous frame was executing:

#1  0x00004000012fc070 in std::locale::_Impl::_Impl(unsigned long) () from /lib64/libstdc++.so.6

More specifically, it calls memset:

   0x00004000012fc064 <+68>:    mov     w1, #0x0                        // #0
   0x00004000012fc068 <+72>:    str     x4, [x19, #16]
   0x00004000012fc06c <+76>:    bl      0x4000012c4200 <memset@plt>
=> 0x00004000012fc070 <+80>:    str     x0, [x19, #8]

BL doesn't set EPSR.B either.

I wonder if PLT code might be causing issues here.
I'm trying to get access to real hardware in order to track this.

Comment 4 Nick Clifton 2024-08-22 10:23:47 UTC
Hi Tulio,

  Which build of libasan is involved ?

  Looking at libasan-14.1.1-7.fc40.aarch64.rpm I see that the plt stubs do indeed appear to be missing the "bti c" instruction at their entry points.  eg:
  
0000000000041070 <memset@plt>:
   41070:	f0000ad0 	adrp	x16, 19c000 <__interceptor_pthread_attr_getguardsize@@Base+0xd6460>
   41074:	f9412a11 	ldr	x17, [x16, #592]
   41078:	91094210 	add	x16, x16, #0x250
   4107c:	d61f0220 	br	x17

  I am not sure what is happening here, but I will ask ARM for help.

Cheers
  Nick

Comment 5 Tulio Magno Quites Machado Filho 2024-08-22 12:41:15 UTC
I was able to reproduce it with libasan-14.2.1-1.fc40.aarch64

Comment 6 Tulio Magno Quites Machado Filho 2024-08-22 17:48:39 UTC
I've just found out a piece of inline asm that is used to generate __interceptor_trampoline_memset and that it does not have support for PACBTI:
https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=libsanitizer/interception/interception.h;h=58e969378a90822ce727dc24af74944f225da189;hb=HEAD#l193

So, I'm reassigning this to GCC.

Comment 7 Bill Roberts 2024-09-04 16:28:36 UTC
Replicated and can confirm that the ASM_TAIL_CALL macro is the b instruction which is not a valid landing pad.

Comment 8 Bill Roberts 2024-09-04 16:35:57 UTC
I also can confirm that adding the proper landing pad there works, will clean up and upstream. Thanks Tullo, that triage work saved me a lot of time

Comment 9 Bill Roberts 2024-09-04 21:49:03 UTC
So libsanitizer is out of the clang tree, they have it fixed here: https://github.com/llvm/llvm-project/pull/84061

Comment 10 Fedora Update System 2024-09-13 13:25:10 UTC
FEDORA-2024-d47aa726a4 (gcc-14.2.1-3.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-d47aa726a4

Comment 11 Fedora Update System 2024-09-13 13:25:55 UTC
FEDORA-2024-c69055a3b6 (gcc-14.2.1-3.fc40) has been submitted as an update to Fedora 40.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-c69055a3b6

Comment 12 Fedora Update System 2024-09-14 02:29:37 UTC
FEDORA-2024-d47aa726a4 has been pushed to the Fedora 41 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-d47aa726a4`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-d47aa726a4

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 13 Fedora Update System 2024-09-14 03:16:34 UTC
FEDORA-2024-c69055a3b6 has been pushed to the Fedora 40 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-c69055a3b6`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-c69055a3b6

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 14 Fedora Update System 2024-09-17 00:15:28 UTC
FEDORA-2024-d47aa726a4 (gcc-14.2.1-3.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 15 Fedora Update System 2024-09-18 19:59:51 UTC
FEDORA-2024-c69055a3b6 (gcc-14.2.1-3.fc40) has been pushed to the Fedora 40 stable repository.
If problem still persists, please make note of it in this bug report.