Bug 2196250 - Kernel versions newer than 6.2.12 cause crashes in Mono builds in Yocto
Summary: Kernel versions newer than 6.2.12 cause crashes in Mono builds in Yocto
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 38
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL: https://github.com/DynamicDevices/met...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-05-08 13:06 UTC by Zoltan Boszormenyi
Modified: 2024-05-21 14:57 UTC (History)
17 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2024-05-21 14:57:55 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Zoltan Boszormenyi 2023-05-08 13:06:21 UTC
I am using Fedora as a build host for creating Yocto based images.

Recently I upgraded to Fedora 38 and my builds were working with the initial state of F38.

After recent upgrades, my builds started failing.

First, it occurred in running cert-sync for an in-house package. This runs the host binary of Mono.

After cleaning up the buildroot, building mono-native also started failing with crashes in CSC.

The Mono crash dump is:
===========================================================================

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x4a068d - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x4a0a0e - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x4515be - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x41ac98 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x42089f - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x7fc9e6681b70 - /lib64/libc.so.6 : 
	0x49c2f5 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x4561b7 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x41c01c - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x603e8a - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x606b27 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x572d2d - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x423575 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x42a47f - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : mono_main
	0x41a565 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : 
	0x7fc9e666bb4a - /lib64/libc.so.6 : 
	0x7fc9e666bc0b - /lib64/libc.so.6 : __libc_start_main
	0x41a395 - /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/mono : _start

=================================================================
	Telemetry Dumper:
=================================================================
No threads attached to runtime.

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 170946]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fc9e671ec73 in wait4 () from /lib64/libc.so.6
  Id   Target Id                                        Frame 
* 1    Thread 0x7fc9e6643240 (LWP 170895) "mono"        0x00007fc9e671ec73 in wait4 () from /lib64/libc.so.6
  2    Thread 0x7fc9e5bff6c0 (LWP 170946) "SGen worker" 0x00007fc9e66cd1d9 in __futex_abstimed_wait_common () from /lib64/libc.so.6

Thread 2 (Thread 0x7fc9e5bff6c0 (LWP 170946) "SGen worker"):
#0  0x00007fc9e66cd1d9 in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007fc9e66cfb79 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6
#2  0x00000000006b47d5 in mono_os_cond_wait (mutex=0x843a40 <lock>, cond=0x843a00 <work_cond>) at ../../mono/utils/mono-os-mutex.h:219
#3  get_work (job=<synthetic pointer>, do_idle=<synthetic pointer>, work_context=<synthetic pointer>, worker_index=0) at sgen-thread-pool.c:165
#4  thread_func (data=<optimized out>) at sgen-thread-pool.c:196
#5  0x00007fc9e66d0907 in start_thread () from /lib64/libc.so.6
#6  0x00007fc9e6756870 in clone3 () from /lib64/libc.so.6

Thread 1 (Thread 0x7fc9e6643240 (LWP 170895) "mono"):
#0  0x00007fc9e671ec73 in wait4 () from /lib64/libc.so.6
#1  0x00000000004a0956 in dump_native_stacktrace (signal=signal@entry=0x7a8288 "SIGSEGV", mctx=mctx@entry=0x7fffe31cb620) at mini-posix.c:1113
#2  0x00000000004a0a0e in mono_dump_native_crash_info (signal=signal@entry=0x7a8288 "SIGSEGV", mctx=mctx@entry=0x7fffe31cb620, info=info@entry=0x7fffe31cb930) at mini-posix.c:1157
#3  0x00000000004515be in mono_handle_native_crash (signal=0x7a8288 "SIGSEGV", mctx=0x7fffe31cb620, info=0x7fffe31cb930) at mini-exceptions.c:3423
#4  0x000000000041ac98 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7fffe31cb930, context=0x7fffe31cb800, debug_fault_addr=0x0) at mini-runtime.c:3369
#5  0x000000000042089f in mono_sigsegv_signal_handler (_dummy=<optimized out>, _info=<optimized out>, context=<optimized out>) at mini-runtime.c:3457
#6  <signal handler called>
#7  mono_arch_create_specific_trampoline (arg1=0x0, tramp_type=MONO_TRAMPOLINE_VCALL, domain=<optimized out>, code_len=code_len@entry=0x7fffe31cbe0c) at tramp-amd64.c:618
#8  0x00000000004561b7 in mono_create_specific_trampoline (arg1=arg1@entry=0x0, tramp_type=tramp_type@entry=MONO_TRAMPOLINE_VCALL, domain=0x20d5a00, code_len=code_len@entry=0x0) at mini-trampolines.c:1333
#9  0x000000000041c01c in mini_get_vtable_trampoline (vt=<optimized out>, slot_index=0) at mini-runtime.c:3556
#10 0x0000000000603e8a in mono_class_create_runtime_vtable (error=0x7fffe31cc0d0, klass=0x21052d0, domain=0x20d5a00) at object.c:2241
#11 mono_class_vtable_checked (domain=domain@entry=0x20d5a00, klass=klass@entry=0x21052d0, error=error@entry=0x7fffe31cc0d0) at object.c:1949
#12 0x0000000000606b27 in mono_object_new_pinned_handle (domain=domain@entry=0x20d5a00, klass=0x21052d0, error=error@entry=0x7fffe31cc0d0) at object.c:5836
#13 0x0000000000572d2d in mono_runtime_init_checked (domain=domain@entry=0x20d5a00, start_cb=start_cb@entry=0x41b495 <mono_thread_start_cb>, attach_cb=attach_cb@entry=0x41b46a <mono_thread_attach_cb>, error=error@entry=0x7fffe31cc0d0) at appdomain.c:333
#14 0x0000000000423575 in mini_init (filename=<optimized out>, runtime_version=runtime_version@entry=0x0) at mini-runtime.c:4420
#15 0x000000000042a47f in mono_main (argc=<optimized out>, argv=<optimized out>) at driver.c:2610
#16 0x000000000041a565 in mono_main_with_options (argv=<optimized out>, argc=<optimized out>) at /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/main.c:54
#17 main (argc=<optimized out>, argv=<optimized out>) at /home/zozo/dtd-yocto-4.2/tmp-sicom-glibc/work/x86_64-linux/mono-native/6.12.0.182-r0/mono-6.12.0.182/mono/mini/main.c:402
[Inferior 1 (process 170895) detached]

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x49c2f5):0x49c2e5  48 83 c0 01 48 83 f8 01 77 72 48 29 dd 8d 45 fb  H...H...wrH)..E.
0x49c2f5  c6 03 e8 88 43 01 0f b6 cc 88 4b 02 89 c2 c1 ea  ....C.....K.....
0x49c305  10 88 53 03 c1 e8 18 88 43 04 4d 85 ff 75 5e c6  ..S.....C.M..u^.
0x49c315  43 05 04 44 89 6b 06 48 8d 6b 0a 48 29 dd 49 63  C..D.k.H.k.H).Ic

===========================================================================

The crash is always identical and reproducible, but sporadic in the sense that it's not always the same DLL whose build crashes. Sometimes it starts happening earlier, sometimes later.

The build succeeds when using kernel 6.2.12-300, but crashes with
6.2.13-300, 6.2.14-300 and 6.3.1-200.

Reproducible: Always

Steps to Reproduce:
1. Set up a Yocto buildroot with meta-mono in the used layers
2. Try to build mono-native
3. 
Actual Results:  
Building the mono-native recipe fails, with crashes in CSC.exe

Expected Results:  
The build should succeed.

At first, I reported the issue against meta-mono at https://github.com/DynamicDevices/meta-mono/issues/180

Comment 1 Zoltan Boszormenyi 2023-05-09 08:14:50 UTC
Another ticket (not by me) against Mono: https://github.com/mono/mono/issues/21651

Comment 2 Aoife Moloney 2024-05-21 14:57:55 UTC
Fedora Linux 38 entered end-of-life (EOL) status on 2024-05-21.

Fedora Linux 38 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.