Hide Forgot
After switching the minimum architecture level for the C compiler (-mcpu in %__global_cflags, defined by redhat-rpm-config) from z9-109 to z10 (or anything newer - z196, zEC12) mdoc started to crash during the build. from a local rebuild ... MCS [net_4_x] mono-symbolicate.exe SymbolManager.cs(115,43): warning CS0168: The variable `e' is declared but never used Compilation succeeded - 1 warning(s) MCS [net_4_x] linkeranalyzer.exe MDOC [net_4_x] cs-errors.tree Stacktrace: at <unknown> <0xffffffff> at System.Configuration.ConfigurationElement.get_Properties () <0x0004c> at System.Configuration.ElementInformation..ctor (System.Configuration.ConfigurationElement,System.Configuration.PropertyInformation) <0x00114> at System.Configuration.ConfigurationElement.InitFromProperty (System.Configuration.PropertyInformation) <0x0005c> at System.Configuration.ConfigurationElementCollection.InitFromProperty (System.Configuration.PropertyInformation) <0x001ea> at System.Configuration.PropertyInformation.get_Value () <0x00114> at System.Configuration.ConfigurationElement.get_Item (string) <0x00098> at System.Configuration.ConfigurationElement.get_Item (System.Configuration.ConfigurationProperty) <0x00046> at System.Diagnostics.TraceSection.get_Listeners () <0x0003e> at System.Diagnostics.SystemDiagnosticsSection.InitializeDefault () <0x00042> at System.Configuration.ConfigurationElement.Reset (System.Configuration.ConfigurationElement) <0x000bc> at System.Configuration.Configuration.GetSectionInstance (System.Configuration.SectionInfo,bool) <0x00646> at System.Configuration.Configuration.GetSectionInstance (System.Configuration.SectionInfo,bool) <0x00430> at System.Configuration.ConfigurationSectionCollection.get_Item (string) <0x001e6> at System.Configuration.Configuration.GetSection (string) <0x00102> at System.Configuration.ClientConfigurationSystem.System.Configuration.Internal.IInternalConfigSystem.GetSection (string) <0x0004e> at System.Configuration.ConfigurationManager.GetSection (string) <0x0004a> at System.Configuration.PrivilegedConfigurationManager.GetSection (string) <0x00026> at System.Diagnostics.DiagnosticsConfiguration.GetConfigSection () <0x00036> at System.Diagnostics.DiagnosticsConfiguration.Initialize () <0x000d8> at System.Diagnostics.DiagnosticsConfiguration.get_SwitchSettings () <0x00020> at System.Diagnostics.Switch.InitializeConfigSettings () <0x00068> at System.Diagnostics.Switch.InitializeWithStatus () <0x0012a> at System.Diagnostics.Switch.get_SwitchSetting () <0x0003e> at System.Diagnostics.BooleanSwitch.get_Enabled () <0x00026> at System.Xml.Serialization.TempAssembly.LoadGeneratedAssembly (System.Type,string,System.Xml.Serialization.XmlSerializerImplementation&) <0x0011e> at System.Xml.Serialization.XmlSerializer..ctor (System.Type,string) <0x003a8> at System.Xml.Serialization.XmlSerializer..ctor (System.Type) <0x00030> at Monodoc.Providers.ErrorProvider.ReadConfig (string) <0x00060> at Monodoc.Providers.ErrorProvider..ctor (string) <0x00040> at Mono.Documentation.MDocAssembler.<Run>m__1 (string) <0x00050> at System.Linq.Enumerable/WhereSelectListIterator`2<TSource_REF, TResult_REF>.MoveNext () <0x0023c> at System.Collections.Generic.List`1<T_REF>.InsertRange (int,System.Collections.Generic.IEnumerable`1<T_REF>) <0x0042c> at System.Collections.Generic.List`1<T_REF>.AddRange (System.Collections.Generic.IEnumerable`1<T_REF>) <0x0003c> at Mono.Documentation.MDocAssembler.Run (System.Collections.Generic.IEnumerable`1<string>) <0x00e9e> at Mono.Documentation.MDoc.Run (string[]) <0x00bb6> at Mono.Documentation.MDoc.Main (string[]) <0x00080> at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <0x0015c> Native stacktrace: /home/sharkcz/mono/mono-4.6.2/mono/mini/mono(+0xd3a72) [0x2aa0a1d3a72] /home/sharkcz/mono/mono-4.6.2/mono/mini/mono(+0x42f3a) [0x2aa0a142f3a] [0x3ffff0f9370] /home/sharkcz/mono/mono-4.6.2/mono/mini/mono(mono_metadata_get_inflated_signature+0x46) [0x2aa0a299c66] /home/sharkcz/mono/mono-4.6.2/mono/mini/mono(+0x17314a) [0x2aa0a27314a] /home/sharkcz/mono/mono-4.6.2/mono/mini/mono(+0x7e8be) [0x2aa0a17e8be] /home/sharkcz/mono/mono-4.6.2/mono/mini/mono(+0x11b33e) [0x2aa0a21b33e] /home/sharkcz/mono/mono-4.6.2/mono/mini/mono(+0x11d346) [0x2aa0a21d346] /home/sharkcz/mono/mono-4.6.2/mono/mini/mono(+0x41dca) [0x2aa0a141dca] /home/sharkcz/mono/mono-4.6.2/mono/mini/mono(+0xd588e) [0x2aa0a1d588e] /home/sharkcz/mono/mono-4.6.2/mono/mini/mono(+0xd5fbc) [0x2aa0a1d5fbc] [0x3ffb29660e2] Debug info from gdb: (lldb) command source -s 0 '/tmp/mono-lldb-commands.OxjGNR' Executing commands in '/tmp/mono-lldb-commands.OxjGNR'. (lldb) process attach --pid 13829 Process 13829 stopped * thread #1: tid = 13829, 0x000003ffb2613b26 libpthread.so.0`__waitpid + 110, name = 'Main', stop reason = signal SIGSTOP frame #0: 0x000003ffb2613b26 libpthread.so.0`__waitpid + 110 libpthread.so.0`__waitpid: -> 0x3ffb2613b26 <+110>: clg %r2, 0(%r13) 0x3ffb2613b2c <+116>: jh 4396744260470 ; <+190> 0x3ffb2613b30 <+120>: lr %r11, %r2 0x3ffb2613b32 <+122>: lgr %r2, %r1 thread #2: tid = 13844, 0x000003ffb260e3ea libpthread.so.0`__pthread_cond_wait + 402, name = 'SGen worker', stop reason = signal SIGSTOP frame #0: 0x000003ffb260e3ea libpthread.so.0`__pthread_cond_wait + 402 libpthread.so.0`__pthread_cond_wait: -> 0x3ffb260e3ea <+402>: lgr %r6, %r2 0x3ffb260e3ee <+406>: lgr %r2, %r1 0x3ffb260e3f2 <+410>: clg %r6, 8(%r13) 0x3ffb260e3f8 <+416>: jh 4396744238578 ; <+922> thread #3: tid = 13845, 0x000003ffb2611916 libpthread.so.0`do_futex_wait.constprop.1 + 70, name = 'Finalizer', stop reason = signal SIGSTOP frame #0: 0x000003ffb2611916 libpthread.so.0`do_futex_wait.constprop.1 + 70 libpthread.so.0`do_futex_wait.constprop.1: -> 0x3ffb2611916 <+70>: clg %r2, 0(%r13) 0x3ffb261191c <+76>: jh 4396744251712 ; <+112> 0x3ffb2611920 <+80>: lgr %r2, %r1 0x3ffb2611924 <+84>: brasl %r14, 4396744254912 ; __pthread_disable_asynccancel Executable module set to "/home/sharkcz/mono/mono-4.6.2/mono/mini/mono-sgen". Architecture set to: s390x-ibm-linux. (lldb) thread list Process 13829 stopped * thread #1: tid = 13829, 0x000003ffb2613b26 libpthread.so.0`__waitpid + 110, name = 'Main', stop reason = signal SIGSTOP thread #2: tid = 13844, 0x000003ffb260e3ea libpthread.so.0`__pthread_cond_wait + 402, name = 'SGen worker', stop reason = signal SIGSTOP thread #3: tid = 13845, 0x000003ffb2611916 libpthread.so.0`do_futex_wait.constprop.1 + 70, name = 'Finalizer', stop reason = signal SIGSTOP (lldb) thread backtrace all * thread #1: tid = 13829, 0x000003ffb2613b26 libpthread.so.0`__waitpid + 110, name = 'Main', stop reason = signal SIGSTOP * frame #0: 0x000003ffb2613b26 libpthread.so.0`__waitpid + 110 frame #1: 0x000002aa0a1d3b6a mono-sgen`mono_handle_native_sigsegv(signal=<unavailable>, ctx=<unavailable>, info=<unavailable>) + 546 at mini-exceptions.c:2427 frame #2: 0x000002aa0a142f3a mono-sgen`mono_sigsegv_signal_handler(_dummy=<unavailable>, _info=0x000003ffff0f9378, context=0x000003ffff0f93f8) + 370 at mini-runtime.c:2873 thread #2: tid = 13844, 0x000003ffb260e3ea libpthread.so.0`__pthread_cond_wait + 402, name = 'SGen worker', stop reason = signal SIGSTOP frame #0: 0x000003ffb260e3ea libpthread.so.0`__pthread_cond_wait + 402 frame #1: 0x000002aa0a36519a mono-sgen`thread_func + 18 at mono-os-mutex.h:107 frame #2: 0x000002aa0a365188 mono-sgen`thread_func(thread_data=0x0000000000000000) + 520 at sgen-thread-pool.c:110 frame #3: 0x000003ffb2607f12 libpthread.so.0`start_thread + 210 thread #3: tid = 13845, 0x000003ffb2611916 libpthread.so.0`do_futex_wait.constprop.1 + 70, name = 'Finalizer', stop reason = signal SIGSTOP frame #0: 0x000003ffb2611916 libpthread.so.0`do_futex_wait.constprop.1 + 70 frame #1: 0x000003ffb2611a00 libpthread.so.0`__new_sem_wait_slow.constprop.0 + 136 frame #2: 0x000002aa0a2e9f02 mono-sgen`finalizer_thread + 12 at mono-os-semaphore.h:166 frame #3: 0x000002aa0a2e9ef6 mono-sgen`finalizer_thread + 14 at mono-coop-semaphore.h:40 frame #4: 0x000002aa0a2e9ee8 mono-sgen`finalizer_thread(unused=<unavailable>) + 744 at gc.c:761 frame #5: 0x000002aa0a2c5c7c mono-sgen`start_wrapper + 428 at threads.c:740 frame #6: 0x000002aa0a2c5ad0 mono-sgen`start_wrapper(data=<unavailable>) + 64 at threads.c:788 frame #7: 0x000002aa0a39b0a4 mono-sgen`inner_start_thread(arg=<unavailable>) + 228 at mono-threads-posix.c:92 frame #8: 0x000003ffb2607f12 libpthread.so.0`start_thread + 210 (lldb) detach Process 13829 detached (lldb) (lldb) quit ================================================================= Got a SIGSEGV while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application. ================================================================= /bin/sh: line 1: 13829 Aborted (core dumped) MONO_PATH="./../class/lib/net_4_x:$MONO_PATH" /home/sharkcz/mono/mono-4.6.2/runtime/mono-wrapper ./../class/lib/net_4_x/mdoc.exe --debug assemble -o cs-errors -f error cs-errors.config make[7]: *** [Makefile:149: cs-errors.tree] Error 134 make[6]: *** [../build/rules.make:221: do-all] Error 2 make[5]: *** [build/rules.make:248: all-recursive] Error 1 make[4]: *** [Makefile:49: profile-do--net_4_x--all] Error 2 make[3]: *** [Makefile:45: profiles-do--all] Error 2 make[2]: *** [Makefile:543: all-local] Error 2 make[2]: Leaving directory '/home/sharkcz/mono/mono-4.6.2/runtime' make[1]: *** [Makefile:512: all-recursive] Error 1 make[1]: Leaving directory '/home/sharkcz/mono/mono-4.6.2' make: *** [Makefile:441: all] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.sU4ejs (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.sU4ejs (%build) Could not execute local: Non zero exit Build in koji fails the same way (eg. http://s390.koji.fedoraproject.org/koji/taskinfo?taskID=2409201). Also fails with both hardening enabled (our default) and disabled. Version-Release number of selected component (if applicable): mono-4.6.2-1.fc26 Build environment: Fedora Rawhide - locally or koji chroot
when running mdoc under gdb: [sharkcz@devel11 docs]$ gdb /home/sharkcz/mono/mono-4.6.2/mono/mini/mono GNU gdb (GDB) Fedora 7.12-29.fc26 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "s390x-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /home/sharkcz/mono/mono-4.6.2/mono/mini/mono...done. (gdb) set args ./../class/lib/net_4_x/mdoc.exe --debug assemble -o cs-errors -f error cs-errors.config (gdb) run Starting program: /home/sharkcz/mono/mono-4.6.2/mono/mini/mono ./../class/lib/net_4_x/mdoc.exe --debug assemble -o cs-errors -f error cs-errors.config Missing separate debuginfos, use: dnf debuginfo-install glibc-2.24.90-12.fc26.s390x glibc-2.24.90-13.fc26.s390x [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x3fff67ff910 (LWP 4443)] [New Thread 0x3fff4bff910 (LWP 4444)] Thread 1 "Main" received signal SIGSEGV, Segmentation fault. mono_metadata_get_inflated_signature (sig=sig@entry=0x2aa0075e8b0, context=context@entry=0x3f3851ec00000000) at metadata.c:2837 2837 helper.context.class_inst = context->class_inst; (gdb) where #0 mono_metadata_get_inflated_signature (sig=sig@entry=0x2aa0075e8b0, context=context@entry=0x3f3851ec00000000) at metadata.c:2837 #1 0x000002aa0017314a in mono_method_get_signature_checked (method=method@entry=0x3ffffffcf80, image=0x2aa004e8250, image@entry=0x3ffda1fd368, token=token@entry=167772356, context=context@entry=0x3f3851ec00000000, error=error@entry=0x2aa0084cc28) at loader.c:720 #2 0x000002aa0007e8be in mono_method_to_ir (cfg=cfg@entry=0x2aa0084c7a0, method=method@entry=0x2aa0073f7d8, start_bblock=<optimized out>, start_bblock@entry=0x0, end_bblock=0x2aa007f53b0, end_bblock@entry=0x0, return_var=return_var@entry=0x0, inline_args=<optimized out>, inline_offset=<optimized out>, is_virtual_call=<optimized out>) at method-to-ir.c:9319 #3 0x000002aa0011b33e in mini_method_compile (method=method@entry=0x2aa0073f7d8, opts=opts@entry=370223487, domain=domain@entry=0x2aa004231e0, flags=flags@entry=JIT_FLAG_RUN_CCTORS, parts=parts@entry=0, aot_method_index=<optimized out>) at mini.c:3516 #4 0x000002aa0011d346 in mono_jit_compile_method_inner (method=method@entry=0x2aa0073f7d8, target_domain=target_domain@entry=0x2aa004231e0, opt=opt@entry=370223487, error=error@entry=0x3ffffffcf80) at mini.c:4197 #5 0x000002aa00041dca in mono_jit_compile_method_with_opt (method=method@entry=0x2aa0073f7d8, opt=370223487, error=error@entry=0x3ffffffcf80) at mini-runtime.c:1910 #6 0x000002aa00042662 in mono_jit_compile_method (method=method@entry=0x2aa0073f7d8, error=error@entry=0x3ffffffcf80) at mini-runtime.c:1954 #7 0x000002aa000d588e in common_call_trampoline (regs=regs@entry=0x3ffffffd0f0, code=code@entry=0x3fffddb033c "\247\t", m=m@entry=0x2aa0073f7d8, vt=vt@entry=0x0, vtable_slot=<optimized out>, vtable_slot@entry=0x0, error=0x3ffffffcf80) at mini-trampolines.c:702 #8 0x000002aa000d5fbc in mono_magic_trampoline (regs=0x3ffffffd0f0, code=0x3fffddb033c "\247\t", arg=0x2aa0073f7d8, tramp=<optimized out>) at mini-trampolines.c:828 #9 0x000003fffdfe60e2 in ?? () #10 0x000003fffddb033c in ?? () #11 0x000003fffddcf0f8 in ?? () #12 0x000003fffddceeea in ?? () #13 0x000003fffddce874 in ?? () #14 0x000003fffddce6f2 in ?? () #15 0x000003fffddce416 in ?? () #16 0x000003fffdde4000 in ?? () #17 0x000003fffdde381a in ?? () #18 0x000003fffdde34ba in ?? () #19 0x000003fffdde32f2 in ?? () #20 0x000003fffdde323a in ?? () #21 0x000003fffdde2fce in ?? () #22 0x000003fffdde29ee in ?? () #23 0x000003fffdde254e in ?? () #24 0x000003fffddfd010 in ?? () #25 0x000003fffdfd4e30 in ?? () #26 0x000003fffdfd368a in ?? () PC not saved
So it's more likely a gcc compiler issue than mono issue. After switching the default optimization for the build from -O2 to -O1 the build passes (no crash in mdoc). The system compiler is gcc-6.2.1-2.fc26.s390x
And the fun now begins ...
next step - method-to-ir.c is miscompiled with -O2 -mcpu=z10
3 places were identified as requiring -O1 for the mdoc tool to function correctly, the ir-emit.h header with a number of inline functions and macros and the mini_emit_inst_for_method() + link_bblock() functions --- method-to-ir.c.orig 2016-11-14 09:48:51.000000000 +0100 +++ method-to-ir.c 2016-11-24 16:56:19.742489485 +0100 @@ -68,7 +68,9 @@ #include "trace.h" +#pragma GCC optimize("O1") #include "ir-emit.h" +#pragma GCC optimize("O2") #include "jit-icalls.h" #include "jit.h" @@ -547,6 +549,7 @@ MONO_ADD_INS (cfg->cbb, ins); \ } while (0) +#pragma GCC optimize("O1") /* * * link_bblock: Links two basic blocks * @@ -609,6 +612,7 @@ } } +#pragma GCC optimize("O2") void mono_link_bblock (MonoCompile *cfg, MonoBasicBlock *from, MonoBasicBlock* to) { @@ -5969,6 +5973,7 @@ return NULL; } +#pragma GCC optimize("O1") static MonoInst* mini_emit_inst_for_method (MonoCompile *cfg, MonoMethod *cmethod, MonoMethodSignature *fsig, MonoInst **args) { @@ -6869,6 +6874,7 @@ return mono_arch_emit_inst_for_method (cfg, cmethod, fsig, args); } +#pragma GCC optimize("O2") /* * This entry point could be used later for arbitrary method * redirection.
Created attachment 1223934 [details] script to reproduce the mdoc call outside the build system
the full command line for the source file should be gcc -DHAVE_CONFIG_H -I. -I../.. -I../../libgc/include -DGC_LINUX_THREADS -D_GNU_SOURCE -D_REENTRANT -DUSE_MMAP -DUSE_MUNMAP -g -Wall -Wunused -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Wpointer-arith -Wno-cast-qual -Wwrite-strings -Wno-switch -Wno-switch-enum -Wno-unused-value -Wno-attributes -DUSE_COMPILER_TLS -I../.. -I../../eglib/src -I../../eglib/src -fvisibility=hidden -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -march=z10 -mtune=z10 -fno-strict-aliasing -std=gnu99 -fno-strict-aliasing -fwrapv -DMONO_DLL_EXPORT -Wno-unused-but-set-variable -g -Wall -Wunused -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Wpointer-arith -Wno-cast-qual -Wwrite-strings -Wno-switch -Wno-switch-enum -Wno-unused-value -Wno-attributes -mbackchain -D__USE_STRING_INLINES -Werror-implicit-function-declaration -c method-to-ir.c -fPIC -DPIC -o .libs/libmini_la-method-to-ir.o
Created attachment 1223936 [details] preprocessed source file
for the record - using -mno-lra doesn't help when compiling the method-to-ir.c file
building with gcc7 and the new defaults (-march=zEC12 -mtune=z13) gives same error, the mono package is back on -march=z9-109 -mtune=z10 to be buildable, using -march=z9-109 -mtune=z13 gives even more interesting results
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle. Changing version to '26'.
The problem is still here, rechecked with gcc-7.2.1-2.fc26.s390x building mono-4.8.0-12.fc28
This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle. Changing version to '28'.
Starting with F-28 (gcc8?) Mono is broken even more on s390x, the workaround is to built with -O1 globally until we know more.
This message is a reminder that Fedora 28 is nearing its end of life. On 2019-May-28 Fedora will stop maintaining and issuing updates for Fedora 28. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '28'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 28 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
For the record - not sure if there was a change in GCC (>=9) or in Mono (more likely), but the problem went away.