Bug 1692646

Summary: Mono 5 does not build for s390x and just hangs at one point
Product: [Fedora] Fedora Reporter: Timotheus Pokorra <mailinglists>
Component: monoAssignee: Xavier Lamien <lxtnow>
Status: CLOSED CANTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: rawhideCC: aoliva, arjun.is, bugproxy, chkr, codonell, dan, dj, fweimer, hannsj_uhl, itamar, john.j5live, law, lxtnow, mfabian, neale, pfrankli, rhughes, rstrode, rth, sandmann, siddhesh
Target Milestone: ---   
Target Release: ---   
Hardware: s390x   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-27 11:18:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 467765, 1436896    

Description Timotheus Pokorra 2019-03-26 06:17:41 UTC
I tried this for F30 and for rawhide, to build Mono 5.18.1.

eg. this scratch build https://koji.fedoraproject.org/koji/taskinfo?taskID=33772449

the build log ends with these lines:

if test -w /builddir/build/BUILD/mono-5.18.1.0/mcs; then :; else chmod -R +w /builddir/build/BUILD/mono-5.18.1.0/mcs; fi
cd /builddir/build/BUILD/mono-5.18.1.0/mcs && make --no-print-directory -s NO_DIR_CHECK=1 PROFILES='net_4_x xbuild_12 xbuild_14           ' CC='gcc' all-profiles
mkdir -p -- build/deps
BUILDSTDERR: make[6]: mcs: Command not found
BUILDSTDERR: make[6]: *** [build/profiles/basic.make:121: build/deps/basic-profile-check.exe] Error 127
BUILDSTDERR: *** The runtime 'mono' doesn't appear to be usable.
BUILDSTDERR: *** Trying the 'monolite-linux/179F7FD7-9CE8-424E-84AF-0DF5B0B8B469' directory.

Then the build hangs.

But on other architectures, it continues after that to use the mcs from monolite:

if test -w /builddir/build/BUILD/mono-5.18.1.0/mcs; then :; else chmod -R +w /builddir/build/BUILD/mono-5.18.1.0/mcs; fi
cd /builddir/build/BUILD/mono-5.18.1.0/mcs && make --no-print-directory -s NO_DIR_CHECK=1 PROFILES='net_4_x xbuild_12 xbuild_14           ' CC='gcc' all-profiles
mkdir -p -- build/deps
BUILDSTDERR: make[6]: mcs: Command not found
BUILDSTDERR: make[6]: *** [build/profiles/basic.make:121: build/deps/basic-profile-check.exe] Error 127
BUILDSTDERR: *** The runtime 'mono' doesn't appear to be usable.
BUILDSTDERR: *** Trying the 'monolite-linux/179F7FD7-9CE8-424E-84AF-0DF5B0B8B469' directory.
BUILDSTDERR: warning CS8001: SDK path could not be resolved
BUILDSTDERR: .//build/gensources.cs(421,13): warning CS0219: The variable `parsedTarget' is assigned but its value is never used
Compilation succeeded - 2 warning(s)

I looked at https://github.com/mono/mono/issues/9009 and tried CFLAGS -no-pie, but that did not make a difference.

Comment 1 Dan Horák 2019-03-26 21:01:33 UTC
I have reproduced the hang locally, will attach a backtrace tomorrow.

Comment 2 Timotheus Pokorra 2019-03-27 05:06:52 UTC
Thanks for looking into this! Let me know if I can do anything.

Comment 3 Dan Horák 2019-03-27 09:16:07 UTC
from F-30 with glibc-2.29-8.fc30.s390x

The C code of Mono is built with "-O1 -march=z9-109 -mtune=z10" (workarounds taken from mono 4.8 packages), see https://src.fedoraproject.org/rpms/mono/blob/master/f/mono.spec#_355

(gdb) attach 54988
Attaching to process 54988
[New LWP 55006]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x000003ff8b791c9a in __lll_lock_wait () from /lib64/libpthread.so.0
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.29-8.fc30.s390x
(gdb) thread apply all bt

Thread 2 (Thread 0x3ff8afff910 (LWP 55006)):
#0  0x000003ff8b78e2b2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000002aa24355c04 in mono_os_cond_wait (mutex=0x2aa244e0f48 <lock>, cond=0x2aa244e0f70 <work_cond>) at ../../mono/utils/mono-os-mutex.h:173
#2  get_work (job=<synthetic pointer>, do_idle=<synthetic pointer>, work_context=<synthetic pointer>, worker_index=<optimized out>) at sgen-thread-pool.c:165
#3  thread_func (data=<optimized out>) at sgen-thread-pool.c:196
#4  0x000003ff8b788236 in start_thread () from /lib64/libpthread.so.0
#5  0x000003ff8b67a256 in thread_start () from /lib64/libc.so.6

Thread 1 (Thread 0x3ff8baf8760 (LWP 54988)):
#0  0x000003ff8b791c9a in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x000003ff8b78ab6c in pthread_mutex_lock () from /lib64/libpthread.so.0
#2  0x000003ff8ba93e28 in _dl_open () from /lib/ld64.so.1
#3  0x000003ff8b80116e in dlopen_doit () from /lib64/libdl.so.2
#4  0x000003ff8b6bcbda in _dl_catch_exception () from /lib64/libc.so.6
#5  0x000003ff8b6bccae in _dl_catch_error () from /lib64/libc.so.6
#6  0x000003ff8b801940 in _dlerror_run () from /lib64/libdl.so.2
#7  0x000003ff8b801204 in dlopen@@GLIBC_2.2 () from /lib64/libdl.so.2
#8  0x000002aa24361894 in mono_dl_open_file (file=file@entry=0x2aa26ce6540 "/home/sharkcz/mono/mono-5.18.1.0/mcs/class/lib/monolite-linux/179F7FD7-9CE8-424E-84AF-0DF5B0B8B469/mscorlib.dll.so", flags=flags@entry=257) at mono-dl-posix.c:80
#9  0x000002aa24360e66 in mono_dl_open (name=name@entry=0x2aa26ce6540 "/home/sharkcz/mono/mono-5.18.1.0/mcs/class/lib/monolite-linux/179F7FD7-9CE8-424E-84AF-0DF5B0B8B469/mscorlib.dll.so", flags=flags@entry=1, error_msg=error_msg@entry=0x3ffde47c0e8) at mono-dl.c:152
#10 0x000002aa240bbc32 in load_aot_module (assembly=0x2aa26ce7cb0, user_data=<optimized out>) at aot-runtime.c:2095
#11 0x000002aa241bdc0c in mono_assembly_invoke_load_hook (ass=ass@entry=0x2aa26ce7cb0) at assembly.c:1717
#12 0x000002aa241bea96 in mono_assembly_load_from_predicate (image=image@entry=0x2aa26ce6920, fname=fname@entry=0x2aa26ce2cd0 "/home/sharkcz/mono/mono-5.18.1.0/mcs/class/lib/monolite-linux/179F7FD7-9CE8-424E-84AF-0DF5B0B8B469/mscorlib.dll", asmctx=asmctx@entry=MONO_ASMCTX_DEFAULT, predicate=predicate@entry=0x0, user_data=user_data@entry=0x0, status=0x3ffde47c68c) at assembly.c:2848
#13 0x000002aa241c1c18 in mono_assembly_open_predicate (filename=filename@entry=0x2aa26ce6360 "/home/sharkcz/mono/mono-5.18.1.0/mcs/class/lib/monolite-linux/179F7FD7-9CE8-424E-84AF-0DF5B0B8B469/mscorlib.dll", asmctx=<optimized out>, asmctx@entry=MONO_ASMCTX_DEFAULT, predicate=predicate@entry=0x0, user_data=user_data@entry=0x0, requesting_assembly=requesting_assembly@entry=0x0, status=0x3ffde47c68c) at assembly.c:2341
#14 0x000002aa241c21ce in load_in_path (basename=basename@entry=0x2aa243c08b6 "mscorlib.dll", search_path=0x2aa26cde440, status=status@entry=0x3ffde47c68c, asmctx=asmctx@entry=MONO_ASMCTX_DEFAULT, predicate=predicate@entry=0x0, user_data=0x0) at assembly.c:762
#15 0x000002aa241c22f8 in mono_assembly_load_corlib (runtime=0x2aa243c0da8 <supported_runtimes>, status=status@entry=0x3ffde47c68c) at assembly.c:4054
#16 0x000002aa241b8c16 in mono_init_internal (filename=filename@entry=0x3ffde47e3c0 ".//class/lib/monolite-linux/179F7FD7-9CE8-424E-84AF-0DF5B0B8B469/mcs.exe", exe_filename=exe_filename@entry=0x3ffde47e3c0 ".//class/lib/monolite-linux/179F7FD7-9CE8-424E-84AF-0DF5B0B8B469/mcs.exe", runtime_version=runtime_version@entry=0x0) at domain.c:571
#17 0x000002aa241b9b20 in mono_init_from_assembly (domain_name=domain_name@entry=0x3ffde47e3c0 ".//class/lib/monolite-linux/179F7FD7-9CE8-424E-84AF-0DF5B0B8B469/mcs.exe", filename=filename@entry=0x3ffde47e3c0 ".//class/lib/monolite-linux/179F7FD7-9CE8-424E-84AF-0DF5B0B8B469/mcs.exe") at domain.c:830
#18 0x000002aa24038e72 in mini_init (filename=0x3ffde47e3c0 ".//class/lib/monolite-linux/179F7FD7-9CE8-424E-84AF-0DF5B0B8B469/mcs.exe", runtime_version=runtime_version@entry=0x0) at mini-runtime.c:4428
#19 0x000002aa2409cd64 in mono_main (argc=<optimized out>, argv=<optimized out>) at driver.c:2316
#20 0x000002aa2402f562 in mono_main_with_options (argv=<optimized out>, argc=<optimized out>) at main.c:50
#21 main (argc=<optimized out>, argv=<optimized out>) at main.c:405

Comment 4 Dan Horák 2019-03-27 09:16:57 UTC
switching to glibc for their opinion ...

Comment 5 Dan Horák 2019-03-27 09:29:45 UTC
and adding Neale for his insight as the main Mono/s390x person

Comment 6 Dan Horák 2019-03-27 11:18:51 UTC
Let's scratch that even when there is likely a bug somewhere in Mono. Seems I got Mono built on s390x with the help from https://github.com/mono/mono/issues/9009#issuecomment-451269331