Bug 1500863 - Ruby test suite randomly fails on AArch64
Summary: Ruby test suite randomly fails on AArch64
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: ruby
Version: 33
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jeroen van Meeuwen
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-11 15:36 UTC by Vít Ondruch
Modified: 2020-08-11 13:03 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Ruby 13758 None None None 2017-10-11 15:38:24 UTC

Description Vít Ondruch 2017-10-11 15:36:02 UTC
Description of problem:
I often, but randomly, observer following bug in Ruby test suite:

~~~
  1) Failure:
TestRubyOptions#test_segv_setproctitle [/builddir/build/BUILD/ruby-2.5.0-r59376/test/ruby/test_rubyoptions.rb:633]:
1. [2/2] Assertion for "stderr"
   | Expected /
   |         \[NOTE\]\n
   |         You\smay\shave\sencountered\sa\sbug\sin\sthe\sRuby\sinterpreter\sor\sextension\slibraries.\n
   |         Bug\sreports\sare\swelcome.\n
   |         (?:.*\n)?
   |         For\sdetails:\shttp:\/\/.*\.ruby-lang\.org\/.*\n
   |         \n
   |         (?:
   |           \[IMPORTANT\]\n
   |           (?:.+\n)+
   |           \n
   |         )?
   |       /x
   | to match
   |   "\n"+
   |   "-- Ruby level backtrace information ----------------------------------------\n"+
   |   "-e:1:in `<main>'\n"+
   |   "-e:1:in `kill'\n\n"+
   |   "-- C level backtrace information -------------------------------------------\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0(rb_print_backtrace+0x20) [0xffff940b5a70] vm_dump.c:671\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0(rb_vm_bugreport+0x8c) [0xffff940b5b0c] vm_dump.c:941\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0(rb_bug_context+0xb8) [0xffff93f92528] error.c:534\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0(sigsegv+0x4c) [0xffff9404c2e4] signal.c:930\n"+
   |   "linux-vdso.so.1 [0xffff942096c0]\n"+
   |   "[0xffff93c230c8]\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0(rb_f_kill+0x2c4) [0xffff9404d1bc] signal.c:498\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0(vm_call_cfunc+0xec) [0xffff940a1e34] vm_insnhelper.c:1889\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0(vm_call_method+0xc8) [0xffff940ace90] ./include/ruby/ruby.h:1966\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0(vm_exec_core+0x1490) [0xffff940a5ab8] insns.def:856\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0(vm_exec+0x84) [0xffff940a97bc] vm_insnhelper.h:231\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0(ruby_exec_internal+0xb4) [0xffff93f959c4] eval.c:244\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0(ruby_exec_node+0x20) [0xffff93f97808] eval.c:308\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0(ruby_run_node+0x24) [0xffff93f995fc] eval.c:300\n"+
   |   "/builddir/build/BUILD/ruby-2.5.0-r59376/ruby(main+0x50) [0xaaaad9048ac0] ./main.c:42\n\n"+
   |   "-- Other runtime information -----------------------------------------------\n\n"+
   |   "* Loaded script: /tmp/test_ruby_test_bug759720170720-12628-14djsbm.rb\n\n"+
   |   "* Loaded features:\n\n"+
   |   "    0 enumerator.so\n"+
   |   "    1 thread.rb\n"+
   |   "    2 rational.so\n"+
   |   "    3 complex.so\n"+
   |   "    4 /builddir/build/BUILD/ruby-2.5.0-r59376/.ext/aarch64-linux/enc/encdb.so\n"+
   |   "    5 /builddir/build/BUILD/ruby-2.5.0-r59376/.ext/aarch64-linux/enc/trans/transdb.so\n\n"+
   |   "* Process memory map:\n\n"+
   |   "aaaad9048000-aaaad9049000 r-xp 00000000 fc:03 400504                     /builddir/build/BUILD/ruby-2.5.0-r59376/ruby\n"+
   |   "aaaad9067000-aaaad9068000 r--p 0000f000 fc:03 400504                     /builddir/build/BUILD/ruby-2.5.0-r59376/ruby\n"+
   |   "aaaad9068000-aaaad9069000 rw-p 00010000 fc:03 400504                     /builddir/build/BUILD/ruby-2.5.0-r59376/ruby\n"+
   |   "aaaafb98e000-aaaafbacb000 rw-p 00000000 00:00 0                          [heap]\n"+
   |   "ffff92ded000-ffff93a26000 r--s 00000000 fc:03 400439                     /builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0\n"+
   |   "ffff93a26000-ffff93a39000 r--s 00000000 fc:03 400504                     /builddir/build/BUILD/ruby-2.5.0-r59376/ruby\n"+
   |   "ffff93a39000-ffff93a4d000 r-xp 00000000 fc:03 1054631                    /usr/lib64/libgcc_s-7-20170718.so.1\n"+
   |   "ffff93a4d000-ffff93a68000 ---p 00014000 fc:03 1054631                    /usr/lib64/libgcc_s-7-20170718.so.1\n"+
   |   "ffff93a68000-ffff93a69000 r--p 0001f000 fc:03 1054631                    /usr/lib64/libgcc_s-7-20170718.so.1\n"+
   |   "ffff93a69000-ffff93a6a000 rw-p 00020000 fc:03 1054631                    /usr/lib64/libgcc_s-7-20170718.so.1\n"+
   |   "ffff93a6a000-ffff93a6c000 r-xp 00000000 fc:03 921755                     /builddir/build/BUILD/ruby-2.5.0-r59376/.ext/aarch64-linux/enc/trans/transdb.so\n"+
   |   "ffff93a6c000-ffff93a89000 ---p 00002000 fc:03 921755                     /builddir/build/BUILD/ruby-2.5.0-r59376/.ext/aarch64-linux/enc/trans/transdb.so\n"+
   |   "ffff93a89000-ffff93a8a000 r--p 0000f000 fc:03 921755                     /builddir/build/BUILD/ruby-2.5.0-r59376/.ext/aarch64-linux/enc/trans/transdb.so\n"+
   |   "ffff93a8a000-ffff93a8b000 rw-p 00000000 00:00 0 \n"+
   |   "ffff93a8b000-ffff93a8d000 r-xp 00000000 fc:03 794844                     /builddir/build/BUILD/ruby-2.5.0-r59376/.ext/aarch64-linux/enc/encdb.so\n"+
   |   "ffff93a8d000-ffff93aaa000 ---p 00002000 fc:03 794844                     /builddir/build/BUILD/ruby-2.5.0-r59376/.ext/aarch64-linux/enc/encdb.so\n"+
   |   "ffff93aaa000-ffff93aab000 r--p 0000f000 fc:03 794844                     /builddir/build/BUILD/ruby-2.5.0-r59376/.ext/aarch64-linux/enc/encdb.so\n"+
   |   "ffff93aab000-ffff93aac000 rw-p 00000000 00:00 0 \n"+
   |   "ffff93aac000-ffff93aad000 ---p 00000000 00:00 0 \n"+
   |   "ffff93aad000-ffff93bcd000 rw-p 00000000 00:00 0 \n"+
   |   "ffff93bcd000-ffff93bcf000 r-xp 00000000 fc:03 1055521                    /usr/lib64/libfreebl3.so\n"+
   |   "ffff93bcf000-ffff93bec000 ---p 00002000 fc:03 1055521                    /usr/lib64/libfreebl3.so\n"+
   |   "ffff93bec000-ffff93bed000 r--p 0000f000 fc:03 1055521                    /usr/lib64/libfreebl3.so\n"+
   |   "ffff93bed000-ffff93bee000 rw-p 00010000 fc:03 1055521                    /usr/lib64/libfreebl3.so\n"+
   |   "ffff93bee000-ffff93d77000 r-xp 00000000 fc:03 1055000                    /usr/lib64/libc-2.25.90.so\n"+
   |   "ffff93d77000-ffff93d8a000 ---p 00189000 fc:03 1055000                    /usr/lib64/libc-2.25.90.so\n"+
   |   "ffff93d8a000-ffff93d8e000 r--p 0018c000 fc:03 1055000                    /usr/lib64/libc-2.25.90.so\n"+
   |   "ffff93d8e000-ffff93d90000 rw-p 00190000 fc:03 1055000                    /usr/lib64/libc-2.25.90.so\n"+
   |   "ffff93d90000-ffff93d94000 rw-p 00000000 00:00 0 \n"+
   |   "ffff93d94000-ffff93e49000 r-xp 00000000 fc:03 1055006                    /usr/lib64/libm-2.25.90.so\n"+
   |   "ffff93e49000-ffff93e63000 ---p 000b5000 fc:03 1055006                    /usr/lib64/libm-2.25.90.so\n"+
   |   "ffff93e63000-ffff93e64000 r--p 000bf000 fc:03 1055006                    /usr/lib64/libm-2.25.90.so\n"+
   |   "ffff93e64000-ffff93e65000 rw-p 000c0000 fc:03 1055006                    /usr/lib64/libm-2.25.90.so\n"+
   |   "ffff93e65000-ffff93e6c000 r-xp 00000000 fc:03 1055524                    /usr/lib64/libcrypt-nss-2.25.90.so\n"+
   |   "ffff93e6c000-ffff93e84000 ---p 00007000 fc:03 1055524                    /usr/lib64/libcrypt-nss-2.25.90.so\n"+
   |   "ffff93e84000-ffff93e85000 r--p 0000f000 fc:03 1055524                    /usr/lib64/libcrypt-nss-2.25.90.so\n"+
   |   "ffff93e85000-ffff93e86000 rw-p 00010000 fc:03 1055524                    /usr/lib64/libcrypt-nss-2.25.90.so\n"+
   |   "ffff93e86000-ffff93eb4000 rw-p 00000000 00:00 0 \n"+
   |   "ffff93eb4000-ffff93eb7000 r-xp 00000000 fc:03 1055004                    /usr/lib64/libdl-2.25.90.so\n"+
   |   "ffff93eb7000-ffff93ed3000 ---p 00003000 fc:03 1055004                    /usr/lib64/libdl-2.25.90.so\n"+
   |   "ffff93ed3000-ffff93ed4000 r--p 0000f000 fc:03 1055004                    /usr/lib64/libdl-2.25.90.so\n"+
   |   "ffff93ed4000-ffff93ed5000 rw-p 00010000 fc:03 1055004                    /usr/lib64/libdl-2.25.90.so\n"+
   |   "ffff93ed5000-ffff93eef000 r-xp 00000000 fc:03 1055014                    /usr/lib64/libpthread-2.25.90.so\n"+
   |   "ffff93eef000-ffff93f04000 ---p 0001a000 fc:03 1055014                    /usr/lib64/libpthread-2.25.90.so\n"+
   |   "ffff93f04000-ffff93f05000 r--p 0001f000 fc:03 1055014                    /usr/lib64/libpthread-2.25.90.so\n"+
   |   "ffff93f05000-ffff93f06000 rw-p 00020000 fc:03 1055014                    /usr/lib64/libpthread-2.25.90.so\n"+
   |   "ffff93f06000-ffff93f0a000 rw-p 00000000 00:00 0 \n"+
   |   "ffff93f0a000-ffff94198000 r-xp 00000000 fc:03 400439                     /builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0\n"+
   |   "ffff94198000-ffff941b2000 ---p 0028e000 fc:03 400439                     /builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0\n"+
   |   "ffff941b2000-ffff941ba000 r--p 00298000 fc:03 400439                     /builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0\n"+
   |   "ffff941ba000-ffff941bb000 rw-p 002a0000 fc:03 400439                     /builddir/build/BUILD/ruby-2.5.0-r59376/libruby.so.2.5.0\n"+
   |   "ffff941bb000-ffff941cb000 rw-p 00000000 00:00 0 \n"+
   |   "ffff941cb000-ffff941ed000 r-xp 00000000 fc:03 1054993                    /usr/lib64/ld-2.25.90.so\n"+
   |   "ffff941fd000-ffff94201000 rw-p 00000000 00:00 0 \n"+
   |   "ffff94206000-ffff94208000 rw-p 00000000 00:00 0 \n"+
   |   "ffff94208000-ffff94209000 r--p 00000000 00:00 0                          [vvar]\n"+
   |   "ffff94209000-ffff9420a000 r-xp 00000000 00:00 0                          [vdso]\n"+
   |   "ffff9420a000-ffff9420b000 r--p 0002f000 fc:03 1054993                    /usr/lib64/ld-2.25.90.so\n"+
   |   "ffff9420b000-ffff9420c000 rw-p 00030000 fc:03 1054993                    /usr/lib64/ld-2.25.90.so\n"+
   |   "ffff9420c000-ffff9420d000 rw-p 00000000 00:00 0 \n"+
   |   "fffffc21f000-fffffca1e000 rw-p 00000000 00:00 0                          [stack]\n"+
   |   "*** Error in `/tmp/test_ruby_test_bug759720170720-12628-14djsbm.rb': double free or corruption (out): 0x0000aaaafba37fe0 ***\n"
   | after 6 patterns with 326 characters.
~~~

For example in this build [1]. As far as I can tell, this happens only on AArch64.


Version-Release number of selected component (if applicable):
Ruby 2.4 as well as development branch of Ruby suffers this issue.


How reproducible:


Steps to Reproduce:
1. Go into the Ruby build directory
2. for i in {1..10}; do echo '*************' $i; make test-all TESTS="-n /test_segv_setproctitle/" || echo ERROR; done > out.log 2>&1
3.

Actual results:
The segfault handler fails.


Expected results:
The segfault handler should print out the expected output.


Additional info:
The thing is, that the exception itself is expected [2], but the handler does not provide the correct output, since it crashes. This [3] is most likely the location where it crashes, which is probably Kernel or glibc method? Not sure


[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=22315390
[2] https://github.com/ruby/ruby/blob/ruby_2_4/test/ruby/test_rubyoptions.rb#L629
[3] https://github.com/ruby/ruby/blob/ruby_2_4/vm_dump.c#L684

Comment 1 Vít Ondruch 2017-10-11 15:38:24 UTC
BTW I reported this issue upstream, but it seem they cannot reproduce this issue on Debian. They appear to be using "uname_srvm: Linux 4.9.23-std-1 #1 SMP Mon Apr 24 13:18:14 UTC 2017 aarch64" and "glibc 2.23"

Comment 2 Florian Weimer 2017-10-11 16:00:03 UTC
Does Ruby call backtrace from a signal handler?  It does on x86-64:

#0  __GI___backtrace (array=0x7f9c5cd12660, size=1024) at ../sysdeps/x86_64/backtrace.c:96
#1  0x00007f9c5ca27715 in rb_print_backtrace () from /lib64/libruby.so.2.4
#2  0x00007f9c5ca2794c in rb_vm_bugreport () from /lib64/libruby.so.2.4
#3  0x00007f9c5c900984 in rb_bug_context () from /lib64/libruby.so.2.4
#4  0x00007f9c5c9bce0e in sigsegv () from /lib64/libruby.so.2.4
#5  <signal handler called>
#6  0x00007f9c5ca16861 in vm_exec_core () from /lib64/libruby.so.2.4
#7  0x00007f9c5ca1b058 in vm_exec () from /lib64/libruby.so.2.4
#8  0x00007f9c5ca1bc11 in eval_string_with_cref () from /lib64/libruby.so.2.4
#9  0x00007f9c5ca1c068 in rb_f_eval () from /lib64/libruby.so.2.4

That's not valid in its own right because backtrace can call malloc, and the SIGSEGV handler might be called from within malloc.

The backtrace issue we had on aarch64 has supposedly been fixed in rawhide:

https://sourceware.org/bugzilla/show_bug.cgi?id=21428

Comment 3 Vít Ondruch 2017-10-12 05:27:46 UTC
(In reply to Florian Weimer from comment #2)
> Does Ruby call backtrace from a signal handler?  It does on x86-64:

Well, I am not an expert, but I guess this is the idea. E.g. if anything goes wrong, including SIGSEGV, provide the backtrace and hence the backtrace is called.

> That's not valid in its own right because backtrace can call malloc, and the
> SIGSEGV handler might be called from within malloc.

But how to get the backtrace in the handler then?

Comment 4 Vít Ondruch 2017-10-19 06:25:08 UTC
This is the upsteram response:

~~~
[ruby-core:83231]
Aktualizováno uživatelem normalperson (Eric Wong) před 7 dnů
Citovat Upravit Odstranit

v.ondruch@tiscali.cz wrote:

    Can anybody help me we answer to glibc maintainers [1]?

     Florian Weimer 2017-10-11 18:00:03 CEST

    Does Ruby call backtrace from a signal handler?  It does on x86-64:

Yes, it calls backtrace() unfortunately. It uses special signal
handlers for SIGILL, SIGSEGV, SIGBUS which sometimes screw up
debugging.

When tracking down some bugs in the past; I've flipped the value
of ruby_enable_coredump in signal.c to disable the nanny sighandlers
and get a real core dump.

(there's a more official way which involves re-running
./configure and using RUBY_DEBUG env; but that's too slow for my
hardware and I'd rather just recompile signal.o)
~~~

Comment 5 Florian Weimer 2017-10-19 06:29:18 UTC
(In reply to Vít Ondruch from comment #4)
> This is the upsteram response:
> 
> ~~~
> [ruby-core:83231]
> Aktualizováno uživatelem normalperson (Eric Wong) před 7 dnů
> Citovat Upravit Odstranit
> 
> v.ondruch@tiscali.cz wrote:
> 
>     Can anybody help me we answer to glibc maintainers [1]?
> 
>      Florian Weimer 2017-10-11 18:00:03 CEST
> 
>     Does Ruby call backtrace from a signal handler?  It does on x86-64:
> 
> Yes, it calls backtrace() unfortunately. It uses special signal
> handlers for SIGILL, SIGSEGV, SIGBUS which sometimes screw up
> debugging.
[…]

Okay, this makes it more of a Ruby bug, unfortunately.

Comment 6 Fedora End Of Life 2018-02-20 15:22:36 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle.
Changing version to '28'.

Comment 7 Ben Cotton 2019-05-02 21:55:35 UTC
This message is a reminder that Fedora 28 is nearing its end of life.
On 2019-May-28 Fedora will stop maintaining and issuing updates for
Fedora 28. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora 'version' of '28'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 28 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 8 Ben Cotton 2019-08-13 16:59:53 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

Comment 9 Ben Cotton 2019-08-13 19:14:17 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

Comment 10 Ben Cotton 2020-08-11 13:03:49 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.


Note You need to log in before you can comment on or make changes to this bug.