Description of problem: Hi, Since the last update of rabbitmq-server, I'm unable to start it under Fedora 32. When I try to start the service I get the following error message: Job for rabbitmq-server.service failed because a fatal signal was delivered causing the control process to dump core. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details. Version-Release number of selected component (if applicable): rabbitmq-server-3.8.3-1.fc32.x86_64 Fedora 32 (5.6.11-300.fc32.x86_64) Actual results: Job for rabbitmq-server.service failed because a fatal signal was delivered causing the control process to dump core. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details. Expected results: Service to start. Additional info: Running "systemctl status rabbitmq-server.service" gives me this: rabbitmq-server.service - RabbitMQ broker Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; disabled; vendor preset: disabled) Active: failed (Result: core-dump) since Thu 2020-05-14 09:23:40 BST; 19s ago Main PID: 162117 (code=dumped, signal=ABRT) May 14 09:23:40 bluedream systemd[1]: Starting RabbitMQ broker... May 14 09:23:40 bluedream rabbitmq-server[162117]: *** stack smashing detected ***: terminated May 14 09:23:40 bluedream systemd[1]: rabbitmq-server.service: Main process exited, code=dumped, status=6/> May 14 09:23:40 bluedream systemd[1]: rabbitmq-server.service: Failed with result 'core-dump'. May 14 09:23:40 bluedream systemd[1]: Failed to start RabbitMQ broker. And "rabbitmq-diagnostics status" returns the following: DIAGNOSTICS =========== attempted to contact: [rabbit@bluedream] rabbit@bluedream: * connected to epmd (port 4369) on bluedream * epmd reports: node 'rabbit' not running at all no other nodes on bluedream * suggestion: start the node Current node details: * node name: 'rabbitmqcli-203239-rabbit@bluedream' * effective user's home directory: /var/lib/rabbitmq * Erlang cookie hash: Dbl4wVUfeG2jUVZAlIqZlQ== Your help is highly appreciated. Many thanks
It might be helpful to collect a core dump (using eg coredumpctl).
Created attachment 1688387 [details] output of coredumpctl dump Thanks for your quick response. I have attached the output of coredumpctl dump >> rabbitmq.dump for you.
This is also the output of coredumpctl debug: PID: 288957 (beam.smp) UID: 969 (rabbitmq) GID: 967 (rabbitmq) Signal: 6 (ABRT) Timestamp: Thu 2020-05-14 10:19:04 BST (1h 8min ago) Command Line: /usr/lib64/erlang/erts-10.7.1/bin/beam.smp -W w -A 128 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/ebin -noshell -noinput -s rabbit boot -sname rabbit@bluedream -boot start_sasl -conf /etc/rabbitmq/rabbitmq.conf -conf_dir /var/lib/rabbitmq/config -conf_script_dir /usr/lib/rabbitmq/bin -conf_schema_dir /var/lib/rabbitmq/schema -conf_advanced /etc/rabbitmq/advanced.config -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit lager_log_root "/var/log/rabbitmq" -rabbit lager_default_file "/var/log/rabbitmq/rabbit" -rabbit lager_upgrade_file "/var/log/rabbitmq/rabbit" -rabbit feature_flags_file "/var/lib/rabbitmq/mnesia/rabbit@bluedream-feature_flags" -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@bluedream-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@bluedream" -ra data_dir "/var/lib/rabbitmq/mnesia/rabbit@bluedream/quorum" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 -- Executable: /usr/lib64/erlang/erts-10.7.1/bin/beam.smp Control Group: /system.slice/rabbitmq-server.service Unit: rabbitmq-server.service Slice: system.slice Boot ID: 0ba53d052ccf4dba8472fd2d40dfc999 Machine ID: 54ea8123d3304bc2ac47ecdba366cd49 Hostname: bluedream Storage: /var/lib/systemd/coredump/core.beam\x2esmp.969.0ba53d052ccf4dba8472fd2d40dfc999.288957.1589447944000000000000.lz4 Message: Process 288957 (beam.smp) of user 969 dumped core. Stack trace of thread 288957: #0 0x00007fb0273d6a25 raise (libc.so.6 + 0x3ca25) #1 0x00007fb0273bf895 abort (libc.so.6 + 0x25895) #2 0x00007fb02741a8c7 __libc_message (libc.so.6 + 0x808c7) #3 0x00007fb0274ab7ea __fortify_fail (libc.so.6 + 0x1117ea) #4 0x00007fb0274ab7b6 __stack_chk_fail (libc.so.6 + 0x1117b6) #5 0x000055cdeb1a0867 erts_save_emu_args (beam.smp + 0x12a867) #6 0x000055cdeb159fbc erl_start (beam.smp + 0xe3fbc) #7 0x000055cdeb0e83ed main (beam.smp + 0x723ed) #8 0x00007fb0273c1042 __libc_start_main (libc.so.6 + 0x27042) #9 0x000055cdeb0e842e _start (beam.smp + 0x7242e) Failed to invoke gdb: No such file or directory
> Failed to invoke gdb: No such file or directory We'd get a lot more detail if: (a) gdb was installed (b) the debuginfo package for rabbitmq-server-3.8.3-1.fc32.x86_64 and whatever glibc you are using was installed When you install gdb and run it, it should tell you what dnf command you have to run to install the debuginfo for (b).
Impressed by your response time! Here are the requested info (output of coredumpctl debug): PID: 288957 (beam.smp) UID: 969 (rabbitmq) GID: 967 (rabbitmq) Signal: 6 (ABRT) Timestamp: Thu 2020-05-14 10:19:04 BST (1h 30min ago) Command Line: /usr/lib64/erlang/erts-10.7.1/bin/beam.smp -W w -A 128 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/ebin -noshell -noinput -s rabbit boot -sname rabbit@bluedream -boot start_sasl -conf /etc/rabbitmq/rabbitmq.conf -conf_dir /var/lib/rabbitmq/config -conf_script_dir /usr/lib/rabbitmq/bin -conf_schema_dir /var/lib/rabbitmq/schema -conf_advanced /etc/rabbitmq/advanced.config -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit lager_log_root "/var/log/rabbitmq" -rabbit lager_default_file "/var/log/rabbitmq/rabbit" -rabbit lager_upgrade_file "/var/log/rabbitmq/rabbit" -rabbit feature_flags_file "/var/lib/rabbitmq/mnesia/rabbit@bluedream-feature_flags" -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@bluedream-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@bluedream" -ra data_dir "/var/lib/rabbitmq/mnesia/rabbit@bluedream/quorum" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 -- Executable: /usr/lib64/erlang/erts-10.7.1/bin/beam.smp Control Group: /system.slice/rabbitmq-server.service Unit: rabbitmq-server.service Slice: system.slice Boot ID: 0ba53d052ccf4dba8472fd2d40dfc999 Machine ID: 54ea8123d3304bc2ac47ecdba366cd49 Hostname: bluedream Storage: /var/lib/systemd/coredump/core.beam\x2esmp.969.0ba53d052ccf4dba8472fd2d40dfc999.288957.1589447944000000000000.lz4 Message: Process 288957 (beam.smp) of user 969 dumped core. Stack trace of thread 288957: #0 0x00007fb0273d6a25 raise (libc.so.6 + 0x3ca25) #1 0x00007fb0273bf895 abort (libc.so.6 + 0x25895) #2 0x00007fb02741a8c7 __libc_message (libc.so.6 + 0x808c7) #3 0x00007fb0274ab7ea __fortify_fail (libc.so.6 + 0x1117ea) #4 0x00007fb0274ab7b6 __stack_chk_fail (libc.so.6 + 0x1117b6) #5 0x000055cdeb1a0867 erts_save_emu_args (beam.smp + 0x12a867) #6 0x000055cdeb159fbc erl_start (beam.smp + 0xe3fbc) #7 0x000055cdeb0e83ed main (beam.smp + 0x723ed) #8 0x00007fb0273c1042 __libc_start_main (libc.so.6 + 0x27042) #9 0x000055cdeb0e842e _start (beam.smp + 0x7242e) GNU gdb (GDB) Fedora 9.1-5.fc32 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/lib64/erlang/erts-10.7.1/bin/beam.smp... Reading symbols from /usr/lib/debug/usr/lib64/erlang/erts-10.7.1/bin/beam.smp-22.3.2-1.fc32.x86_64.debug... [New LWP 288957] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/lib64/erlang/erts-10.7.1/bin/beam.smp -W w -A 128 -MBas ageffcbf -MHas age'. Program terminated with signal SIGABRT, Aborted. #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 50 return ret;
It has to be said it's unlikely I'm going to be able to solve this. I'm just collecting enough info to decide what to do. At the gdb prompt can you get a full stack by typing: t a a bt and then copy and paste the entire output here (or attach it).
What you're doing is appreciated. This is the full stack output of gdb: Thread 1 (Thread 0x7fb027395740 (LWP 288957)): #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007fb0273bf895 in __GI_abort () at abort.c:79 #2 0x00007fb02741a8c7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fb02752bc14 "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:155 #3 0x00007fb0274ab7ea in __GI___fortify_fail (msg=msg@entry=0x7fb02752bbfc "stack smashing detected") at fortify_fail.c:26 #4 0x00007fb0274ab7b6 in __stack_chk_fail () at stack_chk_fail.c:24 #5 0x000055cdeb1a0867 in erts_save_emu_args (argc=<optimized out>, argv=argv@entry=0x7) at beam/utils.c:4400 #6 0x000055cdeb159fbc in early_init (argv=0x7, argc=0x7ffd3204f87c) at beam/erl_init.c:794 #7 erl_start (argc=<optimized out>, argv=0x7) at beam/erl_init.c:1266 #8 0x000055cdeb0e83ed in main (argc=<optimized out>, argv=<optimized out>) at sys/unix/erl_main.c:30
Somewhere in this function: https://github.com/erlang/otp/blob/6d5a5f31c36bbdaad21585d25974177bd1b75e66/erts/emulator/beam/utils.c#L4379 Also looking at the git blame for this function doesn't show any modifications to it in the last many years.
It's a function so whatever that calls it might have changed?
I am able to reproduce this bug locally.
Oh, that's great! Please do let me know if you need any more details and info from me and I'm more than happy to provide them to you. Many thanks :)
argc & argv are really bogus: (gdb) frame 6 #6 0x000055cc12f1efbc in early_init (argv=0x7, argc=0x7ffe29357f8c) at beam/erl_init.c:794 794 erts_save_emu_args(*argc, argv); (gdb) print *argc $26 = 108 (gdb) print argv $27 = (char **) 0x7 (Well, looking at the script that starts rabbitmq I can possibly believe argc, but argv must be wrong.)
Any idea what has triggered this? It was working like a charm yesterday. I think I received an update for rabbitmq-server yesterday evening and starting fresh today this happened. Do you think downgrading rabbitmq would help to narrow it down?
FYI full list of args passed to beam.smp: /usr/lib64/erlang/erts-10.7.1/bin/beam.smp -W w -A 128 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/ebin -noshell -noinput -s rabbit boot -sname rabbit@foo -boot start_sasl -conf /etc/rabbitmq/rabbitmq.conf -conf_dir /var/lib/rabbitmq/config -conf_script_dir /usr/lib/rabbitmq/bin -conf_schema_dir /var/lib/rabbitmq/schema -conf_advanced /etc/rabbitmq/advanced.config -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit lager_log_root "/var/log/rabbitmq" -rabbit lager_default_file "/var/log/rabbitmq/rabbit" -rabbit lager_upgrade_file "/var/log/rabbitmq/rabbit" -rabbit feature_flags_file "/var/lib/rabbitmq/mnesia/rabbit@foo-feature_flags" -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@foo-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@foo" -ra data_dir "/var/lib/rabbitmq/mnesia/rabbit@foo/quorum" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 --
gdb only reproducer is this very long command (on one line): $ gdb --args /usr/lib64/erlang/erts-10.7.1/bin/beam.smp -W w -A 128 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/ebin -noshell -noinput -s rabbit boot -sname rabbit@foo -boot start_sasl -conf /etc/rabbitmq/rabbitmq.conf -conf_dir /var/lib/rabbitmq/config -conf_script_dir /usr/lib/rabbitmq/bin -conf_schema_dir /var/lib/rabbitmq/schema -conf_advanced /etc/rabbitmq/advanced.config -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit lager_log_root "/var/log/rabbitmq" -rabbit lager_default_file "/var/log/rabbitmq/rabbit" -rabbit lager_upgrade_file "/var/log/rabbitmq/rabbit" -rabbit feature_flags_file "/var/lib/rabbitmq/mnesia/rabbit@foo-feature_flags" -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@foo-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@foo" -ra data_dir "/var/lib/rabbitmq/mnesia/rabbit@foo/quorum" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 -- However I do not know what's going on here.
Upstream bug: https://bugs.erlang.org/browse/ERL-1250
This turned out to be a GCC bug (already reported and resolved): https://bugs.erlang.org/browse/ERL-1233 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94734 Can you see if upgrading to a later version of erlang, eg. erlang-22.3.4-1.fc32, fixes this?
Just tried starting rabbitmq-server with erlang-22.3.4-1.fc32 and can confirm that it's working again. Brilliant. Thanks a lot!
Thanks for testing this. *** This bug has been marked as a duplicate of bug 1830137 ***
*** This bug has been marked as a duplicate of bug 1827357 ***