Red Hat Bugzilla – Bug 684392
[abrt] jack-audio-connection-kit-1.9.6-3.fc15: __libc_message: Process /usr/bin/jackd was killed by signal 6 (SIGABRT)
Last modified: 2011-05-03 00:52:24 EDT
abrt version: 1.1.17
Attached file: backtrace, 13103 bytes
cmdline: /usr/bin/jackd -dalsa -dhw:0,0 -r48000 -p512 -n2
comment: It does not matter what command line arguments are given, jackd still crashes.
Attached file: coredump, 11710464 bytes
reason: Process /usr/bin/jackd was killed by signal 6 (SIGABRT)
release: Fedora release 15 (Lovelock)
How to reproduce
1. start jackd from command line with arbitrary sets of parameters.
Created attachment 483871 [details]
I am also experiencing this on 1.9.7 on x86_64 - would provide a backtrace but GDB errors at the moment as well.
Without looking too much into it, I rebuilt it locally without firewire and freebob and it started okay
*** Bug 684530 has been marked as a duplicate of this bug. ***
*** Bug 689099 has been marked as a duplicate of this bug. ***
Created attachment 489780 [details]
Just for clarity - I don't have any firewire devices. Alsa-info attached
Did you configure your jack as described in /usr/share/doc/jack-audio-connection-kit-VERSION/README.Fedora ?
Created attachment 489852 [details]
fedora15:~$ ps -ef|grep pulse
bsjones 7646 3709 0 06:27 pts/0 00:00:00 grep --color=auto pulse
fedora15:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31037
max locked memory (kbytes, -l) 4194304
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 20
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
hmm, how about the jackuser group? Are you a member of it?
Also, did you try uninstalling pulseaudio? Do you get the same crashes without pulseaudio?
oops, I realized that what I said doesn't make sense. You already answered these questions. At this point, since I am not a jack developer, the best thing I can suggest is to file a bug to jack2
Please post here the link if you file a ticket, so we can track the progress and update/patch our packages if necessary.
However, the problem looks to be occurring in libffado . I'll do some more debugging before filing upstream.
I've reported this upstream here: http://subversion.ffado.org/ticket/329
A brute force workaround is to move /usr/lib64/jack/jack_firewire.so so it won't be loaded at runtime. Obviously this will disable firewire support
I experience this bug on a 32 bit system.
I might have a special case as I have disabled firewire in my bios (I never use it).
Brendan's fix works for me as well.
Simply deleting /usr/lib/jack/jack_firewire.so
*** Bug 680802 has been marked as a duplicate of this bug. ***
One of the upstream libffado developers believes there maybe an issue with binutils, specifically in the final linking stage.
The full text of the comment can be found here: http://subversion.ffado.org/ticket/329#comment:17
I am reassigning to binutils for comment..
This certainly sounds like a linker problem, although I guess that there is a small possibility that it is actually a loader problem. Without more information as to exactly what is going wrong it is hard to say any more. But here are a few things that you could try:
* If you build a linker from the current HEAD of the mainline FSF binutils
sources and then link using that, does the problem persist ?
* What about if you use the GOLD linker rather than LD ?
(Configure with --enable-gold to get the gold linker).
* How do the working libffado and the broken libffado differ ?
(Running readelf -a on the two binaries is probably the best way to
compare them, although that will produce a lot of output. Using
readelf -e will produce a much shorter output).
How does the broken version of libffado fail ? Is it another memory corruption problem or something else ?
Created attachment 492237 [details]
diff <(readelf -e libffado-debian-ld.so) <(readelf -e libffado-fedora-ld.so)
Upstream has attached the two libraries for comparison to this ticket: http://subversion.ffado.org/ticket/329#comment:19
The error is free() invalid pointer upon delete
I also pulled binutils-184.108.40.206.8-2 from koji and tried building against that with the same result. I will try your other recommendations.
If I symlink ld to ld.gold in my path and rebuild as perform the final link step I get a working libffado.so
This was done using the from the libffado revision in F15 (patched as per spec), not the upstream trunk.
rpmbuild -ba ~/rpmbuild/SPECS/libffado.spec
ln -s /usr/bin/ld.gold /tmp/bin/ld
g++ -o src/libffado.so -pthread -Wl,-soname=libffado.so.2 -shared src/devicemanager.os src/ffado.os src/ffadodevice.os src/debugmodule/debugmodule.os src/DeviceStringParser.os src/libieee1394/ARMHandler.os src/libieee1394/configrom.os src/libieee1394/csr1212.os src/libieee1394/CycleTimerHelper.os src/libieee1394/ieee1394service.os src/libieee1394/IEC61883.os src/libieee1394/IsoHandlerManager.os src/libstreaming/StreamProcessorManager.os src/libstreaming/util/cip.os src/libstreaming/generic/StreamProcessor.os src/libstreaming/generic/Port.os src/libstreaming/generic/PortManager.os src/libutil/cmd_serialize.os src/libutil/DelayLockedLoop.os src/libutil/IpcRingBuffer.os src/libutil/PacketBuffer.os src/libutil/Configuration.os src/libutil/OptionContainer.os src/libutil/PosixMessageQueue.os src/libutil/PosixSharedMemory.os src/libutil/PosixMutex.os src/libutil/PosixThread.os src/libutil/ringbuffer.os src/libutil/StreamStatistics.os src/libutil/SystemTimeSource.os src/libutil/TimestampedBuffer.os src/libutil/Watchdog.os src/libcontrol/Element.os src/libcontrol/BasicElements.os src/libcontrol/MatrixMixer.os src/libcontrol/CrossbarRouter.os src/libcontrol/ClockSelect.os src/libcontrol/Nickname.os src/libutil/serialize_libxml.os src/bebob/bebob_avdevice.os src/bebob/bebob_avdevice_subunit.os src/bebob/bebob_avplug.os src/bebob/bebob_dl_bcd.os src/bebob/bebob_dl_codes.os src/bebob/bebob_dl_mgr.os src/bebob/bebob_functionblock.os src/bebob/bebob_mixer.os src/bebob/focusrite/focusrite_generic.os src/bebob/focusrite/focusrite_saffire.os src/bebob/focusrite/focusrite_saffirepro.os src/bebob/focusrite/focusrite_cmd.os src/bebob/terratec/terratec_device.os src/bebob/terratec/terratec_cmd.os src/bebob/edirol/edirol_fa101.os src/bebob/edirol/edirol_fa66.os src/bebob/esi/quatafire610.os src/bebob/mackie/onyxmixer.os src/fireworks/fireworks_device.os src/fireworks/fireworks_control.os src/fireworks/fireworks_firmware.os src/fireworks/efc/efc_avc_cmd.os src/fireworks/efc/efc_cmd.os src/fireworks/efc/efc_cmds_hardware.os src/fireworks/efc/efc_cmds_hardware_ctrl.os src/fireworks/efc/efc_cmds_flash.os src/fireworks/efc/efc_cmds_mixer.os src/fireworks/efc/efc_cmds_monitor.os src/fireworks/efc/efc_cmds_ioconfig.os src/fireworks/fireworks_session_block.os src/fireworks/audiofire/audiofire_device.os src/oxford/oxford_device.os src/libstreaming/amdtp-oxford/AmdtpOxfordReceiveStreamProcessor.os src/motu/motu_avdevice.os src/motu/motu_controls.os src/motu/motu_mark3_controls.os src/motu/motu_mixerdefs.os src/motu/motu_mark3_mixerdefs.os src/motu/motu_mixer.os src/libstreaming/motu/MotuPort.os src/libstreaming/motu/MotuPortInfo.os src/libstreaming/motu/MotuReceiveStreamProcessor.os src/libstreaming/motu/MotuTransmitStreamProcessor.os src/dice/dice_avdevice.os src/dice/dice_eap.os src/dice/focusrite/focusrite_eap.os src/dice/focusrite/saffire_pro40.os src/dice/focusrite/saffire_pro24.os src/libavc/streamformat/avc_extended_stream_format.os src/libavc/musicsubunit/avc_descriptor_music.os src/libavc/musicsubunit/avc_musicsubunit.os src/libavc/audiosubunit/avc_audiosubunit.os src/libavc/audiosubunit/avc_descriptor_audio.os src/libavc/audiosubunit/avc_function_block.os src/libavc/descriptors/avc_descriptor_cmd.os src/libavc/descriptors/avc_descriptor.os src/libavc/general/avc_extended_subunit_info.os src/libavc/general/avc_unit_info.os src/libavc/general/avc_generic.os src/libavc/general/avc_subunit_info.os src/libavc/general/avc_connect.os src/libavc/general/avc_signal_format.os src/libavc/general/avc_extended_cmd_generic.os src/libavc/general/avc_extended_plug_info.os src/libavc/general/avc_plug_info.os src/libavc/general/avc_unit.os src/libavc/general/avc_subunit.os src/libavc/general/avc_plug.os src/libavc/general/avc_vendor_dependent_cmd.os src/libavc/avc_definitions.os src/libavc/ccm/avc_signal_source.os src/libstreaming/amdtp/AmdtpPort.os src/libstreaming/amdtp/AmdtpPortInfo.os src/libstreaming/amdtp/AmdtpReceiveStreamProcessor.os src/libstreaming/amdtp/AmdtpTransmitStreamProcessor.os src/genericavc/avc_avdevice.os src/genericavc/stanton/scs.os -lm -lpthread -liec61883 -lraw1394 -lconfig++ -lxml++-2.6 -lxml2 -lglibmm-2.4 -lgobject-2.0 -lsigc-2.0 -lgthread-2.0 -lrt -lglib-2.0
su -c 'cp src/libffado.so /usr/lib64/libffado.so.2.999.0'
Fedora Bugzappers volunteer triage team
"On further thought, the problem is probably not that the constructors aren't executed, but that their execution order changes: GCC http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46770 "
Created attachment 492382 [details]
Disable initfini_array support.
It looks like a problem with the linker's support for .init_array and .fini_array sections. Are you able to build and install a local binutils rpm ? If so, please could you try downloading the binutils source rpm, applying the uploaded patch to the binutils.spec file and then creating a test linker with which you can link jackd. if this works then it confirms the cause the problem, but not the actual nitty gritty of what is wrong. For that I will need to look at the libffado.so files built by the Debian linker and the (unpatched) Fedora linker.
No problem. I rebuilt binutils based on the spec in F15 with disabled initfini-array flag. Your patch was done on the version in rawhide I think.
Linking with this version's ld was successful. The binary is here http://bsjones.fedorapeople.org/libffado.so.2.999.0-binutils-disable-initfini-array.tar.gz
And the rebuilt binutils http://bsjones.fedorapeople.org/binutils-220.127.116.11.6-2.fc15.x86_64.rpm http://bsjones.fedorapeople.org/binutils-18.104.22.168.6-2.fc15.src.rpm http://bsjones.fedorapeople.org/binutils-debuginfo-22.214.171.124.6-2.fc15.x86_64.rpm
> The binary is here
Thanks for that. Can you also make a broken binary for me, so that I can compare the two and find out exactly which constructors/destructors are being called in the wrong order.
Using binutils 126.96.36.199.6-1.fc15
Thanks for the binaries. Unfortunately I am baffled. The ordering of the constructor functions in the .init_array section of the enabled binary is exactly the same as the order in the .ctors section of the disabled binary. Plus there are no extra constructors in either binary. So in theory they should exhibit exactly the same behaviour. (There are no destructors in either binary).
> The error is free() invalid pointer upon delete
Can you find out any more about this error please ? For example, which function is executing the free() and from where should the corresponding malloc() have been called ?
I am beginning to wonder if this is a loader problem rather than a linker problem. Maybe the loader is not executing the .init_array section properly. (This seems rather unlikely as surely lots of binaries would break if this were true).
Created attachment 492846 [details]
Created attachment 492848 [details]
Created attachment 492849 [details]
The error occurs when DebugModuleManager::~DebugModuleManager is called from ffado.cpp . It contains a vector of DebugModules which are iterated through - a delete on the dereferenced vector produces the crash.
It is pretty easy to reproduce - install jack-audio-connection-kit, it will pull in libffado. Calling /usr/bin/jackd without any parameters will reproduce the crash.
The upstream thread may provide some clues. The binaries the maintained used to debug this problem are there also.
And here's the relevant part of the bt:
Starting program: /usr/bin/jackd
[Thread debugging using libthread_db enabled]
Copyright 2001-2005 Paul Davis and others.
Copyright 2004-2011 Grame.
jackdmp comes with ABSOLUTELY NO WARRANTY
This is free software, and you are welcome to redistribute it
under certain conditions; see the file COPYING for details
[New Thread 0x7ffff6bdd700 (LWP 7152)]
Cleaning up leftover debug module: DeviceManager
Program received signal SIGABRT, Aborted.
0x0000003dea2362c5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
#0 0x0000003dea2362c5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x0000003dea237bdb in abort () at abort.c:92
#2 0x0000003dea2722c3 in __libc_message (do_abort=2, fmt=0x3dea35ca28 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:186
#3 0x0000003dea2787ba in malloc_printerr (action=3, str=0x3dea359a5b "free(): invalid pointer", ptr=<optimized out>) at malloc.c:6283
#4 0x00007ffff77e358e in DebugModuleManager::~DebugModuleManager (this=0x7ffff6bde010, __in_chrg=<optimized out>) at src/debugmodule/debugmodule.cpp:272
#5 0x00007ffff77d7076 in exitfunc () at src/ffado.cpp:61
#6 0x0000003de9a13b68 in _dl_close_worker (map=<optimized out>) at dl-close.c:266
#7 0x0000003de9a1462e in _dl_close (_map=0x60ff00) at dl-close.c:754
#8 0x0000003de9a0e8a6 in _dl_catch_error (objname=0x60fee0, errstring=0x60fee8, mallocedp=0x60fed8, operate=0x3deaa00fe0 <dlclose_doit>, args=0x60ff00) at dl-error.c:178
#9 0x0000003deaa0152f in _dlerror_run (operate=0x3deaa00fe0 <dlclose_doit>, args=0x60ff00) at dlerror.c:164
#10 0x0000003deaa0100f in __dlclose (handle=0x1bed) at dlclose.c:48
#11 0x00007ffff7dadf23 in jack_get_descriptor (drivers=0x0, sofile=0x608073 "jack_firewire.so", symbol=<optimized out>) at ../common/JackDriverLoader.cpp:462
#12 0x00007ffff7daefae in jack_drivers_load (drivers=0x0) at ../common/JackDriverLoader.cpp:621
#13 0x00007ffff7db2aca in jackctl_drivers_load (server_ptr=0x606470) at ../common/JackControlAPI.cpp:293
#14 jackctl_server_create (on_device_acquire=0x403a70 <audio_acquire(char const*)>, on_device_release=0x403b80 <audio_release(char const*)>) at ../common/JackControlAPI.cpp:801
#15 0x000000000040247d in main (argc=1, argv=0x7fffffffe118) at ../common/Jackdmp.cpp:253
I am still struggling^H^H^H^H^H investigating this.
One quick question - in DebugModuleManager::~DebugModuleManager() wouldn't it make more sense to walk the m_debugModules vector backwards ? Ie treating the vector as a stack rather than a queue ? (I assume that you will tell me that the modules are meant to be independent of each other and so the order of their deletion is irrelevant, but can this be guaranteed ?)
As an aside I tried compiling the ctor_test.cpp app from:
and it works with and without using init_array sections.
Thanks for looking into this Nick. Some of these questions may be better directed to the developers upstream (I'm not involved in the ffado project). In the upstream comments a number of different methods of dealing with the vector were tried.
Given that this is pretty important to the audio SIG wrt F15, do you have any suggestions for a possible work around (e.g. patching the libffado spec to use the goldlinker / or simply removing the call to ~DebugModuleManager?)
Let me know if there is anything I can do.
Using GOLD is definitely the easiest and best workaround for now. (It also has the advantage that GOLD should be faster than LD).
I'll continue to look into this in the background, but at the moment I am stumped as to why putting the constructors into the .init_array section is breaking libffado.
*** Bug 698903 has been marked as a duplicate of this bug. ***
OS Release: Fedora release 15 (Lovelock)
(In reply to comment #25)
> Hi Brendan,
> Thanks for the binaries. Unfortunately I am baffled. The ordering of the
> constructor functions in the .init_array section of the enabled binary is
> exactly the same as the order in the .ctors section of the disabled binary.
> Plus there are no extra constructors in either binary. So in theory they
> should exhibit exactly the same behaviour. (There are no destructors in either
Hmm, not that I really know anything about the subject, but http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46770#c1 suggests that .init_array and .ctors are processed in opposite order:
"Using .init_array/.fini_array instead of .ctors/.dtors removes the need for the
associated (relative) relocations, and avoids the backwards disk seeks on
startup (since while .ctors are processed backwards, .init_array is processed
So if the ordering is the same, but they are processed differently, this would explain differences, wouldn't it?
libffado-2.1.0-0.3.20110426.svn1983.fc15 has been submitted as an update for Fedora 15.
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libffado-2.1.0-0.3.20110426.svn1983.fc15'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
libffado-2.1.0-0.3.20110426.svn1983.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.