abrt version: 1.1.17 architecture: x86_64 Attached file: backtrace, 13103 bytes cmdline: /usr/bin/jackd -dalsa -dhw:0,0 -r48000 -p512 -n2 comment: It does not matter what command line arguments are given, jackd still crashes. component: jack-audio-connection-kit Attached file: coredump, 11710464 bytes crash_function: __libc_message executable: /usr/bin/jackd kernel: 2.6.38-0.rc8.git0.1.fc15.x86_64 package: jack-audio-connection-kit-1.9.6-3.fc15 rating: 4 reason: Process /usr/bin/jackd was killed by signal 6 (SIGABRT) release: Fedora release 15 (Lovelock) time: 1299828119 uid: 500 How to reproduce ----- 1. start jackd from command line with arbitrary sets of parameters.
Created attachment 483871 [details] File: backtrace
I am also experiencing this on 1.9.7 on x86_64 - would provide a backtrace but GDB errors at the moment as well. Without looking too much into it, I rebuilt it locally without firewire and freebob and it started okay
*** Bug 684530 has been marked as a duplicate of this bug. ***
*** Bug 689099 has been marked as a duplicate of this bug. ***
Created attachment 489780 [details] alsa-info Just for clarity - I don't have any firewire devices. Alsa-info attached
Hi Brendan, Did you configure your jack as described in /usr/share/doc/jack-audio-connection-kit-VERSION/README.Fedora ?
Created attachment 489852 [details] jackd backtrace
I did: fedora15:~$ ps -ef|grep pulse bsjones 7646 3709 0 06:27 pts/0 00:00:00 grep --color=auto pulse fedora15:~$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 31037 max locked memory (kbytes, -l) 4194304 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 20 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
hmm, how about the jackuser group? Are you a member of it? Also, did you try uninstalling pulseaudio? Do you get the same crashes without pulseaudio?
oops, I realized that what I said doesn't make sense. You already answered these questions. At this point, since I am not a jack developer, the best thing I can suggest is to file a bug to jack2 http://trac.jackaudio.org/ Please post here the link if you file a ticket, so we can track the progress and update/patch our packages if necessary.
No problem. However, the problem looks to be occurring in libffado . I'll do some more debugging before filing upstream.
I've reported this upstream here: http://subversion.ffado.org/ticket/329 A brute force workaround is to move /usr/lib64/jack/jack_firewire.so so it won't be loaded at runtime. Obviously this will disable firewire support
I experience this bug on a 32 bit system. I might have a special case as I have disabled firewire in my bios (I never use it). Brendan's fix works for me as well. Simply deleting /usr/lib/jack/jack_firewire.so solves it.
*** Bug 680802 has been marked as a duplicate of this bug. ***
One of the upstream libffado developers believes there maybe an issue with binutils, specifically in the final linking stage. The full text of the comment can be found here: http://subversion.ffado.org/ticket/329#comment:17 I am reassigning to binutils for comment..
Hi Brendan, This certainly sounds like a linker problem, although I guess that there is a small possibility that it is actually a loader problem. Without more information as to exactly what is going wrong it is hard to say any more. But here are a few things that you could try: * If you build a linker from the current HEAD of the mainline FSF binutils sources and then link using that, does the problem persist ? * What about if you use the GOLD linker rather than LD ? (Configure with --enable-gold to get the gold linker). * How do the working libffado and the broken libffado differ ? (Running readelf -a on the two binaries is probably the best way to compare them, although that will produce a lot of output. Using readelf -e will produce a much shorter output). How does the broken version of libffado fail ? Is it another memory corruption problem or something else ? Cheers Nick
Created attachment 492237 [details] diff <(readelf -e libffado-debian-ld.so) <(readelf -e libffado-fedora-ld.so) Upstream has attached the two libraries for comparison to this ticket: http://subversion.ffado.org/ticket/329#comment:19 The error is free() invalid pointer upon delete I also pulled binutils-2.21.51.0.8-2 from koji and tried building against that with the same result. I will try your other recommendations.
If I symlink ld to ld.gold in my path and rebuild as perform the final link step I get a working libffado.so This was done using the from the libffado revision in F15 (patched as per spec), not the upstream trunk. rpmbuild -ba ~/rpmbuild/SPECS/libffado.spec mkdir /tmp/bin ln -s /usr/bin/ld.gold /tmp/bin/ld export PATH=/tmp/bin:$PATH cd ~/rpmbuild/BUILD/libffado-2.1.0 g++ -o src/libffado.so -pthread -Wl,-soname=libffado.so.2 -shared src/devicemanager.os src/ffado.os src/ffadodevice.os src/debugmodule/debugmodule.os src/DeviceStringParser.os src/libieee1394/ARMHandler.os src/libieee1394/configrom.os src/libieee1394/csr1212.os src/libieee1394/CycleTimerHelper.os src/libieee1394/ieee1394service.os src/libieee1394/IEC61883.os src/libieee1394/IsoHandlerManager.os src/libstreaming/StreamProcessorManager.os src/libstreaming/util/cip.os src/libstreaming/generic/StreamProcessor.os src/libstreaming/generic/Port.os src/libstreaming/generic/PortManager.os src/libutil/cmd_serialize.os src/libutil/DelayLockedLoop.os src/libutil/IpcRingBuffer.os src/libutil/PacketBuffer.os src/libutil/Configuration.os src/libutil/OptionContainer.os src/libutil/PosixMessageQueue.os src/libutil/PosixSharedMemory.os src/libutil/PosixMutex.os src/libutil/PosixThread.os src/libutil/ringbuffer.os src/libutil/StreamStatistics.os src/libutil/SystemTimeSource.os src/libutil/TimestampedBuffer.os src/libutil/Watchdog.os src/libcontrol/Element.os src/libcontrol/BasicElements.os src/libcontrol/MatrixMixer.os src/libcontrol/CrossbarRouter.os src/libcontrol/ClockSelect.os src/libcontrol/Nickname.os src/libutil/serialize_libxml.os src/bebob/bebob_avdevice.os src/bebob/bebob_avdevice_subunit.os src/bebob/bebob_avplug.os src/bebob/bebob_dl_bcd.os src/bebob/bebob_dl_codes.os src/bebob/bebob_dl_mgr.os src/bebob/bebob_functionblock.os src/bebob/bebob_mixer.os src/bebob/focusrite/focusrite_generic.os src/bebob/focusrite/focusrite_saffire.os src/bebob/focusrite/focusrite_saffirepro.os src/bebob/focusrite/focusrite_cmd.os src/bebob/terratec/terratec_device.os src/bebob/terratec/terratec_cmd.os src/bebob/edirol/edirol_fa101.os src/bebob/edirol/edirol_fa66.os src/bebob/esi/quatafire610.os src/bebob/mackie/onyxmixer.os src/fireworks/fireworks_device.os src/fireworks/fireworks_control.os src/fireworks/fireworks_firmware.os src/fireworks/efc/efc_avc_cmd.os src/fireworks/efc/efc_cmd.os src/fireworks/efc/efc_cmds_hardware.os src/fireworks/efc/efc_cmds_hardware_ctrl.os src/fireworks/efc/efc_cmds_flash.os src/fireworks/efc/efc_cmds_mixer.os src/fireworks/efc/efc_cmds_monitor.os src/fireworks/efc/efc_cmds_ioconfig.os src/fireworks/fireworks_session_block.os src/fireworks/audiofire/audiofire_device.os src/oxford/oxford_device.os src/libstreaming/amdtp-oxford/AmdtpOxfordReceiveStreamProcessor.os src/motu/motu_avdevice.os src/motu/motu_controls.os src/motu/motu_mark3_controls.os src/motu/motu_mixerdefs.os src/motu/motu_mark3_mixerdefs.os src/motu/motu_mixer.os src/libstreaming/motu/MotuPort.os src/libstreaming/motu/MotuPortInfo.os src/libstreaming/motu/MotuReceiveStreamProcessor.os src/libstreaming/motu/MotuTransmitStreamProcessor.os src/dice/dice_avdevice.os src/dice/dice_eap.os src/dice/focusrite/focusrite_eap.os src/dice/focusrite/saffire_pro40.os src/dice/focusrite/saffire_pro24.os src/libavc/streamformat/avc_extended_stream_format.os src/libavc/musicsubunit/avc_descriptor_music.os src/libavc/musicsubunit/avc_musicsubunit.os src/libavc/audiosubunit/avc_audiosubunit.os src/libavc/audiosubunit/avc_descriptor_audio.os src/libavc/audiosubunit/avc_function_block.os src/libavc/descriptors/avc_descriptor_cmd.os src/libavc/descriptors/avc_descriptor.os src/libavc/general/avc_extended_subunit_info.os src/libavc/general/avc_unit_info.os src/libavc/general/avc_generic.os src/libavc/general/avc_subunit_info.os src/libavc/general/avc_connect.os src/libavc/general/avc_signal_format.os src/libavc/general/avc_extended_cmd_generic.os src/libavc/general/avc_extended_plug_info.os src/libavc/general/avc_plug_info.os src/libavc/general/avc_unit.os src/libavc/general/avc_subunit.os src/libavc/general/avc_plug.os src/libavc/general/avc_vendor_dependent_cmd.os src/libavc/avc_definitions.os src/libavc/ccm/avc_signal_source.os src/libstreaming/amdtp/AmdtpPort.os src/libstreaming/amdtp/AmdtpPortInfo.os src/libstreaming/amdtp/AmdtpReceiveStreamProcessor.os src/libstreaming/amdtp/AmdtpTransmitStreamProcessor.os src/genericavc/avc_avdevice.os src/genericavc/stanton/scs.os -lm -lpthread -liec61883 -lraw1394 -lconfig++ -lxml++-2.6 -lxml2 -lglibmm-2.4 -lgobject-2.0 -lsigc-2.0 -lgthread-2.0 -lrt -lglib-2.0 su -c 'cp src/libffado.so /usr/lib64/libffado.so.2.999.0' gdb jackd run -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
Quoting cladish: "On further thought, the problem is probably not that the constructors aren't executed, but that their execution order changes: GCC http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46770 "
Created attachment 492382 [details] Disable initfini_array support.
Hi Brendan, It looks like a problem with the linker's support for .init_array and .fini_array sections. Are you able to build and install a local binutils rpm ? If so, please could you try downloading the binutils source rpm, applying the uploaded patch to the binutils.spec file and then creating a test linker with which you can link jackd. if this works then it confirms the cause the problem, but not the actual nitty gritty of what is wrong. For that I will need to look at the libffado.so files built by the Debian linker and the (unpatched) Fedora linker. Cheers Nick
No problem. I rebuilt binutils based on the spec in F15 with disabled initfini-array flag. Your patch was done on the version in rawhide I think. Linking with this version's ld was successful. The binary is here http://bsjones.fedorapeople.org/libffado.so.2.999.0-binutils-disable-initfini-array.tar.gz And the rebuilt binutils http://bsjones.fedorapeople.org/binutils-2.21.51.0.6-2.fc15.x86_64.rpm http://bsjones.fedorapeople.org/binutils-2.21.51.0.6-2.fc15.src.rpm http://bsjones.fedorapeople.org/binutils-debuginfo-2.21.51.0.6-2.fc15.x86_64.rpm
Hi Brendan, > The binary is here Thanks for that. Can you also make a broken binary for me, so that I can compare the two and find out exactly which constructors/destructors are being called in the wrong order. Cheers Nick
http://bsjones.fedorapeople.org/libffado.so.2.999.0-binutils-enable-initfini-array.tar.gz Using binutils 2.21.51.0.6-1.fc15
Hi Brendan, Thanks for the binaries. Unfortunately I am baffled. The ordering of the constructor functions in the .init_array section of the enabled binary is exactly the same as the order in the .ctors section of the disabled binary. Plus there are no extra constructors in either binary. So in theory they should exhibit exactly the same behaviour. (There are no destructors in either binary). > The error is free() invalid pointer upon delete Can you find out any more about this error please ? For example, which function is executing the free() and from where should the corresponding malloc() have been called ? I am beginning to wonder if this is a loader problem rather than a linker problem. Maybe the loader is not executing the .init_array section properly. (This seems rather unlikely as surely lots of binaries would break if this were true). Cheers Nick
Created attachment 492846 [details] debugmodule.cpp
Created attachment 492848 [details] debugmodule.h
Created attachment 492849 [details] ffado.cpp
The error occurs when DebugModuleManager::~DebugModuleManager is called from ffado.cpp . It contains a vector of DebugModules which are iterated through - a delete on the dereferenced vector produces the crash. It is pretty easy to reproduce - install jack-audio-connection-kit, it will pull in libffado. Calling /usr/bin/jackd without any parameters will reproduce the crash. The upstream thread may provide some clues. The binaries the maintained used to debug this problem are there also.
And here's the relevant part of the bt: (gdb) run Starting program: /usr/bin/jackd [Thread debugging using libthread_db enabled] jackdmp 1.9.7 Copyright 2001-2005 Paul Davis and others. Copyright 2004-2011 Grame. jackdmp comes with ABSOLUTELY NO WARRANTY This is free software, and you are welcome to redistribute it under certain conditions; see the file COPYING for details [New Thread 0x7ffff6bdd700 (LWP 7152)] Cleaning up leftover debug module: DeviceManager <snip> Program received signal SIGABRT, Aborted. 0x0000003dea2362c5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); (gdb) backtrace #0 0x0000003dea2362c5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x0000003dea237bdb in abort () at abort.c:92 #2 0x0000003dea2722c3 in __libc_message (do_abort=2, fmt=0x3dea35ca28 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:186 #3 0x0000003dea2787ba in malloc_printerr (action=3, str=0x3dea359a5b "free(): invalid pointer", ptr=<optimized out>) at malloc.c:6283 #4 0x00007ffff77e358e in DebugModuleManager::~DebugModuleManager (this=0x7ffff6bde010, __in_chrg=<optimized out>) at src/debugmodule/debugmodule.cpp:272 #5 0x00007ffff77d7076 in exitfunc () at src/ffado.cpp:61 #6 0x0000003de9a13b68 in _dl_close_worker (map=<optimized out>) at dl-close.c:266 #7 0x0000003de9a1462e in _dl_close (_map=0x60ff00) at dl-close.c:754 #8 0x0000003de9a0e8a6 in _dl_catch_error (objname=0x60fee0, errstring=0x60fee8, mallocedp=0x60fed8, operate=0x3deaa00fe0 <dlclose_doit>, args=0x60ff00) at dl-error.c:178 #9 0x0000003deaa0152f in _dlerror_run (operate=0x3deaa00fe0 <dlclose_doit>, args=0x60ff00) at dlerror.c:164 #10 0x0000003deaa0100f in __dlclose (handle=0x1bed) at dlclose.c:48 #11 0x00007ffff7dadf23 in jack_get_descriptor (drivers=0x0, sofile=0x608073 "jack_firewire.so", symbol=<optimized out>) at ../common/JackDriverLoader.cpp:462 #12 0x00007ffff7daefae in jack_drivers_load (drivers=0x0) at ../common/JackDriverLoader.cpp:621 #13 0x00007ffff7db2aca in jackctl_drivers_load (server_ptr=0x606470) at ../common/JackControlAPI.cpp:293 #14 jackctl_server_create (on_device_acquire=0x403a70 <audio_acquire(char const*)>, on_device_release=0x403b80 <audio_release(char const*)>) at ../common/JackControlAPI.cpp:801 #15 0x000000000040247d in main (argc=1, argv=0x7fffffffe118) at ../common/Jackdmp.cpp:253
Hi Brendan, I am still struggling^H^H^H^H^H investigating this. One quick question - in DebugModuleManager::~DebugModuleManager() wouldn't it make more sense to walk the m_debugModules vector backwards ? Ie treating the vector as a stack rather than a queue ? (I assume that you will tell me that the modules are meant to be independent of each other and so the order of their deletion is irrelevant, but can this be guaranteed ?) As an aside I tried compiling the ctor_test.cpp app from: http://subversion.ffado.org/ticket/329#comment:19 and it works with and without using init_array sections. Cheers Nick
Thanks for looking into this Nick. Some of these questions may be better directed to the developers upstream (I'm not involved in the ffado project). In the upstream comments a number of different methods of dealing with the vector were tried. Given that this is pretty important to the audio SIG wrt F15, do you have any suggestions for a possible work around (e.g. patching the libffado spec to use the goldlinker / or simply removing the call to ~DebugModuleManager?) Let me know if there is anything I can do. Brendan
Hi Brendan, Using GOLD is definitely the easiest and best workaround for now. (It also has the advantage that GOLD should be faster than LD). I'll continue to look into this in the background, but at the moment I am stumped as to why putting the constructors into the .init_array section is breaking libffado. Cheers Nick
*** Bug 698903 has been marked as a duplicate of this bug. ***
Package: jack-audio-connection-kit-1.9.7-1.fc15 Architecture: x86_64 OS Release: Fedora release 15 (Lovelock) Comment ----- Starting ardour
(In reply to comment #25) > Hi Brendan, > > Thanks for the binaries. Unfortunately I am baffled. The ordering of the > constructor functions in the .init_array section of the enabled binary is > exactly the same as the order in the .ctors section of the disabled binary. > Plus there are no extra constructors in either binary. So in theory they > should exhibit exactly the same behaviour. (There are no destructors in either > binary). Hmm, not that I really know anything about the subject, but http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46770#c1 suggests that .init_array and .ctors are processed in opposite order: "Using .init_array/.fini_array instead of .ctors/.dtors removes the need for the associated (relative) relocations, and avoids the backwards disk seeks on startup (since while .ctors are processed backwards, .init_array is processed forward)" So if the ordering is the same, but they are processed differently, this would explain differences, wouldn't it?
libffado-2.1.0-0.3.20110426.svn1983.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/libffado-2.1.0-0.3.20110426.svn1983.fc15
Package libffado-2.1.0-0.3.20110426.svn1983.fc15: * should fix your issue, * was pushed to the Fedora 15 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing libffado-2.1.0-0.3.20110426.svn1983.fc15' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/libffado-2.1.0-0.3.20110426.svn1983.fc15 then log in and leave karma (feedback).
libffado-2.1.0-0.3.20110426.svn1983.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.