Bug 684392

Summary: [abrt] jack-audio-connection-kit-1.9.6-3.fc15: __libc_message: Process /usr/bin/jackd was killed by signal 6 (SIGABRT)
Product: [Fedora] Fedora Reporter: Albert Cervin <albert>
Component: binutilsAssignee: Nick Clifton <nickc>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 15CC: brendan.jones.it, green, jakub, mariolinux, nickc, nphilipp, oget.fedora, patsev.anton, vonbrand
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: abrt_hash:cf249db0b3004e0df137c2e242eca5eef586e8ca
Fixed In Version: libffado-2.1.0-0.3.20110426.svn1983.fc15 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-05-03 00:52:24 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
File: backtrace
none
alsa-info
none
jackd backtrace
none
diff <(readelf -e libffado-debian-ld.so) <(readelf -e libffado-fedora-ld.so)
none
Disable initfini_array support.
none
debugmodule.cpp
none
debugmodule.h
none
ffado.cpp none

Description Albert Cervin 2011-03-12 03:15:05 EST
abrt version: 1.1.17
architecture: x86_64
Attached file: backtrace, 13103 bytes
cmdline: /usr/bin/jackd -dalsa -dhw:0,0 -r48000 -p512 -n2
comment: It does not matter what command line arguments are given, jackd still crashes.
component: jack-audio-connection-kit
Attached file: coredump, 11710464 bytes
crash_function: __libc_message
executable: /usr/bin/jackd
kernel: 2.6.38-0.rc8.git0.1.fc15.x86_64
package: jack-audio-connection-kit-1.9.6-3.fc15
rating: 4
reason: Process /usr/bin/jackd was killed by signal 6 (SIGABRT)
release: Fedora release 15 (Lovelock)
time: 1299828119
uid: 500

How to reproduce
-----
1. start jackd from command line with arbitrary sets of parameters.
Comment 1 Albert Cervin 2011-03-12 03:15:08 EST
Created attachment 483871 [details]
File: backtrace
Comment 2 Brendan Jones 2011-04-04 08:53:25 EDT
I am also experiencing this on 1.9.7 on x86_64 - would provide a backtrace but GDB errors at the moment as well. 

Without looking too much into it, I rebuilt it locally without firewire and freebob and it started okay
Comment 3 Brendan Jones 2011-04-04 08:54:43 EDT
*** Bug 684530 has been marked as a duplicate of this bug. ***
Comment 4 Brendan Jones 2011-04-04 08:55:31 EDT
*** Bug 689099 has been marked as a duplicate of this bug. ***
Comment 5 Brendan Jones 2011-04-04 10:08:16 EDT
Created attachment 489780 [details]
alsa-info

Just for clarity - I don't have any firewire devices. Alsa-info attached
Comment 6 Orcan Ogetbil 2011-04-04 10:57:36 EDT
Hi Brendan,
Did you configure your jack as described in /usr/share/doc/jack-audio-connection-kit-VERSION/README.Fedora ?
Comment 7 Brendan Jones 2011-04-04 16:27:30 EDT
Created attachment 489852 [details]
jackd backtrace
Comment 8 Brendan Jones 2011-04-04 16:29:07 EDT
I did:

fedora15:~$ ps -ef|grep pulse
bsjones   7646  3709  0 06:27 pts/0    00:00:00 grep --color=auto pulse
fedora15:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31037
max locked memory       (kbytes, -l) 4194304
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 20
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
Comment 9 Orcan Ogetbil 2011-04-04 22:22:29 EDT
hmm, how about the jackuser group? Are you a member of it?

Also, did you try uninstalling pulseaudio? Do you get the same crashes without pulseaudio?
Comment 10 Orcan Ogetbil 2011-04-04 22:27:30 EDT
oops, I realized that what I said doesn't make sense. You already answered these questions. At this point, since I am not a jack developer, the best thing I can suggest is to file a bug to jack2
   http://trac.jackaudio.org/

Please post here the link if you file a ticket, so we can track the progress and update/patch our packages if necessary.
Comment 11 Brendan Jones 2011-04-04 22:42:02 EDT
No problem.

However, the problem looks to be occurring in libffado . I'll do some more debugging before filing upstream.
Comment 12 Brendan Jones 2011-04-05 18:10:59 EDT
I've reported this upstream here: http://subversion.ffado.org/ticket/329

A brute force workaround is to move /usr/lib64/jack/jack_firewire.so so it won't be loaded at runtime. Obviously this will disable firewire support
Comment 13 Mark van Rossum 2011-04-06 04:40:52 EDT
I experience this bug on a 32 bit system.
I might have a special case as I have disabled firewire in my bios (I never use it).

Brendan's fix works for me as well.
Simply deleting /usr/lib/jack/jack_firewire.so
solves it.
Comment 14 Brendan Jones 2011-04-06 10:30:51 EDT
*** Bug 680802 has been marked as a duplicate of this bug. ***
Comment 15 Brendan Jones 2011-04-14 09:44:33 EDT
One of the upstream libffado developers believes there maybe an issue with binutils, specifically in the final linking stage.

The full text of the comment can be found here: http://subversion.ffado.org/ticket/329#comment:17


I am reassigning to binutils for comment..
Comment 16 Nick Clifton 2011-04-14 11:09:30 EDT
Hi Brendan,

  This certainly sounds like a linker problem, although I guess that there is a small possibility that it is actually a loader problem.  Without more information as to exactly what is going wrong it is hard to say any more.  But here are a few things that you could try:

  * If you build a linker from the current HEAD of the mainline FSF binutils 
    sources and then link using that, does the problem persist ?

  * What about if you use the GOLD linker rather than LD ?
    (Configure with --enable-gold to get the gold linker).

  * How do the working libffado and the broken libffado differ ?
    (Running readelf -a on the two binaries is probably the best way to
    compare them, although that will produce a lot of output.  Using 
    readelf -e will produce a much shorter output).

How does the broken version of libffado fail ?  Is it another memory corruption problem or something else ?

Cheers
  Nick
Comment 17 Brendan Jones 2011-04-14 17:43:09 EDT
Created attachment 492237 [details]
diff <(readelf -e libffado-debian-ld.so) <(readelf -e libffado-fedora-ld.so)


Upstream has attached the two libraries for comparison to this ticket: http://subversion.ffado.org/ticket/329#comment:19


The error is free() invalid pointer upon delete

I also pulled binutils-2.21.51.0.8-2 from koji and tried building against that with the same result. I will try your other recommendations.
Comment 18 Brendan Jones 2011-04-14 20:37:23 EDT
If I symlink ld to ld.gold in my path and rebuild as perform the final link step I get a working libffado.so


This was done using the from the libffado revision in F15 (patched as per spec), not the upstream trunk.

rpmbuild -ba ~/rpmbuild/SPECS/libffado.spec
mkdir /tmp/bin
ln -s /usr/bin/ld.gold /tmp/bin/ld
export PATH=/tmp/bin:$PATH
cd ~/rpmbuild/BUILD/libffado-2.1.0
g++ -o src/libffado.so -pthread -Wl,-soname=libffado.so.2 -shared src/devicemanager.os src/ffado.os src/ffadodevice.os src/debugmodule/debugmodule.os src/DeviceStringParser.os src/libieee1394/ARMHandler.os src/libieee1394/configrom.os src/libieee1394/csr1212.os src/libieee1394/CycleTimerHelper.os src/libieee1394/ieee1394service.os src/libieee1394/IEC61883.os src/libieee1394/IsoHandlerManager.os src/libstreaming/StreamProcessorManager.os src/libstreaming/util/cip.os src/libstreaming/generic/StreamProcessor.os src/libstreaming/generic/Port.os src/libstreaming/generic/PortManager.os src/libutil/cmd_serialize.os src/libutil/DelayLockedLoop.os src/libutil/IpcRingBuffer.os src/libutil/PacketBuffer.os src/libutil/Configuration.os src/libutil/OptionContainer.os src/libutil/PosixMessageQueue.os src/libutil/PosixSharedMemory.os src/libutil/PosixMutex.os src/libutil/PosixThread.os src/libutil/ringbuffer.os src/libutil/StreamStatistics.os src/libutil/SystemTimeSource.os src/libutil/TimestampedBuffer.os src/libutil/Watchdog.os src/libcontrol/Element.os src/libcontrol/BasicElements.os src/libcontrol/MatrixMixer.os src/libcontrol/CrossbarRouter.os src/libcontrol/ClockSelect.os src/libcontrol/Nickname.os src/libutil/serialize_libxml.os src/bebob/bebob_avdevice.os src/bebob/bebob_avdevice_subunit.os src/bebob/bebob_avplug.os src/bebob/bebob_dl_bcd.os src/bebob/bebob_dl_codes.os src/bebob/bebob_dl_mgr.os src/bebob/bebob_functionblock.os src/bebob/bebob_mixer.os src/bebob/focusrite/focusrite_generic.os src/bebob/focusrite/focusrite_saffire.os src/bebob/focusrite/focusrite_saffirepro.os src/bebob/focusrite/focusrite_cmd.os src/bebob/terratec/terratec_device.os src/bebob/terratec/terratec_cmd.os src/bebob/edirol/edirol_fa101.os src/bebob/edirol/edirol_fa66.os src/bebob/esi/quatafire610.os src/bebob/mackie/onyxmixer.os src/fireworks/fireworks_device.os src/fireworks/fireworks_control.os src/fireworks/fireworks_firmware.os src/fireworks/efc/efc_avc_cmd.os src/fireworks/efc/efc_cmd.os src/fireworks/efc/efc_cmds_hardware.os src/fireworks/efc/efc_cmds_hardware_ctrl.os src/fireworks/efc/efc_cmds_flash.os src/fireworks/efc/efc_cmds_mixer.os src/fireworks/efc/efc_cmds_monitor.os src/fireworks/efc/efc_cmds_ioconfig.os src/fireworks/fireworks_session_block.os src/fireworks/audiofire/audiofire_device.os src/oxford/oxford_device.os src/libstreaming/amdtp-oxford/AmdtpOxfordReceiveStreamProcessor.os src/motu/motu_avdevice.os src/motu/motu_controls.os src/motu/motu_mark3_controls.os src/motu/motu_mixerdefs.os src/motu/motu_mark3_mixerdefs.os src/motu/motu_mixer.os src/libstreaming/motu/MotuPort.os src/libstreaming/motu/MotuPortInfo.os src/libstreaming/motu/MotuReceiveStreamProcessor.os src/libstreaming/motu/MotuTransmitStreamProcessor.os src/dice/dice_avdevice.os src/dice/dice_eap.os src/dice/focusrite/focusrite_eap.os src/dice/focusrite/saffire_pro40.os src/dice/focusrite/saffire_pro24.os src/libavc/streamformat/avc_extended_stream_format.os src/libavc/musicsubunit/avc_descriptor_music.os src/libavc/musicsubunit/avc_musicsubunit.os src/libavc/audiosubunit/avc_audiosubunit.os src/libavc/audiosubunit/avc_descriptor_audio.os src/libavc/audiosubunit/avc_function_block.os src/libavc/descriptors/avc_descriptor_cmd.os src/libavc/descriptors/avc_descriptor.os src/libavc/general/avc_extended_subunit_info.os src/libavc/general/avc_unit_info.os src/libavc/general/avc_generic.os src/libavc/general/avc_subunit_info.os src/libavc/general/avc_connect.os src/libavc/general/avc_signal_format.os src/libavc/general/avc_extended_cmd_generic.os src/libavc/general/avc_extended_plug_info.os src/libavc/general/avc_plug_info.os src/libavc/general/avc_unit.os src/libavc/general/avc_subunit.os src/libavc/general/avc_plug.os src/libavc/general/avc_vendor_dependent_cmd.os src/libavc/avc_definitions.os src/libavc/ccm/avc_signal_source.os src/libstreaming/amdtp/AmdtpPort.os src/libstreaming/amdtp/AmdtpPortInfo.os src/libstreaming/amdtp/AmdtpReceiveStreamProcessor.os src/libstreaming/amdtp/AmdtpTransmitStreamProcessor.os src/genericavc/avc_avdevice.os src/genericavc/stanton/scs.os -lm -lpthread -liec61883 -lraw1394 -lconfig++ -lxml++-2.6 -lxml2 -lglibmm-2.4 -lgobject-2.0 -lsigc-2.0 -lgthread-2.0 -lrt -lglib-2.0

su -c 'cp src/libffado.so /usr/lib64/libffado.so.2.999.0'

gdb jackd 
run



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 19 Brendan Jones 2011-04-15 07:11:30 EDT
Quoting cladish:

"On further thought, the problem is probably not that the constructors aren't executed, but that their execution order changes: GCC http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46770 "
Comment 20 Nick Clifton 2011-04-15 09:57:00 EDT
Created attachment 492382 [details]
Disable initfini_array support.
Comment 21 Nick Clifton 2011-04-15 10:00:35 EDT
Hi Brendan,

  It looks like a problem with the linker's support for .init_array and .fini_array sections.  Are you able to build and install a local binutils rpm ?  If so, please could you try downloading the binutils source rpm, applying the uploaded patch to the binutils.spec file and then creating a test linker with which you can link jackd.  if this works then it confirms the cause the problem, but not the actual nitty gritty of what is wrong.  For that I will need to look at the libffado.so files built by the Debian linker and the (unpatched) Fedora linker.

Cheers
  Nick
Comment 22 Brendan Jones 2011-04-15 17:41:31 EDT
No problem. I rebuilt binutils based on the spec in F15 with disabled initfini-array flag. Your patch was done on the version in rawhide I think.

Linking with this version's ld was successful. The binary is here http://bsjones.fedorapeople.org/libffado.so.2.999.0-binutils-disable-initfini-array.tar.gz 

And the rebuilt binutils http://bsjones.fedorapeople.org/binutils-2.21.51.0.6-2.fc15.x86_64.rpm http://bsjones.fedorapeople.org/binutils-2.21.51.0.6-2.fc15.src.rpm http://bsjones.fedorapeople.org/binutils-debuginfo-2.21.51.0.6-2.fc15.x86_64.rpm
Comment 23 Nick Clifton 2011-04-16 04:04:41 EDT
Hi Brendan,

> The binary is here

  Thanks for that.  Can you also make a broken binary for me, so that I can compare the two and find out exactly which constructors/destructors are being called in the wrong order.

Cheers
  Nick
Comment 24 Brendan Jones 2011-04-16 09:27:28 EDT
http://bsjones.fedorapeople.org/libffado.so.2.999.0-binutils-enable-initfini-array.tar.gz

Using binutils 2.21.51.0.6-1.fc15
Comment 25 Nick Clifton 2011-04-18 05:59:32 EDT
Hi Brendan,

  Thanks for the binaries.  Unfortunately I am baffled.  The ordering of the constructor functions in the .init_array section of the enabled binary is exactly the same as the order in the .ctors section of the disabled binary.  Plus there are no extra constructors in either binary.  So in theory they should exhibit exactly the same behaviour.  (There are no destructors in either binary).


> The error is free() invalid pointer upon delete

Can you find out any more about this error please ?  For example, which function is executing the free() and from where should the corresponding malloc() have been called ?

I am beginning to wonder if this is a loader problem rather than a linker problem.  Maybe the loader is not executing the .init_array section properly.  (This seems rather unlikely as surely lots of binaries would break if this were true).

Cheers
  Nick
Comment 26 Brendan Jones 2011-04-18 06:19:07 EDT
Created attachment 492846 [details]
debugmodule.cpp
Comment 27 Brendan Jones 2011-04-18 06:19:46 EDT
Created attachment 492848 [details]
debugmodule.h
Comment 28 Brendan Jones 2011-04-18 06:20:36 EDT
Created attachment 492849 [details]
ffado.cpp
Comment 29 Brendan Jones 2011-04-18 06:31:14 EDT
The error occurs when DebugModuleManager::~DebugModuleManager is called from ffado.cpp . It contains a vector of DebugModules which are iterated through - a delete on the dereferenced vector produces the crash. 

It is pretty easy to reproduce - install jack-audio-connection-kit, it will pull in libffado. Calling /usr/bin/jackd without any parameters will reproduce the crash. 

The upstream thread may provide some clues. The binaries the maintained used to debug this problem are there also.
Comment 30 Brendan Jones 2011-04-18 08:42:31 EDT
And here's the relevant part of the bt:
(gdb) run
Starting program: /usr/bin/jackd 
[Thread debugging using libthread_db enabled]
jackdmp 1.9.7
Copyright 2001-2005 Paul Davis and others.
Copyright 2004-2011 Grame.
jackdmp comes with ABSOLUTELY NO WARRANTY
This is free software, and you are welcome to redistribute it
under certain conditions; see the file COPYING for details
[New Thread 0x7ffff6bdd700 (LWP 7152)]
Cleaning up leftover debug module: DeviceManager

<snip>
Program received signal SIGABRT, Aborted.
0x0000003dea2362c5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) backtrace
#0  0x0000003dea2362c5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x0000003dea237bdb in abort () at abort.c:92
#2  0x0000003dea2722c3 in __libc_message (do_abort=2, fmt=0x3dea35ca28 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:186
#3  0x0000003dea2787ba in malloc_printerr (action=3, str=0x3dea359a5b "free(): invalid pointer", ptr=<optimized out>) at malloc.c:6283
#4  0x00007ffff77e358e in DebugModuleManager::~DebugModuleManager (this=0x7ffff6bde010, __in_chrg=<optimized out>) at src/debugmodule/debugmodule.cpp:272
#5  0x00007ffff77d7076 in exitfunc () at src/ffado.cpp:61
#6  0x0000003de9a13b68 in _dl_close_worker (map=<optimized out>) at dl-close.c:266
#7  0x0000003de9a1462e in _dl_close (_map=0x60ff00) at dl-close.c:754
#8  0x0000003de9a0e8a6 in _dl_catch_error (objname=0x60fee0, errstring=0x60fee8, mallocedp=0x60fed8, operate=0x3deaa00fe0 <dlclose_doit>, args=0x60ff00) at dl-error.c:178
#9  0x0000003deaa0152f in _dlerror_run (operate=0x3deaa00fe0 <dlclose_doit>, args=0x60ff00) at dlerror.c:164
#10 0x0000003deaa0100f in __dlclose (handle=0x1bed) at dlclose.c:48
#11 0x00007ffff7dadf23 in jack_get_descriptor (drivers=0x0, sofile=0x608073 "jack_firewire.so", symbol=<optimized out>) at ../common/JackDriverLoader.cpp:462
#12 0x00007ffff7daefae in jack_drivers_load (drivers=0x0) at ../common/JackDriverLoader.cpp:621
#13 0x00007ffff7db2aca in jackctl_drivers_load (server_ptr=0x606470) at ../common/JackControlAPI.cpp:293
#14 jackctl_server_create (on_device_acquire=0x403a70 <audio_acquire(char const*)>, on_device_release=0x403b80 <audio_release(char const*)>) at ../common/JackControlAPI.cpp:801
#15 0x000000000040247d in main (argc=1, argv=0x7fffffffe118) at ../common/Jackdmp.cpp:253
Comment 31 Nick Clifton 2011-04-19 08:04:57 EDT
Hi Brendan,

  I am still struggling^H^H^H^H^H investigating this.

  One quick question - in DebugModuleManager::~DebugModuleManager() wouldn't it make more sense to walk the  m_debugModules vector backwards ?  Ie treating the vector as a stack rather than a queue ?  (I assume that you will tell me that the modules are meant to be independent of each other and so the order of their deletion is irrelevant, but can this be guaranteed ?)

  As an aside I tried compiling the ctor_test.cpp app from:

    http://subversion.ffado.org/ticket/329#comment:19

  and it works with and without using init_array sections.

Cheers
  Nick
Comment 32 Brendan Jones 2011-04-19 09:21:51 EDT
Thanks for looking into this Nick. Some of these questions may be better directed to the developers upstream (I'm not involved in the ffado project). In the upstream comments a number of different methods of dealing with the vector were tried. 

Given that this is pretty important to the audio SIG wrt F15, do you have any suggestions for a possible work around (e.g. patching the libffado spec to use the goldlinker / or simply removing the call to ~DebugModuleManager?)


Let me know if there is anything I can do.


Brendan
Comment 33 Nick Clifton 2011-04-19 10:47:10 EDT
Hi Brendan,

  Using GOLD is definitely the easiest and best workaround for now.  (It also has the advantage that GOLD should be faster than LD).

  I'll continue to look into this in the background, but at the moment I am stumped as to why putting the constructors into the .init_array section is breaking libffado.

Cheers
  Nick
Comment 34 Orcan Ogetbil 2011-04-22 07:47:59 EDT
*** Bug 698903 has been marked as a duplicate of this bug. ***
Comment 35 mariolinux 2011-04-22 12:02:14 EDT
Package: jack-audio-connection-kit-1.9.7-1.fc15
Architecture: x86_64
OS Release: Fedora release 15 (Lovelock)


Comment
-----
Starting ardour
Comment 36 Nils Philippsen 2011-04-27 09:12:38 EDT
(In reply to comment #25)
> Hi Brendan,
> 
>   Thanks for the binaries.  Unfortunately I am baffled.  The ordering of the
> constructor functions in the .init_array section of the enabled binary is
> exactly the same as the order in the .ctors section of the disabled binary. 
> Plus there are no extra constructors in either binary.  So in theory they
> should exhibit exactly the same behaviour.  (There are no destructors in either
> binary).

Hmm, not that I really know anything about the subject, but http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46770#c1 suggests that .init_array and .ctors are processed in opposite order:

"Using .init_array/.fini_array instead of .ctors/.dtors removes the need for the
associated (relative) relocations, and avoids the backwards disk seeks on
startup (since while .ctors are processed backwards, .init_array is processed
forward)"

So if the ordering is the same, but they are processed differently, this would explain differences, wouldn't it?
Comment 37 Fedora Update System 2011-04-28 06:07:10 EDT
libffado-2.1.0-0.3.20110426.svn1983.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/libffado-2.1.0-0.3.20110426.svn1983.fc15
Comment 38 Fedora Update System 2011-04-28 15:03:23 EDT
Package libffado-2.1.0-0.3.20110426.svn1983.fc15:
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libffado-2.1.0-0.3.20110426.svn1983.fc15'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/libffado-2.1.0-0.3.20110426.svn1983.fc15
then log in and leave karma (feedback).
Comment 39 Fedora Update System 2011-05-03 00:52:18 EDT
libffado-2.1.0-0.3.20110426.svn1983.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.