Please run:

# echo CoreDumpDirectory /tmp > /etc/httpd/conf.d/coredump.conf
# service httpd restart

reproduce the problem, then do

# gdb /usr/sbin/httpd /tmp/core.xxxxx 
...
(gdb) bt
(gdb) bt.full

and attach the gdb output to this bug report.

Hm.
I am not able to make apache create a coredump of the process.
If I do a manual kill -11 of one of the processes a dump is created:

[Mon Jul 07 13:00:08 2008] [notice] child pid 27826 exit signal Segmentation
fault (11), possible coredump in /tmp

However, the segfault that happens on the php-script will not generate one:
[Mon Jul 07 13:00:32 2008] [notice] child pid 27827 exit signal Segmentation
fault (11)

Trying to attach gdb directly to one of the child processes before the segfault
will only make apache not serve any pages with this process.

Any other tips?

Hmmm, weird.  Did you set the process running with "run" after you attached with
gdb and then tried to reproduce?

(it may take N tries for N children running)

that should be "cont" not run, of course.

Created attachment 311144 [details]
bt full

Got it now, thanks.
Installed a few more debuginfo-packages as well-

Not sure how much of this you need. I gave up pressing enter after about 10
MB..

Thanks; not much use in the end since it looks like a trashed stack :(

Can you give a precise sequence of instructions which I'd need to follow to be
able to reproduce the bug here (i.e. checkout from horde cvs, configure as
follows, blah)

Actually if you can try just "bt" not "bt full" on that core dump too, that
would be useful; I've seen "bt full" get confused in cases where plain "bt" does
not in the past.

Output from "bt"

#0  0x000000004254415f in _zend_hash_quick_add_or_update (ht=0x7f877eaef228,
arKey=0x7f877d29a448 "nodes", nKeyLength=6, h=6953829379134,
pData=0x7fff8229b080, nDataSize=8, pDest=0x7fff8229b0e0, flag=1)
    at /usr/src/debug/php-5.2.6/Zend/zend_hash.c:271
#1  0x00000000425a5e25 in ZEND_RECV_SPEC_HANDLER (execute_data=0x7fff8229b3d0)
at /usr/src/debug/php-5.2.6/Zend/zend_execute.c:160
#2  0x000000004255b454 in execute (op_array=0x7f877d330a80) at
/usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:92
#3  0x000000004256f83e in zend_do_fcall_common_helper_SPEC
(execute_data=0x7fff8229b850) at /usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:234
#4  0x000000004255b454 in execute (op_array=0x7f877d330a80) at
/usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:92
#5  0x000000004256f83e in zend_do_fcall_common_helper_SPEC
(execute_data=0x7fff8229bcd0) at /usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:234
#6  0x000000004255b454 in execute (op_array=0x7f877d330a80) at
/usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:92
#7  0x000000004256f83e in zend_do_fcall_common_helper_SPEC
(execute_data=0x7fff8229c150) at /usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:234
#8  0x000000004255b454 in execute (op_array=0x7f877d330a80) at
/usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:92
#9  0x000000004256f83e in zend_do_fcall_common_helper_SPEC
(execute_data=0x7fff8229c5d0) at /usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:234
#10 0x000000004255b454 in execute (op_array=0x7f877d330a80) at
/usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:92
#11 0x000000004256f83e in zend_do_fcall_common_helper_SPEC
(execute_data=0x7fff8229ca50) at /usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:234
#12 0x000000004255b454 in execute (op_array=0x7f877d330a80) at
/usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:92
#13 0x000000004256f83e in zend_do_fcall_common_helper_SPEC
(execute_data=0x7fff8229ced0) at /usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:234
#14 0x000000004255b454 in execute (op_array=0x7f877d330a80) at
/usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:92
#15 0x000000004256f83e in zend_do_fcall_common_helper_SPEC
(execute_data=0x7fff8229d350) at /usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:234
#16 0x000000004255b454 in execute (op_array=0x7f877d330a80) at
/usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:92
#17 0x000000004256f83e in zend_do_fcall_common_helper_SPEC
(execute_data=0x7fff8229d7d0) at /usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:234
#18 0x000000004255b454 in execute (op_array=0x7f877d330a80) at
/usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:92
#19 0x000000004256f83e in zend_do_fcall_common_helper_SPEC
(execute_data=0x7fff8229dc50) at /usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:234
#20 0x000000004255b454 in execute (op_array=0x7f877d330a80) at
/usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:92
#21 0x000000004256f83e in zend_do_fcall_common_helper_SPEC
(execute_data=0x7fff8229e0d0) at /usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:234
#22 0x000000004255b454 in execute (op_array=0x7f877d330a80) at
/usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:92
#23 0x000000004256f83e in zend_do_fcall_common_helper_SPEC
(execute_data=0x7fff8229e550) at /usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:234
#24 0x000000004255b454 in execute (op_array=0x7f877d330a80) at
/usr/src/debug/php-5.2.6/Zend/zend_vm_execute.h:92

This horde-installation might be a bit unregular, as I started with installing
the rpm-packages while the box was still F8.

After a while I needed some new features, and decided to go for the cvs-version
instead.  So I did a cvs checkout in the same dir as the rpm-packages were
installed.
Then I removed the rpms with "--justdb".

However I do not think this is the problem, since the installation worked fine
for several months after that.

Some more info after digging a bit.
If i comment out this code in templates/portal/sidebar.inc the segfault does not
happen:
  <?php $tree->renderTree() ?>

The renderTree() seems to come from the pear-package Horde/Tree.php

That fonctions simply does

    function renderTree($static = false)
    {   
        echo $this->getTree($static);
    }

If i add a 
 print_r($this); 
I can see that this is a javascript-object, so I guess the getTree()-function is
from pear/Horde/Tree/javascript.php

After stepping trough the getTree() function in that file it seems to be
 $this->_buildIndents($this->_root_nodes); 
that is the problem.

_root_nodes contains this:

    [_root_nodes] => Array
        (
            [0] => horde
            [1] => imp
            [2] => dimp
            [3] => organizing
            [4] => myaccount
            [5] => website
            [6] => administration
            [7] => options
            [8] => logout
        )

The function _buildIndents is back in Tree.php, and seems rather simple:

    function _buildIndents($nodes, $indent = 0)
    {   
        foreach ($nodes as $id) {
            $this->_nodes[$id]['indent'] = $indent;
            if (!empty($this->_nodes[$id]['children'])) {
                $this->_buildIndents($this->_nodes[$id]['children'], $indent + 1);
            }
        }
    }

As this is a recursive function, I tried to remove the recursion (the three
lines starting with if (!empty...
That stopped the segfault.

I am attaching a complete print_r() of "$this" for further debugging.

Created attachment 311350 [details]
print_r of $this

Ah, actually this looks just like a recursion bug in the application code - PHP
will crash like that in this case.

The:

            [imp] => Array

array has a:

                            [2] => imp

link in its children node, I'd guess that's the same object and the cause of the
recursion.

You are right.
This is causing recursion.  I have now followed the codepath and tried to figure
out why this happens, and the code involved does not seem to have changed the
last 6 months.

The code that adds "imp" as a child of "imp" is on line 86 in

http://cvs.horde.org/co.php/imp/lib/Block/tree_folders.php?r=1.71

If I remove that code ($tree->addNode()), the sidebar is displayed.  

But this problem has only existed in a few weeks (I waited a bit before I
reported it because I hoped someone would either discover it and fix it, or it
should be reported by someone else.

So it seems to be that the real problem is somewhere else.

Thanks for the help and feedback so far anyway

Reported this to the horde-bugtracker as well.