RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 876686 - thin server crashing
Summary: thin server crashing
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: ruby
Version: 6.3
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: rc
: ---
Assignee: Vít Ondruch
QA Contact: BaseOS QE - Apps
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-11-14 17:42 UTC by Dave Johnson
Modified: 2015-09-07 13:20 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-09-07 13:20:22 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
core dump (25.81 MB, application/x-gzip)
2012-11-19 15:34 UTC, John Eckersberg 		no flags Details
unoptimized core dump (26.49 MB, application/x-gzip)
2012-11-28 14:49 UTC, John Eckersberg 		no flags Details
unoptimized core dump (with segfault) (26.48 MB, application/x-gzip)
2012-11-30 17:37 UTC, John Eckersberg 		no flags Details
View All

Description Dave Johnson 2012-11-14 17:42:19 UTC 
Description of problem:
===========================
With the latest build, v1.1 rc1, automation seems to be triggering the frontend to go service unavailable.  Checking the thin.log I see

/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/inflector/inflections.rb:151: [BUG] Segmentation fault

I have been able to hit this a few time now accross several automation runs.  


Version-Release number of selected component (if applicable):
===================================================================
aeolus-all-0.13.24-1.el6cf.noarch
aeolus-conductor-0.13.24-1.el6cf.noarch
aeolus-conductor-daemons-0.13.24-1.el6cf.noarch
aeolus-conductor-doc-0.13.24-1.el6cf.noarch
aeolus-configure-2.8.11-1.el6cf.noarch
rubygem-activesupport-3.0.10-4.el6cf.noarch
rubygem-aeolus-cli-0.7.7-1.el6cf.noarch
rubygem-aeolus-image-0.3.0-12.el6.noarch
rubygem-thin-1.2.11-3.el6.x86_64
Comment 2 Dave Johnson 2012-11-15 19:51:51 UTC 
Reproduced this again with the v1.1 rc1 build however the failure was slightly different...

In this case, the thin server dies but does not generate a segfault upon failure, it basically just terminates the process.

Checking thin.log shows:
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:124: [BUG] rb_gc_mark(): unknown data type 0x12(0x43bef00) non object

Speaking with John, this confirms his theory that there appears to be a garbage collection issue
Comment 3 Mike Orazi 2012-11-16 15:20:46 UTC 
Removing this from 1.1 since we haven't been able to reproduce it very frequently.  We should still keep tracking it and if a core suddenly shows up, we'll reconsider bringing it into 1.1.
Comment 5 John Eckersberg 2012-11-19 15:34:07 UTC 
Created attachment 647839 [details]
core dump

Top of the backtrace:

#0  0x00007fb8d14fd8a5 in raise () from /lib64/libc.so.6
#1  0x00007fb8d14ff085 in abort () from /lib64/libc.so.6
#2  0x00007fb8d236fb63 in rb_bug (fmt=0x7fb8d240834c "Segmentation fault") at error.c:213
#3  0x00007fb8d23d802c in sigsegv (sig=11) at signal.c:634
#4  <signal handler called>
#5  gc_mark (ptr=6646579027747566960, lev=1) at gc.c:804
#6  0x00007fb8d23931a2 in gc_mark_children (ptr=<value optimized out>, lev=1) at gc.c:1063
#7  0x00007fb8d2370e32 in blk_mark (data=0x6553f90) at eval.c:8526
#8  0x00007fb8d23931a2 in gc_mark_children (ptr=<value optimized out>, lev=1) at gc.c:1063
#9  0x00007fb8d2392d4f in mark_locations_array (x=<value optimized out>, n=148130) at gc.c:690
#10 0x00007fb8d23935f7 in garbage_collect () at gc.c:1493
#11 0x00007fb8d239426f in ruby_xmalloc (size=<value optimized out>) at gc.c:153
#12 0x00007fb8d2370742 in frame_dup (frame=0x66adf80) at eval.c:8575
#13 0x00007fb8d2371881 in blk_copy_prev (block=0x66a9ac0) at eval.c:8594
#14 0x00007fb8d2373878 in proc_alloc (klass=140431782514520, proc=0) at eval.c:8820
#15 0x00007fb8d2396d17 in rb_hash_initialize (argc=0, argv=0x0, hash=140431590685280) at hash.c:321
Comment 6 John Eckersberg 2012-11-19 15:35:29 UTC 
Forgot to mention, this is on ruby-1.8.7.352-7.el6_2.x86_64
Comment 7 John Eckersberg 2012-11-19 16:03:15 UTC 
I see a couple of upstream 1.8.7 patches that might be relevant to this:

https://github.com/ruby/ruby/commit/a4e76099bf44ae7d75bd24c3c8233d710e590fac
https://github.com/ruby/ruby/commit/698d21a276f862da823c7a6536f9c15afe602e19
Comment 9 Vít Ondruch 2012-11-21 12:57:29 UTC 
As long as you are not able to provide better reproducer, I am afraid I cannot help.

It would be really helpful to understand at least roughly where in the Ruby code the SEGV happened and what was going on.

Btw the callstack seems to be 750 levels deep, which seems to be quite enough.
Comment 10 John Eckersberg 2012-11-28 14:49:03 UTC 
Created attachment 653579 [details]
unoptimized core dump

This is a core dump obtained from an unoptimized build of ruby-1.8.7.352-7.el6_3.  The build is available here:

http://people.redhat.com/jeckersb/rpms/ruby-gcdebug/

This did not segfault.  Instead the gc detected something was wrong and aborted with:

rb_gc_mark(): unknown data type 0x20(0x43b2a30) non object

(I pieced that line together from the core; the original log is gone).

We've seen this failure case in addition to a segfault with the optimized build, so I'm reasonably sure this is representative of the problem.
Comment 11 John Eckersberg 2012-11-28 15:15:33 UTC 
It's marking the stack when everything goes wrong.

#13 0x00007f2e931b2437 in garbage_collect () at gc.c:1493
1493        rb_gc_mark_locations((VALUE*)STACK_END, rb_gc_stack_start);

Exactly where on the stack?

#11 0x00007f2e931b0bd1 in mark_locations_array (x=0x7fffdfe78578, n=159451) at gc.c:690
690                 gc_mark(v, 0);

And with a bit more context...

686         VALUE v;
687         while (n--) {
688             v = *x;
689             if (is_pointer_to_heap((void *)v)) {
690                 gc_mark(v, 0);
691             }
692             x++;
693         }

OK, so whatever is at 0x7fffdfe78578 should be a pointer to a ruby object.  Let's walk up the stack until we find that frame:

(gdb) info frame
Stack level 146, frame at 0x7fffdfe78970:
 rip = 0x7f2e9319e596 in proc_invoke (eval.c:8970); saved rip 0x7f2e9319e761
 called by frame at 0x7fffdfe789a0, caller of frame at 0x7fffdfe78410
 source language c.
 Arglist at 0x7fffdfe78960, args: proc=139837825852640, args=139837825847080, self=6, klass=0
 Locals at 0x7fffdfe78960, Previous frame's sp is 0x7fffdfe78970
 Saved registers:
  rbx at 0x7fffdfe78948, rbp at 0x7fffdfe78960, r12 at 0x7fffdfe78950, r13 at 0x7fffdfe78958, rip at 0x7fffdfe78968
(gdb) info locals
_tag = {buf = {{context = {uc_flags = 140736949879920, uc_link = 0x7f2e93186fb3, uc_stack = {ss_sp = 0x9dfe79060, ss_flags = -538475336, ss_size = 4473}, uc_mcontext = {
          gregs = {139838001426152, 72340172838076673, 139837825847640, 140736949880000, 5521, 0, 9, 0, 140736949879856, 86157552, 140736949881184, 86157360, 140736949880864, 
            140736949880032, 0, 140736949879824, 139838013170962, 140736949880096, 139838013095566, 8589934592, 139837825847640, 2, 70977}, fpregs = 0x7fffdfe785d8, 
          __reserved1 = {139837825847640, 140736949880128, 139838013095834, 139837825853440, 139837825847640, 140736949883184, 139838013102127, 139837899355800}}, 
        uc_sigmask = {__val = {0, 139837899395120, 139837827586440, 139837827586440, 139837827004680, 139837899395520, 139837827586440, 206158430224, 140736949881008, 
            140736949880816, 139837827004680, 1, 102, 140733193388064, 0, 139837853836600}}, __fpregs_mem = {cwd = 895, swd = 65535, ftw = 0, fop = 65535, 
          rip = 9295595906256601087, rdp = 50593808, mxcsr = 8100, mxcr_mask = 32558, _st = {{significand = {61976, 35116, 32558, 0}, exponent = 0, padding = {0, 0, 0}}, {
              significand = {0, 0, 0, 0}, exponent = 39288, padding = {34799, 32558, 0}}, {significand = {65535, 65535, 65535, 65535}, exponent = 64104, padding = {34808, 
                32558, 0}}, {significand = {14600, 34817, 32558, 0}, exponent = 10824, padding = {34928, 32558, 0}}, {significand = {34432, 57319, 32767, 0}, exponent = 35361, 
              padding = {57319, 32767, 0}}, {significand = {102, 0, 1, 0}, exponent = 65535, padding = {65535, 0, 0}}, {significand = {20157, 747, 64200, 29385}, 
              exponent = 35208, padding = {35164, 32558, 0}}, {significand = {39368, 34799, 32558, 0}, exponent = 34560, padding = {57319, 32767, 0}}}, _xmm = {{element = {
                2467921150, 32558, 2303877952, 32558}}, {element = {0, 32558, 2473109512, 2}}, {element = {0, 0, 3756492496, 0}}, {element = {7481, 0, 2280626632, 32558}}, {
              element = {2303878192, 32558, 4, 0}}, {element = {2280626352, 32558, 2470888784, 32558}}, {element = {7481, 0, 3756495568, 32767}}, {element = {2467873138, 
                32558, 3756492576, 32767}}, {element = {0, 0, 2354113792, 32558}}, {element = {0, 0, 2280626632, 32558}}, {element = {0, 0, 3756498912, 32767}}, {element = {
                3756498992, 32767, 3, 0}}, {element = {4, 0, 2280626232, 32558}}, {element = {0, 0, 3756497904, 32767}}, {element = {9036937, 32558, 2303096272, 32558}}, {
              element = {2303095552, 32558, 2292621792, 32558}}}, padding = {2292621792, 32558, 2280624032, 32558, 3756492704, 32767, 2468027339, 32558, 2280624032, 32558, 
            3756494288, 32767, 2467920086, 32558, 2303095472, 32558, 2354113632, 32558, 0, 0, 2280626232, 0, 64215, 0}}}, status = 0}}, frame = 0x7fffdfe78ea0, 
  iter = 0x7fffdfe788e0, tag = 5, retval = 4, scope = 0x7f2e87ef91d0, dst = 0, prev = 0x7fffdfe79d70, blkid = 0}
[trimmed]

The address is in this struct somewhere, because:

(gdb) p &_tag
$266 = (struct tag *) 0x7fffdfe78430
(gdb) p (&_tag)+1
$267 = (struct tag *) 0x7fffdfe78820

And recall we're looking for 0x7fffdfe78578.  By trial and error, we find it here:

(gdb) p &(_tag->buf->context->uc_sigmask->__val[4])
$268 = (long unsigned int *) 0x7fffdfe78578

uc_sigmask is of type sigset_t, backed internally by __val which is of type long unsigned int[128].  It's just a 1024-bit signal mask.  It just so happens that in the middle of this bit mask is a value that looks like a pointer into the heap.  It looks like that value was in memory from some previous frame, since I don't see anywhere that it gets initialized explicitly.  In any case, it's definitely not an explicit heap pointer.  This in itself doesn't look like a bug, because it should be perfectly fine to conservatively mark things in the heap that aren't really needed.  However it's suspicious to me given the context.

I'm still trying to make sense of where it all goes wrong while marking the heap, stay tuned for that.
Comment 12 John Eckersberg 2012-11-29 15:33:00 UTC 
Here's a rough backtrace I pieced together from the second coredump, based on the techniques described here:  http://libc.st/ruby/2009/02/23/inspecting-dead-ruby-process-with-gdb.html

/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/associations.rb:1850
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/associations.rb:1850
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/associations.rb:1845
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation/query_methods.rb:246
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation/query_methods.rb:233
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation/query_methods.rb:176
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation/query_methods.rb:174
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation/query_methods.rb:149
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation/query_methods.rb:150
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation.rb:64
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation.rb:64
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation.rb:64
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation.rb:62
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation/finder_methods.rb:341
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation/finder_methods.rb:341
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation/finder_methods.rb:343
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation/finder_methods.rb:124
/usr/share/aeolus-conductor/app/models/deployment.rb:349
/usr/share/aeolus-conductor/app/models/deployment.rb:346
/usr/share/aeolus-conductor/app/helpers/mustache_helper.rb:82
/usr/share/aeolus-conductor/app/helpers/mustache_helper.rb:82
/usr/share/aeolus-conductor/app/helpers/mustache_helper.rb:82
/usr/share/aeolus-conductor/app/helpers/mustache_helper.rb:86
/usr/share/aeolus-conductor/app/helpers/mustache_helper.rb:92
/usr/share/aeolus-conductor/app/views/deployments/_list.html.haml:27
/usr/share/aeolus-conductor/app/views/deployments/_list.html.haml:27
/usr/share/aeolus-conductor/app/views/deployments/_list.html.haml:27
/usr/share/aeolus-conductor/app/views/deployments/_list.html.haml:28
/usr/share/aeolus-conductor/app/views/deployments/_list.html.haml:28
/usr/share/aeolus-conductor/app/views/deployments/_list.html.haml:28
/usr/share/aeolus-conductor/app/views/deployments/_list.html.haml:28
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:361
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:566
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:574
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:341
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:340
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:339
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:61
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:61
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:61
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:26
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:26
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:26
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:148
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:253
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:258
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:148
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:147
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:591
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:361
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:566
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:574
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:341
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:340
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:339
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:61
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:61
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:61
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:88
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:88
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:86
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/helpers/form_tag_helper.rb:571
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/helpers/form_tag_helper.rb:571
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/helpers/form_tag_helper.rb:570
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/helpers/form_tag_helper.rb:52
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/helpers/form_tag_helper.rb:49
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:151
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:151
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:151
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:141
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:122
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:122
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:121
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:1
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:1
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:1
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:1
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:42
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:135
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:128
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:124
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:333
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:328
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:262
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:27
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:18
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:260
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:260
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:250
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:371
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:16
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:88
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:93
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:3
/usr/share/aeolus-conductor/app/helpers/application_helper.rb:70
/usr/share/aeolus-conductor/app/views/deployments/_list.html.haml:26
/usr/share/aeolus-conductor/app/views/deployments/_list.html.haml:26
/usr/share/aeolus-conductor/app/views/deployments/_list.html.haml:1
/usr/share/aeolus-conductor/app/views/deployments/_list.html.haml:1
/usr/share/aeolus-conductor/app/views/deployments/_list.html.haml:46
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:135
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:128
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:124
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:333
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:328
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:262
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:27
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:18
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:260
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:260
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:250
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:371
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:16
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:88
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:93
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:3
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:13
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:14
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:14
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:13
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:14
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:1
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:135
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:128
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:124
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:333
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:328
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:262
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:27
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:18
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:260
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:260
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:250
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:371
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:16
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:88
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:93
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:3
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:17
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:18
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:18
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:11
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:21
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:1
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:24
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:135
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:128
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:124
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:59
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:59
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:59
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:27
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:18
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:53
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:16
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:13
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:3
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:116
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:108
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/renderers.rb:48
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/compatibility.rb:55
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/compatibility.rb:54
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:99
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:93
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:93
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:93
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rendering.rb:16
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:40
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:40
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/core_ext/benchmark.rb:5
/usr/lib/ruby/1.8/benchmark.rb:306
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/core_ext/benchmark.rb:5
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/core_ext/benchmark.rb:5
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/core_ext/benchmark.rb:5
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:40
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:40
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:79
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/railties/controller_runtime.rb:15
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/railties/controller_runtime.rb:22
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:39
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:39
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:41
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/implicit_render.rb:11
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/implicit_render.rb:4
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/base.rb:151
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rendering.rb:10
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:18
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:461
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:429
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:410
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:402
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:94
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:93
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:20
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rescue.rb:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rescue.rb:20
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:30
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:30
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:27
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:18
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:29
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:18
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/base.rb:111
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:40
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:44
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal.rb:135
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rack_delegation.rb:12
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal.rb:178
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:63
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:19
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:150
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:143
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:93
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:64
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:92
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:87
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:141
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:135
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:492
/usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:35
/usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:34
/usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:34
/usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:31
/usr/lib/ruby/gems/1.8/gems/sass-3.1.4/lib/sass/../sass/plugin/rack.rb:50
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/best_standards_support.rb:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/best_standards_support.rb:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/best_standards_support.rb:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/head.rb:16
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/methodoverride.rb:13
/usr/lib/ruby/gems/1.8/gems/rack-restful_submit-1.2.1/lib/rack/rack-restful_submit.rb:12
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/params_parser.rb:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/flash.rb:178
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/flash.rb:194
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/session/abstract_store.rb:149
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/session/abstract_store.rb:148
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/cookies.rb:302
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/cookies.rb:302
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/cookies.rb:302
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:32
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/query_cache.rb:27
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/query_cache.rb:32
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:12
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:8
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:31
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:34
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:354
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:361
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:45
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:416
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:416
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:402
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:44
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:48
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/sendfile.rb:102
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/sendfile.rb:102
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/sendfile.rb:102
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/remote_ip.rb:47
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/show_exceptions.rb:47
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/show_exceptions.rb:47
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/show_exceptions.rb:47
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/show_exceptions.rb:46
/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/rack/logger.rb:12
/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/rack/logger.rb:16
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/runtime.rb:17
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/runtime.rb:17
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/runtime.rb:16
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/lock.rb:34
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/lock.rb:32
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/lock.rb:42
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/lock.rb:42
/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:169
/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:77
/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:76
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:52
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:48
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:46
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:43
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:57
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:63
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:91
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:57
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:57
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:41
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:47
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:249
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:236
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:61
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:51
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/server.rb:150
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/controllers/controller.rb:39
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:163
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:159
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/bin/thin:5
/usr/bin/thin:9
Comment 13 John Eckersberg 2012-11-30 17:37:52 UTC 
Created attachment 655160 [details]
unoptimized core dump (with segfault)

Last core, I swear :)

This one is dying in a different place than the other two (in those, the top of the stack was in activerecord):

/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node_set.rb:237
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:431
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:430
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node_set.rb:239
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node_set.rb:238
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node_set.rb:237
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:429
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:429
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:380
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:218
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:209
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:169
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:167
/usr/lib/ruby/1.8/net/http.rb:543
/usr/lib/ruby/1.8/net/http.rb:542
/usr/lib/ruby/1.8/net/http.rb:539
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:166
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:138
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:137
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:60
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:58
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:31
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:30
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/resource.rb:54
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/resource.rb:49
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:379
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:354
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:428
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:427
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:445
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:445
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:445
/usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.5.0/lib/deltacloud.rb:446
/usr/share/aeolus-conductor/app/models/instance_task.rb:70
/usr/share/aeolus-conductor/app/models/instance_task.rb:70
/usr/share/aeolus-conductor/app/models/instance_task.rb:66
/usr/share/aeolus-conductor/app/models/instance.rb:166
/usr/share/aeolus-conductor/app/models/instance.rb:168
/usr/share/aeolus-conductor/app/models/instance.rb:166
/usr/share/aeolus-conductor/app/helpers/mustache_helper.rb:44
/usr/share/aeolus-conductor/app/helpers/mustache_helper.rb:44
/usr/share/aeolus-conductor/app/views/instances/_list.html.haml:32
/usr/share/aeolus-conductor/app/views/instances/_list.html.haml:32
/usr/share/aeolus-conductor/app/views/instances/_list.html.haml:32
/usr/share/aeolus-conductor/app/views/instances/_list.html.haml:33
/usr/share/aeolus-conductor/app/views/instances/_list.html.haml:33
/usr/share/aeolus-conductor/app/views/instances/_list.html.haml:33
/usr/share/aeolus-conductor/app/views/instances/_list.html.haml:33
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:361
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:566
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:574
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:341
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:340
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:339
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:61
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:61
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:61
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:26
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:26
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:25
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:26
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:148
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:253
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:258
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:148
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:147
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:591
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:361
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:566
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:574
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:341
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:340
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:339
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:61
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:61
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:61
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:88
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:88
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:86
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/helpers/form_tag_helper.rb:571
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/helpers/form_tag_helper.rb:571
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/helpers/form_tag_helper.rb:570
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/helpers/form_tag_helper.rb:52
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/helpers/form_tag_helper.rb:49
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:151
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:151
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:151
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:141
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:122
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:122
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/xss_mods.rb:121
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:1
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:1
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:1
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:1
/usr/share/aeolus-conductor/app/views/layouts/_filter_table.html.haml:42
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:135
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:128
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:124
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:333
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:328
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:262
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:27
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:18
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:260
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:260
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:250
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:371
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:16
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:88
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:93
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:3
/usr/share/aeolus-conductor/app/helpers/application_helper.rb:70
/usr/share/aeolus-conductor/app/views/instances/_list.html.haml:31
/usr/share/aeolus-conductor/app/views/instances/_list.html.haml:31
/usr/share/aeolus-conductor/app/views/instances/_list.html.haml:1
/usr/share/aeolus-conductor/app/views/instances/_list.html.haml:1
/usr/share/aeolus-conductor/app/views/instances/_list.html.haml:50
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:135
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:128
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:124
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:333
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:328
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:262
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:27
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:18
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:260
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:260
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:250
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:371
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:16
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:88
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:93
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:3
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:13
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:14
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:14
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:13
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:14
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:1
/usr/share/aeolus-conductor/app/views/layouts/_tabpanel.html.haml:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:135
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:128
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:124
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:333
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:328
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:262
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:27
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:18
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:260
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:260
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:250
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/partials.rb:371
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:16
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:88
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers.rb:93
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:11
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:3
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:17
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:18
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:18
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:11
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:21
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:1
/usr/share/aeolus-conductor/app/views/pools/index.html.haml:24
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:135
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:128
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:127
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/template.rb:124
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:59
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:59
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:59
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:27
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:18
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:53
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_view/render/rendering.rb:16
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:13
/usr/lib/ruby/gems/1.8/gems/haml-3.1.2/lib/haml/helpers/action_view_mods.rb:3
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:116
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:108
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/renderers.rb:48
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/compatibility.rb:55
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/compatibility.rb:54
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:99
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:93
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:93
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:93
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rendering.rb:16
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:40
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:40
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/core_ext/benchmark.rb:5
/usr/lib/ruby/1.8/benchmark.rb:306
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/core_ext/benchmark.rb:5
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/core_ext/benchmark.rb:5
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/core_ext/benchmark.rb:5
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:40
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:40
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:79
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/railties/controller_runtime.rb:15
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/railties/controller_runtime.rb:22
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:39
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:39
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:41
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/implicit_render.rb:11
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/implicit_render.rb:4
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/base.rb:151
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rendering.rb:10
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:18
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:461
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:429
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:410
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:402
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:94
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:93
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:20
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rescue.rb:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rescue.rb:20
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:30
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:30
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:27
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:18
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:56
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:29
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:18
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/base.rb:111
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:40
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:44
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal.rb:135
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rack_delegation.rb:12
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal.rb:178
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:63
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:19
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:150
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:143
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:93
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:64
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:92
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:87
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:141
/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:135
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:492
/usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:35
/usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:34
/usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:34
/usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:31
/usr/lib/ruby/gems/1.8/gems/sass-3.1.4/lib/sass/../sass/plugin/rack.rb:50
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/best_standards_support.rb:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/best_standards_support.rb:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/best_standards_support.rb:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/head.rb:16
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/methodoverride.rb:13
/usr/lib/ruby/gems/1.8/gems/rack-restful_submit-1.2.1/lib/rack/rack-restful_submit.rb:12
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/params_parser.rb:17
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/flash.rb:178
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/flash.rb:194
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/session/abstract_store.rb:149
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/session/abstract_store.rb:148
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/cookies.rb:302
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/cookies.rb:302
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/cookies.rb:302
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:32
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/query_cache.rb:27
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/query_cache.rb:32
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:12
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:8
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:31
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:34
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:354
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:361
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:45
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:416
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:416
/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:402
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:44
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:48
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/sendfile.rb:102
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/sendfile.rb:102
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/sendfile.rb:102
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/remote_ip.rb:47
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/show_exceptions.rb:47
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/show_exceptions.rb:47
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/show_exceptions.rb:47
/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/show_exceptions.rb:46
/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/rack/logger.rb:12
/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/rack/logger.rb:16
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/runtime.rb:17
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/runtime.rb:17
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/runtime.rb:16
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/lock.rb:34
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/lock.rb:32
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/lock.rb:42
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/lock.rb:42
/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:169
/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:77
/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:76
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:52
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:48
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:46
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:43
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:57
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:63
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:91
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:57
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:57
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:41
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:47
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:249
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:236
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:61
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:51
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/server.rb:150
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/controllers/controller.rb:39
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:163
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:159
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/bin/thin:5
/usr/bin/thin:9


However, the common trend between all three is:

- The interpreter is in proc_alloc/blk_copy_prev/frame_dup when it runs out of heap and forces gc

- During stack mark in garbage collect, it's chasing nonsense values into the heap.  Not 100% positive on the first core since optimizations are masking a lot of it.  Definitely on the 2nd core as I described above, and definitely on this one as I'll get into in a separate comment after this...

- Eventually it segfaults and/or aborts somewhere under blk_mark
Comment 14 John Eckersberg 2012-12-01 02:20:53 UTC 
Real quick, since I promised it in comment #13, here's the phantom stack gc in the third dump.  This one's a bit more involved, but equally unneeded:

Frame 12 shown here is scanning the stack as in the prior two examples:

#12 0x00007f23b1d20bd1 in mark_locations_array (x=0x7fff54d958f8, n=151663) at gc.c:690

0x7fff54d958f8 is in frame 213 (rb_eval), however it's not any of the local variables in scope or saved registers.  To figure out what it's allocated for in this frame, I found the offset from rbp and then disassembled rb_eval to find where it's used.

(gdb) p/x $rbp - (void *)0x7fff54d958f8
$82 = 0x88
(gdb) disassemble /m
[snip]
3965                int noex;
[snip]
3986                    noex = NOEX_PRIVATE;
   0x00007f23b1d00972 <+18516>: movl   $0x2,-0x88(%rbp)

3987                }
3988                else if (SCOPE_TEST(SCOPE_PROTECTED)) {
   0x00007f23b1d0097e <+18528>: mov    0x2d67d0(%rip),%eax        # 0x7f23b1fd7154 <scope_vmode>
   0x00007f23b1d00984 <+18534>: and    $0x2,%eax
   0x00007f23b1d00987 <+18537>: test   %eax,%eax
   0x00007f23b1d00989 <+18539>: je     0x7f23b1d00997 <rb_eval+18553>

3989                    noex = NOEX_PROTECTED;
   0x00007f23b1d0098b <+18541>: movl   $0x4,-0x88(%rbp)
   0x00007f23b1d00995 <+18551>: jmp    0x7f23b1d009a1 <rb_eval+18563>

3990                }
3991                else {
3992                    noex = NOEX_PUBLIC;
   0x00007f23b1d00997 <+18553>: movl   $0x0,-0x88(%rbp)
[snip]


So the stack space for noex only gets written to in the NODE_DEFN case, otherwise it'll contain the remains from a previously finished stack frame.  In this case, it held a pointer into the heap so gc starts marking and eventually crashes.
Comment 15 Vít Ondruch 2012-12-05 11:39:25 UTC 
What about the nokogiri-1.5.0.beta.4? It is quite old version. Have you tried with more recent one?
Comment 16 John Eckersberg 2012-12-11 21:08:15 UTC 
Nope, only with 1.5.0.beta.4.  I'll see if I can't throw together a newer build for el6 and then try to reproduce with it.
Comment 17 John Eckersberg 2012-12-13 18:25:03 UTC 
I've rebuilt nokogiri from f18 (1.5.5) and added it here - http://people.redhat.com/jeckersb/rpms/ruby-gcdebug/

I'll ask Dave to include that in the test automation, see if we can reproduce it still.
Comment 18 John Eckersberg 2012-12-17 14:53:19 UTC 
Reproduced with newer nokogiri.  It actually crashed twice within 90 minutes.  I'll spare the huge core dumps, but here's the gc stack from both.

First one segfaulted with: /usr/lib/ruby/gems/1.8/gems/mustache-0.99.4/lib/mustache/template.rb:77: [BUG] Segmentation fault

#0  0x00007f92d82968a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007f92d8298085 in abort () at abort.c:92
#2  0x00007f92d910bc97 in rb_bug (fmt=0x7f92d91cd703 "Segmentation fault") at error.c:213
#3  0x00007f92d91951ed in sigsegv (sig=11) at signal.c:634
#4  <signal handler called>
#5  0x00007f92d9199364 in st_foreach (table=0x21, func=0x7f92d9138c2f <mark_entry>, arg=2) at st.c:485
#6  0x00007f92d9138c89 in mark_tbl (tbl=0x21, lev=2) at gc.c:722
#7  0x00007f92d9139179 in gc_mark_children (ptr=70068688, lev=2) at gc.c:997
#8  0x00007f92d9138ed0 in gc_mark (ptr=70068688, lev=1) at gc.c:820
#9  0x00007f92d913933e in gc_mark_children (ptr=140268499718720, lev=1) at gc.c:1063
#10 0x00007f92d9138ed0 in gc_mark (ptr=140268499718720, lev=0) at gc.c:820
#11 0x00007f92d9138f00 in rb_gc_mark (ptr=140268499718720) at gc.c:827
#12 0x00007f92d91252ef in blk_mark (data=0x3f0fb80) at eval.c:8526
#13 0x00007f92d9139297 in gc_mark_children (ptr=140268499712800, lev=2) at gc.c:1031
#14 0x00007f92d9138ed0 in gc_mark (ptr=140268499712800, lev=1) at gc.c:820
#15 0x00007f92d913933e in gc_mark_children (ptr=140268499712040, lev=1) at gc.c:1063
#16 0x00007f92d9138ed0 in gc_mark (ptr=140268499712040, lev=0) at gc.c:820
#17 0x00007f92d9138bd1 in mark_locations_array (x=0x7fffc1422300, n=175384) at gc.c:690
#18 0x00007f92d9138c2d in rb_gc_mark_locations (start=0x7fffc13fa740, end=0x7fffc1578bc8) at gc.c:703
#19 0x00007f92d913a437 in garbage_collect () at gc.c:1493
#20 0x00007f92d9137efe in ruby_xmalloc (size=26) at gc.c:153

Second one aborted with: /usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.5/lib/nokogiri/xml/node.rb:151: [BUG] rb_gc_mark(): unknown data type 0x28(0x4de0630) non object

#0  0x00000034122328a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x0000003412234085 in abort () at abort.c:92
#2  0x0000003412e30c97 in rb_bug (fmt=0x3412edc6d0 "rb_gc_mark(): unknown data type 0x%lx(0x%lx) %s") at error.c:213
#3  0x0000003412e5e3d1 in gc_mark_children (ptr=81659440, lev=2) at gc.c:1080
#4  0x0000003412e5ded0 in gc_mark (ptr=81659440, lev=1) at gc.c:820
#5  0x0000003412e5e33e in gc_mark_children (ptr=140070068033480, lev=1) at gc.c:1063
#6  0x0000003412e5ded0 in gc_mark (ptr=140070068033480, lev=0) at gc.c:820
#7  0x0000003412e5df00 in rb_gc_mark (ptr=140070068033480) at gc.c:827
#8  0x0000003412e4a2ef in blk_mark (data=0x5084c40) at eval.c:8526
#9  0x0000003412e5e297 in gc_mark_children (ptr=140070067797280, lev=1) at gc.c:1031
#10 0x0000003412e5ded0 in gc_mark (ptr=140070067797280, lev=0) at gc.c:820
#11 0x0000003412e5dbd1 in mark_locations_array (x=0x7fffa69fcdb0, n=181320) at gc.c:690
#12 0x0000003412e5dc2d in rb_gc_mark_locations (start=0x7fffa69db800, end=0x7fffa6b5eff8) at gc.c:703
#13 0x0000003412e5f437 in garbage_collect () at gc.c:1493
#14 0x0000003412e5cefe in ruby_xmalloc (size=12288) at gc.c:153

For completeness, here's all of the packages installed that match /ruby.*x86_64/:

ruby-1.8.7.352-7.el6_3.gcdebug.x86_64
ruby-augeas-0.3.0-1.el6.x86_64
ruby-devel-1.8.7.352-7.el6_3.gcdebug.x86_64
rubygem-curb-0.7.10-3.el6.x86_64
rubygem-eventmachine-0.12.10-7.el6.x86_64
rubygem-hpricot-0.8.4-2.el6.x86_64
rubygem-json-1.4.6-10.el6.x86_64
rubygem-nokogiri-1.5.5-2.el6cf.x86_64
rubygem-pg-0.11.0-5.el6.x86_64
rubygem-thin-1.2.11-3.el6.x86_64
ruby-irb-1.8.7.352-7.el6_3.gcdebug.x86_64
ruby-libs-1.8.7.352-7.el6_3.gcdebug.x86_64
ruby-rdoc-1.8.7.352-7.el6_3.gcdebug.x86_64
ruby-shadow-1.4.1-13.el6.x86_64
Comment 19 RHEL Program Management 2013-10-14 00:16:03 UTC 
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unable to address this
request at this time.

Red Hat invites you to ask your support representative to
propose this request, if appropriate, in the next release of
Red Hat Enterprise Linux.
Comment 20 Vít Ondruch 2015-09-07 13:20:22 UTC 
I am sorry, but since there is no reliable reproducer, I am closing this ticket. Please reopen if you have better reproducer.
Note You need to log in before you can comment on or make changes to this bug.