Anyone!

Everett,

You must be patient.  We have many competing priorities and are working to get these bugs prioritized and resolved.  I am sorry it took a month for the bug to get to a person who is capable of properly fixing it.  That is why filing bugs via issue tracker is more effective.  Support is better at diagnosing which engineers should receive bugs.

Regards
-steve

Some more core's from aisexec.

ururim> imeall.sh "ls -ld /var/lib/openais/core*"
----tururim----
ssh tururim "ls -ld /var/lib/openais/core*"
ls: /var/lib/openais/core*: No such file or directory
----turuturu----
ssh turuturu "ls -ld /var/lib/openais/core*"
ls: /var/lib/openais/core*: No such file or directory
----uipi----
ssh uipi "ls -ld /var/lib/openais/core*"
-rw------- 1 root root 148443136 Sep 30 11:26 /var/lib/openais/core.18609
-rw------- 1 root root 141422592 Aug 27 11:41 /var/lib/openais/core.18617
-rw------- 1 root root 139124736 Sep 18 14:10 /var/lib/openais/core.18644
-rw------- 1 root root 138498048 Sep 30 15:02 /var/lib/openais/core.18671
----uiracu----
ssh uiracu "ls -ld /var/lib/openais/core*"
-rw------- 1 root root  27512832 Jul  8 17:05 /var/lib/openais/core.12890
-rw------- 1 root root 141565952 Aug 27 11:20 /var/lib/openais/core.18601
----uirapuru----
ssh uirapuru "ls -ld /var/lib/openais/core*"
-rw------- 1 root root 141709312 Aug 27 12:02 /var/lib/openais/core.18608
-rw------- 1 root root 138944512 Sep 10 18:36 /var/lib/openais/core.18625


uipi> gdb /usr/sbin/aisexec /var/lib/openais/core.18671
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
...
...
...
Loaded symbols for /usr/libexec/lcrso/service_evs.lcrso
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Core was generated by `aisexec'.
Program terminated with signal 6, Aborted.
(gdb) bt
#0  0x0000003904030215 in raise () from /lib64/libc.so.6
#1  0x0000003904031cc0 in abort () from /lib64/libc.so.6
#2  0x000000390406a7fb in __libc_message () from /lib64/libc.so.6
#3  0x00000039040e823f in __stack_chk_fail () from /lib64/libc.so.6
#4  0x0000000000411fb6 in message_handler_orf_token (instance=0x2aaaaaaae010, msg=<value optimized out>, 
    msg_len=<value optimized out>, endian_conversion_needed=<value optimized out>) at totemsrp.c:3453
#5  0x2f90000000001f6b in ?? ()
#6  0x80d600007fff4e9b in ?? ()
#7  0x19c0000000000040 in ?? ()
#8  0x00007fff4e9b2fc0 in ?? ()
#9  0x000000001f6b6850 in ?? ()
#10 0x00007fff4e9b2f90 in ?? ()
#11 0x00000000004080d6 in net_deliver_fn (handle=<value optimized out>, fd=<value optimized out>, 
    revents=<value optimized out>, data=0x9c4300007fff4e9b) at totemnet.c:676
#12 0x0000000000405b60 in poll_run (handle=0) at aispoll.c:402
#13 0x00000000004184f2 in main (argc=<value optimized out>, argv=<value optimized out>) at main.c:609
(gdb) 

uipi> gdb /usr/sbin/aisexec /var/lib/openais/core.18609
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/libexec/lcrso/objdb.lcrso...Reading symbols from /usr/lib/debug/usr/libexec/lcrso/objdb.lcrso.debug...done.
done.

...
...


#0  0x0000003904030215 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003904030215 in raise () from /lib64/libc.so.6
#1  0x0000003904031cc0 in abort () from /lib64/libc.so.6
#2  0x000000390406a7fb in __libc_message () from /lib64/libc.so.6
#3  0x00000039040e823f in __stack_chk_fail () from /lib64/libc.so.6
#4  0x0000000000411fb6 in message_handler_orf_token (instance=0x2aaaaaaae010, msg=<value optimized out>, 
    msg_len=<value optimized out>, endian_conversion_needed=<value optimized out>) at totemsrp.c:3453
#5  0xc840000000000b7f in ?? ()
#6  0x80d600007fff5276 in ?? ()
#7  0x04f0000000000040 in ?? ()
#8  0x00007fff5276c870 in ?? ()
#9  0x000000000b7fc850 in ?? ()
#10 0x00007fff5276c840 in ?? ()
#11 0x00000000004080d6 in net_deliver_fn (handle=<value optimized out>, fd=<value optimized out>, 
    revents=<value optimized out>, data=0xb7fc85000007fff) at totemnet.c:676
#12 0x0000000000405b60 in poll_run (handle=0) at aispoll.c:402
#13 0x00000000004184f2 in main (argc=<value optimized out>, argv=<value optimized out>) at main.c:609
(gdb)

Do you need anymore information?

Reassigned to Honza.

Honza,

When your investigating customer application, please run openais via valgrind to determine this issue.

Created attachment 366311 [details]
Proposed patch

I was able to reproduce very strange behavior setting high drop of mcast. I'm almost sure, that patch included will solve this problem.

Main problem is in orf_token_mcast function, when message_item's iovlen is MAXIOVS, so it will not fit to sort_queue_item (because item 0 is reserved, so we need +1 larger iovec array) and one memcpy will overwrite stack in pretty ugly way.

This patch increase the size of sort_queue_item iovec array + 1, so there should be no overwrite and no fall.

There are conditions where a multicast fills up the MAXIOVS array entries?

(In reply to comment #7)
> There are conditions where a multicast fills up the MAXIOVS array entries?  

It doesn't fills up MAXIOVS. It will refill by one MAXIOVS entries. And destroys stack.

Created attachment 368230 [details]
Proposed patch, take 2

Patch sent to ml.

Everet,
there is patch which should fix problem. Can you please test it, and report back, if it works or not?

Honza,

Patch #2 is for a different bz # (the stack protector segfault).  Do you believe it also affects this bugzilla, or should customer also test first patch in conjunction with second?

Regards
-steve

(In reply to comment #15)
> Honza,
> 
> Patch #2 is for a different bz # (the stack protector segfault).  Do you
> believe it also affects this bugzilla, or should customer also test first patch
> in conjunction with second?
> 
> Regards
> -steve  

?????

I really don't understand. Patch #1 and #2 has same effect. It's still same bugzilla, but little different solution (one is increase selected iovec array by 1 and second is increase all iovec arrays by 5).

I'll see if the folks at the Customer site can test .

Has this patch been cut for RHEL 5.3 and RHEL 5.4?
If so, do you have a link to the patches?

(In reply to comment #18)
> Has this patch been cut for RHEL 5.3 and RHEL 5.4?
> If so, do you have a link to the patches?  

It's from 5.4, but should be applicable to 5.3 without any problems.

(In reply to comment #18)
> Has this patch been cut for RHEL 5.3 and RHEL 5.4?
> If so, do you have a link to the patches?  

Everet,
I sent to John package I built, and that package contains both patches (this bz and https://bugzilla.redhat.com/show_bug.cgi?id=519159). I hope that make your life little easier.

Hi,

We were told that even tough these bugs were identified on RH 5.3 we still should install the corrected packages on RH5.4, is that so?

We've been trying to install the 3 rpm packages that solve both bugs (525280 and 519159) on a RH 5.4 machine on our cluster.

There seem to be some version conflicts:
> tururim> rpm -U openais-0.80.6-8.el5.1.jf.x86_64.rpm openais-debuginfo-0.80.6-8.el5.1.jf.x86_64.rpm openais-devel-0.80.6-8.el5.1.jf.x86_64.rpm
>
>         package openais-0.80.6-8.el5_4.1.x86_64 (which is newer than openais-0.80.6-8.el5.1.jf.x86_64) is already installed
>
>         package openais-debuginfo-0.80.6-8.el5.1.jf.x86_64 is already installed
>
> tururim> rpm -qa |grep openais
> openais-debuginfo-0.80.6-8.el5.1.jf
> openais-0.80.6-8.el5_4.1

What would be the correct procedure to upgrade these packages? Can we use the --force or --nodeps options?

I'm sorry, I overlooked and sent to you correct packages, but with bad name. Sending you and John correct one.

Regards,
  Honza

(In reply to comment #21)
> Hi,
> 
> We were told that even tough these bugs were identified on RH 5.3 we still
> should install the corrected packages on RH5.4, is that so?
> 
> We've been trying to install the 3 rpm packages that solve both bugs (525280
> and 519159) on a RH 5.4 machine on our cluster.
> 
> There seem to be some version conflicts:
> > tururim> rpm -U openais-0.80.6-8.el5.1.jf.x86_64.rpm openais-debuginfo-0.80.6-8.el5.1.jf.x86_64.rpm openais-devel-0.80.6-8.el5.1.jf.x86_64.rpm
> >
> >         package openais-0.80.6-8.el5_4.1.x86_64 (which is newer than openais-0.80.6-8.el5.1.jf.x86_64) is already installed
> >
> >         package openais-debuginfo-0.80.6-8.el5.1.jf.x86_64 is already installed
> >
> > tururim> rpm -qa |grep openais
> > openais-debuginfo-0.80.6-8.el5.1.jf
> > openais-0.80.6-8.el5_4.1
> 
> What would be the correct procedure to upgrade these packages? Can we use the
> --force or --nodeps options?

Joaquim or Everett,

Can we get some feedback on whether or not these patches resolved your issue in RHEL5.4?  We need to make a decision as to whether or not to include these in the next release of RHEL and the deadline is approaching (in the next few days).  Without your feedback we may be unable to include these patches.

Thanks

Hi,

I'm sorry to inform you that at this moment we're still waiting for the customer to install this patch on their environment.

BR

(In reply to comment #24)
> Hi,
> 
> I'm sorry to inform you that at this moment we're still waiting for the
> customer to install this patch on their environment.
> 
> BR    

It looks like the customer VIVO is not going to follow up on this issue.
Have the patches regarding, GFS2 and openais been applied to RHEL 5.4
for formal distribution?

If so, what might they be?  The way I understand it, is that the plock_ownership actually works in RHEL 5.4 and increase lock performance.  I am also under the impression that 'central_processing' has issues in RHEL 5.4 as well. So, this means we will need patches for 'openais' as noted in this bug report and if possible patches for 'central processing' option as well.  We have one customer site who is working with RHEL 5.4, so we have an opportunity to apply patches for all of these issues at this site, if those patches can be identified, and created.

Here are the 2 lines in cluster.conf that manage the settings for the features we are talking about in RHEL clustering:


	<rm central_processing="1" log_facility="local4" log_level="7">
	<gfs_controld plock_ownership="0" plock_rate_limit="0"/>

Everett

Joaquim:

Is it possible you can use Cluster #1 to test to see if this issue has been resolved?

What do you need to complete this task?

We have RedHat's attention and you should follow through on the request if at all possible.

Everettt

Everett,

Right now both Vivo's clusters are on production, so I don't think we'll be able to do any more testing on them. They're both running 5.3 using.

On the next weeks we expect to get a third cluster running and this one may be available for testing the patches, it is just too early for me to be sure when.

I'll relay this conversation to the project manager and see if we can arrange this test.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0180.html