I attempted an fs relabel, but that failed (segfaulted) and hosed up my machines. I then reinstalled all 4 to the latest tree (RHEL6.0-20100901.0), once again updated the selinux policies, and was again able to recreate this problem. 

[root@taft-01 ~]# rpm -q corosync
corosync-1.2.3-20.el6.x86_64

The core file from corosync looks like corosync is not allowed to open files via selinux.  There are AVCs in the audit log.  I have attached the audit log.

Created attachment 446037 [details]
Audit log showing the AVCs

Turn on the allow_corosync_rw_tmpfs boolean

# setsebool -P allow_corosync_rw_tmpfs 1

Dan,

Question - is this something all deployments will have to do at install time?

Regards
-steve

I don't know.  I don't setup clustering.  In reality the problem here is some random domain is creating a tmpfs_t that we don't know about.  This should not happen.  We can change the booleans default value to on, but this would only affect fresh install machines.

I attempted the setsebool listed in comment #4, and now instead of coredumping, the cmd just hangs.

Sep  8 18:19:07 taft-01 qarshd[4331]: Running cmdline: cman_tool version -r
[HUNG over night]

[root@taft-01 ~]# strace -p 4332
attach: ptrace(PTRACE_ATTACH, ...): Operation not permitted

Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x0000003bee4a666e in waitpid () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install corosynclib-1.2.3-20.el6.x86_64 glibc-2.12-1.6.el6.x86_64 libxml2-2.7.6-1.el6.x86_64 zlib-1.2.3-25.el6.x86_64
(gdb) where
#0  0x0000003bee4a666e in waitpid () from /lib64/libc.so.6
#1  0x0000003bee43e6e9 in do_system () from /lib64/libc.so.6
#2  0x00000000004032e4 in version (comline=0x7fff53d97880)
    at /usr/src/debug/cluster-3.0.12/cman/cman_tool/main.c:799
#3  0x0000000000404b4f in main (argc=<value optimized out>, 
    argv=<value optimized out>, envp=0x7fff53d97b28)
    at /usr/src/debug/cluster-3.0.12/cman/cman_tool/main.c:1250
(gdb) up
#1  0x0000003bee43e6e9 in do_system () from /lib64/libc.so.6
(gdb) up
#2  0x00000000004032e4 in version (comline=0x7fff53d97880)
    at /usr/src/debug/cluster-3.0.12/cman/cman_tool/main.c:799
799			result = system("/usr/bin/ccs_sync");


Looks like cman_tool is blocked inside waitpid.  In this case, ccs_sync is not finishing, blocking the cman_tool from finishing.

ricci is not compiled with debug symbols.  That needs to be fixed.  Here is the backtrace of ccs_sync blocking on ricci (libs?).  This causes cman_tool to block in waitpid, causing comment #7.

0x0000003bee4d41d0 in __read_nocancel () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.6.el6.x86_64 libxml2-2.7.6-1.el6.x86_64 nspr-4.8.6-1.el6.x86_64 nss-3.12.7-2.el6.x86_64 nss-softokn-3.12.7-1.1.el6.x86_64 nss-softokn-freebl-3.12.7-1.1.el6.x86_64 nss-util-3.12.7-1.el6.x86_64 sqlite-3.6.20-1.el6.x86_64 zlib-1.2.3-25.el6.x86_64
(gdb) where
#0  0x0000003bee4d41d0 in __read_nocancel () from /lib64/libc.so.6
#1  0x0000003bee471538 in _IO_new_file_underflow () from /lib64/libc.so.6
#2  0x0000003bee4676b1 in getdelim () from /lib64/libc.so.6
#3  0x0000003bee4dce52 in getpass () from /lib64/libc.so.6
#4  0x000000000040345c in ricci_conn_handler_want_send_auth ()
#5  0x0000000000402056 in main ()

btw getpass is deprecated and has been removed from any posix standards.

GETPASS(3)                 Linux Programmer’s Manual                GETPASS(3)

NAME
       getpass - get a password

SYNOPSIS
       #include <unistd.h>

       char *getpass( const char *prompt);

DESCRIPTION
       This function is obsolete.  Do not use it.
CONFORMING TO
       Present in SUSv2, but marked LEGACY.  Removed in POSIX.1-2001.

it's not hung in comment #7, it's just waiting for the initial passwd (which i didn't see any output for in our auto scripts). the original issue appears to be fixed with setsebool.

the setsebool required should probably be in tech notes somewhere.

Corey,

Regarding comment #13, I believe our plan of action at this point is to remove the boolean entirely which will remove the selinux AVCs from components connecting to corosync.  Dan can handle bz from here and we should test the zero day errata he suggests with the full stack on a clean install.

Regards
-steve

Thank you for your bug report. This issue was evaluated for inclusion
in the current release of Red Hat Enterprise Linux. Unfortunately, we
are unable to address this request in the current release. Because we
are in the final stage of Red Hat Enterprise Linux 6 development, only
significant, release-blocking issues involving serious regressions and
data corruption can be considered.

If you believe this issue meets the release blocking criteria as
defined and communicated to you by your Red Hat Support representative,
please ask your representative to file this issue as a blocker for the
current release. Otherwise, ask that it be evaluated for inclusion in
the next minor release of Red Hat Enterprise Linux.

Could it be a problem with qarshd? 

Corey, 
how did you start cman?

service cman start 

or

qarsh hostname service cman start

Miroslav we want to remove the boolean because third party apps could create the tmpfs_t files.  I was thinking we could change it to be allowed unless the unoconfined policy was disabled.

Not quite kosher but something like

optional_policy(`
	gen_require(`
		attribute unconfined_services;
	')	

	fs_manage_tmpfs_files(corosync_t)
	init_manage_script_status_files(corosync_t)
')

########################################
## <summary>
##	Manage init script
##	status files.
## </summary>
## <param name="domain">
##	<summary>
##	Domain allowed access.
##	</summary>
## </param>
#
interface(`init_manage_script_status_files',`
	gen_require(`
		type initrc_state_t;
	')

	manage_files_pattern($1, initrc_state_t, initrc_state_t)
')

Ok, it will work.

Fixed in selinux-policy-3.7.19-55.el6.noarch.

*** Bug 634557 has been marked as a duplicate of this bug. ***

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, the "allow_corosync_rw_tmpfs" boolean allowed third party applications to create, write and read generic tmpfs files. To prevent this, the boolean has been removed, and unless the unconfined policy is disabled, generic tmpfs files can now be managed using Corosync.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0526.html