It looks that the SELinux policy does not allow ctdb to set up a public cluster IP address.

The main purpose of ctdb is to manage a cluster-wide tdb database, but additionally it has basic cluster manager functionality like starting the samba deamon and setting up and managing public cluster IP addresses. The latter seems to fail is SELinux is enabled.

Reassigning to selinux-policy.

We need to write policy for ctdb in order to allow samba to talk to it.  It also looks like ctdb is leaking an open file descriptor to anon_inodefs

#============= ifconfig_t ==============
allow ifconfig_t anon_inodefs_t:file { read write };

#============= iptables_t ==============
allow iptables_t anon_inodefs_t:file write;


/var/ctdb should be in the payload, although it would  be better if this was in /var/lib?

 /var/run/ctdbd should also be in the payload.

Created attachment 511918 [details]
Here is the initial policy.

tar xvf /tmp/ctdbd.tgz
cd /tmp/
sh ctdbd.sh
service ctdp restart

And start collecting AVC's

Created attachment 513209 [details]
audit logs when starting ctdb with initial policy

I found one typo in the initial policy in ctdbd.te, %s/ctdpd/ctdbd/g

After fixing that the policy built and installed.  Attached is the new audit logs from starting the service while it manages samba.

Created attachment 513226 [details]
Updated policy

Created attachment 513441 [details]
updated logs for updated policy

type=AVC msg=audit(1310760603.538:900): avc:  denied  { read write } for  pid=978 comm="ctdbd" name=".ctdb_socket_lock" dev=dm-0 ino=1831435 scontext=unconfined_u:system_r:ctdbd_t:s0 tcontext=unconfined_u:object_r:initrc_tmp_t:s0 tclass=file

If you stop the ctdbd service, will /tmp/.ctdb_socket_lock delete? Since it should be labeled as ctdbd_tmp_t.

We have in the ctdbd.fc

/var/ctdbd(/.*)?                gen_context(system_u:object_r:ctdbd_var_lib_t,s0)

You need to run 

chcon -R -t ctdbd_var_lib_t /var/ctdbd
 
or change it in the ctdbd.fc file and run restorecon -R -v /var/ctdb

Other problem is /mnt/ctdb0 and /mnt/share0. We need to find a label for these location.

Could you explain me what is purpose of these?

Created attachment 513949 [details]
updated ctdbd.te

Updated policy.

Also Nate,
please run

echo "-w /etc/shadow -p wa" >> /etc/audit/audit.rules
service auditd restart


Will give us full paths.

(In reply to comment #11)
> Other problem is /mnt/ctdb0 and /mnt/share0. We need to find a label for these
> location.
> 
> Could you explain me what is purpose of these?

/mnt/ctdb0 is a GFS2 file system being used for CTDB.
/mnt/share0 is a GFS2 file system being shared by Samba.

Created attachment 514018 [details]
New set of audit logs

I found one syntax error in your updated ctdbd.te

Compiling  ctdbd module
/usr/bin/checkmodule:  loading policy configuration from tmp/ctdbd.tmp
ctdbd.te":21:ERROR 'syntax error' at token 'files_spool_file' on line 4693:
type ctdbd_spool_t;
files_spool_file(ctdbd_spool_t)
/usr/bin/checkmodule:  error(s) encountered while parsing configuration
make: *** [tmp/ctdbd.mod] Error 1

I commented out the files_spool_file line since I don't think ctdb uses a spool directory.  I also fixed the paths in ctdbd.fc to point to /var/ctdb instead of /var/ctdbd.

Here are the updated audit logs with the new ctdbd.te and added auditd options.

Created attachment 514171 [details]
updated ctdbd policy

Ok, could you try to execute

# chcon -R -t samba_share_t /mnt/share0
# chcon -R -t ctdbd_var_lib_t /mnt/ctdb0
# chcon -R -t ctdbd_var_lib_t /etc/ctdb 

and test the updated policy.

You should see only leak file desciptors and samba related AVC msgs.

Created attachment 514200 [details]
Next set of audit logs

Here is the latest set of logs.  I had to comment out the line that contained shorewall_t in ctdbd.te which wasn't defined in my policy.

Great, we are close. Just

allow ctdbd_t tmp_t:file { read write open lock };

/tmp/.ctdb_socket_lock is still mislabeled. Could you remove it and try to restart ctdbd.

Created attachment 515753 [details]
Next set of audit logs

I've rebuilt the systems since last run so I've had to patch together the latest policy.  Can you send a new tarball next time?

I added fixex to F16. I believe the policy is ready for RHEL6 backport.

Fixed in selinux-policy-3.7.19-106.el6

I modified /etc/sysconfig/ctdb in such a way that CTDB_SYSLOG=yes.

----
type=SYSCALL msg=audit(08/04/2011 11:35:44.620:27248) : arch=i386 syscall=socketcall(socket) success=yes exit=20 a0=1 a1=bf9ed460 a2=945d810 a3=945d878 items=0 ppid=1 pid=10698 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=9 comm=ctdbd exe=/usr/sbin/ctdbd subj=unconfined_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(08/04/2011 11:35:44.620:27248) : avc:  denied  { create } for  pid=10698 comm=ctdbd scontext=unconfined_u:system_r:ctdbd_t:s0 tcontext=unconfined_u:system_r:ctdbd_t:s0 tclass=udp_socket 
----
type=SYSCALL msg=audit(08/04/2011 11:35:44.622:27249) : arch=i386 syscall=socketcall(sendto) success=yes exit=54 a0=b a1=bf9ed460 a2=945d810 a3=14 items=0 ppid=1 pid=10698 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=9 comm=ctdbd exe=/usr/sbin/ctdbd subj=unconfined_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(08/04/2011 11:35:44.622:27249) : avc:  denied  { write } for  pid=10698 comm=ctdbd scontext=unconfined_u:system_r:ctdbd_t:s0 tcontext=unconfined_u:system_r:ctdbd_t:s0 tclass=udp_socket 
----
type=SYSCALL msg=audit(08/04/2011 11:35:44.628:27250) : arch=i386 syscall=socketcall(bind) success=yes exit=0 a0=2 a1=bf9ed4f0 a2=945d800 a3=945a498 items=0 ppid=10698 pid=10741 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=9 comm=ctdbd exe=/usr/sbin/ctdbd subj=unconfined_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(08/04/2011 11:35:44.628:27250) : avc:  denied  { node_bind } for  pid=10741 comm=ctdbd saddr=127.0.0.1 src=4379 scontext=unconfined_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=udp_socket 
type=AVC msg=audit(08/04/2011 11:35:44.628:27250) : avc:  denied  { name_bind } for  pid=10741 comm=ctdbd src=4379 scontext=unconfined_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:ctdb_port_t:s0 tclass=udp_socket 
type=AVC msg=audit(08/04/2011 11:35:44.628:27250) : avc:  denied  { bind } for  pid=10741 comm=ctdbd scontext=unconfined_u:system_r:ctdbd_t:s0 tcontext=unconfined_u:system_r:ctdbd_t:s0 tclass=udp_socket 
----
type=SYSCALL msg=audit(08/04/2011 11:35:44.729:27251) : arch=i386 syscall=socketcall(recv) success=yes exit=73 a0=a a1=bf9dd3e0 a2=bf9ed43c a3=0 items=0 ppid=10698 pid=10741 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=9 comm=ctdbd exe=/usr/sbin/ctdbd subj=unconfined_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(08/04/2011 11:35:44.729:27251) : avc:  denied  { read } for  pid=10741 comm=ctdbd laddr=127.0.0.1 lport=4379 scontext=unconfined_u:system_r:ctdbd_t:s0 tcontext=unconfined_u:system_r:ctdbd_t:s0 tclass=udp_socket 
----

Created attachment 516789 [details]
Next set of audit logs

Latest set of audit logs from `service ctdb start` with samba managed by ctdb.  
Running selinux-policy-3.7.19-106.el6

I am fixing

allow ctdbd_t ctdb_port_t:tcp_socket name_connect;
allow smbd_t ctdbd_tmp_t:sock_file { write getattr };


But still remains some issues which needs to be fixed in ctdbd package

#672641

Fixed in selinux-policy-3.7.19-107.el6

I can't find the ctdb types in the latest selinux-policy package.

[root@dash-03 targeted]# rpm -q selinux-policy
selinux-policy-3.7.19-107.el6.noarch
[root@dash-03 targeted]# seinfo -t | grep ctdb
[root@dash-03 targeted]#

I see

# seinfo -t | grep ctdb
   ctdbd_var_lib_t
   ctdbd_var_run_t
   ctdbd_initrc_exec_t
   ctdbd_exec_t
   ctdbd_log_t
   ctdb_client_packet_t
   ctdbd_t
   ctdb_port_t
   ctdbd_tmp_t
   ctdbd_spool_t
   ctdb_server_packet_t

# semodule -l |grep ctdb
ctdbd	1.0.0

# seinfo -xaunconfined_domain_type |grep ctdbd
ctdbd_t

# rpm -qa *selinux-policy*
selinux-policy-targeted-3.7.19-107.el6.noarch
selinux-policy-minimum-3.7.19-107.el6.noarch
selinux-policy-doc-3.7.19-107.el6.noarch
selinux-policy-3.7.19-107.el6.noarch
selinux-policy-mls-3.7.19-107.el6.noarch

Could you try to reinstall the policy and make sure nothing blows up.

It appears to be a problem with the qarshd policy.  When including "qemu_domtrans_unconfined(qarshd)" it causes the policy load to fail.

Running Transaction
  Updating   : selinux-policy-3.7.19-107.el6.noarch                                                  1/4
  Updating   : selinux-policy-targeted-3.7.19-107.el6.noarch                                         2/4
libsepol.print_missing_requirements: qarshd's global requirements were not met: type/attribute unconfined_qemu_t (No such file or directory).
libsemanage.semanage_link_sandbox: Link packages failed (No such file or directory).
semodule:  Failed!
  Cleanup    : selinux-policy-targeted-3.7.19-106.el6.noarch                                         3/4
  Cleanup    : selinux-policy-3.7.19-106.el6.noarch                                                  4/4
Installed products updated.

After filtering out that domtrans I'm back in business.

FYI, there are currently 35 domtrans rules I need to filter out in order for the qarshd policy to load.  Should I file bugs for these?

Fill a new bug please. Thanks.

What is qarshd?  Why is it running unconfined virtual machines?  Does it work with svirt/libvirt?

I removed 

qemu_domtrans_unconfined()


AFAIK they try to include all domtrans interfaces into qarshd policy to make sure all these interfaces are correct.


This issue is also covered by SEWatch tool which tries to compile/load all interfaces. I just need to clean up a script which does these tests.

qarshd is the server part of our QA remote shell.  It was written to be more transparent than ssh for testing purposes.  When adding SELinux support we decided to auto-generate the policy so we wouldn't have to edit the policy ever time we start using it to test something new.

As a side effect of auto-generating the policy we find that some of the domtrans interfaces don't work and need to be filtered out to get a working policy.

Ok nice feature...

I found another command that doesn't work when selinux is enabled with clustering on in samba.  smbcontrol needs access to ctdb.socket.


type=AVC msg=audit(1317220776.455:50562): avc:  denied  { write } for  pid=11780 comm="smbcontrol" name="ctdb.socket" dev=dm-0 ino=1831426 scontext=unconfined_u:unconfined_r:smbcontrol_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:ctdbd_tmp_t:s0 tclass=sock_file

That is a different bug.

Miroslav lets add

optional_policy(`
	ctdbd_stream_connect(smbcontrol_t)
')

I think the rest of the bug reported here is fixed in the latest RHEL6 policy?

I cloned off the new AVC to a new bug.  Moving back to VERIFIED.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1511.html