Bug 767640

+++ This bug was initially created as a clone of Bug #759403 +++

Created attachment 539567 [details]
Part of /var/log/audit/audit.log.

Description of problem:
  OpenMPI job submited to parallel universe fails on generation ssh keys 
  (in /usr/libexec/condor/sshd.sh - around line 50).

  When selinux is set to 'Permissive', ssh keys are correctly created.

Version-Release number of selected component (if applicable):
  condor-7.6.5-0.8.el5.i386

How reproducible:
  100%

Steps to Reproduce:
  1. Submit OpenMPI job to paralel universe (bug 537232 comment 2)
  2. After job finish, check output file of the job and /var/log/audit/audit.log
  
Actual results:
  In output file is:
    ssh keygenerator /usr/bin/ssh-keygen returned error 0 exiting
  For audit.log see attachment.

Expected results:
  Keys are correctly created.
  
Additional info:
  About 0 printed as error code in output message is bug 759154.

--- Additional comment from tstclair on 2011-12-13 10:20:05 EST ---

Is this a new issue with recent builds or has this always existed and has recently been exposed?

--- Additional comment from dwalsh on 2011-12-13 11:07:06 EST ---

SELinux is blocking the output of the ssh-keygen command from 

/var/lib/condor/execute/dir_26678/_condor_stdout

Any reason this is not in a directory like /var/log/condor?

--- Additional comment from dwalsh on 2011-12-13 11:08:51 EST ---

Miroslav, this is another reason we should have ssh_keygen_t as an unconfined domain in RHEL5.

--- Additional comment from matt on 2011-12-13 11:14:36 EST ---

(In reply to comment #2)
> SELinux is blocking the output of the ssh-keygen command from 
> 
> /var/lib/condor/execute/dir_26678/_condor_stdout
> 
> Any reason this is not in a directory like /var/log/condor?

Yes. This is part of a job starting on an execution resource. The job's scratch space is under /var/lib/condor/execute in a dynamically created (and destroyed) directory.

--- Additional comment from dwalsh on 2011-12-13 13:52:45 EST ---

So content in /var/lib/condor/execute is all log type stuff?

If you execute

# chcon -R -t var_log_t /var/lib/condor/execute

Does your scripts work?

--- Additional comment from matt on 2011-12-13 14:02:18 EST ---

(In reply to comment #5)
> So content in /var/lib/condor/execute is all log type stuff?

There are no restrictions of what could be in the scratch dir - data files, logs, programs with exec bit set (common), named pipes, you name it.

The dir tends to be the cwd for the running job.

--- Additional comment from dwalsh on 2011-12-13 14:54:20 EST ---

Then I would suggest condor should move it to /var/run/condor in the future, if it does not need to survive a reboot.  Then we can add a label for that directory and allow processes to write inherited files with that label.

--- Additional comment from matt on 2011-12-13 15:35:02 EST ---

(In reply to comment #7)
> Then I would suggest condor should move it to /var/run/condor in the future, if
> it does not need to survive a reboot.  Then we can add a label for that
> directory and allow processes to write inherited files with that label.

Given that we'll have to remain in /var/lib/condor/execute for now, how can we proceed?

--- Additional comment from tstclair on 2011-12-13 16:40:06 EST ---

As a note re: comment #7 

The data should persist on a reboot.  There are scenarios where it would be useful.

--- Additional comment from dwalsh on 2011-12-13 16:46:34 EST ---

Number one would be to get ssh_keygen_t to be an unconfined domain as I told Miroslav.  Have you tried labeling /var/lib/condor/execute as I asked?  Did this fix the problem?

--- Additional comment from dahorak on 2011-12-14 03:00:09 EST ---

(In reply to comment #1)
> Is this a new issue with recent builds or has this always existed and has
> recently been exposed?
I tested it only on latest version for MRG 2.1 (7.6.5-0.8), so I don't know if it is new issue or no.

--- Additional comment from dahorak on 2011-12-14 05:30:16 EST ---

(In reply to comment #5)
> If you execute
> 
> # chcon -R -t var_log_t /var/lib/condor/execute
> 
> Does your scripts work?

In this case in output file is now:
  Saving the key failed: /var/lib/condor/execute/dir_14779/tmp/hostkey.
  ssh keygenerator /usr/bin/ssh-keygen returned error 0 exiting
and in error file:
  open /var/lib/condor/execute/dir_14779/tmp/hostkey failed: Permission denied.

(version condor-7.6.5-0.9.el5.i386)

--- Additional comment from mgrepl on 2011-12-14 06:20:04 EST ---

I am making ssh_keygen_t as unconfined domain as we have it in RHEL6.

--- Additional comment from matt on 2011-12-14 09:33:14 EST ---

-
mpi.sub -

universe = parallel
cmd = ompiscript
args = summpi
log = mpi.log
output = mpi.out
error = mpi.err
machine_count = 1
environment = LD_LIBRARY_PATH=/usr/lib/openmpi/1.4-gcc/lib
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT
queue
-

-
Change to ompiscript only to set OMPIDIR=/usr/lib/openmpi/1.4-gcc/bin
-

-
# ls -alZR /var/lib/condor/execute
/var/lib/condor/execute:
drwxr-xr-x  condor condor system_u:object_r:var_lib_t      .
drwxr-xr-x  condor condor system_u:object_r:var_lib_t      ..

# wc -l /var/log/audit/audit.log
141 /var/log/audit/audit.log

(condor_submit, wait for fail)

# wc -l /var/log/audit/audit.log
146 /var/log/audit/audit.log

# tail -n5 /var/log/audit/audit.log
type=AVC msg=audit(1323872136.048:101): avc:  denied  { append } for  pid=7766 comm="ssh-keygen" path="/home/matt/mpi.out" dev=dm-0 ino=1835039 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file
type=AVC msg=audit(1323872136.048:101): avc:  denied  { append } for  pid=7766 comm="ssh-keygen" path="/home/matt/mpi.err" dev=dm-0 ino=1835040 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file
type=SYSCALL msg=audit(1323872136.048:101): arch=40000003 syscall=11 success=yes exit=0 a0=8cf0bb0 a1=8ced0a8 a2=8cf0e28 a3=0 items=0 ppid=7755 pid=7766 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)
type=AVC msg=audit(1323872136.780:102): avc:  denied  { write } for  pid=7766 comm="ssh-keygen" name="tmp" dev=dm-0 ino=7110723 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:var_lib_t:s0 tclass=dir
type=SYSCALL msg=audit(1323872136.780:102): arch=40000003 syscall=5 success=no exit=-13 a0=400207e0 a1=8241 a2=180 a3=8241 items=0 ppid=7755 pid=7766 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)

# chcon -R -t var_log_t /var/lib/condor/execute
# ls -alZR /var/lib/condor/execute
/var/lib/condor/execute:
drwxr-xr-x  condor condor system_u:object_r:var_log_t      .
drwxr-xr-x  condor condor system_u:object_r:var_lib_t      ..

# wc -l /var/log/audit/audit.log
146 /var/log/audit/audit.log

(condor_submit, wait for fail)

# wc -l /var/log/audit/audit.log
151 /var/log/audit/audit.log

# tail -n5 /var/log/audit/audit.log
type=AVC msg=audit(1323872246.793:103): avc:  denied  { append } for  pid=7813 comm="ssh-keygen" path="/home/matt/mpi.out" dev=dm-0 ino=1835039 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file
type=AVC msg=audit(1323872246.793:103): avc:  denied  { append } for  pid=7813 comm="ssh-keygen" path="/home/matt/mpi.err" dev=dm-0 ino=1835040 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file
type=SYSCALL msg=audit(1323872246.793:103): arch=40000003 syscall=11 success=yes exit=0 a0=82a8bb0 a1=82a50a8 a2=82a8e28 a3=0 items=0 ppid=7802 pid=7813 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)
type=AVC msg=audit(1323872247.790:104): avc:  denied  { write } for  pid=7813 comm="ssh-keygen" name="tmp" dev=dm-0 ino=7110723 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:var_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1323872247.790:104): arch=40000003 syscall=5 success=no exit=-13 a0=400207e0 a1=8241 a2=180 a3=8241 items=0 ppid=7802 pid=7813 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)

# setenforce permissive

# wc -l /var/log/audit/audit.log
154 /var/log/audit/audit.log

# tail -n3 /var/log/audit/audit.log
type=MAC_STATUS msg=audit(1323872299.005:105): enforcing=0 old_enforcing=1 auid=0 ses=3
type=SYSCALL msg=audit(1323872299.005:105): arch=40000003 syscall=4 success=yes exit=1 a0=3 a1=bff68394 a2=1 a3=bff68394 items=0 ppid=7008 pid=7826 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=3 comm="setenforce" exe="/usr/sbin/setenforce" subj=root:system_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=USER_AVC msg=audit(1323872299.005:106): user pid=5261 uid=81 auid=4294967295 subj=system_u:system_r:system_dbusd_t:s0 msg='avc:  received setenforce notice (enforcing=0) : exe="?" (sauid=81, hostname=?, addr=?, terminal=?)'

(condor_submit, wait for success: output has "The sum of 1 to 1000 is: 500500", *accidentally run twice, audit.log contains permissive data for two runs*)

# wc -l /var/log/audit/audit.log
162 /var/log/audit/audit.log

# tail -n8 /var/log/audit/audit.log
type=AVC msg=audit(1323872353.759:107): avc:  denied  { append } for  pid=7852 comm="ssh-keygen" path="/home/matt/mpi.out" dev=dm-0 ino=1835039 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file
type=SYSCALL msg=audit(1323872353.759:107): arch=40000003 syscall=11 success=yes exit=0 a0=9eadbb0 a1=9eaa0a8 a2=9eade28 a3=0 items=0 ppid=7841 pid=7852 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)
type=AVC msg=audit(1323872354.322:108): avc:  denied  { write } for  pid=7852 comm="ssh-keygen" name="tmp" dev=dm-0 ino=7110723 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:var_log_t:s0 tclass=dir
type=AVC msg=audit(1323872354.322:108): avc:  denied  { add_name } for  pid=7852 comm="ssh-keygen" name="hostkey" scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:var_log_t:s0 tclass=dir
type=AVC msg=audit(1323872354.322:108): avc:  denied  { create } for  pid=7852 comm="ssh-keygen" name="hostkey" scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:var_log_t:s0 tclass=file
type=SYSCALL msg=audit(1323872354.322:108): arch=40000003 syscall=5 success=yes exit=3 a0=400207e0 a1=8241 a2=180 a3=8241 items=0 ppid=7841 pid=7852 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)
type=AVC msg=audit(1323872354.323:109): avc:  denied  { write } for  pid=7852 comm="ssh-keygen" path="/var/lib/condor/execute/dir_7840/tmp/hostkey" dev=dm-0 ino=7110724 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:var_log_t:s0 tclass=file
type=SYSCALL msg=audit(1323872354.323:109): arch=40000003 syscall=4 success=yes exit=1675 a0=3 a1=40040000 a2=68b a3=68b items=0 ppid=7841 pid=7852 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)
-

This run should demonstrate: 0) the output file is not always in /var/lib/condor/execute/dir_XYZ, see failures on /home/matt/mpi.out; 1) in permissive, the hostkey file is also necessary; 2) chcon to var_log_t has no impact

Side note, when the job is running (sleep 45 inserted into ompiscript) -

# ls -alZR /var/lib/condor/execute/
/var/lib/condor/execute/:
drwxr-xr-x  condor condor system_u:object_r:var_log_t      .
drwxr-xr-x  condor condor system_u:object_r:var_lib_t      ..
drwxr-xr-x  matt   matt   root:object_r:var_log_t          dir_7231

/var/lib/condor/execute/dir_7231:
drwxr-xr-x  matt   matt   root:object_r:var_log_t          .
drwxr-xr-x  condor condor system_u:object_r:var_log_t      ..
-rwx------  matt   matt   root:object_r:var_log_t          chirp.config
-rw-r--r--  condor condor root:object_r:var_log_t          .job.ad
-rw-r--r--  condor condor root:object_r:var_log_t          .machine.ad

--- Additional comment from matt on 2011-12-14 10:03:44 EST ---

# restorecon -R /var/lib/condor/execute/

# ls -alZ /var/lib/condor/execute/
drwxr-xr-x  condor condor system_u:object_r:var_lib_t      .
drwxr-xr-x  condor condor system_u:object_r:var_lib_t      ..

# setenforce enforcing

# cat > mykeygen.te
policy_module(mykeygen, 1.0)
gen_require(`
type ssh_keygen_t;
')
unconfined_domain(ssh_keygen_t)

# make -f /usr/share/selinux/devel/Makefile
Compiling targeted mykeygen module
/usr/bin/checkmodule:  loading policy configuration from tmp/mykeygen.tmp
/usr/bin/checkmodule:  policy configuration loaded
/usr/bin/checkmodule:  writing binary representation (version 6) to tmp/mykeygen.mod
Creating targeted mykeygen.pp policy package
rm tmp/mykeygen.mod.fc tmp/mykeygen.mod

# semodule -i mykeygen.pp

# getenforce 
Enforcing

# wc -l /var/log/audit/audit.log
214 /var/log/audit/audit.log

(submit, wait for success)

# wc -l /var/log/audit/audit.log
214 /var/log/audit/audit.log

This confirms that making ssh_keygen_t unconfined allows the ompiscript to run.

Note, this was done on,

# lsb_release  -a
LSB Version:	:core-4.0-ia32:core-4.0-noarch:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-ia32:printing-4.0-noarch
Distributor ID:	RedHatEnterpriseServer
Description:	Red Hat Enterprise Linux Server release 5.7 (Tikanga)
Release:	5.7
Codename:	Tikanga

With, these installed,

# rpm -qa | grep selinux
libselinux-devel-1.33.4-5.7.el5
libselinux-utils-1.33.4-5.7.el5
libselinux-python-1.33.4-5.7.el5
selinux-policy-2.4.6-316.el5
selinux-policy-targeted-2.4.6-316.el5
selinux-policy-devel-2.4.6-316.el5
libselinux-1.33.4-5.7.el5

Open question: when the ssh_keygen_t policy introduced the breakage.