759403 – Selinux disallow creating ssh keys for OpenMPI job (sshd.sh script)

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 759403 - Selinux disallow creating ssh keys for OpenMPI job (sshd.sh script)

Summary: Selinux disallow creating ssh keys for OpenMPI job (sshd.sh script)

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	selinux-policy
Sub Component:
Version:	6.2
Hardware:	All
OS:	Linux
Priority:	medium
Severity:	unspecified
Target Milestone:	rc
Target Release:	6.3
Assignee:	Miroslav Grepl
QA Contact:	Daniel Horák
Docs Contact:
URL:
Whiteboard:
Depends On:	767633 767640
Blocks:	784580 788658
TreeView+	depends on / blocked

Reported:	2011-12-02 10:15 UTC by Daniel Horák
Modified:	2012-06-20 12:29 UTC (History)
CC List:	10 users (show)
Fixed In Version:	N/A
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	767633 767640 784580 (view as bug list)
Environment:
Last Closed:	2012-06-20 12:29:19 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
Part of /var/log/audit/audit.log. (1.37 KB, text/plain) 2011-12-02 10:15 UTC, Daniel Horák	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	767633	1	None	None	None	2021-01-20 06:05:38 UTC
Red Hat Bugzilla	767640	0	high	CLOSED	EL6 selinux-policy ssh_keygen_t prevents OpenMPI job from running	2021-02-22 00:41:40 UTC
Red Hat Product Errata	RHBA-2012:0780	0	normal	SHIPPED_LIVE	selinux-policy bug fix and enhancement update	2012-06-19 20:34:59 UTC

Internal Links: 767633 767640

Description Daniel Horák 2011-12-02 10:15:21 UTC

Created attachment 539567 [details]
Part of /var/log/audit/audit.log.

Description of problem:
  OpenMPI job submited to parallel universe fails on generation ssh keys 
  (in /usr/libexec/condor/sshd.sh - around line 50).

  When selinux is set to 'Permissive', ssh keys are correctly created.

Version-Release number of selected component (if applicable):
  condor-7.6.5-0.8.el5.i386

How reproducible:
  100%

Steps to Reproduce:
  1. Submit OpenMPI job to paralel universe (bug 537232 comment 2)
  2. After job finish, check output file of the job and /var/log/audit/audit.log
  
Actual results:
  In output file is:
    ssh keygenerator /usr/bin/ssh-keygen returned error 0 exiting
  For audit.log see attachment.

Expected results:
  Keys are correctly created.
  
Additional info:
  About 0 printed as error code in output message is bug 759154.

Comment 1 Timothy St. Clair 2011-12-13 15:20:05 UTC

Is this a new issue with recent builds or has this always existed and has recently been exposed?

Comment 2 Daniel Walsh 2011-12-13 16:07:06 UTC

SELinux is blocking the output of the ssh-keygen command from 

/var/lib/condor/execute/dir_26678/_condor_stdout

Any reason this is not in a directory like /var/log/condor?

Comment 3 Daniel Walsh 2011-12-13 16:08:51 UTC

Miroslav, this is another reason we should have ssh_keygen_t as an unconfined domain in RHEL5.

Comment 4 Matthew Farrellee 2011-12-13 16:14:36 UTC

(In reply to comment #2)
> SELinux is blocking the output of the ssh-keygen command from 
> 
> /var/lib/condor/execute/dir_26678/_condor_stdout
> 
> Any reason this is not in a directory like /var/log/condor?

Yes. This is part of a job starting on an execution resource. The job's scratch space is under /var/lib/condor/execute in a dynamically created (and destroyed) directory.

Comment 5 Daniel Walsh 2011-12-13 18:52:45 UTC

So content in /var/lib/condor/execute is all log type stuff?

If you execute

# chcon -R -t var_log_t /var/lib/condor/execute

Does your scripts work?

Comment 6 Matthew Farrellee 2011-12-13 19:02:18 UTC

(In reply to comment #5)
> So content in /var/lib/condor/execute is all log type stuff?

There are no restrictions of what could be in the scratch dir - data files, logs, programs with exec bit set (common), named pipes, you name it.

The dir tends to be the cwd for the running job.

Comment 7 Daniel Walsh 2011-12-13 19:54:20 UTC

Then I would suggest condor should move it to /var/run/condor in the future, if it does not need to survive a reboot.  Then we can add a label for that directory and allow processes to write inherited files with that label.

Comment 8 Matthew Farrellee 2011-12-13 20:35:02 UTC

(In reply to comment #7)
> Then I would suggest condor should move it to /var/run/condor in the future, if
> it does not need to survive a reboot.  Then we can add a label for that
> directory and allow processes to write inherited files with that label.

Given that we'll have to remain in /var/lib/condor/execute for now, how can we proceed?

Comment 9 Timothy St. Clair 2011-12-13 21:40:06 UTC

As a note re: comment #7 

The data should persist on a reboot.  There are scenarios where it would be useful.

Comment 10 Daniel Walsh 2011-12-13 21:46:34 UTC

Number one would be to get ssh_keygen_t to be an unconfined domain as I told Miroslav.  Have you tried labeling /var/lib/condor/execute as I asked?  Did this fix the problem?

Comment 11 Daniel Horák 2011-12-14 08:00:09 UTC

(In reply to comment #1)
> Is this a new issue with recent builds or has this always existed and has
> recently been exposed?
I tested it only on latest version for MRG 2.1 (7.6.5-0.8), so I don't know if it is new issue or no.

Comment 12 Daniel Horák 2011-12-14 10:30:16 UTC

(In reply to comment #5)
> If you execute
> 
> # chcon -R -t var_log_t /var/lib/condor/execute
> 
> Does your scripts work?

In this case in output file is now:
  Saving the key failed: /var/lib/condor/execute/dir_14779/tmp/hostkey.
  ssh keygenerator /usr/bin/ssh-keygen returned error 0 exiting
and in error file:
  open /var/lib/condor/execute/dir_14779/tmp/hostkey failed: Permission denied.

(version condor-7.6.5-0.9.el5.i386)

Comment 13 Miroslav Grepl 2011-12-14 11:20:04 UTC

I am making ssh_keygen_t as unconfined domain as we have it in RHEL6.

Comment 14 Matthew Farrellee 2011-12-14 14:33:14 UTC

-
mpi.sub -

universe = parallel
cmd = ompiscript
args = summpi
log = mpi.log
output = mpi.out
error = mpi.err
machine_count = 1
environment = LD_LIBRARY_PATH=/usr/lib/openmpi/1.4-gcc/lib
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT
queue
-

-
Change to ompiscript only to set OMPIDIR=/usr/lib/openmpi/1.4-gcc/bin
-

-
# ls -alZR /var/lib/condor/execute
/var/lib/condor/execute:
drwxr-xr-x  condor condor system_u:object_r:var_lib_t      .
drwxr-xr-x  condor condor system_u:object_r:var_lib_t      ..

# wc -l /var/log/audit/audit.log
141 /var/log/audit/audit.log

(condor_submit, wait for fail)

# wc -l /var/log/audit/audit.log
146 /var/log/audit/audit.log

# tail -n5 /var/log/audit/audit.log
type=AVC msg=audit(1323872136.048:101): avc:  denied  { append } for  pid=7766 comm="ssh-keygen" path="/home/matt/mpi.out" dev=dm-0 ino=1835039 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file
type=AVC msg=audit(1323872136.048:101): avc:  denied  { append } for  pid=7766 comm="ssh-keygen" path="/home/matt/mpi.err" dev=dm-0 ino=1835040 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file
type=SYSCALL msg=audit(1323872136.048:101): arch=40000003 syscall=11 success=yes exit=0 a0=8cf0bb0 a1=8ced0a8 a2=8cf0e28 a3=0 items=0 ppid=7755 pid=7766 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)
type=AVC msg=audit(1323872136.780:102): avc:  denied  { write } for  pid=7766 comm="ssh-keygen" name="tmp" dev=dm-0 ino=7110723 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:var_lib_t:s0 tclass=dir
type=SYSCALL msg=audit(1323872136.780:102): arch=40000003 syscall=5 success=no exit=-13 a0=400207e0 a1=8241 a2=180 a3=8241 items=0 ppid=7755 pid=7766 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)

# chcon -R -t var_log_t /var/lib/condor/execute
# ls -alZR /var/lib/condor/execute
/var/lib/condor/execute:
drwxr-xr-x  condor condor system_u:object_r:var_log_t      .
drwxr-xr-x  condor condor system_u:object_r:var_lib_t      ..

# wc -l /var/log/audit/audit.log
146 /var/log/audit/audit.log

(condor_submit, wait for fail)

# wc -l /var/log/audit/audit.log
151 /var/log/audit/audit.log

# tail -n5 /var/log/audit/audit.log
type=AVC msg=audit(1323872246.793:103): avc:  denied  { append } for  pid=7813 comm="ssh-keygen" path="/home/matt/mpi.out" dev=dm-0 ino=1835039 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file
type=AVC msg=audit(1323872246.793:103): avc:  denied  { append } for  pid=7813 comm="ssh-keygen" path="/home/matt/mpi.err" dev=dm-0 ino=1835040 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file
type=SYSCALL msg=audit(1323872246.793:103): arch=40000003 syscall=11 success=yes exit=0 a0=82a8bb0 a1=82a50a8 a2=82a8e28 a3=0 items=0 ppid=7802 pid=7813 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)
type=AVC msg=audit(1323872247.790:104): avc:  denied  { write } for  pid=7813 comm="ssh-keygen" name="tmp" dev=dm-0 ino=7110723 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:var_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1323872247.790:104): arch=40000003 syscall=5 success=no exit=-13 a0=400207e0 a1=8241 a2=180 a3=8241 items=0 ppid=7802 pid=7813 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)

# setenforce permissive

# wc -l /var/log/audit/audit.log
154 /var/log/audit/audit.log

# tail -n3 /var/log/audit/audit.log
type=MAC_STATUS msg=audit(1323872299.005:105): enforcing=0 old_enforcing=1 auid=0 ses=3
type=SYSCALL msg=audit(1323872299.005:105): arch=40000003 syscall=4 success=yes exit=1 a0=3 a1=bff68394 a2=1 a3=bff68394 items=0 ppid=7008 pid=7826 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=3 comm="setenforce" exe="/usr/sbin/setenforce" subj=root:system_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=USER_AVC msg=audit(1323872299.005:106): user pid=5261 uid=81 auid=4294967295 subj=system_u:system_r:system_dbusd_t:s0 msg='avc:  received setenforce notice (enforcing=0) : exe="?" (sauid=81, hostname=?, addr=?, terminal=?)'

(condor_submit, wait for success: output has "The sum of 1 to 1000 is: 500500", *accidentally run twice, audit.log contains permissive data for two runs*)

# wc -l /var/log/audit/audit.log
162 /var/log/audit/audit.log

# tail -n8 /var/log/audit/audit.log
type=AVC msg=audit(1323872353.759:107): avc:  denied  { append } for  pid=7852 comm="ssh-keygen" path="/home/matt/mpi.out" dev=dm-0 ino=1835039 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file
type=SYSCALL msg=audit(1323872353.759:107): arch=40000003 syscall=11 success=yes exit=0 a0=9eadbb0 a1=9eaa0a8 a2=9eade28 a3=0 items=0 ppid=7841 pid=7852 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)
type=AVC msg=audit(1323872354.322:108): avc:  denied  { write } for  pid=7852 comm="ssh-keygen" name="tmp" dev=dm-0 ino=7110723 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:var_log_t:s0 tclass=dir
type=AVC msg=audit(1323872354.322:108): avc:  denied  { add_name } for  pid=7852 comm="ssh-keygen" name="hostkey" scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:var_log_t:s0 tclass=dir
type=AVC msg=audit(1323872354.322:108): avc:  denied  { create } for  pid=7852 comm="ssh-keygen" name="hostkey" scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:var_log_t:s0 tclass=file
type=SYSCALL msg=audit(1323872354.322:108): arch=40000003 syscall=5 success=yes exit=3 a0=400207e0 a1=8241 a2=180 a3=8241 items=0 ppid=7841 pid=7852 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)
type=AVC msg=audit(1323872354.323:109): avc:  denied  { write } for  pid=7852 comm="ssh-keygen" path="/var/lib/condor/execute/dir_7840/tmp/hostkey" dev=dm-0 ino=7110724 scontext=root:system_r:ssh_keygen_t:s0 tcontext=root:object_r:var_log_t:s0 tclass=file
type=SYSCALL msg=audit(1323872354.323:109): arch=40000003 syscall=4 success=yes exit=1675 a0=3 a1=40040000 a2=68b a3=68b items=0 ppid=7841 pid=7852 auid=0 uid=501 gid=501 euid=501 suid=501 fsuid=501 egid=501 sgid=501 fsgid=501 tty=(none) ses=1 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=root:system_r:ssh_keygen_t:s0 key=(null)
-

This run should demonstrate: 0) the output file is not always in /var/lib/condor/execute/dir_XYZ, see failures on /home/matt/mpi.out; 1) in permissive, the hostkey file is also necessary; 2) chcon to var_log_t has no impact

Side note, when the job is running (sleep 45 inserted into ompiscript) -

# ls -alZR /var/lib/condor/execute/
/var/lib/condor/execute/:
drwxr-xr-x  condor condor system_u:object_r:var_log_t      .
drwxr-xr-x  condor condor system_u:object_r:var_lib_t      ..
drwxr-xr-x  matt   matt   root:object_r:var_log_t          dir_7231

/var/lib/condor/execute/dir_7231:
drwxr-xr-x  matt   matt   root:object_r:var_log_t          .
drwxr-xr-x  condor condor system_u:object_r:var_log_t      ..
-rwx------  matt   matt   root:object_r:var_log_t          chirp.config
-rw-r--r--  condor condor root:object_r:var_log_t          .job.ad
-rw-r--r--  condor condor root:object_r:var_log_t          .machine.ad

Comment 15 Matthew Farrellee 2011-12-14 15:03:44 UTC

# restorecon -R /var/lib/condor/execute/

# ls -alZ /var/lib/condor/execute/
drwxr-xr-x  condor condor system_u:object_r:var_lib_t      .
drwxr-xr-x  condor condor system_u:object_r:var_lib_t      ..

# setenforce enforcing

# cat > mykeygen.te
policy_module(mykeygen, 1.0)
gen_require(`
type ssh_keygen_t;
')
unconfined_domain(ssh_keygen_t)

# make -f /usr/share/selinux/devel/Makefile
Compiling targeted mykeygen module
/usr/bin/checkmodule:  loading policy configuration from tmp/mykeygen.tmp
/usr/bin/checkmodule:  policy configuration loaded
/usr/bin/checkmodule:  writing binary representation (version 6) to tmp/mykeygen.mod
Creating targeted mykeygen.pp policy package
rm tmp/mykeygen.mod.fc tmp/mykeygen.mod

# semodule -i mykeygen.pp

# getenforce 
Enforcing

# wc -l /var/log/audit/audit.log
214 /var/log/audit/audit.log

(submit, wait for success)

# wc -l /var/log/audit/audit.log
214 /var/log/audit/audit.log

This confirms that making ssh_keygen_t unconfined allows the ompiscript to run.

Note, this was done on,

# lsb_release  -a
LSB Version:	:core-4.0-ia32:core-4.0-noarch:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-ia32:printing-4.0-noarch
Distributor ID:	RedHatEnterpriseServer
Description:	Red Hat Enterprise Linux Server release 5.7 (Tikanga)
Release:	5.7
Codename:	Tikanga

With, these installed,

# rpm -qa | grep selinux
libselinux-devel-1.33.4-5.7.el5
libselinux-utils-1.33.4-5.7.el5
libselinux-python-1.33.4-5.7.el5
selinux-policy-2.4.6-316.el5
selinux-policy-targeted-2.4.6-316.el5
selinux-policy-devel-2.4.6-316.el5
libselinux-1.33.4-5.7.el5

Open question: when the ssh_keygen_t policy introduced the breakage.

Comment 27 Daniel Horák 2012-02-23 14:23:42 UTC

Tested and verified on RHEL 6.3 i386 and x86_64 with following packages:
# rpm -qa | grep selinux-policy
  selinux-policy-targeted-3.7.19-137.el6.noarch
  selinux-policy-3.7.19-137.el6.noarch

Whole test report is in Bug 788658 Comment 7.

>>> VERIFIED

Comment 29 errata-xmlrpc 2012-06-20 12:29:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0780.html

Note You need to log in before you can comment on or make changes to this bug.