Bug 992726

Summary: Numerous SELinux issues on Condor
Product: [Fedora] Fedora Reporter: Miroslav Grepl <mgrepl>
Component: condorAssignee: Timothy St. Clair <tstclair>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 19CC: bbockelm, b.m.a.g.piette, dominick.grift, dwalsh, ltoscano, matt, mgrepl, tomspur, tstclair
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: condor-8.1.0-0.3 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 984061 Environment:
Last Closed: 2015-02-18 11:13:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 984061    
Bug Blocks:    

Description Miroslav Grepl 2013-08-05 06:03:58 UTC
+++ This bug was initially created as a clone of Bug #984061 +++

Description of problem:
F18 SELinux policy is preventing updated package from running.

Version-Release number of selected component (if applicable):
F18 & F19 Latest

How reproducible:
100%

Steps to Reproduce:
1. Download and install latest condor from koji-testing (http://koji.fedoraproject.org/koji/buildinfo?buildID=428879) 
2. systemctl start condor.service
3. warnings/errors depending on you level

Actual results:
Numerous warnings and errors

Expected results:
Starts up

--- Additional comment from Timothy St. Clair on 2013-07-12 12:12:54 EDT ---

See also: https://bugzilla.redhat.com/show_bug.cgi?id=920543

--- Additional comment from Miroslav Grepl on 2013-07-16 09:25:01 EDT ---

What AVC are you getting?

--- Additional comment from Timothy St. Clair on 2013-07-22 11:54:03 EDT ---



--- Additional comment from Timothy St. Clair on 2013-07-22 12:39:16 EDT ---

Contains the startup log, but I'm guessing there are more, but this is the first batch.

--- Additional comment from Timothy St. Clair on 2013-07-24 12:31:04 EDT ---

Any traction here?  I would like to push out a condor update to F19, and there have been other bug reports coming in.

--- Additional comment from Daniel Walsh on 2013-07-24 17:27:17 EDT ---

ce1e00335b8ac8248370ce5fb512b4a8f62f1760 allows this in git.

--- Additional comment from Bernard Piette on 2013-07-25 06:54:22 EDT ---

Various error logs when trying to install condor-8.1.0-0.2.

--- Additional comment from Timothy St. Clair on 2013-07-25 09:53:54 EDT ---

Bernard - re: comment #7 

Are your logs captured from an updated policy in comment #6?

---------------------------------------------------------------
Miroslav -

We do this dance every time there is a policy update.  Doesn't it make sense to have a series of "troublesome" packages to eval against prior to updating policy?  I would be happy to write up the process for a very simple smoke test.  Or if you had some type of test-bed I would be happy to try and jig into it.

--- Additional comment from Bernard Piette on 2013-07-25 10:05:40 EDT ---

(In reply to Timothy St. Clair from comment #8)
> Bernard - re: comment #7 
> 
> Are your logs captured from an updated policy in comment #6?
> 
> ---------------------------------------------------------------
> Miroslav -
> 
> We do this dance every time there is a policy update.  Doesn't it make sense
> to have a series of "troublesome" packages to eval against prior to updating
> policy?  I would be happy to write up the process for a very simple smoke
> test.  Or if you had some type of test-bed I would be happy to try and jig
> into it.

Miroslav,

I am new to Bugzilla. I maintain 200 Fedora PCs running condor, so can test things easily. What does comment 6 mean? What is that updated policy or how do I get it installed?

Bernard

--- Additional comment from Miroslav Grepl on 2013-07-26 05:06:36 EDT ---

It meas it has been fixed on 

https://git.fedorahosted.org/git/selinux-policy.git

and will be a part of the next selinux-policy F19 build.

--- Additional comment from Bernard Piette on 2013-07-30 11:54:12 EDT ---

(In reply to Miroslav Grepl from comment #10)
> It meas it has been fixed on 
> 
> https://git.fedorahosted.org/git/selinux-policy.git
> 
> and will be a part of the next selinux-policy F19 build.

I did one more test with the following selinux rpms installed: 

libselinux-devel-2.1.13-15.fc19.x86_64
selinux-policy-doc-3.12.1-66.fc19.noarch
libselinux-2.1.13-15.fc19.x86_64
libselinux-utils-2.1.13-15.fc19.x86_64
libselinux-python-2.1.13-15.fc19.x86_64
selinux-policy-devel-3.12.1-66.fc19.noarch
selinux-policy-3.12.1-66.fc19.noarch
libselinux-2.1.13-15.fc19.i686
selinux-policy-targeted-3.12.1-66.fc19.noarch

# rpm -Uvh condor-8.1.0-0.2.fc19.x86_64.rpm condor-classads-8.1.0-0.2.fc19.x86_64.rpm condor-procd-8.1.0-0.2.fc19.x86_64.rpm

Updating / installing...
   1:condor-procd-8.1.0-0.2.fc19      ################################# [ 33%]
   2:condor-classads-8.1.0-0.2.fc19   ################################# [ 67%]
   3:condor-8.1.0-0.2.fc19            ################################# [100%]
libsemanage.dbase_llist_set: record not found in the database
libsemanage.dbase_llist_set: could not set record value
Could not change boolean condor_domain_can_network_connect
Could not change policy booleans


Then 
# systemctl start condor
# tail /var/log/messages
Jul 30 16:53:01 hopf systemd[1]: Starting Condor Distributed High-Throughput-Computing...
Jul 30 16:53:01 hopf systemd[1]: Started Condor Distributed High-Throughput-Computing.
Jul 30 16:53:01 hopf systemd[1]: condor.service: main process exited, code=exited, status=4/NOPERMISSION
Jul 30 16:53:01 hopf systemd[1]: condor.service: control process exited, code=exited status=1
Jul 30 16:53:01 hopf systemd[1]: Unit condor.service entered failed state.
Jul 30 16:53:01 hopf condor_master[3501]: ERROR "Failed to determine my IP address using NETWORK_INTERFACE=*" at line 232 in file /builddir/build/BUILD/condor-8.1.0/src/condor_utils/my_hostname.cpp
Jul 30 16:53:01 hopf condor_off[3504]: Can't find address for local master
Jul 30 16:53:01 hopf condor_off[3504]: Perhaps you need to query another pool.

Bernard

--- Additional comment from Fedora Update System on 2013-08-02 09:28:10 EDT ---

selinux-policy-3.12.1-69.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/selinux-policy-3.12.1-69.fc19

--- Additional comment from Fedora Update System on 2013-08-02 17:54:13 EDT ---

Package selinux-policy-3.12.1-69.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing selinux-policy-3.12.1-69.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-14089/selinux-policy-3.12.1-69.fc19
then log in and leave karma (feedback).

--- Additional comment from Bernard Piette on 2013-08-03 12:12:15 EDT ---

After installing selinux-policy-3.12.1-69.fc19:

hopf:# rpm -Uvh condor-8.1.0-0.2.fc19.x86_64.rpm condor-classads-8.1.0-0.2.fc19.x86_64.rpm condor_error.txt condor-procd-8.1.0-0.2.fc19.x86_64.rpm

Preparing...                          ################################# [100%]
Updating / installing...
   1:condor-procd-8.1.0-0.2.fc19      ################################# [ 33%]
   2:condor-classads-8.1.0-0.2.fc19   ################################# [ 67%]
   3:condor-8.1.0-0.2.fc19            ################################# [100%]
libsemanage.dbase_llist_set: record not found in the database
libsemanage.dbase_llist_set: could not set record value
Could not change boolean condor_domain_can_network_connect
Could not change policy booleans


SO there still is an selinux problem. with the boolean condor_domain_can_network_connect.

This being said, when I start condor :

hopf:# systemctl start condor

the daemon now start, but not the condor_schedd:

hopf:# ps aux | fgrep condor
condor   23108  0.0  0.0  88672  4368 ?        Ss   17:03   0:00 /usr/sbin/condor_master -f
root     23109  0.0  0.0  23336  3620 ?        S    17:03   0:00 condor_procd -A /var/run/condor/procd_pipe -R 10000000 -S 60 -C 988
condor   23112  0.0  0.0  88600  4380 ?        Ss   17:03   0:00 condor_collector -f
root     23176  0.0  0.0 107960   676 pts/2    S+   17:09   0:00 fgrep --color=auto condor

hopf:# tail -f /var/log/messages
Aug  3 17:03:47 hopf systemd[1]: Starting Condor Distributed High-Throughput-Computing...
Aug  3 17:03:47 hopf systemd[1]: Started Condor Distributed High-Throughput-Computing.

hopf:# condor_q
Error: 

Extra Info: You probably saw this error because the condor_schedd is not 
running on the machine you are trying to query. If the condor_schedd is not 
running, the Condor system will not be able to find an address and port to 
connect to and satisfy this request. Please make sure the Condor daemons are 
running and try again.
 
Extra Info: If the condor_schedd is running on the machine you are trying to 
query and you still see the error, the most likely cause is that you have 
setup a personal Condor, you have not defined SCHEDD_NAME in your 
condor_config file, and something is wrong with your SCHEDD_ADDRESS_FILE 
setting. You must define either or both of those settings in your config 
file, or you must use the -name option to condor_q. Please see the Condor 
manual for details on SCHEDD_NAME and SCHEDD_ADDRESS_FILE. 

So some progress has been made, but there are still problems.

Bernard

--- Additional comment from Fedora Update System on 2013-08-04 18:59:02 EDT ---

selinux-policy-3.12.1-69.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

--- Additional comment from Miroslav Grepl on 2013-08-05 02:03:03 EDT ---

Fixed in selinux-policy-3.12.1-70.fc19.

The problem is there is no  condor_domain_can_network_connect boolean but condor_tcp_network_connect.

Comment 1 Miroslav Grepl 2013-08-05 06:06:01 UTC
Could you please change a rpm scriptlet to use 

condor_tcp_network_connect

boolean.

Comment 2 Bernard Piette 2013-08-08 19:35:35 UTC
Using a clean system, I have installed selinux-policy-3.12.1-69.fc19 and condor-8.1.0-0.2.fc19. I then had to turn condor_tcp_network_connect on by hand.

After that the condor daemons did not start yet. I had to compile and install the following 4 selinux modules which I think should be installed by the condor rpm:

condor_master_fix.te:
#############################################################
module condor_master_fix 1.0;

require {
	type initrc_tmp_t;
	type condor_log_t;
	type condor_master_t;
	type net_conf_t;
	type condor_schedd_tmp_t;
	type krb5_conf_t;
	type condor_startd_tmp_t;
	type proc_t;
	class capability fowner;
	class dir getattr;
	class file { rename setattr read write getattr unlink open };
}

#============= condor_master_t ==============

allow condor_master_t condor_log_t:file { write rename unlink read setattr };
allow condor_master_t condor_schedd_tmp_t:dir getattr;
allow condor_master_t condor_startd_tmp_t:dir getattr;
allow condor_master_t initrc_tmp_t:dir getattr;
allow condor_master_t krb5_conf_t:file getattr;
allow condor_master_t net_conf_t:file { read getattr open };
allow condor_master_t proc_t:file { read getattr open };
allow condor_master_t self:capability fowner;
#############################################################

condor_schedd_fix.te:
#############################################################
module condor_schedd_fix 1.0;

require {
	type condor_log_t;
	type condor_master_t;
	type condor_schedd_tmp_t;
	type nfs_t;
	type etc_runtime_t;
	type home_root_t;
	type condor_schedd_t;
	class capability fowner;
	class file { read link append setattr };
	class dir { read getattr open search };
}

#============= condor_schedd_t ==============
allow condor_schedd_t condor_log_t:file read;
allow condor_schedd_t condor_log_t:file setattr;
allow condor_schedd_t etc_runtime_t:dir { read getattr open };
allow condor_schedd_t etc_runtime_t:file { link append };
allow condor_schedd_t home_root_t:dir search;
allow condor_schedd_t nfs_t:dir search;
allow condor_schedd_t self:capability fowner;
#############################################################

condor_collector_fix.te :
#############################################################
module condor_collector_fix 1.0;

require {
	type condor_collector_t;
	type condor_log_t;
        type etc_runtime_t;
        class dir { getattr search };
	class file { write rename unlink setattr };
}

#============= condor_collector_t ==============
allow condor_collector_t condor_log_t:file unlink;
allow condor_collector_t condor_log_t:file { write rename setattr };
allow condor_collector_t etc_runtime_t:dir { getattr search };
#############################################################

condor_negotiator_fix.te :
#############################################################
module condor_negotiator_fix 1.0;

require {
	type condor_negotiator_t;
	type condor_log_t;
        type etc_runtime_t;
	class file { setattr write };
        class dir { add_name search write };
}

#============= condor_negotiator_t ==============
allow condor_negotiator_t condor_log_t:file { setattr write };
allow condor_negotiator_t etc_runtime_t:dir { add_name search write };
allow condor_negotiator_t etc_runtime_t:file write;
#############################################################

After restarting condor, all the condor daemins runs now, but submitting a condor job still fails:

% condor_submit condor_job.txt

ERROR: Can't find address of local schedd



There are no errors in the system log files nor the condor ones.

The default configuration files is thus not functional yet.

Comment 3 Daniel Walsh 2013-08-09 12:42:15 UTC
Can you attach the audit log that you used to generate that policy.

Comment 4 Bernard Piette 2013-08-09 12:55:36 UTC
Unfortunately I do not have a detailed log of what I did. I proceeded by first creating the following script file:

MakeFixMod.csh:
#########################################
#!/bin/tcsh

if(("$1" == "") || ("$2" == "")) then
  echo "SYNTAX: MakeFixMod.csh module_name "
  echo "Example   MakeMod.csh condor_master"
endif

set MOD=$1

grep $MOD /var/log/audit/audit.log | audit2allow -m $MOD"_fix" > $MOD"_fix.te
checkmodule -M -m $MOD"_fix.te" -o $MOD"_fix.mod"
semodule_package -o $MOD"_fix.pp" -m $MOD"_fix.mod"
semodule -i $MOD"_fix.pp"
##########################################

then :

STEP A:
# systemctl start condor.service
# MakeFixMod.csh condor_collector 
# MakeFixMod.csh condor_master
# MakeFixMod.csh condor_schedd
# MakeFixMod.csh condor_negitiator

STEP B:
# systemctl restart condor.service
# grep condor /var/log/audit/audit.log | audit2allow

check for other policies that still need to be set and add them to the for created condor policies, compile the modified policy files into modules and load them. Then go back to STEP B until no more policies were needed. 

As I did this, condor managed to go a fit further each time, hence the need to do this iteratively.

I hope this answers your question.

Comment 5 Miroslav Grepl 2013-08-19 12:12:39 UTC
Could you attach compressed /var/log/audit/audit.log (or mail me).

Comment 6 Bernard Piette 2013-08-22 10:45:15 UTC
Condor 8.1.0-0.2 still does not work.

There are still various selinux issues, but with selinux turned off, condor does not work either.

Installed packages: condor-procd-8.1.0-0.2.fc19.x86_64 condor-classads-8.1.0-0.2.fc19.x86_64 condor-8.1.0-0.2.fc19.x86_64

Using the default installation and
% setenforce 0
% systemctl start condor

% ps aux | fgrep condor 
condor   28342  0.0  0.0  89024  6108 ?        Ss   11:15   0:00 /usr/sbin/condor_master -f
root     28343  0.0  0.0  23460  4104 ?        S    11:15   0:00 condor_procd -A /var/run/condor/procd_pipe -R 10000000 -S 60 -C 988
condor   28344  0.0  0.0  89288  6308 ?        Ss   11:15   0:00 condor_collector -f
condor   28345  0.0  0.0  89500  6488 ?        Ss   11:15   0:00 condor_negotiator -f
condor   28347  0.0  0.0  89844  6696 ?        Ss   11:15   0:00 condor_startd -f
root     28591  0.0  0.0 107960   680 pts/7    S+   11:33   0:00 fgrep --color=auto condor

(In other words the scheduler does not starts. )

% tail /var/log/condor/SchedLog
08/22/13 11:35:31 (pid:28691) ******************************************************
08/22/13 11:35:31 (pid:28691) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
08/22/13 11:35:31 (pid:28691) ** /usr/sbin/condor_schedd
08/22/13 11:35:31 (pid:28691) ** SubsystemInfo: name=SCHEDD type=SCHEDD(5) class=DAEMON(1)
08/22/13 11:35:31 (pid:28691) ** Configuration: subsystem:SCHEDD local:<NONE> class:DAEMON
08/22/13 11:35:31 (pid:28691) ** $CondorVersion: 8.1.0 Jul 15 2013 BuildID: RH-8.1.0-0.2.fc19 PRE-RELEASE-UWCS $
08/22/13 11:35:31 (pid:28691) ** $CondorPlatform: X86_64-Fedora_19 $
08/22/13 11:35:31 (pid:28691) ** PID = 28691
08/22/13 11:35:31 (pid:28691) ** Log last touched 8/22 11:35:18
08/22/13 11:35:31 (pid:28691) ******************************************************
08/22/13 11:35:31 (pid:28691) Using config source: /etc/condor/condor_config
08/22/13 11:35:31 (pid:28691) Using local config sources: 
08/22/13 11:35:31 (pid:28691)    /etc/condor/config.d/00personal_condor.config
08/22/13 11:35:31 (pid:28691) DaemonCore: command socket at <129.234.21.14:41942>
08/22/13 11:35:31 (pid:28691) DaemonCore: private command socket at <129.234.21.14:41942>
08/22/13 11:35:31 (pid:28691) History file rotation is enabled.
08/22/13 11:35:31 (pid:28691)   Maximum history file size is: 20971520 bytes
08/22/13 11:35:31 (pid:28691)   Number of rotated history files is: 2
08/22/13 11:35:32 (pid:28691) Failed to execute /usr/sbin/condor_shadow.std, ignoring
08/22/13 11:35:32 (pid:28691) About to rotate ClassAd log /var/lib/condor/spool/job_queue.log
08/22/13 11:35:32 (pid:28691) 2.0: JobLeaseDuration remaining: 36
08/22/13 11:35:32 (pid:28691) directory_util::rec_touch_file: Directory /var/lock/condor/local cannot be created (Permission denied) 
08/22/13 11:35:32 (pid:28691) Starting add_shadow_birthdate(2.0)
Stack dump for process 28691 at timestamp 1377167732 (4 frames)
/lib64/libcondor_utils_8_1_0.so(dprintf_dump_stack+0x72)[0x7f6305a85972]
/lib64/libcondor_utils_8_1_0.so(+0x17b5f7)[0x7f6305b205f7]
/lib64/libpthread.so.0[0x3086e0efa0]
[0x7fff4cba11d0]

(The condor master tries to restart the scheduler every 30 seconds resulting the above error message)

The access write for /var/lock/condor (which actualy is  /run/lock/condor) :
drwxrwxr-x. 2 condor condor 60 Aug 21 13:03 condor/

The failure to create /var/lock/condor/local is not a selinux issue as it is turned off, nor an access write one. This looks like a bug in schedd.

The file /usr/sbin/condor_shadow.std does not exist (should it?)

% tail /var/log/messages 
Aug 22 11:16:42 hopf kernel: [79994.644748] condor_schedd[28432]: segfault at 7fff67479930 ip 00007fff67479930 sp 00007fff67475740 error 15

Once these problems are solved, we will be able to solve the selinux problems.

Comment 7 Miroslav Grepl 2013-08-22 10:55:05 UTC
I am going to check your audit.log which you sent me.

Comment 8 Bernard Piette 2013-08-22 11:01:36 UTC
(In reply to Miroslav Grepl from comment #7)
> I am going to check your audit.log which you sent me.

Before we try to address the selinux issues which should try to get condor working with selinux turned off. As described in Comment 6 there are still plain condor issues.

Comment 9 Matthew Farrellee 2013-08-22 11:04:46 UTC
BTW, I'm not seeing that startup error...


$ getenforce                      
Enforcing

$ ls -ald /var/lock/condor    
ls: cannot access /var/lock/condor: No such file or directory

$ ls -ald /var/run/condor     
ls: cannot access /var/run/condor: No such file or directory

$ sudo yum install -y condor          
Loaded plugins: auto-update-debuginfo, langpacks, refresh-packagekit
Resolving Dependencies
--> Running transaction check
---> Package condor.x86_64 0:8.1.0-0.2.fc19 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=======================================================================================
 Package          Arch             Version                     Repository         Size
=======================================================================================
Installing:
 condor           x86_64           8.1.0-0.2.fc19              updates           4.2 M

Transaction Summary
=======================================================================================
Install  1 Package
Total download size: 4.2 M
Installed size: 13 M
Downloading packages:
condor-8.1.0-0.2.fc19.x86_64.rpm                                | 4.2 MB  00:00:13     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : condor-8.1.0-0.2.fc19.x86_64                                        1/1 
libsemanage.dbase_llist_set: record not found in the database (No such file or directory).
libsemanage.dbase_llist_set: could not set record value (No such file or directory).
Could not change boolean condor_domain_can_network_connect
Could not change policy booleans
  Verifying  : condor-8.1.0-0.2.fc19.x86_64                                        1/1 
Installed:
  condor.x86_64 0:8.1.0-0.2.fc19                                                       
Complete!

$ sudo systemctl start condor
$ sudo systemctl status condor
condor.service - Condor Distributed High-Throughput-Computing
   Loaded: loaded (/usr/lib/systemd/system/condor.service; disabled)
   Active: active (running) since Thu 2013-08-22 07:00:55 EDT; 3s ago
 Main PID: 29950 (condor_master)
   CGroup: name=systemd:/system/condor.service
           ├─29950 /usr/sbin/condor_master -f
           ├─29953 condor_procd -A /var/run/condor/procd_pipe -R 10000000 -S 60 -C 9...
           ├─29954 condor_collector -f
           ├─29955 condor_negotiator -f
           ├─29956 condor_schedd -f
           ├─29961 condor_startd -f
           ├─29986 /usr/sbin/condor_starter -classad
           └─29987 /usr/bin/java -classpath /usr/share/condor:/usr/share/condor/scim...
Aug 22 07:00:55 eeyore.local systemd[1]: Started Condor Distributed High-Throughpu...g.

$ ls -ald /var/run/condor     
0 drwxrwxr-x. 2 condor condor 80 Aug 22 07:00 /var/run/condor/

$ ls -al /var/run/condor 
total 0
0 drwxr-xr-x. 48 root   root   1360 Aug 22 07:00 ../
0 prw-------.  1 condor root      0 Aug 22 07:00 procd_pipe.watchdog|
0 prw-------.  1 condor root      0 Aug 22 07:00 procd_pipe|
0 drwxrwxr-x.  2 condor condor   80 Aug 22 07:00 ./

$ ls -ald /var/lock/condor
0 drwxrwxr-x. 2 condor condor 60 Aug 22 07:00 /var/lock/condor/

$ ls -al /var/lock/condor 
total 0
0 drwxr-xr-x. 8 root   root   160 Aug 22 07:00 ../
0 drwxrwxr-x. 2 condor condor  60 Aug 22 07:00 ./
0 -rw-------. 1 condor condor   0 Aug 22 07:00 InstanceLock

$ pstree | grep condor
        |-condor_master-+-condor_collecto
        |               |-condor_negotiat
        |               |-condor_procd
        |               |-condor_schedd
        |               `-condor_startd

$ condor_q    
-- Submitter: eeyore.local : <192.168.1.103:37532> : eeyore.local
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended

Comment 10 Miroslav Grepl 2013-08-22 11:04:49 UTC
Then the bug should be cloned.

Comment 11 Miroslav Grepl 2013-08-22 11:31:57 UTC
I updated condor rules.

Comment 12 Bernard Piette 2013-08-22 11:51:09 UTC
(In reply to Matthew Farrellee from comment #9)
> BTW, I'm not seeing that startup error...
> 

Interesting, we need to find out what differs between our systems.
Condor wont install on its own for me.

> $ sudo yum install -y condor          
> Loaded plugins: auto-update-debuginfo, langpacks, refresh-packagekit
> Resolving Dependencies
> --> Running transaction check
> ---> Package condor.x86_64 0:8.1.0-0.2.fc19 will be installed
> --> Finished Dependency Resolution
> Dependencies Resolved
> =============================================================================
> ==========
>  Package          Arch             Version                     Repository   
> Size
> =============================================================================
> ==========
> Installing:
>  condor           x86_64           8.1.0-0.2.fc19              updates      
> 4.2 M
> 
> Transaction Summary


When I try to install condor I am forced to install condor-classads and condor_procd (refgardless of selinux being on or off).

:# yum install condor
Loaded plugins: langpacks, refresh-packagekit, verify
google-chrome                                            |  951 B     00:00     
google-talkplugin                                        |  951 B     00:00     
maths                                                    | 2.9 kB     00:00 !!! 
maths_extra                                              | 2.9 kB     00:00 !!! 
rpmfusion-free-updates                                   | 3.3 kB     00:00     
rpmfusion-nonfree-updates                                | 3.3 kB     00:00     
updates/19/x86_64/metalink                               |  27 kB     00:00     
updates                                                  | 4.6 kB     00:00     
(1/2): updates/19/x86_64/group_gz                          | 385 kB   00:00     
(2/2): updates/19/x86_64/primary_db                        | 7.3 MB   00:00     
(1/2): updates/19/x86_64/updateinfo                        | 682 kB   00:00     
(2/2): updates/19/x86_64/pkgtags                           | 463 kB   00:00     
Resolving Dependencies
--> Running transaction check
---> Package condor.x86_64 0:8.1.0-0.2.fc19 will be installed
--> Processing Dependency: condor-procd = 8.1.0-0.2.fc19 for package: condor-8.1.0-0.2.fc19.x86_64
--> Processing Dependency: condor-classads = 8.1.0-0.2.fc19 for package: condor-8.1.0-0.2.fc19.x86_64
--> Processing Dependency: libclassad.so.5()(64bit) for package: condor-8.1.0-0.2.fc19.x86_64
--> Running transaction check
---> Package condor-classads.x86_64 0:8.1.0-0.2.fc19 will be installed
---> Package condor-procd.x86_64 0:8.1.0-0.2.fc19 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                Arch          Version                Repository    Size
================================================================================
Installing:
 condor                 x86_64        8.1.0-0.2.fc19         maths        4.2 M
Installing for dependencies:
 condor-classads        x86_64        8.1.0-0.2.fc19         maths        209 k
 condor-procd           x86_64        8.1.0-0.2.fc19         maths         96 k

Transaction Summary
================================================================================
Install  1 Package (+2 Dependent packages)


Why do we see that difference? THIS IS MOST ODD! Did you have condor_procd and condor-classads already installed?

Also which version of selinux is installed on your system?

Comment 13 Bernard Piette 2013-08-22 12:04:51 UTC
I have just realised that in my previous comment, yum picked the file from my loca yum repo (which I use to maintain 200 PCs). This does not make any difference if I exclude that repos though (the rpms are identical):

:# yum install condor --disablerepo=maths
Loaded plugins: langpacks, refresh-packagekit, verify
Resolving Dependencies
--> Running transaction check
---> Package condor.x86_64 0:8.1.0-0.2.fc19 will be installed
--> Processing Dependency: condor-procd = 8.1.0-0.2.fc19 for package: condor-8.1.0-0.2.fc19.x86_64
--> Processing Dependency: condor-classads = 8.1.0-0.2.fc19 for package: condor-8.1.0-0.2.fc19.x86_64
--> Processing Dependency: libclassad.so.5()(64bit) for package: condor-8.1.0-0.2.fc19.x86_64
--> Running transaction check
---> Package condor-classads.x86_64 0:8.1.0-0.2.fc19 will be installed
---> Package condor-procd.x86_64 0:8.1.0-0.2.fc19 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package               Arch         Version                 Repository     Size
================================================================================
Installing:
 condor                x86_64       8.1.0-0.2.fc19          updates       4.2 M
Installing for dependencies:
 condor-classads       x86_64       8.1.0-0.2.fc19          updates       209 k
 condor-procd          x86_64       8.1.0-0.2.fc19          updates        96 k

Transaction Summary
================================================================================
Install  1 Package (+2 Dependent packages)

Comment 14 Matthew Farrellee 2013-08-22 12:24:47 UTC
(In reply to Bernard Piette from comment #12)

> Why do we see that difference? THIS IS MOST ODD! Did you have condor_procd
> and condor-classads already installed?

They are.

$ rpm -q condor-classads condor-procd
condor-classads-8.1.0-0.2.fc19.x86_64
condor-procd-8.1.0-0.2.fc19.x86_64


> Also which version of selinux is installed on your system?

$ rpm -qa | grep selinux
libselinux-devel-2.1.13-15.fc19.x86_64
libselinux-utils-2.1.13-15.fc19.x86_64
libselinux-2.1.13-15.fc19.x86_64
libselinux-2.1.13-15.fc19.i686
libselinux-python-2.1.13-15.fc19.x86_64
selinux-policy-targeted-3.12.1-69.fc19.noarch
selinux-policy-devel-3.12.1-69.fc19.noarch
selinux-policy-3.12.1-69.fc19.noarch

Comment 15 Bernard Piette 2013-08-22 12:36:33 UTC
(In reply to Matthew Farrellee from comment #14)

> $ rpm -q condor-classads condor-procd
> condor-classads-8.1.0-0.2.fc19.x86_64
> condor-procd-8.1.0-0.2.fc19.x86_64

and what is the output of rpm -qa | fgrep condor for you?

For me 
:# rpm -qa | fgrep condor 
condor-8.1.0-0.2.fc19.x86_64
condor-classads-8.1.0-0.2.fc19.x86_64
condor-procd-8.1.0-0.2.fc19.x86_64

and

:# rpm -qa | grep selinux
libselinux-2.1.13-15.fc19.x86_64
libselinux-devel-2.1.13-15.fc19.x86_64
libselinux-utils-2.1.13-15.fc19.x86_64
libselinux-python-2.1.13-15.fc19.x86_64
selinux-policy-doc-3.12.1-71.fc19.noarch
selinux-policy-3.12.1-71.fc19.noarch
selinux-policy-targeted-3.12.1-71.fc19.noarch
selinux-policy-devel-3.12.1-71.fc19.noarch
libselinux-2.1.13-15.fc19.i686

(but I am not concerned about selinux at this stage as that can be fixed using semodules)

Comment 16 Bernard Piette 2013-08-22 12:36:50 UTC
More on comment 6 above: it turns out that condor_schedd does starts when condor is started but it crashes as soon as a job is submitted. I had a job in the condor queue which I forgot to remove.

So 
:# setenforce 0
:# systemctl stop condor
:# /bin/rm -f /var/lib/condor/spool/job*
:# systemctl start condor

:# ps aux | fgrep condor
condor   29445  0.0  0.0  89020  6032 ?        Ss   13:17   0:00 /usr/sbin/condor_master -f
root     29446  0.0  0.0  23460  4092 ?        S    13:17   0:00 condor_procd -A /var/run/condor/procd_pipe -R 10000000 -S 60 -C 988
condor   29447  0.0  0.0  89132  6240 ?        Ss   13:17   0:00 condor_collector -f
condor   29448  0.0  0.0  89256  6064 ?        Ss   13:17   0:00 condor_negotiator -f
condor   29449  0.0  0.0  90224  6664 ?        Ss   13:17   0:00 condor_schedd -f
condor   29450  0.0  0.0  89844  6472 ?        Ss   13:17   0:00 condor_startd -f
root     29520  0.0  0.0 107960   676 pts/5    S+   13:20   0:00 fgrep --color=auto condor


user% condor_submit condor_job.txt
Submitting job(s).
1 job(s) submitted to cluster 3.

user% condor_q

-- Failed to fetch ads from: <129.234.12.34:58489> : hopf
CEDAR:6001:Failed to connect to <129.234.12.34:58489>


condor_job.txt:
####################################################
executable     = /user/bin/sleep
universe       = vanilla 
#input          = nothing.data                
arguments      =  10
output         = condor_output_hopf.log                
error          = condor_error_hopf.log             
log            = condor_log_hopf.log 
####################################################


:# ps aux | fgrep condor
condor   29445  0.0  0.0  89020  6096 ?        Ss   13:17   0:00 /usr/sbin/condor_master -f
root     29446  0.0  0.0  23460  4096 ?        S    13:17   0:00 condor_procd -A /var/run/condor/procd_pipe -R 10000000 -S 60 -C 988
condor   29447  0.0  0.0  89132  6244 ?        Ss   13:17   0:00 condor_collector -f
condor   29448  0.0  0.0  89432  6316 ?        Ss   13:17   0:00 condor_negotiator -f
condor   29450  0.0  0.0  89844  6576 ?        Ss   13:17   0:00 condor_startd -f
root     29536  0.0  0.0 107960   676 pts/5    S+   13:21   0:00 fgrep --color=auto condor


(condor_schedd has died)
:# tail -100 /var/log/condor/SchedLog
******************************************************
08/22/13 13:23:44 (pid:29580) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
08/22/13 13:23:44 (pid:29580) ** /usr/sbin/condor_schedd
08/22/13 13:23:44 (pid:29580) ** SubsystemInfo: name=SCHEDD type=SCHEDD(5) class=DAEMON(1)
08/22/13 13:23:44 (pid:29580) ** Configuration: subsystem:SCHEDD local:<NONE> class:DAEMON
08/22/13 13:23:44 (pid:29580) ** $CondorVersion: 8.1.0 Jul 15 2013 BuildID: RH-8.1.0-0.2.fc19 PRE-RELEASE-UWCS $
08/22/13 13:23:44 (pid:29580) ** $CondorPlatform: X86_64-Fedora_19 $
08/22/13 13:23:44 (pid:29580) ** PID = 29580
08/22/13 13:23:44 (pid:29580) ** Log last touched 8/22 13:23:03
08/22/13 13:23:44 (pid:29580) ******************************************************
08/22/13 13:23:44 (pid:29580) Using config source: /etc/condor/condor_config
08/22/13 13:23:44 (pid:29580) Using local config sources: 
08/22/13 13:23:44 (pid:29580)    /etc/condor/config.d/00personal_condor.config
08/22/13 13:23:44 (pid:29580) DaemonCore: command socket at <129.234.21.14:51810>
08/22/13 13:23:44 (pid:29580) DaemonCore: private command socket at <129.234.21.14:51810>
08/22/13 13:23:44 (pid:29580) History file rotation is enabled.
08/22/13 13:23:44 (pid:29580)   Maximum history file size is: 20971520 bytes
08/22/13 13:23:44 (pid:29580)   Number of rotated history files is: 2
08/22/13 13:23:45 (pid:29580) Failed to execute /usr/sbin/condor_shadow.std, ignoring
08/22/13 13:23:45 (pid:29580) About to rotate ClassAd log /var/lib/condor/spool/job_queue.log
08/22/13 13:23:45 (pid:29580) 1.0: JobLeaseDuration remaining: 1081
08/22/13 13:23:45 (pid:29580) directory_util::rec_touch_file: Directory /var/lock/condor/local cannot be created (Permission denied) 
08/22/13 13:23:45 (pid:29580) Starting add_shadow_birthdate(1.0)
Stack dump for process 29580 at timestamp 1377174225 (4 frames)
/lib64/libcondor_utils_8_1_0.so(dprintf_dump_stack+0x72)[0x7f6f46aea972]
/lib64/libcondor_utils_8_1_0.so(+0x17b5f7)[0x7f6f46b855f7]
/lib64/libpthread.so.0[0x3086e0efa0]
[0x7fffb0f7a1a0]

Comment 17 Matthew Farrellee 2013-08-22 13:06:26 UTC
There is no diff in rpm output wrt condor packages.

I recommend you start from a fresh system and separate the schedd bug out from this selinux bug.

Comment 18 Bernard Piette 2013-08-23 09:31:56 UTC
Installed Fedora 19 from DVD.

# yum update 

# rpm -q selinux-policy
selinux-policy-3.12.1-71.fc19.noarch
selinux-policy-targeted-3.12.1-71.fc19.noarch

# yum install condor
(install many dependent packages)

# rpm -qa | fgrep condor
condor-8.1.0-0.2.fc19.x86_64
condor-classads-8.1.0-0.2.fc19.x86_64
condor-procd-8.1.0-0.2.fc19.x86_64

# systemctl enable condor
# systemctl start condor
# ps aux | fgrep condor
condor    2868  0.3  0.0  96872  4416 ?        Ss   09:28   0:00 /usr/sbin/condor_master -f
root      2869  0.3  0.0  23964  3072 ?        S    09:28   0:00 condor_procd -A /var/run/condor/procd_pipe -R 10000000 -S 60 -C 990
condor    2872  0.0  0.0  92780  4468 ?        Ss   09:28   0:00 condor_collector -f
root      3120  0.0  0.0 107964   660 pts/1    S+   09:28   0:00 fgrep --color=auto condor

NO condor_negotiator, NO condor_schedd , NO condor_startd

# tail /var/log/messages 
Aug 23 09:16:08 hopf setroubleshoot: SELinux is preventing /usr/sbin/condor_master from read access on the file hosts. For complete SELinux messages. run sealert -l 17eff763-7c56-49d3-bbb3-d21af42f5861
...
Aug 23 09:16:06 hopf setroubleshoot: SELinux is preventing /usr/sbin/condor_master from read access on the file meminfo. For complete SELinux messages. run sealert -l f820579e-eafd-4a47-b64d-f4f41e048e11
...
Aug 23 09:16:06 hopf setroubleshoot: SELinux is preventing /usr/sbin/condor_master from read access on the file cpuinfo. For complete SELinux messages. run sealert -l f820579e-eafd-4a47-b64d-f4f41e048e11
Aug 23 09:16:06 hopf setroubleshoot: SELinux is preventing /usr/sbin/condor_master from read access on the file resolv.conf. For complete SELinux messages. run sealert -l 17eff763-7c56-49d3-bbb3-d21af42f5861
...
Aug 23 09:16:07 hopf setroubleshoot: SELinux is preventing /usr/sbin/condor_master from write access on the file .master_address.new. For complete SELinux messages. run sealert -l 274a39f3-92d2-47bf-95b5-0cefb5d7ff6a
...
Aug 23 09:16:07 hopf setroubleshoot: SELinux is preventing /usr/sbin/condor_master from setattr access on the file MasterLog. For complete SELinux messages. run sealert -l a47020cd-71f5-4972-9529-8550ca6b36ce
Aug 23 09:16:07 hopf setroubleshoot: SELinux is preventing /usr/sbin/condor_master from read access on the file stat. For complete SELinux messages. run sealert -l f820579e-eafd-4a47-b64d-f4f41e048e11
...
Aug 23 09:17:07 hopf setroubleshoot: SELinux is preventing /usr/sbin/condor_collector from setattr access on the file CollectorLog. For complete SELinux messages. run sealert -l e6342827-fc60-4b19-a9dd-d2160b1c4774

# yum install policycoreutils-devel

# fgrep condor /var/log/audit/audit.log | audit2allow
#============= condor_collector_t ==============
allow condor_collector_t condor_log_t:file { write setattr };

#============= condor_master_t ==============
allow condor_master_t condor_log_t:file { write setattr };
allow condor_master_t net_conf_t:file read;
allow condor_master_t proc_t:file read;

SO THERE ARE STILL SELINUX ISSUES WHICH CAN PROBABLY BE FIXED BY CREATING 
SEMODULES

FIRST WE MUST CHECK IF CONDOR WORKS WITH SELINUX TRUNED OFF
CHECK https://bugzilla.redhat.com/show_bug.cgi?id=1000106 FOR THE DETAILS

Comment 19 Miroslav Grepl 2013-08-23 09:46:58 UTC
I am going to do a new build today for testing.

Comment 20 Fedora End Of Life 2015-01-09 22:35:16 UTC
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 21 Fedora End Of Life 2015-02-18 11:13:24 UTC
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.