Bug 1009677

Summary: foreman cannot remove its own pid files if crash/unclean exit
Product: Red Hat Satellite Reporter: Corey Welton <cwelton>
Component: ProvisioningAssignee: Ohad Levy <ohadlevy>
Status: CLOSED CURRENTRELEASE QA Contact: Katello QA List <katello-qa-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.0.2CC: bkearney, cwelton, dcleal, mmccune
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-20 15:28:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Welton 2013-09-18 20:33:11 UTC
Description of problem:
I'm not sure of an exact repro case just yet (although you might simply be able to reboot a server while foreman is running), however attempting to restart the foreman instance first appears to work, but later is shown to have not started.  The logs tell us it cannot remove the pid file.

Version-Release number of selected component (if applicable):

Satellite-6.0.2-RHEL-6-20130913.1

How reproducible:


Steps to Reproduce:
1.  Cause an unclean stoppage of foreman 
2.  Attempt to restart (foreman service restart or katello-service restart)
3.  Observe the [OK] for foreman processes.
4. 'service foreman status'

Actual results:
[root@ivanova ~]# service foreman status
Foreman dead but pid file exists
 dead but pid file exists

Observed in the thin process logs:

/opt/rh/ruby193/root/usr/share/gems/gems/thin-1.3.1/lib/thin/daemonizing.rb:158:in `delete': Permission denied - tmp/pids/thin.5501.pid (Errno::EACCES)
        from /opt/rh/ruby193/root/usr/share/gems/gems/thin-1.3.1/lib/thin/daemonizing.rb:158:in `remove_pid_file'
        from /opt/rh/ruby193/root/usr/share/gems/gems/thin-1.3.1/lib/thin/daemonizing.rb:59:in `block in daemonize'


Expected results:
* Foreman can restart itself cleanly (and remove its pid files)
* Foreman shouldn't tell us [OK] if it hasn't actually started
* I'm not really sure we should be seeing pid files in multiple locations ('/var/run/foreman/pids/', '/usr/share/foreman/tmp/pids/')

Additional info:

Manual removal of pids _may_ solve things, but that's messy

Comment 2 Corey Welton 2013-09-18 21:00:45 UTC
/usr/share/foreman/tmp/pids/ is owned by root.  modifying it to foreman.foreman allows foreman to remove the pids.

Comment 3 Dominic Cleal 2013-09-19 07:06:22 UTC
I can't reproduce this upstream using ruby193-rubygem-thin-1.3.1-7.el6.x86_64.

It appears that the PID file should be changed while thin is running as root, so I can't see what would prevent it.  Have you checked for any SELinux AVCs at the time of the permission denied message?

[root@rhel6b ~]# service foreman start
Starting foreman: Starting server on 0.0.0.0:1234 ... 
                                                           [  OK  ]
[root@rhel6b ~]# ll ~foreman/tmp/pids/
total 4
-rw-r--r--. 1 root root 4 Sep 19 06:59 thin.1234.pid
[root@rhel6b ~]# pkill -9 thin
[root@rhel6b ~]# service foreman status
Foreman dead but pid file exists
[root@rhel6b ~]# service foreman start
Starting foreman: Starting server on 0.0.0.0:1234 ... 
>> Deleting stale PID file tmp/pids/thin.1234.pid
                                                           [  OK  ]
[root@rhel6b ~]# cat ~foreman/tmp/pids/thin.1234.pid; echo
6858
[root@rhel6b ~]# ps auxww | grep [t]hin
foreman   6858 90.7 12.0 442984 122924 ?       Sl   06:59   0:12 thin server (0.0.0.0:1234)
[root@rhel6b ~]# cat /var/log/foreman/thin.1234.log 
>> Writing PID to tmp/pids/thin.1234.pid
>> Changing process privilege to foreman:foreman
>> Thin web server (v1.3.1 codename Triple Espresso)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:1234, CTRL+C to stop
>> Writing PID to tmp/pids/thin.1234.pid
>> Changing process privilege to foreman:foreman
>> Thin web server (v1.3.1 codename Triple Espresso)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:1234, CTRL+C to stop


Regarding the two paths, they should be equivalent.  /usr/share/foreman/tmp is defined as a symlink to /var/run/foreman in the RPM, else that's a bug.

I'd like to treat this as low priority anyway, since we plan on adopting Passenger for MDP3 and this isn't an issue under normal operation.

Comment 4 Mike McCune 2013-10-23 23:36:17 UTC
+1 to moving this to MDP3 and just switching over to Passenger

Comment 5 Dominic Cleal 2014-01-20 15:28:49 UTC
We're using Passenger in MDP3, so httpd will be the primary service instead.