Bug 1009677 - foreman cannot remove its own pid files if crash/unclean exit
foreman cannot remove its own pid files if crash/unclean exit
Status: CLOSED CURRENTRELEASE
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Provisioning (Show other bugs)
6.0.2
Unspecified Unspecified
unspecified Severity high (vote)
: Unspecified
: --
Assigned To: Ohad Levy
Katello QA List
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-18 16:33 EDT by Corey Welton
Modified: 2016-09-26 10:27 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-20 10:28:49 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Welton 2013-09-18 16:33:11 EDT
Description of problem:
I'm not sure of an exact repro case just yet (although you might simply be able to reboot a server while foreman is running), however attempting to restart the foreman instance first appears to work, but later is shown to have not started.  The logs tell us it cannot remove the pid file.

Version-Release number of selected component (if applicable):

Satellite-6.0.2-RHEL-6-20130913.1

How reproducible:


Steps to Reproduce:
1.  Cause an unclean stoppage of foreman 
2.  Attempt to restart (foreman service restart or katello-service restart)
3.  Observe the [OK] for foreman processes.
4. 'service foreman status'

Actual results:
[root@ivanova ~]# service foreman status
Foreman dead but pid file exists
 dead but pid file exists

Observed in the thin process logs:

/opt/rh/ruby193/root/usr/share/gems/gems/thin-1.3.1/lib/thin/daemonizing.rb:158:in `delete': Permission denied - tmp/pids/thin.5501.pid (Errno::EACCES)
        from /opt/rh/ruby193/root/usr/share/gems/gems/thin-1.3.1/lib/thin/daemonizing.rb:158:in `remove_pid_file'
        from /opt/rh/ruby193/root/usr/share/gems/gems/thin-1.3.1/lib/thin/daemonizing.rb:59:in `block in daemonize'


Expected results:
* Foreman can restart itself cleanly (and remove its pid files)
* Foreman shouldn't tell us [OK] if it hasn't actually started
* I'm not really sure we should be seeing pid files in multiple locations ('/var/run/foreman/pids/', '/usr/share/foreman/tmp/pids/')

Additional info:

Manual removal of pids _may_ solve things, but that's messy
Comment 2 Corey Welton 2013-09-18 17:00:45 EDT
/usr/share/foreman/tmp/pids/ is owned by root.  modifying it to foreman.foreman allows foreman to remove the pids.
Comment 3 Dominic Cleal 2013-09-19 03:06:22 EDT
I can't reproduce this upstream using ruby193-rubygem-thin-1.3.1-7.el6.x86_64.

It appears that the PID file should be changed while thin is running as root, so I can't see what would prevent it.  Have you checked for any SELinux AVCs at the time of the permission denied message?

[root@rhel6b ~]# service foreman start
Starting foreman: Starting server on 0.0.0.0:1234 ... 
                                                           [  OK  ]
[root@rhel6b ~]# ll ~foreman/tmp/pids/
total 4
-rw-r--r--. 1 root root 4 Sep 19 06:59 thin.1234.pid
[root@rhel6b ~]# pkill -9 thin
[root@rhel6b ~]# service foreman status
Foreman dead but pid file exists
[root@rhel6b ~]# service foreman start
Starting foreman: Starting server on 0.0.0.0:1234 ... 
>> Deleting stale PID file tmp/pids/thin.1234.pid
                                                           [  OK  ]
[root@rhel6b ~]# cat ~foreman/tmp/pids/thin.1234.pid; echo
6858
[root@rhel6b ~]# ps auxww | grep [t]hin
foreman   6858 90.7 12.0 442984 122924 ?       Sl   06:59   0:12 thin server (0.0.0.0:1234)
[root@rhel6b ~]# cat /var/log/foreman/thin.1234.log 
>> Writing PID to tmp/pids/thin.1234.pid
>> Changing process privilege to foreman:foreman
>> Thin web server (v1.3.1 codename Triple Espresso)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:1234, CTRL+C to stop
>> Writing PID to tmp/pids/thin.1234.pid
>> Changing process privilege to foreman:foreman
>> Thin web server (v1.3.1 codename Triple Espresso)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:1234, CTRL+C to stop


Regarding the two paths, they should be equivalent.  /usr/share/foreman/tmp is defined as a symlink to /var/run/foreman in the RPM, else that's a bug.

I'd like to treat this as low priority anyway, since we plan on adopting Passenger for MDP3 and this isn't an issue under normal operation.
Comment 4 Mike McCune 2013-10-23 19:36:17 EDT
+1 to moving this to MDP3 and just switching over to Passenger
Comment 5 Dominic Cleal 2014-01-20 10:28:49 EST
We're using Passenger in MDP3, so httpd will be the primary service instead.

Note You need to log in before you can comment on or make changes to this bug.