Bug 975700 - Apache based cartridge can not be started/restarted if httpd.pid file is corrupted
Summary: Apache based cartridge can not be started/restarted if httpd.pid file is corr...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 1.x
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: ---
Assignee: Rob Millner
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-06-19 07:23 UTC by Xiaoli Tian
Modified: 2013-11-18 00:48 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-06-24 14:57:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
broker log (13.41 KB, text/plain)
2013-06-19 07:25 UTC, Xiaoli Tian
no flags Details

Description Xiaoli Tian 2013-06-19 07:23:50 UTC
Description of problem:
If httpd.pid file is corrupted, apached based cartridge like php and perl can not be started.

[downloadphpapp-domxq1.dev.rhcloud.com run]\> echo >httpd.pid 

[downloadphpapp-domxq1.dev.rhcloud.com run]\> gear start
Starting gear...
CLIENT_MESSAGE: Starting Apache+mod_php HTTPD server
(20014)Internal error: Error retrieving pid file run/httpd.pid
Remove it before continuing if it is corrupted.
An error occurred executing 'gear start' (exit code: 1)
Error message: Failed to execute: 'control start' for /var/lib/openshift/ad509918d8ac11e2a0f322000a9a98e8/php

For more details about the problem, try running the command again with the '--trace' option.

Version-Release number of selected component (if applicable):
devenv_3383

How reproducible:
Always

Steps to Reproduce:
1. ssh to the app and echo >httpd.pid 
2. run gear start/restart or rhc app start/restart $appname
3. remove httpd.pid
4. run gear start/restart


Actual results:
After step 2:
[root@ip-10-154-152-232 ~]# rhc app-start downloadphpapp
Unable to complete the requested operation due to: Failed to correctly execute all parallel operations - --DEBUG--
Failed to execute: 'control start' for /var/lib/openshift/ad509918d8ac11e2a0f322000a9a98e8/php

(20014)Internal error: Error retrieving pid file run/httpd.pid
Remove it before continuing if it is corrupted.

--RESULT--

--MESSAGE--
Starting Apache+mod_php HTTPD server

--ERROR--

--APP INFO--

--CART COMMANDS--

--CART PROPERTIES--
{}
--DATA--

--EXIT CODE--
1
--USER ACTIONABLE--
false
.
Reference ID: 8ad8d6b7741359386257b61be498d845


After step 4:
[downloadphpapp-domxq1.dev.rhcloud.com run]\> rm httpd.pid 
[downloadphpapp-domxq1.dev.rhcloud.com run]\> ls -lh
total 0
[downloadphpapp-domxq1.dev.rhcloud.com run]\> gear start
Starting gear...
CLIENT_MESSAGE: Starting Apache+mod_php HTTPD server
(98)Address already in use: make_sock: could not bind to address 127.0.252.1:8080
no listening sockets available, shutting down
Unable to open logs
An error occurred executing 'gear start' (exit code: 1)
Error message: Failed to execute: 'control start' for /var/lib/openshift/ad509918d8ac11e2a0f322000a9a98e8/php

[downloadphpapp-domxq1.dev.rhcloud.com run]\> netstat  |grep 127.0.252.1
[downloadphpapp-domxq1.dev.rhcloud.com run]\> 



Expected results:
It should be started/restarted successfully even if  httpd.pid is  corrupted.

Additional info:

Part of broker log( check whole log in attchment pls)
2013-06-19 03:13:08.723 [DEBUG] DEBUG: Output of parallel execute: [{:tag=>{"op_id"=>"51c15a02f054168bee00016c"}, :gear=>"ad509918d8ac11e2a0f322000a9a98e8", :job=>{:cartridge=>"openshift-origin-node", :action=>"start", :args=>{"--with-app-uuid"=>"ad509918d8ac11e2a0f322000a9a98e8", "--with-app-name"=>"downloadphpapp", "--with-container-uuid"=>"ad509918d8ac11e2a0f322000a9a98e8", "--with-container-name"=>"downloadphpapp", "--with-namespace"=>"domxq1", "--with-request-id"=>"8ad8d6b7741359386257b61be498d845", "--cart-name"=>"php-5.3", "--component-name"=>"openshift-php-5.3", "--with-software-version"=>"5.3", "--cartridge-vendor"=>"openshift"}}, :result_stdout=>"Failed to execute: 'control start' for /var/lib/openshift/ad509918d8ac11e2a0f322000a9a98e8/php\nCLIENT_MESSAGE: Starting Apache+mod_php HTTPD server\n\n(20014)Internal error: Error retrieving pid file run/httpd.pid\nRemove it before continuing if it is corrupted.\n", :result_stderr=>"", :result_exit_code=>1}], exitcode: 0, from: ip-10-154-152-232  (Request ID: 8ad8d6b7741359386257b61be498d845) (pid:2165)
2013-06-19 03:13:08.728 [DEBUG] DEBUG: MCollective Response Time (execute_parallel): 1.723373855s  (Request ID: 8ad8d6b7741359386257b61be498d845) (pid:2165)
2013-06-19 03:13:08.728 [DEBUG] DEBUG: server results: Failed to execute: 'control start' for /var/lib/openshift/ad509918d8ac11e2a0f322000a9a98e8/php (pid:2165)

Comment 1 Xiaoli Tian 2013-06-19 07:25:03 UTC
Created attachment 762768 [details]
broker log

Comment 2 Rob Millner 2013-06-19 23:27:54 UTC
Pull request:
https://github.com/openshift/origin-server/pull/2909

Comment 3 openshift-github-bot 2013-06-20 01:54:05 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/d1335eb6779c8e7906fe6ffb8908f788969b6e11
Bug 975700 - check the httpd pid file for corruption and attempt to fix it.

Comment 4 Xiaoli Tian 2013-06-20 07:30:18 UTC
Verified it on devenv_3390 , the httpd.pid file could be fixed while restart or start

for php:
[phpapp1-domx1.dev.rhcloud.com run]\> cat httpd.pid 
11904
[phpapp1-domx1.dev.rhcloud.com run]\> echo >httpd.pid 
# rhc app-restart phpapp1
RESULT:
phpapp1 restarted
[phpapp1-domx1.dev.rhcloud.com 51c2ad9392950e091f000001]\> cat php/run/httpd.pid 
11904


For perl:

[perlappp-domx1.dev.rhcloud.com 622987635372541576478720]\> cat perl/run/httpd.pid 
10762
[perlappp-domx1.dev.rhcloud.com 622987635372541576478720]\> echo > perl/run/httpd.pid 
[perlappp-domx1.dev.rhcloud.com 622987635372541576478720]\> cat perl/run/httpd.pid 

[perlappp-domx1.dev.rhcloud.com 622987635372541576478720]\> gear start
Starting gear...
httpd (pid 10762) already running
[perlappp-domx1.dev.rhcloud.com 622987635372541576478720]\> cat perl/run/httpd.pid 
10762


Note You need to log in before you can comment on or make changes to this bug.