Bug 853855 - Sometimes can't create ruby-1.8 app successfully on INT&STAGE
Sometimes can't create ruby-1.8 app successfully on INT&STAGE
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Ram Ranganathan
libra bugs
:
: 854502 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-03 03:18 EDT by Jianwei Hou
Modified: 2015-05-14 18:59 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-09-17 17:30:13 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
error_log (8.41 KB, application/octet-stream)
2012-09-03 03:18 EDT, Jianwei Hou
no flags Details
screen capture (27.27 KB, image/png)
2012-09-03 03:18 EDT, Jianwei Hou
no flags Details
gears status after creation (66.91 KB, image/png)
2012-09-05 04:39 EDT, joycezhang
no flags Details
gears status after scaling up (70.64 KB, image/png)
2012-09-05 04:39 EDT, joycezhang
no flags Details

  None (edit)
Description Jianwei Hou 2012-09-03 03:18:07 EDT
Created attachment 609274 [details]
error_log

Description of problem:
After upgrade on INT, a ruby-1.8 application is broken, can't access the app's website, check app's status from CLI, it shows the app is running

Version-Release number of selected component (if applicable):
On INT

How reproducible:
sometimes(didn'r reproduce on devenv)

Steps to Reproduce:
1. Create applications to cover all cartridges, and note down there status
2. Upgrage server to latest and migrate
3. Check all prepared data 
  
Actual results:
a ruby-1.8 application doesn't work properly

Check status from command line
hjw@my apps$ rhc app status -a ruby18
Password: ******


RESULT:
Total Accesses: 7
Total kBytes: 1
Uptime: 2606
ReqPerSec: .00268611
BytesPerSec: .392939
BytesPerReq: 146.286
BusyWorkers: 1
IdleWorkers: 0
Scoreboard: W....

But when accessing website, got 404 not found error.
see attachments

Expected results:
All apps should work properly

Additional info:
Attached screenshot and error logs
If you want to find out more detailed information, my application is at:  http://ruby18-jhou.int.rhcloud.com/
Comment 1 Jianwei Hou 2012-09-03 03:18:41 EDT
Created attachment 609276 [details]
screen capture
Comment 2 Jianwei Hou 2012-09-03 05:29:57 EDT
Caught this error again when creating new ruby-1.8 application on INT

The website returns 404 error
The logfile contains the following message:

[Mon Sep 03 05:21:46 2012] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:libra_t:s0:c1,c753
[Mon Sep 03 05:21:46 2012] [error] *** Passenger could not be initialized because of this error: Unable to start the Phusion Passenger watchdog (/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/agents/PassengerWatchdog): Permission denied (13)
[Mon Sep 03 05:21:46 2012] [notice] mod_bw : Memory Allocated 32 bytes (each conf takes 32 bytes)
[Mon Sep 03 05:21:46 2012] [notice] mod_bw : Version 0.8 - Initialized [1 Confs]
[Mon Sep 03 05:21:46 2012] [error] *** Passenger could not be initialized because of this error: Unable to start the Phusion Passenger watchdog (/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/agents/PassengerWatchdog): Permission denied (13)
[Mon Sep 03 05:21:46 2012] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.4 configured -- resuming normal operations
[Mon Sep 03 05:22:43 2012] [error] [client 127.3.119.129] Attempt to serve directory: /var/lib/stickshift/b51e842b512041b1be9082e661431104/app-root/runtime/repo/public/
[Mon Sep 03 05:22:43 2012] [error] [client 127.3.119.129] File does not exist: /var/lib/stickshift/b51e842b512041b1be9082e661431104/app-root/runtime/repo/public/favicon.ico

[r1-xtianint1.int.rhcloud.com logs]\> ls -l /usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/agents/PassengerWatchdog
-rwxr-xr-x. 1 root root 1242282 Aug 28 10:38 /usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/agents/PassengerWatchdog
[r1-xtianint1.int.rhcloud.com logs]\> file /usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/agents/PassengerWatchdog
/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/agents/PassengerWatchdog: regular file, no read permission
[r1-xtianint1.int.rhcloud.com logs]\> /usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/agents/PassengerWatchdog
bash: /usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/agents/PassengerWatchdog: Permission denied
Comment 3 Ram Ranganathan 2012-09-05 01:05:57 EDT
Need more info -- are there any selinux issues?? If so, can we get /var/log/audit/audit.log information.  This could also be an app issue.

Can't reproduce this on devenv -- Can you try w/ another app/new app??  

Also if possible ssh into the app and see if you can get to the ruby application directly. 

ssh b51e842b512041b1be9082e661431104@r1-xtianint1.int.rhcloud.com

and then run  curl -vvv $OPENSHIFT_INTERNAL_IP:8080 and see if that works -- if not, see what errors are in the log files for the app or maybe you can try pushing the sample app  and see if that works. 

Am unable to reproduce this.
Comment 4 Jianwei Hou 2012-09-05 03:09:43 EDT
(In reply to comment #3)
> Need more info -- are there any selinux issues?? If so, can we get
> /var/log/audit/audit.log information.  This could also be an app issue.
> 
> Can't reproduce this on devenv -- Can you try w/ another app/new app??  
> 
> Also if possible ssh into the app and see if you can get to the ruby
> application directly. 
> 
> ssh b51e842b512041b1be9082e661431104@r1-xtianint1.int.rhcloud.com
> 
> and then run  curl -vvv $OPENSHIFT_INTERNAL_IP:8080 and see if that works --
> if not, see what errors are in the log files for the app or maybe you can
> try pushing the sample app  and see if that works. 
> 
> Am unable to reproduce this.

I can not access /var/log/audit/audit.log on INT, But I've found a way to reproduce.
Note: only on INT(not on devenv or STG) and only with a ruby-1.8 application.

Steps:
1. Create a ruby-1.8 application(please follow the below command, the order of the arguments must follow the order to reproduce)
   rhc app create -p ****** -n --no-dns -t ruby-1.8 -a ruby18app
2. After creation, access app's website -- returns 404 error
3. ssh into application(ssh d5477bc3dd494f2e82f0ffd64c96acd9@ruby18app-xtianint1.int.rhcloud.com), and run:
   curl -vvv $OPENSHIFT_INTERNAL_IP:8080 -- <title>404 Not Found</title>
4. Check logs:

[ruby18app-xtianint1.int.rhcloud.com logs]\> cat error_log-20120905-000000-EST
[Wed Sep 05 02:52:50 2012] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:libra_t:s0:c2,c341
[Wed Sep 05 02:52:50 2012] [error] *** Passenger could not be initialized because of this error: Unable to start the Phusion Passenger watchdog (/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/agents/PassengerWatchdog): Permission denied (13)
[Wed Sep 05 02:52:50 2012] [notice] mod_bw : Memory Allocated 32 bytes (each conf takes 32 bytes)
[Wed Sep 05 02:52:50 2012] [notice] mod_bw : Version 0.8 - Initialized [1 Confs]
[Wed Sep 05 02:52:50 2012] [error] *** Passenger could not be initialized because of this error: Unable to start the Phusion Passenger watchdog (/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/agents/PassengerWatchdog): Permission denied (13)
[Wed Sep 05 02:52:50 2012] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.4 configured -- resuming normal operations
[Wed Sep 05 02:53:26 2012] [error] [client 127.4.168.1] Attempt to serve directory: /var/lib/stickshift/d5477bc3dd494f2e82f0ffd64c96acd9/app-root/runtime/repo/public/
[Wed Sep 05 02:53:26 2012] [error] [client 127.4.168.1] File does not exist: /var/lib/stickshift/d5477bc3dd494f2e82f0ffd64c96acd9/app-root/runtime/repo/public/favicon.ico
[Wed Sep 05 02:57:52 2012] [error] [client 127.4.168.1] Attempt to serve directory: /var/lib/stickshift/d5477bc3dd494f2e82f0ffd64c96acd9/app-root/runtime/repo/public/
Comment 5 Wei Sun 2012-09-05 04:27:18 EDT
*** Bug 854502 has been marked as a duplicate of this bug. ***
Comment 6 joycezhang 2012-09-05 04:39:21 EDT
Created attachment 609903 [details]
gears status after creation
Comment 7 joycezhang 2012-09-05 04:39:56 EDT
Created attachment 609904 [details]
gears status after scaling up
Comment 8 joycezhang 2012-09-05 04:44:36 EDT
This bug is also reprocuced while creating a scalable ruby-1.8 app on INT.

Please refer to the details as below:

Steps to reproduce:
1. Create a scalable ruby-1.8 app:
rhc app create -a myscapp8 -t ruby-1.8 -s -p xx
2. Check the proxy-status.

Actual results:
The gears are down by default after creation. Please refer to attached "ruby-1.8 scalable app proxy.png".

In addtion, the gears can't be scaled up successfully in this case. Please refer to attached "gears status_after scaling up.png" which shows the status after performing scale up twice.

Thanks.
Comment 9 Ram Ranganathan 2012-09-05 13:46:31 EDT
@zhangjing, yeah the scale ups will show down state because of the inability for passenger to startup on INT. Will debug this in bug week since its working on devenv and STG.  

Points to a configuration issue on INT -- marking it as low.
Comment 10 joycezhang 2012-09-12 02:54:38 EDT
This bug is still reproduced on INT(devenv_2153), the gears are down by default after ruby-1.8 scalable app creation. Thanks.
Comment 11 Rony Gong 2012-09-13 22:15:28 EDT
This bug is also reproduced on STG(devenv-stg_235),sometimes can't create ruby-1.8 app success, so up severity.

Running Command - rm -rf 4rrm2mxrfj && rhc app create -a 4rrm2mxrfj -t ruby-1.8 -l qgong+2@redhat.com -p 111111 -d -r ./4rrm2mxrfj
Submitting form:
rhlogin: qgong+2@redhat.com
debug: true
Contacting https://stg.openshift.redhat.com
Creating application: 4rrm2mxrfj in lg3o888jhf
Contacting https://stg.openshift.redhat.com
Response from server:
DEBUG:


Exit Code: 0
broker_c: ["namespace", "rhlogin", "ssh", "app_uuid", "debug", "alter", "cartridge", "cart_type", "action", "app_name", "api"]
api_c: ["placeholder"]
API version: 1.1.3
DEBUG: '4rrm2mxrfj' creation returned success.
Now your new domain name is being propagated worldwide (this might take a minute)...
Pulling new repo down
git clone ssh://df742e38eeac4cd6a7814faed47dee45@4rrm2mxrfj-lg3o888jhf.stg.rhcloud.com/~/git/4rrm2mxrfj.git/ ./4rrm2mxrfj
Checking if the application is available #1
Checking if the application is available #2
Checking if the application is available #3
Checking if the application is available #4
Checking if the application is available #5
Checking if the application is available #6
Checking if the application is available #7
Application is not available
Unable to access your new application.
Cleaning up git repo
Cleaning up application
Contacting https://stg.openshift.redhat.com
Command Return: 1
Comment 12 Ram Ranganathan 2012-09-14 01:05:26 EDT
@qgong, that is a different issue as the app was created fine but the  DNS propogation timed out and the rhc tools failed waiting to do a git clone.
Comment 13 joycezhang 2012-09-14 02:38:42 EDT
I also met same error as qgong mentioned in comment 11 when testing on STG today, this error is just reproduced for ruby-1.8. Other types app are working fine. Thanks.
Comment 14 Ram Ranganathan 2012-09-14 15:27:11 EDT
The selinux labels on STG and INT  were incorrect for the passenger-3.0.4 rpm helper scripts -- Thomas reset these to the right context and it works fine now. 

This works fine for me w/ 
   http://ruby18i-ramr.dev.rhcloud.com/ 
   http://ruby18-ramr2.stg.rhcloud.com/haproxy-status/

Can you please retest. 

Also, as previously mentioned - the git clone failing is another separate issue -- has to do w/ the DNS entry not propogating in time.
Comment 15 Jianwei Hou 2012-09-17 02:36:18 EDT
Verified on STG and INT

Steps:
1. Create non-scaling/scaling ruby-1.8 application
2. After creation, access app's website
3. Repeat step 1~step 2 more times to make sure problem is fixed. 

Result:
Website can be accessed. For scalable app, no gear down occured.
Repeated more than 10 times, never reproduced again, so move this bug to verified.

Note You need to log in before you can comment on or make changes to this bug.