Bug 867299 - Often meet time out error when add postgresql/mongodb to scalable app
Often meet time out error when add postgresql/mongodb to scalable app
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Ram Ranganathan
libra bugs
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-17 05:19 EDT by Rony Gong
Modified: 2015-05-14 19:00 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-19 14:25:45 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
development.log (5.22 KB, text/plain)
2012-10-17 05:19 EDT, Rony Gong
no flags Details

  None (edit)
Description Rony Gong 2012-10-17 05:19:05 EDT
Created attachment 628621 [details]
development.log

Description of problem:
Always meet time out issue when embed postgresql, but postgresql is added successfully actually and could be accessed.
other dbs(mysql and mongodb) work fine.

Version-Release number of selected component (if applicable):
devenv_2340
rhc-0.99.7.gem

How reproducible:
always

Steps to Reproduce:
1.Creat app like jboss
2.Embed postgresql
 $ rhc cartridge add -a qjboss -c postgresql
3.
  
Actual results:
[qgong@localhost dev]$ rhc cartridge add -a qjboss -c postgresql
Password: 

Adding 'postgresql-8.4' to application 'qjboss'
Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status of your applications.
[qgong@localhost dev]$ echo $?
128
[qgong@localhost dev]$ rhc cartridge add -a qjboss -c postgresql
Password: 

Adding 'postgresql-8.4' to application 'qjboss'
postgresql-8.4 already embedded in 'qjboss'

Expected results:
Adding 'postgresql-8.4' to application 'qjboss'
Success

Useful postgresql-8.4 properties
=============================
   Root User: admin
   Root Password: kTw_c2bji8C7
   Database Name: qjboss
Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT




Additional info:
Comment 1 Ram Ranganathan 2012-10-17 13:48:43 EDT
I'll take a look at this Rob.
Comment 2 Ram Ranganathan 2012-10-17 16:01:45 EDT
Fixed with pull request: 
     https://github.com/openshift/origin-server/pull/702

Waiting for merge.
Comment 3 Rony Gong 2012-10-18 21:25:52 EDT
Verified on devenv_2348
[qgong@localhost ~]$ rhc cartridge add -a qjboss -c postgresql -p xxx
Adding 'postgresql-8.4' to application 'qjboss'
Success

Useful postgresql-8.4 properties
================================
  connection_url = postgresql://127.0.250.129:5432/
  database_name  = qjboss
  username       = admin
  password       = yscGc77SKSwq
Comment 4 Rony Gong 2012-10-23 01:32:32 EDT
Reopen this on devenv_2364, same error just happens when add postgresql to scalable app.
[qgong@localhost dev]$ rhc cartridge add -a qspython -c postgresql-8.4 
Password: 

Adding 'postgresql-8.4' to application 'qspython'
Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain
show' or 'rhc app status' to check the status of your applications.
[qgong@localhost dev]$ rhc cartridge add -a qspython -c postgresql-8.4 
Password: 

Adding 'postgresql-8.4' to application 'qspython'
postgresql-8.4 already embedded in 'qspython'
Comment 5 Ram Ranganathan 2012-10-23 13:49:31 EDT
Works for me with devenv_2367 -- know there were some selinux related issues in the interim releases. Can you please retest and check.  Thanks.
Comment 6 Chandrika Gole 2012-10-23 19:40:32 EDT
Veirfied on scalable and non-scalable app
Comment 7 Jianwei Hou 2012-11-09 06:01:34 EST
This problem is reproduced on STG now

Try to add postgresql-8.4 or mongdb-2.2 to a scalable application, cartridge is added successfully, but CLI received timeout error.

[hjw@rheltest apps]$ time rhc cartridge add mongodb-2.2 -a app1s
Password: ******

Adding 'mongodb-2.2' to application 'app1s'
Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the
status of your applications.

real	1m12.805s
user	0m0.604s
sys	0m0.132s

[hjw@rheltest apps]$ time rhc cartridge add postgresql-8.4 -a app1s
Password: ******

Adding 'postgresql-8.4' to application 'app1s'
Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the
status of your applications.

real	1m12.907s
user	0m0.602s
sys	0m0.124s


[hjw@rheltest apps]$ rhc app show app1s
Password: ******


app1s @ http://app1s-jhou1.stg.rhcloud.com/
===========================================
  Application Info
  ================
    UUID      = 7b4c330e17174ee28767bd1f06d7728b
    SSH URL   = ssh://7b4c330e17174ee28767bd1f06d7728b@app1s-jhou1.stg.rhcloud.com
    Git URL   = ssh://7b4c330e17174ee28767bd1f06d7728b@app1s-jhou1.stg.rhcloud.com/~/git/app1s.git/
    Created   = 5:40 AM
    Gear Size = small
  Cartridges
  ==========
    haproxy-1.4
    mongodb-2.2    = mongodb://a4085b63a6-jhou1.stg.rhcloud.com:47521/
    postgresql-8.4 = postgresql://ba52a61229-jhou1.stg.rhcloud.com:41261/
    php-5.3
  Scaling Info
  ============
    Scaled x2 (minimum: 2, maximum: available gears) with haproxy-1.4 on small gears
Comment 8 Ram Ranganathan 2012-11-09 19:28:26 EST
The problem was the mcollective timeouts changed from 3 mins to 1 min on prod and stage (180 to 60). Thomas has made the required config  changes and is repushing these out  -- can you please test this once he's done.  Probably in the next hour odd. 

Thanks,
Comment 9 Jianwei Hou 2012-11-11 21:40:33 EST
Tested on STG and prod, still reproduced this problem, seems mcollective timeout is still 1 min

Also checked mcollective timeout config on devenv:
[root@ip-10-190-26-94 conf]# pwd
/var/www/openshift/broker/conf
[root@ip-10-190-26-94 conf]# cat openshift-origin-msg-broker-mcollective.conf 
MCOLLECTIVE_DISCTIMEOUT=2
MCOLLECTIVE_TIMEOUT=60
.....

[hjw@rheltest apps]$ time rhc cartridge add mongodb-2.2 -a php1s
Password: ******

Adding 'mongodb-2.2' to application 'php1s'
Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the
status of your applications.

real	1m10.939s
user	0m0.438s
sys	0m0.136s

[hjw@rheltest apps]$ rhc app show php1s
Password: ******


php1s @ http://php1s-jhou1.stg.rhcloud.com/
===========================================
  Application Info
  ================
    SSH URL   = ssh://3f0dcb26bca14b4baadcecf83da004f3@php1s-jhou1.stg.rhcloud.com
    Git URL   = ssh://3f0dcb26bca14b4baadcecf83da004f3@php1s-jhou1.stg.rhcloud.com/~/git/php1s.git/
    Gear Size = small
    Created   = Nov 11  9:13 PM
    UUID      = 3f0dcb26bca14b4baadcecf83da004f3
  Cartridges
  ==========
    mongodb-2.2 = mongodb://ad1c43c49a-jhou1.stg.rhcloud.com:42726/
    php-5.3
    haproxy-1.4
  Scaling Info
  ============
    Scaled x2 (minimum: 2, maximum: available gears) with haproxy-1.4 on small gears
Comment 10 Ram Ranganathan 2012-11-12 21:15:59 EST
@Hou, works fine for me --  could be either the nodes you are on prod were slower or its more than likely your network connection to prod/stage might be slower.
 I guess you may need to bump the timeout to something more than 1 minute.

To remove any network latencies out of the picture, can you please test this by 
running the rhc tools on an amazon instance (say a devenv talking to prod/stage). 
Thanks.

Will fix the devenv/rhc-broker timeout to be 3 mins -- though that is sorta moot as the performance profile on devenvs is a lot different than on stg/prod.
Comment 11 Jianwei Hou 2012-11-13 02:24:47 EST
Thanks Ram, with an --timeout option on stg/prod, the above problem is gone, so this should be caused by my slow network.

I also run the same test on an amazon instance against prod, I still caught timeout issue:

[root@ip-10-70-13-101 openshift]# time rhc cartridge add postgresql-8.4 -a php1s -predhat
Adding 'postgresql-8.4' to application 'php1s'
Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status
of your applications.

real	1m4.228s
user	0m0.900s
sys	0m0.366s

But I think this might be cause by slow network, so I'm not going to reopen this bug now.
If you think this is OK, I'll move this to verified, thanks!
Comment 12 Rony Gong 2012-11-13 03:54:44 EST
Verified on devenv_2458
[qgong@localhost dev]$rhc cartridge add -a qsphp -c postgresql -p 111111
Adding 'postgresql-8.4' to application 'qsphp'
Success
postgresql-8.4
==============
  Properties
  ==========
    Database Name  = qsphp
    Connection URL = postgresql://a78e08b1ec-qgong16.dev.rhcloud.com:35586/
    Username       = admin
    Password       = xT95KUN6-yTz

Note You need to log in before you can comment on or make changes to this bug.