Created attachment 628621 [details] development.log Description of problem: Always meet time out issue when embed postgresql, but postgresql is added successfully actually and could be accessed. other dbs(mysql and mongodb) work fine. Version-Release number of selected component (if applicable): devenv_2340 rhc-0.99.7.gem How reproducible: always Steps to Reproduce: 1.Creat app like jboss 2.Embed postgresql $ rhc cartridge add -a qjboss -c postgresql 3. Actual results: [qgong@localhost dev]$ rhc cartridge add -a qjboss -c postgresql Password: Adding 'postgresql-8.4' to application 'qjboss' Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status of your applications. [qgong@localhost dev]$ echo $? 128 [qgong@localhost dev]$ rhc cartridge add -a qjboss -c postgresql Password: Adding 'postgresql-8.4' to application 'qjboss' postgresql-8.4 already embedded in 'qjboss' Expected results: Adding 'postgresql-8.4' to application 'qjboss' Success Useful postgresql-8.4 properties ============================= Root User: admin Root Password: kTw_c2bji8C7 Database Name: qjboss Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT Additional info:
I'll take a look at this Rob.
Fixed with pull request: https://github.com/openshift/origin-server/pull/702 Waiting for merge.
Verified on devenv_2348 [qgong@localhost ~]$ rhc cartridge add -a qjboss -c postgresql -p xxx Adding 'postgresql-8.4' to application 'qjboss' Success Useful postgresql-8.4 properties ================================ connection_url = postgresql://127.0.250.129:5432/ database_name = qjboss username = admin password = yscGc77SKSwq
Reopen this on devenv_2364, same error just happens when add postgresql to scalable app. [qgong@localhost dev]$ rhc cartridge add -a qspython -c postgresql-8.4 Password: Adding 'postgresql-8.4' to application 'qspython' Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status of your applications. [qgong@localhost dev]$ rhc cartridge add -a qspython -c postgresql-8.4 Password: Adding 'postgresql-8.4' to application 'qspython' postgresql-8.4 already embedded in 'qspython'
Works for me with devenv_2367 -- know there were some selinux related issues in the interim releases. Can you please retest and check. Thanks.
Veirfied on scalable and non-scalable app
This problem is reproduced on STG now Try to add postgresql-8.4 or mongdb-2.2 to a scalable application, cartridge is added successfully, but CLI received timeout error. [hjw@rheltest apps]$ time rhc cartridge add mongodb-2.2 -a app1s Password: ****** Adding 'mongodb-2.2' to application 'app1s' Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status of your applications. real 1m12.805s user 0m0.604s sys 0m0.132s [hjw@rheltest apps]$ time rhc cartridge add postgresql-8.4 -a app1s Password: ****** Adding 'postgresql-8.4' to application 'app1s' Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status of your applications. real 1m12.907s user 0m0.602s sys 0m0.124s [hjw@rheltest apps]$ rhc app show app1s Password: ****** app1s @ http://app1s-jhou1.stg.rhcloud.com/ =========================================== Application Info ================ UUID = 7b4c330e17174ee28767bd1f06d7728b SSH URL = ssh://7b4c330e17174ee28767bd1f06d7728b.rhcloud.com Git URL = ssh://7b4c330e17174ee28767bd1f06d7728b.rhcloud.com/~/git/app1s.git/ Created = 5:40 AM Gear Size = small Cartridges ========== haproxy-1.4 mongodb-2.2 = mongodb://a4085b63a6-jhou1.stg.rhcloud.com:47521/ postgresql-8.4 = postgresql://ba52a61229-jhou1.stg.rhcloud.com:41261/ php-5.3 Scaling Info ============ Scaled x2 (minimum: 2, maximum: available gears) with haproxy-1.4 on small gears
The problem was the mcollective timeouts changed from 3 mins to 1 min on prod and stage (180 to 60). Thomas has made the required config changes and is repushing these out -- can you please test this once he's done. Probably in the next hour odd. Thanks,
Tested on STG and prod, still reproduced this problem, seems mcollective timeout is still 1 min Also checked mcollective timeout config on devenv: [root@ip-10-190-26-94 conf]# pwd /var/www/openshift/broker/conf [root@ip-10-190-26-94 conf]# cat openshift-origin-msg-broker-mcollective.conf MCOLLECTIVE_DISCTIMEOUT=2 MCOLLECTIVE_TIMEOUT=60 ..... [hjw@rheltest apps]$ time rhc cartridge add mongodb-2.2 -a php1s Password: ****** Adding 'mongodb-2.2' to application 'php1s' Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status of your applications. real 1m10.939s user 0m0.438s sys 0m0.136s [hjw@rheltest apps]$ rhc app show php1s Password: ****** php1s @ http://php1s-jhou1.stg.rhcloud.com/ =========================================== Application Info ================ SSH URL = ssh://3f0dcb26bca14b4baadcecf83da004f3.rhcloud.com Git URL = ssh://3f0dcb26bca14b4baadcecf83da004f3.rhcloud.com/~/git/php1s.git/ Gear Size = small Created = Nov 11 9:13 PM UUID = 3f0dcb26bca14b4baadcecf83da004f3 Cartridges ========== mongodb-2.2 = mongodb://ad1c43c49a-jhou1.stg.rhcloud.com:42726/ php-5.3 haproxy-1.4 Scaling Info ============ Scaled x2 (minimum: 2, maximum: available gears) with haproxy-1.4 on small gears
@Hou, works fine for me -- could be either the nodes you are on prod were slower or its more than likely your network connection to prod/stage might be slower. I guess you may need to bump the timeout to something more than 1 minute. To remove any network latencies out of the picture, can you please test this by running the rhc tools on an amazon instance (say a devenv talking to prod/stage). Thanks. Will fix the devenv/rhc-broker timeout to be 3 mins -- though that is sorta moot as the performance profile on devenvs is a lot different than on stg/prod.
Thanks Ram, with an --timeout option on stg/prod, the above problem is gone, so this should be caused by my slow network. I also run the same test on an amazon instance against prod, I still caught timeout issue: [root@ip-10-70-13-101 openshift]# time rhc cartridge add postgresql-8.4 -a php1s -predhat Adding 'postgresql-8.4' to application 'php1s' Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status of your applications. real 1m4.228s user 0m0.900s sys 0m0.366s But I think this might be cause by slow network, so I'm not going to reopen this bug now. If you think this is OK, I'll move this to verified, thanks!
Verified on devenv_2458 [qgong@localhost dev]$rhc cartridge add -a qsphp -c postgresql -p 111111 Adding 'postgresql-8.4' to application 'qsphp' Success postgresql-8.4 ============== Properties ========== Database Name = qsphp Connection URL = postgresql://a78e08b1ec-qgong16.dev.rhcloud.com:35586/ Username = admin Password = xT95KUN6-yTz