| Summary: | Can't connect DB for some scalable apps on STG | ||
|---|---|---|---|
| Product: | OpenShift Online | Reporter: | Yan Du <yadu> |
| Component: | Containers | Assignee: | mfisher |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | libra bugs <libra-bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 2.x | CC: | admiller, agoldste, mfisher, xtian |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-01-24 03:28:43 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Please retest - this should be working now. 1. For new apps on stg, all dbs can be connected normally. 2. For old apps on stg (after migration), seems some of them still have the problem: [perlsdbjks-dystg.stg.rhcloud.com 52735fd1dbd93c7f6900001c]\> mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1 Server version: 5.1.69 Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> exit Bye [perlsdbjks-dystg.stg.rhcloud.com 52735fd1dbd93c7f6900001c]\> psql psql: could not connect to server: Connection refused Is the server running on host "52735fd1dbd93c7f6900002c-dystg.stg.rhcloud.com" and accepting TCP/IP connections on port 47036? [perlsdbjks-dystg.stg.rhcloud.com 52735fd1dbd93c7f6900001c]\> mongo MongoDB shell version: 2.4.6 connecting to: 52735fd1dbd93c7f69000038-dystg.stg.rhcloud.com:39921/admin Mon Nov 4 21:22:53.149 Error: couldn't connect to server 52735fd1dbd93c7f69000038-dystg.stg.rhcloud.com:39921 at src/mongo/shell/mongo.js:147 exception: connect failed After remove the old database and then re-add database to the old apps, the DB can be connected normally. Provide one more broken old apps to help check if it's a issue: mysql could be connected successfully: failed to connect to psql from the haproxy gear, but could connect to psql from psql gear directly: [jbeap1s-bmeng1stg.stg.rhcloud.com 521357452587c8e48f000166]\> mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1 Server version: 5.1.69 Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> exit Bye [jbeap1s-bmeng1stg.stg.rhcloud.com 521357452587c8e48f000166]\> psql psql: could not connect to server: Connection refused Is the server running on host "52135db82587c8e48f00024f-bmeng1stg.stg.rhcloud.com" and accepting TCP/IP connections on port 44366? [52135db82587c8e48f00024f-bmeng1stg.stg.rhcloud.com 52135db82587c8e48f00024f]\> hostname ex-std-node3.stg.rhcloud.com [52135db82587c8e48f00024f-bmeng1stg.stg.rhcloud.com 52135db82587c8e48f00024f]\> psql psql (8.4.18) Type "help" for help. jbeap1s=# $ rhc app-show jbeap1s jbeap1s @ http://jbeap1s-bmeng1stg.stg.rhcloud.com/ (uuid: 521357452587c8e48f000166) ------------------------------------------------------------------------------------ Domain: bmeng1stg Created: Aug 20 7:47 PM Gears: 5 (defaults to small) Git URL: ssh://521357452587c8e48f000166.rhcloud.com/~/git/jbeap1s.git/ SSH: 521357452587c8e48f000166.rhcloud.com jbosseap-6 (JBoss Enterprise Application Platform 6.1.0) -------------------------------------------------------- Scaling: x3 (minimum: 3, maximum: 3) on small gears This gear costs an additional $0.03 per gear after the first 3 gears. haproxy-1.4 (Web Load Balancer) ------------------------------- Gears: Located with jbosseap-6, cron-1.4 cron-1.4 (Cron 1.4) ------------------- Scaling: x3 (minimum: 3, maximum: 3) on small gears mysql-5.1 (MySQL 5.1) --------------------- Gears: 1 small Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/ Database Name: jbeap1s Password: lE1HX3U9K9jn Username: admin7ltD7B9 postgresql-8.4 (PostgreSQL 8.4) ------------------------------- Gears: 1 small Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT Database Name: jbeap1s Password: NCySa-dDNezm Username: adminfsrygr7 $ rhc app-show jbeap1s --gears ID State Cartridges Size SSH URL ------------------------ ------- ------------------------------- ----- --------------------------------------------------------------------------- 521357452587c8e48f000166 started jbosseap-6 haproxy-1.4 cron-1.4 small 521357452587c8e48f000166.rhcloud.com 52142e432587c8adc80007c6 started jbosseap-6 haproxy-1.4 cron-1.4 small 52142e432587c8adc80007c6.rhcloud.com 52774bfedbd93c504f0000ec started jbosseap-6 haproxy-1.4 cron-1.4 small 52774bfedbd93c504f0000ec.rhcloud.com 52135d8ddbd93cdda6000657 started mysql-5.1 small 52135d8ddbd93cdda6000657.rhcloud.com 52135db82587c8e48f00024f started postgresql-8.4 small 52135db82587c8e48f00024f.rhcloud.com FYI: Both mongo and postgresql could be connected successfully in this old apps: $ rhc app-show jbeap2s jbeap2s @ http://jbeap2s-bmeng1stg.stg.rhcloud.com/ (uuid: 5213578fdbd93cdda600057d) ------------------------------------------------------------------------------------ Domain: bmeng1stg Created: Aug 20 7:48 PM Gears: 3 (defaults to small) Git URL: ssh://5213578fdbd93cdda600057d.rhcloud.com/~/git/jbeap2s.git/ SSH: 5213578fdbd93cdda600057d.rhcloud.com jbosseap-6 (JBoss Enterprise Application Platform 6.1.0) -------------------------------------------------------- Scaling: x1 (minimum: 1, maximum: available) on small gears This gear costs an additional $0.03 per gear after the first 3 gears. haproxy-1.4 (Web Load Balancer) ------------------------------- Gears: Located with jbosseap-6, jenkins-client-1 jenkins-client-1 (Jenkins Client) --------------------------------- Gears: Located with jbosseap-6, haproxy-1.4 Job URL: https://jenkins-bmeng1stg.stg.rhcloud.com/job/jbeap2s-build/ mongodb-2.2 (MongoDB 2.2) ------------------------- Gears: 1 small Connection URL: mongodb://$OPENSHIFT_MONGODB_DB_HOST:$OPENSHIFT_MONGODB_DB_PORT/ Database Name: jbeap2s Password: nXPm7BM3jCUI Username: admin postgresql-9.2 (PostgreSQL 9.2) ------------------------------- Gears: 1 small Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT Database Name: jbeap2s Password: pQsSr8meclTX Username: admingizun8w The iptables nat rules for some gears are incorrect and use a previous IP address which is no longer correct. Ops (Adam) has the patch, and a mechanism to determine which nodes should be fixed in STG and PROD. Test on denvenv_stage_554, DBs can be connected again after fix the invalid IP in iptables.nat.rules. Steps: 1. Create a scalable app with mysql, postgresql, mongodb embedded 2. ssh into app and check all DBs can be connected normally 3. ssh into instance and mofify the nodeIP to an invalid value for your app in /etc/openshift/iptables.nat.rules 4. Run below command to make the change take effect # iptables -t nat -F # iptables -F rhc-app-comm # service openshift-iptables-port-proxy start 5. ssh into app and try to connect DB, DB can't be connected. 6. Fix the nodeIP by oo-iptables-port-proxy fixaddr, the invalid Ip in iptables.nat.rules can be fixed after running the command. [root@ip-10-116-237-173 ~]# oo-iptables-port-proxy fixaddr 10.116.237.173 10.116.237.173/32 11.116.237.173/32 7. restart the service as step4 to make the change take effect 8. ssh into app again and try to connect DB, DB can be connected successfully. Release ticket updated. Passing back to Q/E to validate on a STG node. Test on STG, DBs can be connected successfully for both broken old apps and new apps. Move bug to verified. |
Description of problem: Can't connect DB for scalable app on STG [n10sdbjks-dystg.stg.rhcloud.com 52735e2bdbd93c49d100012f]\> psql psql: could not connect to server: Connection timed out Is the server running on host "52735e2bdbd93c49d100013f-dystg.stg.rhcloud.com" and accepting TCP/IP connections on port 65391? [n10sdbjks-dystg.stg.rhcloud.com 52735e2bdbd93c49d100012f]\> mongo MongoDB shell version: 2.4.6 connecting to: 52735e2bdbd93c49d100014b-dystg.stg.rhcloud.com:62551/admin Mon Nov 4 05:18:18.487 Error: couldn't connect to server 52735e2bdbd93c49d100014b-dystg.stg.rhcloud.com:62551 at src/mongo/shell/mongo.js:147 exception: connect failed [n10sdbjks-dystg.stg.rhcloud.com 52735e2bdbd93c49d100012f]\> mysql ERROR 2003 (HY000): Can't connect to MySQL server on '52735e2bdbd93c49d1000133-dystg.stg.rhcloud.com' (110) Version-Release number of selected component (if applicable): stg(devenv_stage_549) How reproducible: Sometimes Steps to Reproduce: 1. create scalable apps with mysql-5.1, postgresql-8.4 and mongodb-2.2 embedded 2. check the app and try to connect the DBs Actual results: 1. Can't connect DB for some scalable apps on STG 2. Control DB or check the status, found DBs is runnung and can be controlled successfully without error. 3. DB can be connected in DB gears [root@dhcp-10-143 n10sdbjks]# rhc cartridge restart -a n10sdbjks -c mongodb-2.2 Restarting mongodb-2.2 ... done [root@dhcp-10-143 n10sdbjks]# rhc cartridge status -a n10sdbjks -c mongodb-2.2 RESULT: MongoDB is running Expected results: DB should be connected normally Additional info: [perlsdbjks-dystg.stg.rhcloud.com 52735fd1dbd93c7f6900001c]\> hostname ex-std-node2.stg.rhcloud.com [n10sdbjks-dystg.stg.rhcloud.com 52735e2bdbd93c49d100012f]\> hostname ex-std-node1.stg.rhcloud.com