Bug 1056961 - Could not connect to DB via proxy port for some scalable apps due to the begin_uid changed on INT/STG after migration
Summary: Could not connect to DB via proxy port for some scalable apps due to the begi...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: ---
Assignee: Jhon Honce
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-23 09:17 UTC by XiuJuan Wang
Modified: 2015-05-14 23:33 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-02-26 19:10:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description XiuJuan Wang 2014-01-23 09:17:48 UTC
Description of problem:
For existed scale perl-5.10(with postgresql-9.2),nodejs-0.6 (with mongodb-2.2), jbossas-7/jbossews-1.0/ruby-1.9(with postgresql-8.4) apps,only connect to DB from DB gear. From others gears,could not connect to DB. For non scalable apps, connect to DB.


Version-Release number of selected component (if applicable):
INT(devenv_4261)

How reproducible:
always

Steps to Reproduce:
1.Create a scalable perl-5.10 app, embed postresql-9.2
2.Upgrage server
3.Restart app or cartridges
4.Show app
5.Show app gears
6.Ssh into app from each gear, and connect psql
7.Ssh into postgresql-9.2 DB gear and show progresses

Actual results:
4.$ rhc app show  perldbjks
perldbjks @ http://perldbjks-migrate.int.rhcloud.com/ (uuid: 52ce07426cec0e946700082d)
--------------------------------------------------------------------------------------
  Domain:     migrate
  Created:    Jan 09 10:19 AM
  Gears:      6 (defaults to small)
  Git URL:    ssh://52ce07426cec0e946700082d.rhcloud.com/~/git/perldbjks.git/
  SSH:        52ce07426cec0e946700082d.rhcloud.com
  Deployment: auto (on git push)
  postgresql-9.2 (PostgreSQL 9.2)
  -------------------------------
    Gears:          1 small
    Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT
    Database Name:  perldbjks
    Password:       LFvWNnkbJnLX
    Username:       admin8pryxmq
  mongodb-2.2 (MongoDB 2.2)
  -------------------------
    Gears:          1 small
    Connection URL: mongodb://$OPENSHIFT_MONGODB_DB_HOST:$OPENSHIFT_MONGODB_DB_PORT/
    Database Name:  perldbjks
    Password:       hcYQ2CkSnhn5
    Username:       admin
  mysql-5.1 (MySQL 5.1)
  ---------------------
    Gears:          1 small
    Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/
    Database Name:  perldbjks
    Password:       i8NsVsrD7IM1
    Username:       adminfn52Icz
  perl-5.10 (Perl 5.10)
  ---------------------
    Scaling: x3 (minimum: 3, maximum: 4) on small gears
  haproxy-1.4 (Web Load Balancer)
  -------------------------------
    Gears: Located with perl-5.10, jenkins-client-1
  jenkins-client-1 (Jenkins Client)
  ---------------------------------
    Gears:   Located with perl-5.10, haproxy-1.4
    Job URL: https://jenkins-migrate.int.rhcloud.com/job/perldbjks-build/
5. show app gears
$ rhc app show  perldbjks --gears
ID                       State   Cartridges                             Size  SSH URL
------------------------ ------- -------------------------------------- ----- -------------------------------------------------------------------------
52ce07426cec0e946700082e started postgresql-9.2                         small 52ce07426cec0e946700082e.rhcloud.com
52ce07426cec0e9467000839 started mongodb-2.2                            small 52ce07426cec0e9467000839.rhcloud.com
52ce07426cec0e9467000843 started mysql-5.1                              small 52ce07426cec0e9467000843.rhcloud.com
52ce07426cec0e946700082d started perl-5.10 haproxy-1.4 jenkins-client-1 small 52ce07426cec0e946700082d.rhcloud.com
52d4a69e03ef643f9b000590 started perl-5.10 haproxy-1.4 jenkins-client-1 small 52d4a69e03ef643f9b000590.rhcloud.com
52d4a69e03ef643f9b000591 started perl-5.10 haproxy-1.4 jenkins-client-1 small 52d4a69e03ef643f9b000591.rhcloud.com
Expected results:

6.Could not connect psql 9.2 except DB gear
[perldbjks-migrate.int.rhcloud.com 52ce07426cec0e946700082d]\> psql 
psql: could not connect to server: Connection refused
	Is the server running on host "52ce07426cec0e946700082e-migrate.int.rhcloud.com" (10.9.143.98) and accepting
	TCP/IP connections on port 38451?

7. Show processes from DB gear
[52ce07426cec0e946700082e-migrate.int.rhcloud.com 52ce07426cec0e946700082e]\> psql 
psql (9.2.4)
Type "help" for help.
perldbjks=# \q
[52ce07426cec0e946700082e-migrate.int.rhcloud.com 52ce07426cec0e946700082e]\> ps -ef
UID         PID   PPID  C STIME TTY          TIME CMD
1084     315050      1  0 01:32 ?        00:00:00 /opt/rh/postgresql92/root/usr/bin/postgres
1084     315074 315050  0 01:32 ?        00:00:00 postgres: logger process                  
1084     315076 315050  0 01:32 ?        00:00:00 postgres: checkpointer process            
1084     315077 315050  0 01:32 ?        00:00:00 postgres: writer process                  
1084     315078 315050  0 01:32 ?        00:00:00 postgres: wal writer process              
1084     434435 434256  0 02:10 ?        00:00:00 sshd: 52ce07426cec0e946700082e@pts/1
1084     434455 434435  2 02:10 pts/1    00:00:00 /bin/bash --init-file /usr/bin/rhcsh -i
1084     435132 434455  4 02:10 pts/1    00:00:00 ps -ef

Expected results:
Should  connect to DB

Additional info:

Comment 1 Michal Fojtik 2014-01-23 15:03:44 UTC
XiuJuan Wang: While I'm trying to get the correct devenv to run to reproduce this issue, in meanwhile could you please try to run:

\conninfo

in the psql command. That should print out details about the current database connection (hostname/user/etc). I wonder if they are different than '52ce07426cec0e946700082e-migrate.int.rhcloud.com'.

Comment 2 Michal Fojtik 2014-01-23 15:42:34 UTC
II'm not able to reproduce this issue on the devenv_4265, so this might be migration issue.

Comment 5 Mrunal Patel 2014-01-24 01:44:58 UTC
QE please note that you might hit this issue for existing apps on INT, if a newer app has a uid greater than an older app by 500. We don't intend to fix that on INT, so we are closing the bug.

Comment 6 XiuJuan Wang 2014-01-24 09:03:09 UTC
Hi Mrunal Patel,
DB on my newer app works well after INT upgrade. The DB UID 5639 , is greater 500 than that broken app. So will ingore this issue. Thanks!

ps -ef
UID         PID   PPID  C STIME TTY          TIME CMD
5639     270895 270879  0 03:58 ?        00:00:00 sshd: 52e0b35903ef6410c500026e@pts/4
5639     270902 270895  1 03:58 pts/4    00:00:00 /bin/bash --init-file /usr/bin/rhcsh -i
5639     271748 270902  0 03:58 pts/4    00:00:00 ps -ef
5639     502813      1  0 Jan23 ?        00:00:00 /opt/rh/postgresql92/root/usr/bin/postgres
5639     502844 502813  0 Jan23 ?        00:00:00 postgres: logger process                  
5639     502858 502813  0 Jan23 ?        00:00:00 postgres: checkpointer process            
5639     502859 502813  0 Jan23 ?        00:00:00 postgres: writer process                  
5639     502860 502813  0 Jan23 ?        00:00:00 postgres: wal writer process

Comment 7 Meng Bo 2014-01-28 09:03:21 UTC
The issue can be reproduced on current STG(devenv-stage_655) as well. 

May be regression issue from the fix of this bug
https://bugzilla.redhat.com/show_bug.cgi?id=1051015

Please make sure this will not affect PROD.




Head gear:
[cpython27ssmin2gearsjkns-cstgmigrate.stg.rhcloud.com 52e0af5d2587c8d8e900033f]\> psql
psql: could not connect to server: Connection refused
	Is the server running on host "52e0af5d2587c8d8e9000340-cstgmigrate.stg.rhcloud.com" (10.34.7.149) and accepting
	TCP/IP connections on port 40301?

DB gear:
[52e0af5d2587c8d8e9000340-cstgmigrate.stg.rhcloud.com 52e0af5d2587c8d8e9000340]\> psql
psql (9.2.4)
Type "help" for help.

cpython27ssmin2gearsjkns=#

The app was created on 
    Created:    Jan 23  2:20 PM

Comment 8 Meng Bo 2014-01-28 09:08:08 UTC
UID for the head gear:
[cpython27ssmin2gearsjkns-cstgmigrate.stg.rhcloud.com 52e0af5d2587c8d8e900033f]\> id -u
6122

UID for the DB gear:
[52e0af5d2587c8d8e9000340-cstgmigrate.stg.rhcloud.com 52e0af5d2587c8d8e9000340]\> id -u
1454

Comment 11 Meng Bo 2014-01-29 10:13:03 UTC
Tested on devenv, upgrade from devenv-stage_660 to latest stage branch (devenv-stage_667).

Create scalable app with db added before upgrade, occupy the UIDs between 1004 and 1499 via useradd.
Create new apps after upgrade and migration, which used the UID 1500, 1501,1502,1503

Check the old app, all the dbs can be connected.

Move the bug to verified.

Comment 12 XiuJuan Wang 2014-02-13 06:08:31 UTC
Reproduce on INT(devenv_4369)

My older scaleble app perldbjks's dbs can be not connected (xiuwang+m:redhat):
Head gear's UID is 4703
PSQL gear UID is  2110
Mysql gear UID is 2713
Mongo Gear UID is 2407

New create apps pys UID is 4603,perl UID is 5374,ruby UID is 5774.

Comment 13 XiuJuan Wang 2014-02-13 07:10:43 UTC
My issue was affected by bug 1064219.
Sorry for mistake it.

Comment 14 XiuJuan Wang 2014-02-13 07:11:44 UTC
Accordding to comment 13, move this bug to Verified.


Note You need to log in before you can comment on or make changes to this bug.