Bug 894976

Summary: Can't connect db from haproxy gear after move db gear of scalable application across district
Product: OKD Reporter: Rony Gong 🔥 <qgong>
Component: PodAssignee: Rajat Chopra <rchopra>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: low Docs Contact:
Priority: medium    
Version: 2.xCC: bhatiam, dmcphers, rchopra
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-14 00:48:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
development.log none

Description Rony Gong 🔥 2013-01-14 08:00:33 UTC 
Created attachment 678067 [details]
development.log

Description of problem:
Can't connect to db any more after move db gear of salable application across district.


Version-Release number of selected component (if applicable):
fork_ami_refctr1_404
rhc-1.3.2

How reproducible:
always

Steps to Reproduce:
1.Setup multi node env, and setup 2 districts to add nodes.
2.Create scalable app(nodejs) and embed db(mongodb).
3.Move this mongodb gear across district.
4.Connect to DB from haproxy gear
  
Actual results:
1.Move the db gear of salable application across district
[root@ip-10-84-219-69 openshift]# oo-admin-move --gear_uuid 50f3a0905b9f935407000162 -i domU-12-31-39-06-CC-FF --allow_change_district
URL: http://qsruby19-qgong12.dev.rhcloud.com
Login: qgong
App UUID: 50f396be5b9f93540700009a
Gear UUID: 50f3a0905b9f935407000162
DEBUG: Source district uuid: 50f376b85b9f93ba08000001
DEBUG: Destination district uuid: 50f39fa35b9f93c042000001
DEBUG: Getting existing app 'qsruby19' status before moving
DEBUG: Gear component 'ruby-1.9' was running
DEBUG: Stopping existing app cartridge 'mongodb-2.2' before moving
DEBUG: Performing cartridge level pre-move for embedded mongodb-2.2 for 'qsruby19' on ip-10-84-219-69
DEBUG: Reserved uid '1008' on district: '50f39fa35b9f93c042000001'
DEBUG: Creating new account for gear '50f3a0905b' on domU-12-31-39-06-CC-FF
DEBUG: Moving content for app 'qsruby19', gear '50f3a0905b' to domU-12-31-39-06-CC-FF
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 31429
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 31429 killed;
DEBUG: Performing cartridge level move for embedded mongodb-2.2 for 'qsruby19' on domU-12-31-39-06-CC-FF
DEBUG: Performing cartridge level post-move for embedded mongodb-2.2 for 'qsruby19' on domU-12-31-39-06-CC-FF
DEBUG: Starting cartridge 'mongodb-2.2' in 'qsruby19' after move on domU-12-31-39-06-CC-FF
DEBUG: Fixing DNS and mongo for gear '50f3a0905b' after move
DEBUG: Changing server identity of '50f3a0905b' from 'ip-10-84-219-69' to 'domU-12-31-39-06-CC-FF'
DEBUG: Deconfiguring old app 'qsruby19' on ip-10-84-219-69 after move
Successfully moved 'qsruby19' with gear uuid '50f3a0905b9f935407000162' from 'ip-10-84-219-69' to 'domU-12-31-39-06-CC-FF'
2. After move, check db connection by ssh into the haproxy gear
[qsruby19-qgong12.dev.rhcloud.com ~]\> mongo
+ mongo
+ test 0 -gt 0
+ uopt='--username admin'
+ '[' -n E3YwbatdlQd4 ']'
+ popt='--password E3YwbatdlQd4'
+ egrep '\-\-host|50f3a0905b-qgong12.dev.rhcloud.com'
+ echo
+ '[' -n 50f3a0905b-qgong12.dev.rhcloud.com ']'
+ hopt=50f3a0905b-qgong12.dev.rhcloud.com:38176/admin
+ unset LD_LIBRARY_PATH
+ /usr/bin/mongo 50f3a0905b-qgong12.dev.rhcloud.com:38176/admin --username admin --password E3YwbatdlQd4
MongoDB shell version: 2.2.0
connecting to: 50f3a0905b-qgong12.dev.rhcloud.com:38176/admin
Mon Jan 14 01:59:25 Error: couldn't connect to server 50f3a0905b-qgong12.dev.rhcloud.com:38176 src/mongo/shell/mongo.js:93
exception: connect failed


Expected results:
After move , should still could connect db successfully.

Additional info:
After move, check db connection by ssh into the db gear, connection well
[50f3a0905b-qgong12.dev.rhcloud.com ~]\> mongo
+ mongo
+ test 0 -gt 0
+ uopt='--username admin'
+ '[' -n E3YwbatdlQd4 ']'
+ popt='--password E3YwbatdlQd4'
+ egrep '\-\-host|127.1.248.1'
+ echo
+ '[' -n '' ']'
+ hopt=127.1.248.1:27017/admin
+ unset LD_LIBRARY_PATH
+ /usr/bin/mongo 127.1.248.1:27017/admin --username admin --password E3YwbatdlQd4
MongoDB shell version: 2.2.0
connecting to: 127.1.248.1:27017/admin
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
Comment 1 Rajat Chopra 2013-02-01 16:53:02 UTC 
Fixed in https://github.com/openshift/origin-server/pull/1272
Comment 2 Rony Gong 🔥 2013-02-04 05:24:03 UTC 
Verified on devenv_2765.
1.In multi distct env, move the db gear across district.
[root@ip-10-114-222-187 openshift]# oo-admin-move --gear_uuid 510f42945a4a0f3f0a00002b -i ip-10-202-43-37 --allow_change_district
      ** Notice: The native BSON extension was not loaded. **

      For optimal performance, use of the BSON extension is recommended.

      To enable the extension make sure ENV['BSON_EXT_DISABLED'] is not set
      and run the following command:

        gem install bson_ext

      If you continue to receive this message after installing, make sure that
      the bson_ext gem is in your load path.

URL: http://qsjbosseap-qgong20.dev.rhcloud.com
Login: qgong
App UUID: 510f42405a4a0f3f0a000005
Gear UUID: 510f42945a4a0f3f0a00002b
DEBUG: Source district uuid: 510f41eb5a4a0fab16000001
DEBUG: Destination district uuid: 510f41fa5a4a0fc116000001
DEBUG: Getting existing app 'qsjbosseap' status before moving
DEBUG: Gear component 'jbossas-7' was running
DEBUG: Stopping existing app cartridge 'mongodb-2.2' before moving
DEBUG: Performing cartridge level pre-move for embedded mongodb-2.2 for 'qsjbosseap' on ip-10-114-222-187
DEBUG: Reserved uid '1000' on district: '510f41fa5a4a0fc116000001'
DEBUG: Creating new account for gear '510f42945a' on ip-10-202-43-37
DEBUG: Moving content for app 'qsjbosseap', gear '510f42945a' to ip-10-202-43-37
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Warning: Permanently added '10.114.222.187' (RSA) to the list of known hosts.
Warning: Permanently added '10.202.43.37' (RSA) to the list of known hosts.
Agent pid 14076
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 14076 killed;
DEBUG: Performing cartridge level move for embedded mongodb-2.2 for 'qsjbosseap' on ip-10-202-43-37
DEBUG: Performing cartridge level post-move for embedded mongodb-2.2 for 'qsjbosseap' on ip-10-202-43-37
DEBUG: Starting cartridge 'mongodb-2.2' in 'qsjbosseap' after move on ip-10-202-43-37
DEBUG: Fixing DNS and mongo for gear '510f42945a' after move
DEBUG: Changing server identity of '510f42945a' from 'ip-10-114-222-187' to 'ip-10-202-43-37'
DEBUG: Deconfiguring old app 'qsjbosseap' on ip-10-114-222-187 after move
Successfully moved 'qsjbosseap' with gear uuid '510f42945a4a0f3f0a00002b' from 'ip-10-114-222-187' to 'ip-10-202-43-37'

2. Check the db connection:
[qsjbosseap-qgong20.dev.rhcloud.com ~]\> mongo
MongoDB shell version: 2.2.2
connecting to: 510f42945a-qgong20.dev.rhcloud.com:38031/admin
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
> show dbs;
admin	0.03125GB
local	(empty)
qsjbosseap	0.03125GB