Bug 965490

Summary: The account can't connect to server any more after migrate the application that embed with switchyard
Product: OpenShift Online Reporter: Rony Gong 🔥 <qgong>
Component: ContainersAssignee: Paul Morie <pmorie>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.xCC: dmcphers, jhou, pmorie, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-11 04:07:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
The app data in mongodb after migration
none
development.log when rhc domain show
none
development.log none

Description Rony Gong 🔥 2013-05-21 10:01:19 UTC
Created attachment 750960 [details]
The app data in mongodb after migration

Description of problem:
The account can't connect to server any more after migrate the application that embed with switchyard

After do migration from v1 to v2. This account couldn't connect the server any more. eg:

rhc setup:
The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server (such as a Web proxy or firewall). Please verify that you can access the OpenShift server
https://int.openshift.redhat.com/broker/rest/domains/qgong1/applications

please see the attachment for the detail development.log



Version-Release number of selected component (if applicable):
migrate from devenv_stag_353  to devenv_3253
migrate from INT(sprint27) to INT(devenv_3250)
How reproducible:
Always

Steps to Reproduce:
1.Create jbosseap/jbossas application, then embed swithchyard
2.Do migration to latest build
3.The use this account, do action: rhc domain show

Actual results:
[openshift@F17x64-openshift ~]$ rhc domain show
The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server (such as a Web proxy or firewall). Please verify that you can access the OpenShift server
https://ec2-107-22-115-240.compute-1.amazonaws.com/broker/rest/domains/qgong111/applications


Expected results:
Could list application successfully.

Additional info:
It could work well again if delete this app's data in application collection in mogondb.

Comment 1 Rony Gong 🔥 2013-05-21 10:03:08 UTC
Created attachment 750962 [details]
development.log when rhc domain show

Comment 2 Bill DeCoste 2013-05-21 19:41:12 UTC
Running the migration script on a single devenv results in a successful 'rhc domain show'

Comment 3 Jianwei Hou 2013-05-22 10:26:19 UTC
This can be reproduced after migrated on devenv_3258

1. Launch devenv-stage_353
2. Prepare test data
3. Add V1 marker to node
#touch /var/lib/openshift/.settings/v1_cartridge_format
4. scp -r ~/devenv-local and /etc/yum.repos.d/* from devenv_3258 instance to same location on devenv-stage_353 instance
5. yum update -y --enablerepo devenv-local
6. Run oo-admin-clear-pending-ops
7. Remove the v1 marker from the node
#rm -f /var/lib/openshift/.settings/v1_cartridge_format
8. Restart mcollective
#service mcollective restart
9. Clear the broker cache
#oo-admin-broker-cache --clear --console
10. Restart the rhc-broker
#server rhc-broker restart
12. Run migrate-mongo-2.0.28 
13. Run rhc-admin-migrate --version 2.0.28

After migration the acount with one ruby-1.8 app and one jbossas+switchyard app can not communicate with server any more, the error was same with the description.

According to development.log, it was blocked at 'get_profile_for_component' method.

Comment 4 Jianwei Hou 2013-05-22 10:26:43 UTC
Created attachment 751652 [details]
development.log

Comment 5 Bill DeCoste 2013-05-22 15:21:26 UTC
Not sure what's going on but have recreated. Doesn't seem to be related to the gear. Seeing nothing in development.log. Can't do anything from rhc.

[root@ip-10-10-42-16 ~]# rhc domain show
The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server
(such as a Web proxy or firewall). Please verify that you can access the OpenShift server https://localhost/broker/rest/api
[root@ip-10-10-42-16 ~]# rhc domain show -d
DEBUG: Using config file /root/.openshift/express.conf
DEBUG: Authenticating with RHC::Auth::Token
DEBUG: Connecting to https://localhost/broker/rest/api
DEBUG: Getting all domains
DEBUG: Client supports API versions 1.1, 1.2, 1.3, 1.4
DEBUG: Using token authentication
DEBUG: Request GET https://localhost/broker/rest/api
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 500  519 ms
DEBUG: Response did not include a message from server: invalid character at "<!DOCTYPE "
The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server
(such as a Web proxy or firewall). Please verify that you can access the OpenShift server https://localhost/broker/rest/api
[root@ip-10-10-42-16 ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd: [Wed May 22 11:03:38 2013] [warn] module passenger_module is already loaded, skipping
[Wed May 22 11:03:38 2013] [warn] NameVirtualHost *:443 has no VirtualHosts
[Wed May 22 11:03:38 2013] [warn] NameVirtualHost *:80 has no VirtualHosts
                                                           [  OK  ]

Comment 6 Paul Morie 2013-05-29 18:02:39 UTC
The problem here is that the name of the switchyard cartridge changed from V1 -> V2.  I'm making changes so that the name and other information in the manifest will be identical.

Comment 7 Paul Morie 2013-05-29 22:09:42 UTC
PR submitted.

Comment 9 Rony Gong 🔥 2013-05-30 09:29:28 UTC
Retest on migration from devenv_stage_353 to devenv_3290 ---Failed
After migration, the account(swithyard added) could connect server well(rhc domain show)

But when check more, can't control the switchyard cartridge well, the server development.log show;
2013-05-30 05:23:55.901 [DEBUG] DEBUG: server results: Failed to get cartridge 'switchyard-0.6' from  in gear 536256880632503637901312: Cartridge directory not found for switchyard-0.6 (pid:19297)
2013-05-30 05:23:55.901 [ERROR] Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support. (pid:19297)

The switchyard cartridge directory deleted after migration actually.

[qgong@dhcp-10-94 openshift-quickstarter]$ rhc cartridge stop -a qjbosseap -c switchyard-0.6
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support..
Reference ID: 061e1becdd28d205f0b4e725f81d8446

Comment 10 Paul Morie 2013-05-30 18:00:43 UTC
Multiple PRs submitted, tldr:

1. The V1 switchyard cartridge _does not_ create a directory for switchyard.  It only adds switchyard to the JBoss AS/EAP module path.
2. In V2, there is a discrete directory which will contain but EAP and AS env vars in its env directory.
3. I had to infer the presence of switchyard in an app in a special case for the migration.  Now, switchyard should be accurately detected and both a switchyard directory and a switchyard-0.6 symlink to that directory will be present in the migrated application.

Comment 11 openshift-github-bot 2013-05-30 21:31:03 UTC
Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/953ee9ba1fcf64f8128adf78a27267797fcec761
Bug 965490: Fix V1 -> V2 migrations for switchyard

Comment 12 Rony Gong 🔥 2013-05-31 07:54:03 UTC
Verified on migration from devenv_stage_353 to devenv_3296
After migration, the account connect server well, and could control switchyard well.
[root@dhcp-10-94 smygaeapp]# rhc cartridge stop  -a qjbosseap -c switchyard-0.6
RESULT:
switchyard-0.6 stopped