Bug 1118703 - [JBossFuse]Fail to create two fuse apps continuously
Summary: [JBossFuse]Fail to create two fuse apps continuously
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Fuse
Version: 2.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Aileen
QA Contact: David Simansky
URL:
Whiteboard:
Depends On: 1109058 1120413 1121815
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-11 09:33 UTC by xjia
Modified: 2017-05-31 18:22 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-31 18:22:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description xjia 2014-07-11 09:33:50 UTC
Description of problem:
When user creates two fuse apps continuously, the password of fuse management console are same.  
And check the warning message when creating app, find the warning messages are not same. 
For instance: when creating fuse1, it prompts "  Setting up fabric container
Generating password
Configuring zookeeper server"
when creating fuse2, it prompts "  Setting up fabric container
Using provided password from domain
Using provided url from domain"

If deleting one fuse app ,and create another fuse app, the password will not be same.

Version-Release number of selected component (if applicable):
fuse-openshift-cartridge-openshift-enterprise-rpm-6.1.0.redhat.385-2-fc18.noarch.rpm
2.1.z/2014-07-07.1

How reproducible:
always

Steps to Reproduce:
1. rhc app create fuse1 fuse-1.0.0
2. rhc app create fuse2 fuse-1.0.0
3. rhc app delete fuse2
4. rhc app create fuse2 fuse-1.0.0

Actual results:
Step1 :
[root@broker ~]# rhc app create fuse1 fuse-1.0.0
Application Options
-------------------
Domain:     jia
Cartridges: fuse-1.0.0
Gear Size:  default
Scaling:    no

Creating application 'fuse1' ... Started Fuse cart
done

  Setting up fabric container
Generating password
Configuring zookeeper server

Fuse created successfully.  Please make note of these credentials:

   Console URL:        http://fuse1-jia.fuse.com
   Console User:       admin
   Console Password:   nDJrRqXwShVs
   Zookeeper URL:      fuse1-jia.fuse.com:58627
   Zookeeper Password: nDJrRqXwShVs

   The Fuse Management Console will be available shortly at: http://fuse1-jia.fuse.com

Fabric( 21063) is started successfully

Waiting for your DNS name to be available ... done

Initialized empty Git repository in /root/fuse1/.git/
reverse mapping checking getaddrinfo for dhcp-128-178.nay.redhat.com [10.66.128.178] failed - POSSIBLE BREAK-IN ATTEMPT!

Your application 'fuse1' is now available.

  URL:        http://fuse1-jia.fuse.com/
  SSH to:     53bf9b4241312e7c530002c4.com
  Git remote: ssh://53bf9b4241312e7c530002c4.com/~/git/fuse1.git/
  Cloned to:  /root/fuse1

Run 'rhc show-app fuse1' for more details about your app.


step 2:
[root@broker ~]# rhc app create fuse2 fuse-1.0.0
Application Options
-------------------
Domain:     jia
Cartridges: fuse-1.0.0
Gear Size:  default
Scaling:    no

Creating application 'fuse2' ... Started Fuse cart
done

  Setting up fabric container
Using provided password from domain
Using provided url from domain

Fuse created successfully.  Please make note of these credentials:

   Console URL:        http://fuse2-jia.fuse.com
   Console User:       admin
   Console Password:   nDJrRqXwShVs
   Zookeeper URL:      fuse1-jia.fuse.com:58627
   Zookeeper Password: nDJrRqXwShVs

   The Fuse Management Console will be available shortly at: http://fuse2-jia.fuse.com

Fabric( 22941) is started successfully
WARNING: Creating fabric containers directly from
         openshift is supported in a limited way
         as such containers can not be fully managed
         by fabric.

Waiting for your DNS name to be available ... done

Initialized empty Git repository in /root/fuse2/.git/
reverse mapping checking getaddrinfo for dhcp-128-178.nay.redhat.com [10.66.128.178] failed - POSSIBLE BREAK-IN ATTEMPT!

Your application 'fuse2' is now available.

  URL:        http://fuse2-jia.fuse.com/
  SSH to:     53bf9b8141312e7c530002dc.com
  Git remote: ssh://53bf9b8141312e7c530002dc.com/~/git/fuse2.git/
  Cloned to:  /root/fuse2

Run 'rhc show-app fuse2' for more details about your app.


Step 4:
[root@broker ~]# rm -rf fuse2 ; rhc app create fuse2 fuse-1.0.0
Application Options
-------------------
Domain:     jia
Cartridges: fuse-1.0.0
Gear Size:  default
Scaling:    no

Creating application 'fuse2' ... Started Fuse cart
done

  Setting up fabric container
Generating password
Configuring zookeeper server

Fuse created successfully.  Please make note of these credentials:

   Console URL:        http://fuse2-jia.fuse.com
   Console User:       admin
   Console Password:   UeZPUanAGJLA
   Zookeeper URL:      fuse2-jia.fuse.com:54102
   Zookeeper Password: UeZPUanAGJLA

   The Fuse Management Console will be available shortly at: http://fuse2-jia.fuse.com

Fabric( 31552) is started successfully

Waiting for your DNS name to be available ... done

Initialized empty Git repository in /root/fuse2/.git/
reverse mapping checking getaddrinfo for dhcp-128-178.nay.redhat.com [10.66.128.178] failed - POSSIBLE BREAK-IN ATTEMPT!

Your application 'fuse2' is now available.

  URL:        http://fuse2-jia.fuse.com/
  SSH to:     53bf9d7541312e7c5300032d.com
  Git remote: ssh://53bf9d7541312e7c5300032d.com/~/git/fuse2.git/
  Cloned to:  /root/fuse2

Run 'rhc show-app fuse2' for more details about your app.

Expected results:
Step 2:
Using different password for each app.

Additional info:

Comment 2 Kurt T Stam 2014-07-15 13:32:04 UTC
We are referencing a system variable in the config file. It seems that the value of this system param is not changing quickly enough. Note that I don't know the true implementation of this; just guessing.

Comment 3 Aileen 2014-07-15 13:57:59 UTC
Is there some way to get around this? What do other cartridges do?

Comment 4 Ben Parees 2014-07-15 14:12:02 UTC
The generate_password referenced here is a function call, not an environment variable:

export OPENSHIFT_FUSE_ZOOKEEPER_PASSWORD=$(generate_password)

it's calling this:

https://github.com/openshift/origin-server/blob/master/node/misc/usr/lib/cartridge_sdk/bash/sdk#L171

So it is not a race condition/timing issue.  There is something else going on here where the 2nd Fuse deployment is considering itself a container, rather than a fabric server.  Note the output during the 2nd deployment:

WARNING: Creating fabric containers directly from
         openshift is supported in a limited way
         as such containers can not be fully managed
         by fabric.

Also notice that the zookeeper URL in the 2nd deployment corresponds to the zookeper URL from the 1st deployment.  So again, the 2nd deployment appears to be picking up domain info and using it.

You can also see that in the output here (again from the 2nd deployment):
Setting up fabric container
Using provided password from domain
Using provided url from domain

Comment 5 Kurt T Stam 2014-07-15 14:35:14 UTC
https://github.com/jboss-fuse/fuse-openshift-cartridge/blob/jboss-fuse-6.1.x-379/bin/control#L71 adds it as a domain level variable. So the second fuse app in the domain is using the same zookeeper (and thus the same passwords). This looks to be a feature. 

Aileen, can you verify that this is intended behavior, and this needs documentation?

Comment 6 Ben Parees 2014-07-15 15:00:21 UTC
After discussions with James, the design was intended to be that if you launched a 2nd fuse container from the fuse cli or fuse webconsole, that container would join the fabric.

The logic to implement that results in the same behavior for the rhc cli (if you launch a 2nd fuse app in the same openshift domain, it will join the exist fabric).

That is why it has the same password.

For now, if users want a 2nd fuse fabric deployed, they will need to create a 2nd domain.

QE should confirm, however, that when deploying multiple fuse apps to the same domain, they do properly function as a cluster of fabric servers.

Comment 7 xjia 2014-07-16 01:00:59 UTC
Hi Ben:
Could you explain why if delete the 2nd fuse app and create 3rd fuse app, the password is changed?
Thanks

Comment 8 Ben Parees 2014-07-16 21:01:54 UTC
No I don't think I can explain that.  But hopefully this whole issue will be resolved once I finish reworking the setup/install scripts.  Unfortunately I've run into an openshift bug that is preventing me from reworking the scripts correctly.

Comment 9 Ben Parees 2014-07-17 19:44:41 UTC
The install/setup/control scripts have been refactored such that the env variable setting should work now.

Note that with this refactoring, the domain env variables are used during install to determine if this fuse app should be added to an existing fabric, or created as a new fabric.  This runs into bug 1120413 in which the domain env variable values are not available to the install script.

What this means for QE:

Until 1120413 is fixed in OSE, every "rhc app create <fuseapp> fuse" will create a standalone fuse instance.

If it is urgent to get the fix for 1120413 into an OSE patch, that can be arranged, just let me know.

The expectation, once 1120413 is fixed, is that additional fuse apps in the same domain will share configuration (including passwords) because it's one clustered environment, not distinct environments.

Comment 10 xjia 2014-07-18 02:17:09 UTC
Could you build an rpm package for this bug? Then i can start to test it. Thanks.

Comment 11 xjia 2014-07-18 07:50:53 UTC
Until bug1120413 is fixed in OSE, QE can verify this bug.

Comment 12 xjia 2014-07-21 07:55:50 UTC
Test it again. This issue is still not fixed.

Create fuse1, check domain.env_vars in mongo:
		{
			"key" : "OPENSHIFT_FUSE_DOMAIN_ZOOKEEPER_URL",
			"value" : "fuse1-jia.dev.rhcloud.com:38032",
			"component_id" : ObjectId("53ccf97c6d011716f9000141")
		},
		{
			"key" : "OPENSHIFT_FUSE_DOMAIN_ZOOKEEPER_PASSWORD",
			"value" : "A7HGZegGHL3L",
			"component_id" : ObjectId("53ccf97c6d011716f9000141")
		}

Create fuse2, check domain.env_vars in mongo(component_id is recovered!):
		{
			"key" : "OPENSHIFT_FUSE_DOMAIN_ZOOKEEPER_URL",
			"value" : "fuse1-jia.dev.rhcloud.com:38032",
			"component_id" : ObjectId("53ccfbc76d011716f900015b")
		},
		{
			"key" : "OPENSHIFT_FUSE_DOMAIN_ZOOKEEPER_PASSWORD",
			"value" : "A7HGZegGHL3L",
			"component_id" : ObjectId("53ccfbc76d011716f900015b")
		}

Now if delete fuse2 app, these variables will be removed. This domain.env_vars might be recorded all the used app's component_id instead of only one component_id.

Comment 13 Ben Parees 2014-07-21 16:01:11 UTC
Matej, please mark this bug ON_QA after respinning the fuse cartridge RPM.  The changes have already been delivered to the OSE branch of the cartridge github project.

Thanks.

Comment 14 Ben Parees 2014-07-21 16:02:15 UTC
xjia:  It sounds like you've hit another bug, in which domain env vars are removed from the domain when a single app is removed from the domain?  If so, please open that as a separate bug against openshift broker.

Comment 15 xjia 2014-07-22 01:21:47 UTC
Create bug1121815 to track #c12 issue.

Comment 16 Ben Parees 2014-07-22 21:10:29 UTC
Per 1121815 being closed as working as designed, I have opened a JIRA against Fuse to track the need for Fuse to potentially modify the behavior, or documentation to account for this:
https://issues.jboss.org/browse/ENTESB-1732

Note that there is another minor bug in the current Fuse cartridge in which the domain env variables are created by every fuse app, instead of just the first one.  I will fix that momentarily, that is the reason deleting fuse2 removes the environment variables.  They should only be removed when fuse1 is removed.

Comment 17 Ben Parees 2014-07-22 21:14:18 UTC
Fix such that domain env vars will be tied to the first fuse app created:
https://github.com/jboss-fuse/fuse-openshift-cartridge/pull/28

Not sure when the RPM will be respun.

Comment 18 David Simansky 2014-07-23 08:47:46 UTC
@Ben
With the latest respin I can't set password manually during app creation. Not sure if it's just combination of the OSE broker bug and this PR fix or something else is wrong. However, when I try to set the env variable it's getting ignored, was working in previous build.


I've tried 2 variants of command:

rhc app create fuse fuse-1 -e OPENSHIFT_FUSE_DOMAIN_ZOOKEEPER_PASSWORD=admin

rhc app create fuse fuse-1 -e OPENSHIFT_FUSE_ZOOKEEPER_PASSWORD=admin -e OPENSHIFT_FUSE_DOMAIN_ZOOKEEPER_PASSWORD=admin

Comment 19 Ben Parees 2014-07-23 14:31:27 UTC
@David

the latest RPM seems to do the right thing for me. (you should NOT set the domain env var, just set the regular one):

[bparees@bparees ~/scratch]$ rhc app create fuse1 fuse -e 
OPENSHIFT_FUSE_ZOOKEEPER_PASSWORD=admin

Using fuse-1.0.0 (JBoss Fuse 6.1.0 EA) for 'fuse'

Application Options
-------------------
Domain:                bparees
Cartridges:            fuse-1.0.0
Gear Size:             default
Scaling:               no
Environment Variables: OPENSHIFT_FUSE_ZOOKEEPER_PASSWORD=admin

Creating application 'fuse1' ... Started Fuse cart
done

  Setting up fabric container
Using provided password from cartridge
Configuring zookeeper server

Fuse created successfully.  Please make note of these credentials:

   Console URL:        http://fuse1-bparees.dev.rhcloud.com
   Console User:       admin
   Console Password:   admin
   Zookeeper URL:      fuse1-bparees.dev.rhcloud.com:38032
   Zookeeper Password: admin

   The Fuse Management Console will be available shortly at: http://fuse1-bparees.dev.rhcloud.com

Cartridge fuse exposed URL tls:fuse1-bparees.dev.rhcloud.com:2303
Fabric( 32020) is started successfully

Your public SSH key must be uploaded to the OpenShift server to access code.
Upload now? (yes|no)
y

Since you do not have any keys associated with your OpenShift account, your new
key will be uploaded as the 'default' key.

Uploading key 'default' ... done

Waiting for your DNS name to be available ... done

Cloning into 'fuse1'...

Your application 'fuse1' is now available.

  URL:        http://fuse1-bparees.dev.rhcloud.com/
  SSH to:     53cffea77981b9cc0b000016.rhcloud.com
  Git remote: ssh://53cffea77981b9cc0b000016.rhcloud.com/~/git/fuse1.git/
  Cloned to:  /home/bparees/scratch/fuse1

Run 'rhc show-app fuse1' for more details about your app.
[bparees@bparees ~/scratch]$

Comment 20 xjia 2014-07-24 03:16:57 UTC
Version:
openshift-origin-cartridge-fuse-6.1.0.redhat.387-3.el6op.noarch.rpm
devenv_5004

Fail to create the second fuse app.
[root@ip-10-186-29-141 ~]# rhc app create fuse2 fuse-1 
Using fuse-1.0.0 (JBoss Fuse 6.1.0 EA) for 'fuse-1'

Application Options
-------------------
Domain:     jia
Cartridges: fuse-1.0.0
Gear Size:  default
Scaling:    no

Creating application 'fuse2' ... 
CART_PROPERTIES: username=admin
CART_PROPERTIES: password=rt9VLVB6LknR
client: JAVA_HOME not set; results may vary
retrying (attempt 1) ...
retrying (attempt 2) ...
retrying (attempt 3) ...
Failed to get the session.

Error log:
2014-07-24 03:16:11.137 [DEBUG] DEBUG: [#<MCollective::RPC::Result:0x000000070e0608 @agent="openshift", @action="cartridge_do", @results={:sender=>"ip-10-186-29-141", :statuscode=>1, :statusmsg=>"cartridge_do_action failed 157. Output Shell command '/sbin/runuser -s /bin/sh 53d0b2130300ece972000001 -c \"exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1002' /bin/sh -c \\\"/var/lib/openshift/53d0b2130300ece972000001/fuse/bin/install --version 1.0.0\\\"\"' returned an error. rc=1\nCLIENT_RESULT: Setting up fabric container\nCLIENT_RESULT: Using provided password from domain\nCLIENT_RESULT: Using provided url from domain\nCLIENT_RESULT: \nCLIENT_RESULT: Fuse created successfully.  Please make note of these credentials:\nCLIENT_RESULT: \nCLIENT_RESULT:    Console URL:        http://fuse2-jia.dev.rhcloud.com\nCLIENT_RESULT:    Console User: [HIDDEN] (Request ID: ea1380d86a71da28d10da4cf2f1247e1) (pid:79)
2014-07-24 03:16:11.148 [DEBUG] DEBUG: MCollective Response Time (execute_direct: configure): 148.667768634s  (Request ID: ea1380d86a71da28d10da4cf2f1247e1) (pid:79)
2014-07-24 03:16:11.148 [DEBUG] DEBUG: server results: Shell command '/sbin/runuser -s /bin/sh 53d0b2130300ece972000001 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1002' /bin/sh -c \"/var/lib/openshift/53d0b2130300ece972000001/fuse/bin/install --version 1.0.0\""' returned an error. rc=1 (pid:79)
2014-07-24 03:16:11.148 [DEBUG] DEBUG: server results: (pid:79)
2014-07-24 03:16:11.148 [ERROR] Failed to get the session. (pid:79)
2014-07-24 03:16:11.149 [ERROR] /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.27.1/lib/openshift/mcollective_application_container_proxy.rb:2763:in `parse_result'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.27.1/lib/openshift/mcollective_application_container_proxy.rb:2944:in `run_cartridge_command'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.27.1/lib/openshift/mcollective_application_container_proxy.rb:816:in `add_component'

Comment 21 Ben Parees 2014-07-25 19:16:25 UTC
What I'm finding is that the fabric join operations simply take way too long and are being timed out by openshift.  Specifically i'm referring to this command which is run during install for the 2nd fuse app:

    ${FUSE_BIN_DIR}/client -r 3 -d 10 -h ${OPENSHIFT_FUSE_IP} -a ${OPENSHIFT_FUSE_SSH_PORT} -u admin -p ${OPENSHIFT_FUSE_ZOOKEEPER_PASSWORD} "wait-for-service io.fabric8.boot.commands.service.Join"

If I comment those commands (both "client" commands) out of the install script and bring up a 2nd fuse app server and then run those commands by hand while SSHed into the gear, the container is joined to the fabric, but it takes multiple minutes to complete.

As such, I think we need the Fuse team to investigate why the operation is taking so long (I don't think it used to).

Comment 22 xjia 2014-08-15 09:39:08 UTC
Could you move this bug to "ON_QA"?
Version:
openshift-origin-cartridge-fuse-6.1.0.redhat.389-1.el6op.noarch.rpm

Verify:
[root@broker ~]# rhc app create fuse1 fuse -g xpaas
Using fuse-1.0.0 (JBoss Fuse 6.1.0 EA) for 'fuse'

Application Options
-------------------
Domain:     xuan
Cartridges: fuse-1.0.0
Gear Size:  xpaas
Scaling:    no

Creating application 'fuse1' ... Started Fuse cart
done

  Setting up fabric container
Generating password
Configuring zookeeper server

Fuse created successfully.  Please make note of these credentials:

   Console URL:        http://fuse1-xuan.fuse.com
   Console User:       admin
   Console Password:   WGGrHr6CyNMV
   Zookeeper URL:      fuse1-xuan.fuse.com:59481
   Zookeeper Password: WGGrHr6CyNMV

   The Fuse Management Console will be available shortly at: http://fuse1-xuan.fuse.com

Cartridge fuse exposed URL tls:fuse1-xuan.fuse.com:2303
Cartridge fuse exposed URL tls:fuse1-xuan.fuse.com:2304
Cartridge fuse exposed URL tls:fuse1-xuan.fuse.com:2305
Cartridge fuse exposed URL tls:fuse1-xuan.fuse.com:2306
Cartridge fuse exposed URL tls:fuse1-xuan.fuse.com:2307
Fabric( 10513) is started successfully

Waiting for your DNS name to be available ... done

Initialized empty Git repository in /root/fuse1/.git/
reverse mapping checking getaddrinfo for dhcp-128-56.nay.redhat.com [10.66.128.56] failed - POSSIBLE BREAK-IN ATTEMPT!

Your application 'fuse1' is now available.

  URL:        http://fuse1-xuan.fuse.com/
  SSH to:     53edd3fe41312ed18a00006a.com
  Git remote: ssh://53edd3fe41312ed18a00006a.com/~/git/fuse1.git/
  Cloned to:  /root/fuse1

Run 'rhc show-app fuse1' for more details about your app.
[root@broker ~]# rhc app create fuse2 fuse -g xpaas --no-git
Using fuse-1.0.0 (JBoss Fuse 6.1.0 EA) for 'fuse'

Application Options
-------------------
Domain:     xuan
Cartridges: fuse-1.0.0
Gear Size:  xpaas
Scaling:    no

Creating application 'fuse2' ... Fuse cart already started
done

  Setting up fabric container
Using provided password from domain
Using provided url from domain

Fuse created successfully.  Please make note of these credentials:

   Console URL:        http://fuse2-xuan.fuse.com
   Console User:       admin
   Console Password:   WGGrHr6CyNMV
   Zookeeper URL:      fuse1-xuan.fuse.com:59481
   Zookeeper Password: WGGrHr6CyNMV

   The Fuse Management Console will be available shortly at: http://fuse2-xuan.fuse.com

Fabric( 12028) is started successfully
WARNING: Creating fabric containers directly from
         openshift is supported in a limited way
         as such containers can not be fully managed
         by fabric.
Cartridge fuse exposed URL tls:fuse2-xuan.fuse.com:2303
Cartridge fuse exposed URL tls:fuse2-xuan.fuse.com:2304
Cartridge fuse exposed URL tls:fuse2-xuan.fuse.com:2305
Cartridge fuse exposed URL tls:fuse2-xuan.fuse.com:2306
Cartridge fuse exposed URL tls:fuse2-xuan.fuse.com:2307

Waiting for your DNS name to be available ... done

Your application 'fuse2' is now available.

  URL:        http://fuse2-xuan.fuse.com/
  SSH to:     53edd41c41312ed18a000084.com
  Git remote: ssh://53edd41c41312ed18a000084.com/~/git/fuse2.git/

Run 'rhc show-app fuse2' for more details about your app.
[root@broker ~]# rhc app create fuse3 fuse -g xpaas --no-git
Using fuse-1.0.0 (JBoss Fuse 6.1.0 EA) for 'fuse'

Application Options
-------------------
Domain:     xuan
Cartridges: fuse-1.0.0
Gear Size:  xpaas
Scaling:    no

Creating application 'fuse3' ... Fuse cart already started
done

  Setting up fabric container
Using provided password from domain
Using provided url from domain

Fuse created successfully.  Please make note of these credentials:

   Console URL:        http://fuse3-xuan.fuse.com
   Console User:       admin
   Console Password:   WGGrHr6CyNMV
   Zookeeper URL:      fuse1-xuan.fuse.com:59481
   Zookeeper Password: WGGrHr6CyNMV

   The Fuse Management Console will be available shortly at: http://fuse3-xuan.fuse.com

Fabric( 16038) is started successfully
WARNING: Creating fabric containers directly from
         openshift is supported in a limited way
         as such containers can not be fully managed
         by fabric.
Cartridge fuse exposed URL tls:fuse3-xuan.fuse.com:2303
Cartridge fuse exposed URL tls:fuse3-xuan.fuse.com:2304
Cartridge fuse exposed URL tls:fuse3-xuan.fuse.com:2305
Cartridge fuse exposed URL tls:fuse3-xuan.fuse.com:2306
Cartridge fuse exposed URL tls:fuse3-xuan.fuse.com:2307

Waiting for your DNS name to be available ... done

Your application 'fuse3' is now available.

  URL:        http://fuse3-xuan.fuse.com/
  SSH to:     53edd45a41312ed18a000098.com
  Git remote: ssh://53edd45a41312ed18a000098.com/~/git/fuse3.git/

Run 'rhc show-app fuse3' for more details about your app.

Comment 23 xjia 2014-08-15 09:49:35 UTC
According to #comment22, verify this bug.

Comment 24 Marek Schmidt 2014-08-27 10:35:24 UTC
Reopening,

This still frequently occurs with the current build (openshift-origin-cartridge-fuse-6.1.0.redhat.390-2.el6op.noarch.rpm ) in our testing environment (OSE 2.1 on OS1 internal cloud)


Application Options
-------------------
Domain:     foobar
Cartridges: fuse-1.0.0
Gear Size:  default
Scaling:    no

Creating application 'fuse2' ... CART_PROPERTIES: username=admin
CART_PROPERTIES: password=YMjyJQRh_ewH
client: JAVA_HOME not set; results may vary
retrying (attempt 1) ...
retrying (attempt 2) ...
retrying (attempt 3) ...
Failed to get the session.

Comment 25 Marek Schmidt 2014-08-27 11:36:51 UTC
The workaround for me is to increase the number of retries on the

${FUSE_BIN_DIR}/client -r 3 -d 10 -h ${OPENSHIFT_FUSE_IP} -a ${OPENSHIFT_FUSE_SSH_PORT} -u admin -p ${OPENSHIFT_FUSE_PASSWORD} "wait-for-service io.fabric8.boot.commands.service.JoinAvailable"

line


It approx. takes 8 retries before it connects in my testing env,  so the current 3 looks over-optimistic... The question is whether the bug is that it takes too long, or if just the number of retries is too low.

Comment 26 Ben Parees 2014-08-27 14:22:54 UTC
discussed with Fuse team, increasing timeout:

https://github.com/jboss-fuse/fuse-openshift-cartridge/pull/55
https://github.com/jboss-fuse/fuse-openshift-cartridge/pull/56

will need new rpm spin to get changes.

Comment 27 Marek Schmidt 2014-08-27 15:44:29 UTC
Is there any particular reason for the retries to be as small as possible? The only downside of having it much larger would be that if indeed something wrong happens, the timeout will occur a bit later..

Can we just set it to e.g. 42 ?

Comment 28 Ben Parees 2014-08-27 16:46:56 UTC
if it's too high we'll hit the openshift timeout, i think it's probably better/more informative if we give up on retries rather than hit that generic limit.

Comment 29 xjia 2014-09-01 07:12:51 UTC
Verify it on multi-nodes env.

Version:
openshift-origin-cartridge-amq-6.1.0.redhat.391-1.el6op.noarch.rpm
openshift-origin-cartridge-fuse-6.1.0.redhat.390-3.el6op.noarch.rpm

Step:
[root@broker ~]# rhc app create fuse01 fuse  --no-git 
Checking for a domain ... none

Applications are grouped into domains - each domain has a unique name (called a namespace) that becomes part of your public application URL. You will not be able to
create an application without completing this step.

Please enter a namespace (letters and numbers only): xuan
Your domain 'xuan' has been successfully created

Using fuse-1.0.0 (JBoss Fuse 6.1.0 EA) for 'fuse'

Application Options
-------------------
Domain:     xuan
Cartridges: fuse-1.0.0
Gear Size:  default
Scaling:    no

Creating application 'fuse01' ... fuse01 already started
done

  Generating password
Configuring zookeeper server

Application fuse01 created successfully.  Please make note of these credentials:

   Console URL:        https://fuse01-xuan.fuse-vsphere.com
   Console User:       admin
   Console Password:   UhDTd7gc81jp
   Zookeeper URL:      fuse01-xuan.fuse-vsphere.com:56696
   Zookeeper Password: UhDTd7gc81jp
   SSL Certificate:    

-----BEGIN CERTIFICATE-----
MIIE7TCCAtWgAwIBAgIEMFaBpjANBgkqhkiG9w0BAQsFADAnMSUwIwYDVQQDExxmdXNlMDEteHVh
bi5mdXNlLXZzcGhlcmUuY29tMB4XDTE0MDkwMTA3MDYwNloXDTI0MDgyOTA3MDYwNlowJzElMCMG
A1UEAxMcZnVzZTAxLXh1YW4uZnVzZS12c3BoZXJlLmNvbTCCAiIwDQYJKoZIhvcNAQEBBQADggIP
ADCCAgoCggIBANADxNkjkY6OfoAYmQMqGhQgE5kY9EEnnO05eV5POkQ6eFV7yQ/xEnowPkw5PaTV
I8N3tk5AlSTXEcFGDGjQ4jUPow1n9vhRj2EZxNa0otfkqGzBNKCTGQ3wBgEOsto8FtheNghki2Qi
VO97kpgTOIUTBF5ihTenl+pTB2ouMfFevK1S0PrsQh7FJlEJ0RF2NyvOSoS09lDD+lP8lg8RFrjz
cOExefDdmlWnBy3SvbYFOWS5+o52us/fq4lWOyE91MrRdZ8nHZs2CnQGUJBJBvjbv+r9VA787U//
VAn49yhXQ3N/1lQflU047IFJWpMQt9qwPbeWgVluuHaekv32QSfw3aQzi1pfMFb+DUhy6ux5lhFe
QXuVlfUBSKfg6Q5PsQVsLBazi7XsIn+vLbbUGNbrD0XKTTxP7qTqHBaUzK5lsHosaC7F/28TKlR0
5tXTcohPTPnpfIvtlU55PTZNDYDF3HJ4rroCaKT/66DmclaInONMrbCefMdki59mfJlqMKfmXfco
sXXOHHdtLgkw5I50xQVU9p/Dw6/GCPVda+whVa69khJY30QjxgJLALnv2Okk66ZEORwps8LSGuSq
uAghApKS8RGLWlgSbdz8ZT/UxTOevkwmD7pdC0nnWeQEbMzMghjRzMvnQ2uCbBOG9rKT9SE7HtW2
73Uo+ksdWN5tAgMBAAGjITAfMB0GA1UdDgQWBBTkXyaCwUhgb00QKUd4cqaax6I9cDANBgkqhkiG
9w0BAQsFAAOCAgEAak6odv95XGCP/WImz7cIEESxQPmD1WLKBEJNJTDRhqynwj/RUVNgFt1yH6Ys
isXFP3NwLnX6+/+p8focO5zviT/LOUQnUj72ZrAmW2zxIpru70j3uExsXXziQtjsAwJ0u/UD3kq8
48xISDbnhsJ1eY0cf14RvWQtT9ZyITW+KHRW5/GKsfkAJ3UpyuWbFIR2Hr4p/VJLq2HxViBAuUYx
CAy25I1owa0cH8TjNc8F0ueBu+Q7lBV3N1zW+689bWecC5tbNJfyyIDdxV9FJMntgpJFb/Yln47g
80miGL8fcwDcELH+O70J7CGDo1Cy9J32uQygl6M919Bjh3w22p0YSyI/sTg8dJ+frIhTbNuBbEEl
7OsgHeI+sSUTLeT+WkKFEunPmSLopjSMxXuJj/U1g1QBNayu5u1cx1xO313TLFbinx4tY/qtx+Aw
2ZvxzaX3bFarB9gZdKdmK+gtmsYixc1vusb83v4m3KQ7VQeTlRsWcdBYm2vubvYuBGcVC5PY+x6j
VRxC9sMEVZrzxlDUPPyAJG673kZDEyB1yhLjGy2QvNHV1vg3CN1YdAnvyRO8KMhGLBu0BVmV/2x9
lpMdxSV5AOTqCzZLbfPxavMsu/e273Ijpy/CYVeKLwA8I0koylbD5tDxq1uI05CPxwzTEVc/Eg+c
WgbjcdD/fLR//bw=
-----END CERTIFICATE-----

   The management console will be available shortly at: https://fuse01-xuan.fuse-vsphere.com

fuse01 ( 14101) is started successfully
Cartridge fuse endpoint OPENSHIFT_FUSE_OPENWIRE_SSL_PORT is exposed at URL tls:fuse01-xuan.fuse-vsphere.com:2303
Cartridge fuse endpoint OPENSHIFT_FUSE_STOMP_SSL_PORT is exposed at URL tls:fuse01-xuan.fuse-vsphere.com:2304
Cartridge fuse endpoint OPENSHIFT_FUSE_AMQP_SSL_PORT is exposed at URL tls:fuse01-xuan.fuse-vsphere.com:2305
Cartridge fuse endpoint OPENSHIFT_FUSE_MQTT_SSL_PORT is exposed at URL tls:fuse01-xuan.fuse-vsphere.com:2306
Cartridge fuse endpoint OPENSHIFT_FUSE_GATEWAY_SSL_PORT is exposed at URL tls:fuse01-xuan.fuse-vsphere.com:2307
Cartridge fuse endpoint OPENSHIFT_FUSE_APP_TLS_PORT_1 is exposed at URL tls:fuse01-xuan.fuse-vsphere.com:2308
Cartridge fuse endpoint OPENSHIFT_FUSE_APP_TLS_PORT_2 is exposed at URL tls:fuse01-xuan.fuse-vsphere.com:2309
Cartridge fuse endpoint OPENSHIFT_FUSE_APP_TLS_PORT_3 is exposed at URL tls:fuse01-xuan.fuse-vsphere.com:2310

Waiting for your DNS name to be available ... done

Your application 'fuse01' is now available.

  URL:        http://fuse01-xuan.fuse-vsphere.com/
  SSH to:     54041ad01018d6ce51000235.com
  Git remote: ssh://54041ad01018d6ce51000235.com/~/git/fuse01.git/

Run 'rhc show-app fuse01' for more details about your app.
[root@broker ~]# rhc app create fuse02 fuse  --no-git 
Using fuse-1.0.0 (JBoss Fuse 6.1.0 EA) for 'fuse'

Application Options
-------------------
Domain:     xuan
Cartridges: fuse-1.0.0
Gear Size:  default
Scaling:    no

Creating application 'fuse02' ... fuse02 already started
done

  Using provided password from domain
Using provided url from domain

Application fuse02 created successfully.  Please make note of these credentials:

   Console URL:        https://fuse01-xuan.fuse-vsphere.com
   Console User:       admin
   Console Password:   UhDTd7gc81jp
   Zookeeper URL:      fuse01-xuan.fuse-vsphere.com:56696
   Zookeeper Password: UhDTd7gc81jp
fuse02 ( 18375) is started successfully

WARNING: Creating fabric containers directly from
         openshift is supported in a limited way
         as such containers can not be fully managed
         by fabric.
Cartridge fuse endpoint OPENSHIFT_FUSE_OPENWIRE_SSL_PORT is exposed at URL tls:fuse02-xuan.fuse-vsphere.com:2303
Cartridge fuse endpoint OPENSHIFT_FUSE_STOMP_SSL_PORT is exposed at URL tls:fuse02-xuan.fuse-vsphere.com:2304
Cartridge fuse endpoint OPENSHIFT_FUSE_AMQP_SSL_PORT is exposed at URL tls:fuse02-xuan.fuse-vsphere.com:2305
Cartridge fuse endpoint OPENSHIFT_FUSE_MQTT_SSL_PORT is exposed at URL tls:fuse02-xuan.fuse-vsphere.com:2306
Cartridge fuse endpoint OPENSHIFT_FUSE_GATEWAY_SSL_PORT is exposed at URL tls:fuse02-xuan.fuse-vsphere.com:2307
Cartridge fuse endpoint OPENSHIFT_FUSE_APP_TLS_PORT_1 is exposed at URL tls:fuse02-xuan.fuse-vsphere.com:2308
Cartridge fuse endpoint OPENSHIFT_FUSE_APP_TLS_PORT_2 is exposed at URL tls:fuse02-xuan.fuse-vsphere.com:2309
Cartridge fuse endpoint OPENSHIFT_FUSE_APP_TLS_PORT_3 is exposed at URL tls:fuse02-xuan.fuse-vsphere.com:2310

Waiting for your DNS name to be available ... done

Your application 'fuse02' is now available.

  URL:        http://fuse02-xuan.fuse-vsphere.com/
  SSH to:     54041b311018d6ce5100024e.com
  Git remote: ssh://54041b311018d6ce5100024e.com/~/git/fuse02.git/

Run 'rhc show-app fuse02' for more details about your app.
[root@broker ~]# rhc app create fuse03 fuse  --no-git 
Using fuse-1.0.0 (JBoss Fuse 6.1.0 EA) for 'fuse'

Application Options
-------------------
Domain:     xuan
Cartridges: fuse-1.0.0
Gear Size:  default
Scaling:    no

Creating application 'fuse03' ... fuse03 already started
done

  Using provided password from domain
Using provided url from domain

Application fuse03 created successfully.  Please make note of these credentials:

   Console URL:        https://fuse01-xuan.fuse-vsphere.com
   Console User:       admin
   Console Password:   UhDTd7gc81jp
   Zookeeper URL:      fuse01-xuan.fuse-vsphere.com:56696
   Zookeeper Password: UhDTd7gc81jp
fuse03 ( 6203) is started successfully

WARNING: Creating fabric containers directly from
         openshift is supported in a limited way
         as such containers can not be fully managed
         by fabric.
Cartridge fuse endpoint OPENSHIFT_FUSE_OPENWIRE_SSL_PORT is exposed at URL tls:fuse03-xuan.fuse-vsphere.com:2303
Cartridge fuse endpoint OPENSHIFT_FUSE_STOMP_SSL_PORT is exposed at URL tls:fuse03-xuan.fuse-vsphere.com:2304
Cartridge fuse endpoint OPENSHIFT_FUSE_AMQP_SSL_PORT is exposed at URL tls:fuse03-xuan.fuse-vsphere.com:2305
Cartridge fuse endpoint OPENSHIFT_FUSE_MQTT_SSL_PORT is exposed at URL tls:fuse03-xuan.fuse-vsphere.com:2306
Cartridge fuse endpoint OPENSHIFT_FUSE_GATEWAY_SSL_PORT is exposed at URL tls:fuse03-xuan.fuse-vsphere.com:2307
Cartridge fuse endpoint OPENSHIFT_FUSE_APP_TLS_PORT_1 is exposed at URL tls:fuse03-xuan.fuse-vsphere.com:2308
Cartridge fuse endpoint OPENSHIFT_FUSE_APP_TLS_PORT_2 is exposed at URL tls:fuse03-xuan.fuse-vsphere.com:2309
Cartridge fuse endpoint OPENSHIFT_FUSE_APP_TLS_PORT_3 is exposed at URL tls:fuse03-xuan.fuse-vsphere.com:2310

Waiting for your DNS name to be available ... done

Your application 'fuse03' is now available.

  URL:        http://fuse03-xuan.fuse-vsphere.com/
  SSH to:     54041b7d1018d6ce51000263.com
  Git remote: ssh://54041b7d1018d6ce51000263.com/~/git/fuse03.git/

Run 'rhc show-app fuse03' for more details about your app.
[root@broker ~]# rhc app create fuse04 fuse  --no-git 
Using fuse-1.0.0 (JBoss Fuse 6.1.0 EA) for 'fuse'

Application Options
-------------------
Domain:     xuan
Cartridges: fuse-1.0.0
Gear Size:  default
Scaling:    no

Creating application 'fuse04' ... fuse04 already started
done

  Using provided password from domain
Using provided url from domain

Application fuse04 created successfully.  Please make note of these credentials:

   Console URL:        https://fuse01-xuan.fuse-vsphere.com
   Console User:       admin
   Console Password:   UhDTd7gc81jp
   Zookeeper URL:      fuse01-xuan.fuse-vsphere.com:56696
   Zookeeper Password: UhDTd7gc81jp
fuse04 ( 10341) is started successfully

WARNING: Creating fabric containers directly from
         openshift is supported in a limited way
         as such containers can not be fully managed
         by fabric.
Cartridge fuse endpoint OPENSHIFT_FUSE_OPENWIRE_SSL_PORT is exposed at URL tls:fuse04-xuan.fuse-vsphere.com:2303
Cartridge fuse endpoint OPENSHIFT_FUSE_STOMP_SSL_PORT is exposed at URL tls:fuse04-xuan.fuse-vsphere.com:2304
Cartridge fuse endpoint OPENSHIFT_FUSE_AMQP_SSL_PORT is exposed at URL tls:fuse04-xuan.fuse-vsphere.com:2305
Cartridge fuse endpoint OPENSHIFT_FUSE_MQTT_SSL_PORT is exposed at URL tls:fuse04-xuan.fuse-vsphere.com:2306
Cartridge fuse endpoint OPENSHIFT_FUSE_GATEWAY_SSL_PORT is exposed at URL tls:fuse04-xuan.fuse-vsphere.com:2307
Cartridge fuse endpoint OPENSHIFT_FUSE_APP_TLS_PORT_1 is exposed at URL tls:fuse04-xuan.fuse-vsphere.com:2308
Cartridge fuse endpoint OPENSHIFT_FUSE_APP_TLS_PORT_2 is exposed at URL tls:fuse04-xuan.fuse-vsphere.com:2309
Cartridge fuse endpoint OPENSHIFT_FUSE_APP_TLS_PORT_3 is exposed at URL tls:fuse04-xuan.fuse-vsphere.com:2310

Waiting for your DNS name to be available ... done

Your application 'fuse04' is now available.

  URL:        http://fuse04-xuan.fuse-vsphere.com/
  SSH to:     54041bba1018d6ce51000278.com
  Git remote: ssh://54041bba1018d6ce51000278.com/~/git/fuse04.git/

Run 'rhc show-app fuse04' for more details about your app.
[root@broker ~]#

Comment 31 Eric Paris 2017-05-31 18:22:11 UTC
We apologize, however, we do not plan to address this report at this time. The majority of our active development is for the v3 version of OpenShift. If you would like for Red Hat to reconsider this decision, please reach out to your support representative. We are very sorry for any inconvenience this may cause.


Note You need to log in before you can comment on or make changes to this bug.