Bug 994130 - Deplying new code via 'git push' sometimes hangs indefinitely
Summary: Deplying new code via 'git push' sometimes hangs indefinitely
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Image
Version: 2.x
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: ---
Assignee: Dan Mace
QA Contact: libra bugs
URL:
Whiteboard:
: 993248 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-08-06 14:40 UTC by Ben Browning
Modified: 2015-06-11 21:18 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-06-11 21:18:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Ben Browning 2013-08-06 14:40:52 UTC
Description of problem:
Sometimes a 'git push' works flawlessly and sometimes it hangs indefinitely. It seems to hang more often when using a scaled application with more than one gear. Here's an example output when trying to push new code to my scaled application using 3 gears:

[bbrowning@f19 ~/tmp/tb3cluster]$ git push
Counting objects: 9, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 469 bytes | 0 bytes/s, done.
Total 5 (delta 3), reused 0 (delta 0)
remote: Stopping jbossas cart
remote: Sending SIGTERM to jboss:29004 ...
remote: CLIENT_RESULT: HAProxy instance is stopped
remote: CLIENT_RESULT: HAProxy instance is started
remote: SSH_CMD: ssh 5200fad44382ec95410000d8.62.29
remote: SSH_CMD: ssh 52010024e0b8cd75a4000050.249.97

The command was hung at this point for over 30 minutes before I decided to CTRL+C it. At this point git thinks the commit didn't get pushed to the remote, but in fact it did because I see my configuration change (this was a change to .openshift/config/standalone.xml) reflected in the running gears.

All 3 gears have my pushed code and were restarted, despite what git and the hung git push think.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Mrunal Patel 2013-08-06 16:41:45 UTC
Hi, Ben,
Have you added any action hooks for the application?

Thanks,
Mrunal

Comment 2 Ben Browning 2013-08-06 17:43:42 UTC
Yes, my application does have a pre_start_jbossas7 hook - https://github.com/openshift-quickstart/torquebox-quickstart/blob/master/.openshift/action_hooks/pre_start_jbossas-7 that calls into https://github.com/openshift-quickstart/torquebox-quickstart/blob/master/.openshift/torquebox.sh

Is there some exit code I need to ensure the hooks have for things to progress smoothly?

Comment 3 Mrunal Patel 2013-08-06 18:34:10 UTC
This may not be the issue, but could help:

https://github.com/openshift-quickstart/torquebox-quickstart/blob/master/.openshift/action_hooks/pre_start_jbossas-7#L22

TORQUEBOX_HOME isn't set in this file, so torquebox is getting installed on
every pre_start if I am reading the code correctly, which could slow down the deploy cycle.

Comment 4 Ben Browning 2013-08-06 18:55:24 UTC
TORQUEBOX_HOME is set via the sourced https://github.com/openshift-quickstart/torquebox-quickstart/blob/master/.openshift/torquebox.sh

The pre_start_jbossas-7 script is executing fine and the new scaled up gear is booting and deploying the application fine. Everything's working great except the 'rhc scale-cartridge' command just hangs instead of exiting when things are scaled up.

Comment 5 Mrunal Patel 2013-08-15 19:03:34 UTC
The steps to download, unzip torquebox and then the bundle install are adding to the long times here. Did your rhc scale-cartridge commands eventually return?

Comment 6 Ben Browning 2013-08-15 20:20:22 UTC
Sometimes the command returns and sometimes it hangs indefinitely. I've waited more than 30 minutes on several occasions just to see if it would ever return and it has not.

The weird thing is as far as I can tell the new code does get pushed to all the gears and everything restarts and comes back up fine. The 'git push' just doesn't return.

Comment 7 Mrunal Patel 2013-08-16 16:03:10 UTC
I tried this a few times yesterday but wasn't able to reproduce the git push hanging. I will keep on trying to see if we can get anything.

Comment 8 Mrunal Patel 2013-08-19 20:10:44 UTC
*** Bug 993248 has been marked as a duplicate of this bug. ***

Comment 9 Mrunal Patel 2013-08-20 14:52:47 UTC
I retried this several times yesterday. It did exceed 240 seconds but the gears did eventually come up. I suggest converting the quickstart into a cartridge using CDK. Also, the code in pre start seems to be re-installing torquebox every time slowing down git pushes/scale ups. As far as the git push hanging, I wasn't able
to reproduce that. I am wondering if it has something to do with your ssh/git settings. I am lowering severity on this one since it should not block the release.

Comment 10 Ben Browning 2013-08-20 15:12:48 UTC
I've seen a hang during scaling up with the Immutant cartridge as well (https://github.com/immutant/openshift-immutant-cart) which is very similar to my quickstart but converted to a cartridge. Thus, I don't think converting to a cartridge will help anything.

The code does not install torquebox on every git push - it only does it once per gear and never again for that gear.

Comment 11 Ben Browning 2013-08-20 15:27:15 UTC
Also, the 240 seconds mentioned - is that some timeout that could just be increased?


Note You need to log in before you can comment on or make changes to this bug.