Bug 985641

Summary: error: failed to execute controlled stop for mysql & error: failed to push some refs to git repo
Product: OpenShift Online Reporter: Robert Main <lenswipe>
Component: ContainersAssignee: Hiro Asari <hasari>
Status: CLOSED INSUFFICIENT_DATA QA Contact: libra bugs <libra-bugs>
Severity: low Docs Contact:
Priority: medium    
Version: 2.xCC: hasari, jkeck, lenswipe, whearn, zhewang
Target Milestone: ---Keywords: SupportQuestion
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-22 20:29:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Robert Main 2013-07-18 01:54:21 UTC
Description of problem:
When pushing to my git repo I recieve a message telling me that it "failed to execute controlled stop for mysql". This happens when I try to push my commit(s) to openshift(full push output below)

Additional info:
$ git push origin master
Counting objects: 13, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 537 bytes, done.
Total 7 (delta 6), reused 0 (delta 0)
remote: An error occurred executing 'gear prereceive' (exit code: 1)
remote: Error message: Failed to execute: 'control stop' for /var/lib/openshift/51c67691e0b8cd2529000066/mysql
remote: 
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/v2_cart_model.rb:1097:in `block in do_control_with_directory'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/v2_cart_model.rb:903:in `process_cartridges'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/v2_cart_model.rb:1064:in `do_control_with_directory'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/v2_cart_model.rb:925:in `do_control'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/v2_cart_model.rb:1317:in `stop_cartridge'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/v2_cart_model.rb:1176:in `block in stop_gear'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/v2_cart_model.rb:79:in `block in each_cartridge'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/v2_cart_model.rb:909:in `block in process_cartridges'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/v2_cart_model.rb:907:in `each'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/v2_cart_model.rb:907:in `process_cartridges'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/v2_cart_model.rb:77:in `each_cartridge'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/v2_cart_model.rb:1175:in `stop_gear'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/application_container.rb:335:in `stop_gear'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.7/lib/openshift-origin-node/model/application_container.rb:462:in `pre_receive'
remote: /usr/bin/gear:115:in `block (3 levels) in <main>'
remote: /usr/bin/gear:64:in `do_command'
remote: /usr/bin/gear:95:in `block (2 levels) in <main>'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:74:in `run!'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'
To ssh://51c67691e0b8cd2529000066.com/~/git/walrussoup.git/
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://51c67691e0b8cd2529000066.com/~/git/walrussoup.git/'

Comment 1 Hiro Asari 2013-07-18 13:33:53 UTC
What is the status of the mysql cartridge?

Can you stop and start the cartridge?

Comment 2 Robert Main 2013-07-18 16:45:08 UTC
Not sure about the status of the mysql cartridge, how do I find out?
I can start and stop the cartridge from command line and the web interface and that preceded by another push fixes the problem momentarily.

Comment 3 Hiro Asari 2013-07-18 17:39:54 UTC
To see the cartridge status:

rhc cartridge status mysql -a walrussoup

Are you saying that you can start and stop the cartridge without a problem and that 'git push' works for a while, but after a while the problem returns?

Since it is not possible to control the DB cartridge on the console, I assume you mean starting/stopping the application.

You said "that preceded", which doesn't seem to fit with the rest of what you are saying.

Comment 4 Robert Main 2013-07-18 18:31:53 UTC
I'm saying I can start/stop the application from the console without a problem. I should have read your post more carefully since you asked me to start/stop the cartridge not the application but someone was talking to me when I was writing the reply hence the slightly odd wording. Apologies.

For now the cartridge status reports the following:
$ rhc cartridge status mysql-5.1 -a walrussoup

RESULT:
MySQL is running

Comment 5 Hiro Asari 2013-07-18 18:45:01 UTC
Can you stop the cartridge with this?

rhc cartridge stop mysql-5.1 -a walrussoup


What happens if you restart your application (instead of the cartridge)? You indicated that `git push` would work for a while after that, correct? (Also, how long is "a while"? Could you be more precise?) And `git push` stops working at some point, yes?

When `git push` is failing, can you ssh to the gear, and see if you can connect to the database?

rhc app ssh walrussoup

mysql

Also, what would 'ps -ef' show at that time?

Comment 6 Hiro Asari 2013-07-18 19:11:55 UTC
Also, do you have any action hook for this app defined?

Comment 7 Robert Main 2013-07-18 21:11:57 UTC
$ rhc cartridge stop mysql-5.1 -a walrussoup

Unable to complete the requested operation due to: Failed to correctly execute
all parallel operations - --DEBUG--

--RESULT--

--MESSAGE--

--ERROR--

--APP INFO--

--CART COMMANDS--

--CART PROPERTIES--
{}
--DATA--

--EXIT CODE--

--USER ACTIONABLE--
false
.
Reference ID: d311ec68d9e12c2ba7b40db37855c5d8


Restarting the app instead of the cartridge works fine, git push doesn't work for "a while"( I may have worded that badly, again - apologies). It's more that it gets that one particular push through and where it needs to be, sometimes an indeterminate number of pushes afterwards succeed, sometimes not.

Next time gish push failed I will ssh to the gear and do that - I'm about to start work on my application again so I'll bear this in mind.

I don't have any action hooks defined for the app as far as I'm aware (I didn't define any anyway).

Comment 8 Hiro Asari 2013-07-25 19:40:29 UTC
Has this recurred? Were you able to determine what sort of state mysql was in while you can't stop mysql? Does $OPENSHIFT_MYSQL_DIR/pid/mysql.pid exist? If so, does it match with match the mysql PID?

In $OPENSHIFT_MYSQL_DIR/bin/control, the stop() function will be invoked when you run 'rhc cartridge stop mysql-5.1 -a walrussoup'. You can invoke this manually, and get a little more debugging information:

bash -x $OPENSHIFT_MYSQL_DIR/bin/control stop

Comment 9 Robert Main 2013-07-25 20:32:30 UTC
Hi there

I currently having company staying so I haven't been able to do much development on my app. Hoping to make another push in the near future though and see if it happens again.

Comment 10 Zhe Wang 2013-08-13 08:33:22 UTC
The bug is sometimes producible in devenv_3641. Specifically, when I pushed changes to a ruby-1.9 app with a MySQL-5.1 cart added via Git, the errors were:

[zhe@fedora rb9]$ git add . && git commit -amp && git push
[master 2e5ceb6] p
 1 file changed, 1 insertion(+), 1 deletion(-)
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 284 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Stopping Ruby cart
remote: Waiting for stop to finish
remote: An error occurred executing 'gear prereceive' (exit code: 1)
remote: Error message: Failed to execute: 'control stop' for /var/lib/openshift/5209b1384a6aa70997000002/mysql
remote: 
remote: For more details about the problem, try running the command again with the '--trace' option.
To ssh://5209b1384a6aa70997000002.rhcloud.com/~/git/rb9.git/
 ! [remote rejected] master -> master (pre-receive hook declined)

Comment 11 Hiro Asari 2013-08-13 14:32:30 UTC
(In reply to Zhe Wang from comment #10)
> The bug is sometimes producible in devenv_3641. Specifically, when I pushed
> changes to a ruby-1.9 app with a MySQL-5.1 cart added via Git, the errors
> were:
> 
> [zhe@fedora rb9]$ git add . && git commit -amp && git push
> [master 2e5ceb6] p
>  1 file changed, 1 insertion(+), 1 deletion(-)
> Counting objects: 5, done.
> Delta compression using up to 4 threads.
> Compressing objects: 100% (3/3), done.
> Writing objects: 100% (3/3), 284 bytes, done.
> Total 3 (delta 2), reused 0 (delta 0)
> remote: Stopping Ruby cart
> remote: Waiting for stop to finish
> remote: An error occurred executing 'gear prereceive' (exit code: 1)
> remote: Error message: Failed to execute: 'control stop' for
> /var/lib/openshift/5209b1384a6aa70997000002/mysql
> remote: 
> remote: For more details about the problem, try running the command again
> with the '--trace' option.
> To
> ssh://5209b1384a6aa70997000002.rhcloud.com/~/git/rb9.git/
>  ! [remote rejected] master -> master (pre-receive hook declined)

Was this app created with 'app create' without '--from-code' but added 'rails' to Gemfile (and Gemfile.lock) later? If so, this error might be a result of the problem with pre-build function in https://github.com/openshift/origin-server/commit/6400c02.

This was fixed by https://github.com/openshift/origin-server/commit/b46b39c, which will be in devenv_3642.

Comment 12 Zhe Wang 2013-08-15 04:26:36 UTC
Hi, Hiro,

The app referred in Comment#10 was a raw one, without building from any code. I simply created a ruby-1.9 app and then added a MySQL cartridge to it. After that, I changed the title of the Welcome page (in config.ru) a bit, and pushed the change via Git.

Hope this info helps.

Thanks,
Zhe Wang

Comment 13 Hiro Asari 2013-08-15 12:43:12 UTC
Zhe, I believe the bare application is working now on devenv. If you see a problem, please open a separate ticket, since it's most likely a separate issue.

Let us now wait for further input from Robert.

Comment 14 Hiro Asari 2013-09-12 20:19:11 UTC
Hi, Robert,

Is there any progress here?

Comment 15 Hiro Asari 2013-10-22 20:29:30 UTC
I'm closing this for now. If there is anything we can do to investigate further, do let us know.

Comment 16 Robert Main 2016-07-04 14:55:45 UTC
Any chance I could stop getting emails about this since the bug is closed?