Bug 974786 - "git push origin" to scaled application errors out on secondary gear after resync looking for git repository
Summary: "git push origin" to scaled application errors out on secondary gear after re...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Dan Mace
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-06-15 19:55 UTC by Boris Mironov
Modified: 2015-05-14 23:21 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-06-24 14:56:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Boris Mironov 2013-06-15 19:55:15 UTC
Description of problem:
"git push origin" to scaled application errors out on secondary gear after resync looking for git repository

Version-Release number of selected component (if applicable):
openshift.com as of Jun 15, 2013

How reproducible:
Always

Steps to Reproduce:
1. "git push origin"
2. watch for output
3.

Actual results:
remote: sent 39543 bytes  received 39850 bytes  3378.43 bytes/sec
remote: total size is 3312371  speedup is 41.72
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//node_modules ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//virtenv ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//../.m2 ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//.openshift ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//deployments ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//perl5lib ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//phplib ']'
remote: + for rpccall in '"${OPENSHIFT_SYNC_GEARS_POST[@]}"'
remote: + ssh 79311893ae2544f68e6d9830121782cc.182.117 'gear remotedeploy'
remote: No such file or directory - /var/lib/openshift/79311893ae2544f68e6d9830121782cc/git/swimming.git
remote: Exit code:
remote: An error occurred executing 'gear postreceive'

Expected results:
deployment completes with application being up

Additional info:

Comment 1 Mrunal Patel 2013-06-19 02:12:11 UTC
Boris,
Did just happen all of a sudden or were there any other errors you saw on a prior git push? Could you ssh into 79311893ae2544f68e6d9830121782cc and ls -lZ ~/git directory and also cd $OPENSHIFT_HOMEDIR and find . -name swimming.git

Thanks,
Mrunal

Comment 2 Boris Mironov 2013-06-19 02:45:29 UTC
Hi Mrunal,

This issue started to happen after migration to cartridge v2.
Right after this migration, my "git push" started to fail with "Argument too long" errors and then some time after I lost ability to establish SSH connection to application at all.

Later when it was posted on openshift.com that migration was over I was still unable to connect via SSH. Nam Duong helped me to regain SSH access by asking RedHat admins to take a look on my gear. After that my "git push" works with few exceptions:

- "app_ctl start" starts application twice
- "git push" fails with error stated in this bug.

So, to make long story short, it is kind of "started to happen all of a sudden", but somehow related to migration to Carteidge v2.

Below is requested output from secondary gear:

> ls -lZ ~/git
drwxr-xr-x. 79311893ae2544f68e6d9830121782cc 79311893ae2544f68e6d9830121782cc unconfined_u:object_r:openshift_var_lib_t:s0:c2,c857 79311893ae.git

> cd $OPENSHIFT_HOMEDIR
> find . -name swimming.git
find: `./.ssh': Permission denied
find: `./.sandbox': Permission denied
find: `./.tmp': Permission denied

Best regards,
Boris

Comment 3 Mrunal Patel 2013-06-19 17:39:25 UTC
Boris,
Thanks for the information. We have identified the issue and are working on a fix.

- Mrunal

Comment 5 openshift-github-bot 2013-06-19 21:54:26 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/b0dedc1552e09d6aee82e11f45619c135ffc8acf
Bug 974786: Scaled gear hot deploy logic fix

Make the `gear` script `--hot-deploy` option require an explicit boolean argument
so that the option overrides a repo check for the hot_deploy marker.

Comment 6 chunchen 2013-06-20 06:31:17 UTC
It's fixed, verified on devenv_3389, please refer to the following results:

1. Create a scalable ruby app with mysql
rhc app create cruby9s0 ruby-1.9 mysql-5.1 -s
2. Add hot_deploy marker
cd cruby9s0
touch .openshift/markers/hot_deploy
3. Git push
git add .; git commit -amp; git push origin

[master 758bebb] p
 0 files changed
 create mode 100644 ok
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 248 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Not stopping cartridge ruby because hot deploy is enabled
remote: Not stopping cartridge haproxy because hot deploy is enabled
remote: Running build on Ruby cart
remote: Not starting cartridge haproxy because hot deploy is enabled
remote: Application will not be stopped during gear sync to to presence of hot_deploy marker
remote: Not starting cartridge ruby because hot deploy is enabled
remote: Hot deploy marker is present. Touching Passenger restart.txt to trigger redeployment.
To ssh://dd0997c6d97011e2b4ca22000a8a1e30.rhcloud.com/~/git/cruby9s0.git/
   3478bde..758bebb  master -> master

Comment 7 Boris Mironov 2013-06-20 12:16:42 UTC
Hello,

Thanks for confirming the bug and making solution for it.

Please note that my application does not have hot_deploy marker. So far it looks for git repository on secondary gear. Please add such configuration to your test cases.

Thanks,
Boris

Comment 8 Boris Mironov 2013-06-21 12:50:43 UTC
Hello,

Symptoms of this bug are similar to bug 971862.

Thanks,
Boris

Comment 9 Boris Mironov 2013-07-01 02:39:07 UTC
Hi Dan,

This bug is closed since Jun 24, but I still experience the issue...

Please advice,
Boris


Note You need to log in before you can comment on or make changes to this bug.