Bug 974786 - "git push origin" to scaled application errors out on secondary gear after resync looking for git repository
"git push origin" to scaled application errors out on secondary gear after re...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Dan Mace
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-15 15:55 EDT by Boris Mironov
Modified: 2015-05-14 19:21 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-24 10:56:02 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Boris Mironov 2013-06-15 15:55:15 EDT
Description of problem:
"git push origin" to scaled application errors out on secondary gear after resync looking for git repository

Version-Release number of selected component (if applicable):
openshift.com as of Jun 15, 2013

How reproducible:
Always

Steps to Reproduce:
1. "git push origin"
2. watch for output
3.

Actual results:
remote: sent 39543 bytes  received 39850 bytes  3378.43 bytes/sec
remote: total size is 3312371  speedup is 41.72
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//node_modules ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//virtenv ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//../.m2 ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//.openshift ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//deployments ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//perl5lib ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//phplib ']'
remote: + for rpccall in '"${OPENSHIFT_SYNC_GEARS_POST[@]}"'
remote: + ssh 79311893ae2544f68e6d9830121782cc@10.90.182.117 'gear remotedeploy'
remote: No such file or directory - /var/lib/openshift/79311893ae2544f68e6d9830121782cc/git/swimming.git
remote: Exit code:
remote: An error occurred executing 'gear postreceive'

Expected results:
deployment completes with application being up

Additional info:
Comment 1 Mrunal Patel 2013-06-18 22:12:11 EDT
Boris,
Did just happen all of a sudden or were there any other errors you saw on a prior git push? Could you ssh into 79311893ae2544f68e6d9830121782cc and ls -lZ ~/git directory and also cd $OPENSHIFT_HOMEDIR and find . -name swimming.git

Thanks,
Mrunal
Comment 2 Boris Mironov 2013-06-18 22:45:29 EDT
Hi Mrunal,

This issue started to happen after migration to cartridge v2.
Right after this migration, my "git push" started to fail with "Argument too long" errors and then some time after I lost ability to establish SSH connection to application at all.

Later when it was posted on openshift.com that migration was over I was still unable to connect via SSH. Nam Duong helped me to regain SSH access by asking RedHat admins to take a look on my gear. After that my "git push" works with few exceptions:

- "app_ctl start" starts application twice
- "git push" fails with error stated in this bug.

So, to make long story short, it is kind of "started to happen all of a sudden", but somehow related to migration to Carteidge v2.

Below is requested output from secondary gear:

> ls -lZ ~/git
drwxr-xr-x. 79311893ae2544f68e6d9830121782cc 79311893ae2544f68e6d9830121782cc unconfined_u:object_r:openshift_var_lib_t:s0:c2,c857 79311893ae.git

> cd $OPENSHIFT_HOMEDIR
> find . -name swimming.git
find: `./.ssh': Permission denied
find: `./.sandbox': Permission denied
find: `./.tmp': Permission denied

Best regards,
Boris
Comment 3 Mrunal Patel 2013-06-19 13:39:25 EDT
Boris,
Thanks for the information. We have identified the issue and are working on a fix.

- Mrunal
Comment 5 openshift-github-bot 2013-06-19 17:54:26 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/b0dedc1552e09d6aee82e11f45619c135ffc8acf
Bug 974786: Scaled gear hot deploy logic fix

Make the `gear` script `--hot-deploy` option require an explicit boolean argument
so that the option overrides a repo check for the hot_deploy marker.
Comment 6 chunchen 2013-06-20 02:31:17 EDT
It's fixed, verified on devenv_3389, please refer to the following results:

1. Create a scalable ruby app with mysql
rhc app create cruby9s0 ruby-1.9 mysql-5.1 -s
2. Add hot_deploy marker
cd cruby9s0
touch .openshift/markers/hot_deploy
3. Git push
git add .; git commit -amp; git push origin

[master 758bebb] p
 0 files changed
 create mode 100644 ok
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 248 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Not stopping cartridge ruby because hot deploy is enabled
remote: Not stopping cartridge haproxy because hot deploy is enabled
remote: Running build on Ruby cart
remote: Not starting cartridge haproxy because hot deploy is enabled
remote: Application will not be stopped during gear sync to to presence of hot_deploy marker
remote: Not starting cartridge ruby because hot deploy is enabled
remote: Hot deploy marker is present. Touching Passenger restart.txt to trigger redeployment.
To ssh://dd0997c6d97011e2b4ca22000a8a1e30@cruby9s0-cdm.dev.rhcloud.com/~/git/cruby9s0.git/
   3478bde..758bebb  master -> master
Comment 7 Boris Mironov 2013-06-20 08:16:42 EDT
Hello,

Thanks for confirming the bug and making solution for it.

Please note that my application does not have hot_deploy marker. So far it looks for git repository on secondary gear. Please add such configuration to your test cases.

Thanks,
Boris
Comment 8 Boris Mironov 2013-06-21 08:50:43 EDT
Hello,

Symptoms of this bug are similar to bug 971862.

Thanks,
Boris
Comment 9 Boris Mironov 2013-06-30 22:39:07 EDT
Hi Dan,

This bug is closed since Jun 24, but I still experience the issue...

Please advice,
Boris

Note You need to log in before you can comment on or make changes to this bug.