Red Hat Bugzilla – Bug 974786
"git push origin" to scaled application errors out on secondary gear after resync looking for git repository
Last modified: 2015-05-14 19:21:49 EDT
Description of problem: "git push origin" to scaled application errors out on secondary gear after resync looking for git repository Version-Release number of selected component (if applicable): openshift.com as of Jun 15, 2013 How reproducible: Always Steps to Reproduce: 1. "git push origin" 2. watch for output 3. Actual results: remote: sent 39543 bytes received 39850 bytes 3378.43 bytes/sec remote: total size is 3312371 speedup is 41.72 remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"' remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//node_modules ']' remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"' remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//virtenv ']' remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"' remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//../.m2 ']' remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"' remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//.openshift ']' remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"' remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//deployments ']' remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"' remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//perl5lib ']' remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"' remote: + '[' -d /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/ruby//phplib ']' remote: + for rpccall in '"${OPENSHIFT_SYNC_GEARS_POST[@]}"' remote: + ssh 79311893ae2544f68e6d9830121782cc@10.90.182.117 'gear remotedeploy' remote: No such file or directory - /var/lib/openshift/79311893ae2544f68e6d9830121782cc/git/swimming.git remote: Exit code: remote: An error occurred executing 'gear postreceive' Expected results: deployment completes with application being up Additional info:
Boris, Did just happen all of a sudden or were there any other errors you saw on a prior git push? Could you ssh into 79311893ae2544f68e6d9830121782cc and ls -lZ ~/git directory and also cd $OPENSHIFT_HOMEDIR and find . -name swimming.git Thanks, Mrunal
Hi Mrunal, This issue started to happen after migration to cartridge v2. Right after this migration, my "git push" started to fail with "Argument too long" errors and then some time after I lost ability to establish SSH connection to application at all. Later when it was posted on openshift.com that migration was over I was still unable to connect via SSH. Nam Duong helped me to regain SSH access by asking RedHat admins to take a look on my gear. After that my "git push" works with few exceptions: - "app_ctl start" starts application twice - "git push" fails with error stated in this bug. So, to make long story short, it is kind of "started to happen all of a sudden", but somehow related to migration to Carteidge v2. Below is requested output from secondary gear: > ls -lZ ~/git drwxr-xr-x. 79311893ae2544f68e6d9830121782cc 79311893ae2544f68e6d9830121782cc unconfined_u:object_r:openshift_var_lib_t:s0:c2,c857 79311893ae.git > cd $OPENSHIFT_HOMEDIR > find . -name swimming.git find: `./.ssh': Permission denied find: `./.sandbox': Permission denied find: `./.tmp': Permission denied Best regards, Boris
Boris, Thanks for the information. We have identified the issue and are working on a fix. - Mrunal
https://github.com/openshift/origin-server/pull/2903
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/b0dedc1552e09d6aee82e11f45619c135ffc8acf Bug 974786: Scaled gear hot deploy logic fix Make the `gear` script `--hot-deploy` option require an explicit boolean argument so that the option overrides a repo check for the hot_deploy marker.
It's fixed, verified on devenv_3389, please refer to the following results: 1. Create a scalable ruby app with mysql rhc app create cruby9s0 ruby-1.9 mysql-5.1 -s 2. Add hot_deploy marker cd cruby9s0 touch .openshift/markers/hot_deploy 3. Git push git add .; git commit -amp; git push origin [master 758bebb] p 0 files changed create mode 100644 ok Counting objects: 3, done. Delta compression using up to 4 threads. Compressing objects: 100% (2/2), done. Writing objects: 100% (2/2), 248 bytes, done. Total 2 (delta 1), reused 0 (delta 0) remote: Not stopping cartridge ruby because hot deploy is enabled remote: Not stopping cartridge haproxy because hot deploy is enabled remote: Running build on Ruby cart remote: Not starting cartridge haproxy because hot deploy is enabled remote: Application will not be stopped during gear sync to to presence of hot_deploy marker remote: Not starting cartridge ruby because hot deploy is enabled remote: Hot deploy marker is present. Touching Passenger restart.txt to trigger redeployment. To ssh://dd0997c6d97011e2b4ca22000a8a1e30@cruby9s0-cdm.dev.rhcloud.com/~/git/cruby9s0.git/ 3478bde..758bebb master -> master
Hello, Thanks for confirming the bug and making solution for it. Please note that my application does not have hot_deploy marker. So far it looks for git repository on secondary gear. Please add such configuration to your test cases. Thanks, Boris
Hello, Symptoms of this bug are similar to bug 971862. Thanks, Boris
Hi Dan, This bug is closed since Jun 24, but I still experience the issue... Please advice, Boris