Bug 1031240

This looks like a cartridge bug. I will take a look.

Hi Michal,

Could you please provide more details into recent changes to this bug report. What is the meaning of "UpcomingRelease" keyword:
 - is it that issue is fixed and will be deployed as part of upcoming release?
 - is it that issue has to wait for upcoming release to be deployed and only then it could be fixed?
 - something else?

Thanks,
Boris

Hi Boris,

It means this will be not fixed in this release cycle. There is a Trello card[1] that address similar problem and we think that these two issues are similar.
Currently we don't allow access to .bundler/config to users as OpenShift manages this file and this file is created/modified during the 'build' operation.

There are two options how to fix this:

1. Reserve the $GEAR_HOME/.bundle/config directory for OpenShift and allow users to specify their customized bundle configuration files in $REPO_DIR/.bundle/config.

2. Provide special ENV variable to allow users pass certain compilation options/etc to the bundler.

3. Or allow users to have full control of the .bundle/config (iow. stop rewriting the users supplied .bundle/config)

I dunno which solution is better and it probably require more discussion...

[1] https://trello.com/c/Hots86Sx/21-make-bundler-in-the-default-ruby-build-configurable-to-support-git-submodules

Hi Michal,

Fair enough.

1. Please take into account following in your discussions:
  - we need a way to install own gems. OpenShift is bundled with Rails 3.2.8. As a user I can not install newer version of Rails into default location due to lack of permissions. This is very outdated. Some people want to experiment with Rail 4
  - above point pushes us to install our set of gems into non-default place, which in turn consumes very limited disk quota on gear
  - we also need a way to preserve installed gems through "git push origin" to avoid installing them each time


2. On my gear $GEAR_HOME is not set. If intention is to have it set to ~/ruby then it should not interfere with my application own .bundle dir (~/app-root/repo/.bundle)


Thanks,
Boris

Boris,

Have you tried to put the newer version of Rails into Gemfile? I'm sure you can install your own Rails version using the Gemfile. But I will try to test installing Rails 4.

Also, what error message you get when you try to install newer version?

Hi Boris,

You can install your own version of Rails. If you delete your .bundle/config file from your git repo and create a Gemfile + Gemfile.lock that specify a newer version of Rails, it should install just fine. I set up an app the other day that has Redmine 2.4, which uses Rails 3.2.15 iirc, and it worked. Please let us know if you're not able to do so.

Additionally, we do preserve previously-installed gems so they don't have to be installed with each git push, unless you specify you want to force a clean build. If this isn't the behavior you're seeing, please let us know and provide as much detail as possible.

Thanks!
Andy

Michal,

I did not try Rails 4 (because Ruby 2.0 is not available yet), but successfully use 3.2.14 via Gemfile. With .bundle/config setting it goes to vendor/bundle directory in my repo.

No error messages are shown during installation of newer Rails or any other gem via Gemfile. I even successfully use gems that are doing native compilation (eg, Nokogiri) using same approach. "rake" is a bit tricky, but still, I use much newer version compared to default one (that unfortunately I can not update via "gem update rake").

Thanks,
Boris

Hi Andy,

If I would remove .bundle/config then how can I install nokogiri gem? Would it go to default gem location that can be updated only during "git push"?

Thanks,
Boris

Boris,

We install gems into app-root/runtime/repo/vendor. Nokogiri will install just fine there. I would recommend you set the environment variable NOKOGIRI_USE_SYSTEM_LIBRARIES=1 to try to reduce the number of inodes used and overall build time.

What do you have in your .bundle/config file?

Andy

Hi Andy,

Here is my .bundle/config file

---
BUNDLE_FROZEN: '1'
BUNDLE_PATH: vendor/bundle
BUNDLE_DISABLE_SHARED_GEMS: '1'
BUNDLE_CLEAN: true


I also create symbolic link like

ln -sf "../../../data/ruby" "app-root/repo/vendor/bundle/ruby"

to preserve gems and avoid their reinstall during "git push". May be this issue is resolved now, but some time ago it was very annoying to wait for all gems to reinstall during endless "git push". So, basically I keep all my gems in my ~/app-root/data/ruby dir.

Thanks,
Boris

Hi Boris,

I would definitely recommend you delete your .bundle/config file, as it is essentially the same as what we create by default (we don't have BUNDLE_CLEAN). Also, I would remove your symbolic link, as we do preserve gems between git pushes. Here is an example of me creating a Gemfile with just rack in it and doing 2 git pushes. You'll see with the first push that it says "Installing rack (1.5.2)", and with the second push, it says "Using rack (1.5.2)".

$ git add Gemfile Gemfile.lock

% git commit -am w
[master 51012ff] w
 2 files changed, 13 insertions(+)
 create mode 100644 Gemfile
 create mode 100644 Gemfile.lock

$ git push
Counting objects: 5, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 439 bytes, done.
Total 4 (delta 1), reused 0 (delta 0)
remote: Stopping Ruby cartridge
remote: [Tue Nov 26 11:29:32 2013] [warn] PassEnv variable SHELL was undefined
remote: [Tue Nov 26 11:29:32 2013] [warn] PassEnv variable USER was undefined
remote: [Tue Nov 26 11:29:32 2013] [warn] PassEnv variable LOGNAME was undefined
remote: Waiting for stop to finish
remote: Building git ref 'master', commit 51012ff
remote: Building Ruby cartridge
remote: Bundling RubyGems based on Gemfile/Gemfile.lock to repo/vendor/bundle with 'bundle install --deployment'
remote: Fetching gem metadata from https://rubygems.org/..........
remote: Installing rack (1.5.2)
remote: Using bundler (1.1.4)
remote: Your bundle is complete! It was installed into ./vendor/bundle
remote: Preparing build for deployment
remote: Deployment id is ee5231df
remote: Activating deployment
remote: Starting Ruby cartridge
remote: Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://[...]/~/git/r.git/
   089a93f..51012ff  master -> master

# make some change
$ vim config.ru

$ git commit -am w
[master e3abfef] w

$ git push
Counting objects: 5, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 285 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Stopping Ruby cartridge
remote: Waiting for stop to finish
remote: Saving away previously bundled RubyGems
remote: Building git ref 'master', commit e3abfef
remote: Building Ruby cartridge
remote: Restoring previously bundled RubyGems (note: you can commit .openshift/markers/force_clean_build at the root of your repo to force a clean bundle)
remote: Bundling RubyGems based on Gemfile/Gemfile.lock to repo/vendor/bundle with 'bundle install --deployment'
remote: Using rack (1.5.2)
remote: Using bundler (1.1.4)
remote: Your bundle is complete! It was installed into ./vendor/bundle
remote: Preparing build for deployment
remote: Deployment id is 1aa14512
remote: Activating deployment
remote: Starting Ruby cartridge
remote: Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://[...]/~/git/r.git/
   51012ff..e3abfef  master -> master

Hi Andy,

Thanks for your suggestions. I will try them shortly. While I will be touching action hooks, do I still need to have "exit 0" in their end to avoid "gear postreceive" issues?

Thanks,
Boris

No, you don't need exit 0. Please let me know if you find otherwise.

Hi Andy,

I removed .bundle/config and updated Gemfile (now it requires rails 3.2.15).
Deployment failed but with different error:

$ git push origin
Counting objects: 13, done.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 922 bytes, done.
Total 7 (delta 5), reused 0 (delta 0)
remote: Executing 'pre_stop_ruby' hook...
remote: Stopping 'check_queue'...
remote: Syncing git content to other proxy gears
remote: Saving away previously bundled RubyGems
remote: Building git ref 'master', commit 2a15fcf
remote: Executing 'pre_build' hook...
remote:   Restoring stored assets with 'ln -sf "../../../../app-root/data/assets" "app-root/repo/public/assets"' in /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b
remote: Building Ruby cartridge
remote: Restoring previously bundled RubyGems (note: you can commit .openshift/markers/force_clean_build at the root of your repo to force a clean bundle)
remote: Bundling RubyGems based on Gemfile/Gemfile.lock to repo/vendor/bundle with 'bundle install --deployment'
remote: Fetching gem metadata from https://rubygems.org/..........
remote: Fetching gem metadata from https://rubygems.org/..
remote: Using rake (10.1.0) 
remote: Using i18n (0.6.5) 
remote: Installing multi_json (1.8.2) 
remote: Installing activesupport (3.2.15) 
remote: Using builder (3.0.4) 
remote: Installing activemodel (3.2.15) 
remote: Using erubis (2.7.0) 
remote: Using journey (1.0.4) 
remote: Using rack (1.4.5) 
remote: Using rack-cache (1.2) 
remote: Using rack-test (0.6.2) 
remote: Using hike (1.2.3) 
remote: Using tilt (1.4.1) 
remote: Using sprockets (2.2.2) 
remote: Installing actionpack (3.2.15) 
remote: Installing mime-types (1.25.1) 
remote: Using polyglot (0.3.3) 
remote: Using treetop (1.4.15) 
remote: Using mail (2.5.4) 
remote: Installing actionmailer (3.2.15) 
remote: Installing arel (3.0.3) 
remote: Installing tzinfo (0.3.38) 
remote: Installing activerecord (3.2.15) 
remote: Installing activeresource (3.2.15) 
remote: Using bcrypt-ruby (3.1.1) 
remote: Using cancan (1.6.10) 
remote: Using coffee-script-source (1.6.3) 
remote: Installing execjs (2.0.2) 
remote: Using coffee-script (2.2.0) 
remote: Using rack-ssl (1.3.3) 
remote: Installing json (1.8.1) with native extensions 
remote: ....
remote: ..
remote: 
remote: ...
remote: ..
remote: 
remote: Using rdoc (3.12.2) 
remote: Using thor (0.14.6) 
remote: Installing railties (3.2.15) 
remote: Using coffee-rails (3.2.2) 
remote: Using columnize (0.3.6) 
remote: Using debugger-linecache (1.2.0) 
remote: Installing debugger-ruby_core_source (1.2.4) 
remote: Installing debugger (1.6.2) with native extensions .....
remote: ...................
remote: ..
remote: 
remote: Using orm_adapter (0.4.0) 
remote: Using warden (1.2.3) 
remote: Using devise (3.0.3) 
remote: Installing docile (1.1.0) 
remote: Using foreigner (1.4.2) 
remote: Using jquery-rails (3.0.4) 
remote: Using libv8 (3.11.8.17) 
remote: Using syntax (1.0.0) 
remote: Using maruku (0.6.1) 
remote: Installing mini_portile (0.5.2) 
remote: Using minitest (4.7.3) 
remote: Using mysql2 (0.3.13) 
remote: Using nokogiri (1.6.0) 
remote: Using bundler (1.1.4) 
remote: Installing rails (3.2.15) 
remote: Using ref (1.0.5) 
remote: Using rolify (3.2.0) 
remote: Using ruby-prof (0.13.0) 
remote: Installing sass (3.2.12) 
remote: Using sass-rails (3.2.6) 
remote: Using simple_form (2.1.0) 
remote: Installing simplecov-html (0.8.0) 
remote: Installing simplecov (0.8.2) 
remote: Using therubyracer (0.11.4) 
remote: Installing turbo-sprockets-rails3 (0.3.10) 
remote: Using uglifier (2.1.2) 
remote: Using webrick (1.3.1) 
remote: Using will_paginate (3.0.4) 
remote: Using yui-compressor (0.11.0) 
remote: Your bundle is complete! It was installed into ./vendor/bundle
remote: Removing mime-types (1.24)
remote: Removing arel (3.0.2)
remote: Removing activesupport (3.2.14)
remote: Removing execjs (1.4.0)
remote: Removing railties (3.2.14)
remote: Removing simplecov-html (0.7.1)
remote: Removing rails (3.2.14)
remote: Removing tzinfo (0.3.37)
remote: Removing actionpack (3.2.14)
remote: Removing simplecov (0.7.1)
remote: Removing json (1.8.0)
remote: Removing multi_json (1.7.9)
remote: Removing sass (3.2.10)
remote: Removing activeresource (3.2.14)
remote: Removing mini_portile (0.5.1)
remote: Removing turbo-sprockets-rails3 (0.3.9)
remote: Removing debugger-ruby_core_source (1.2.3)
remote: Removing debugger (1.6.1)
remote: Removing activemodel (3.2.14)
remote: Removing actionmailer (3.2.14)
remote: Removing activerecord (3.2.14)
remote: Executing 'build' hook...
remote: Preparing build for deployment
remote: Deployment id is 336f5aa1
remote: Distributing deployment to child gears
remote: Result: failure
remote: Distribution status: failure
remote: Distribution failed for the following gears:
remote: 79311893ae (rsync: connection unexpectedly closed (0 bytes received so far) [sender])
remote: Deployment completed with status: failure
remote: postreceive failed
To ssh://swimming-bsmgroup.rhcloud.com/~/git/swimming.git
   005cddb..2a15fcf  master -> master

I believe you've encountered a different bug, unfortunately :-( If you're willing, you can try to scale your ruby cartridge down to 1, and then back up to 2. I think that should fix it and allow you to git push.

I'm trying to complete build & deploy, but constantly running into "disk quota exceeded". What is a proper way to clean deployments or disable them?

Example of error during "ctl_app deploy":

rsync: mkstemp "/var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/app-deployments/2013-11-26_13-41-21.462/repo/vendor/plugins/..gitkeep.ojB1me" failed: Disk quota exceeded (122)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]


My quota is consumed completely during "ctl_app deploy". Right before deployment it shows:

> quota -s
Disk quotas for user 4fcf6f211aeb4279b098ba74635a1e3b (uid 3148): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/mapper/EBSStore01-user_home01
                   500M       0   1024M           24220       0   40000        

It is hard to believe that my app+gems are 15000+ inodes...

I think in my case deployment should be stored in a form of "tar + gzip" to save space and inodes.

If your app ends up installing nokogiri and has other gems, you very well can use up that many inodes. For example, when deploying Redmine 2.4, the app's source is ~ 2000 inodes and the gems are ~ 15000. We made a significant change recently to our deployment system and we now keep an extra copy of your source code and dependencies to support rolling back if needed. As a result, if your deployment is 15000 inodes, it really takes up around 30000 inodes because of the extra copy. We're looking into a fix.

Is there a way to disable deployment history?

Not at this time, no.

I have asked ops to increase your inode quota and it should be increased now. Please try to deploy again.

Hi Andy,

Quota is still the same:

> quota -s
Disk quotas for user 4fcf6f211aeb4279b098ba74635a1e3b (uid 3148): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/mapper/EBSStore01-user_home01
                   503M       0   1024M           24254       0   40000   


I will check it again later tonight.

Thanks,
Boris

I just ran it again using the above id, can you check again please Boris?

Hi Andy & Corey,

I see that quota on my primary has been increased to 80000 inodes. Now I can pass first phase of "git push origin". Surprisingly I was just 500 inodes short before (now it shows 40527 inodes allocated).

New place where "git push origin" fails is right after bundle update:

remote: Your bundle is complete! It was installed into ./vendor/bundle
remote: Executing 'build' hook...
remote: Preparing build for deployment
remote: Deployment id is d5ae1dcc
remote: Distributing deployment to child gears
remote: Result: failure
remote: Distribution status: failure
remote: Distribution failed for the following gears:
remote: 79311893ae (rsync: connection unexpectedly closed (0 bytes received so far) [sender])
remote: Deployment completed with status: failure
remote: postreceive failed
To ssh://4fcf6f211aeb4279b098ba74635a1e3b.com/~/git/swimming.git/
   c33b1da..75f9321  master -> master


I checked my secondary gear and it is well below quota:

> quota -s
Disk quotas for user 79311893ae2544f68e6d9830121782cc (uid 2900):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/mapper/EBSStore01-user_home01
                   575M       0   1024M           21403       0   40000


I was able to manually synchronize repo dir:

> rsync -v -az repo/ 79311893ae2544f68e6d9830121782cc.182.117:app-root/repo/

...
vendor/plugins/
vendor/plugins/.gitkeep

sent 2276070 bytes  received 56882 bytes  358915.69 bytes/sec
total size is 200051882  speedup is 85.75


Best regards,
Boris

Hello,

Any suggestions how to produce more information about this failure? I tried to run "ctl_app deploy --trace" but it did not produce more info than I posted above. What puzzles me is that from SSH session I can successfully do "manual" deployment by running rsync and "ctl_app stop/start" commands. Could it be due to some differences in environment between my SSH session and one that is used by git deployment?

Thanks,
Boris

Hi Boris,

You're still running into the issue I mentioned in comment #16. From your push attempt above:

remote: Distribution failed for the following gears:
remote: 79311893ae (rsync: connection unexpectedly closed (0 bytes received so far) [sender])

We are working on a fix for this, as well as for increasing the default inode quota for gears. I'll let you know as soon as they're ready, and sorry for the delay!

Andy

Want to ask if this bug is still relevant, or it should be closed.

Due to the discussion I think this bug should be closed, and if there are any further problems considering Rails, I would suggest to open a proper bug for it.

Hi Jakub,

The issue is still there (I can not do "git push origin" without massaging it after manually). I agree that initial issue is resolved. Probably the best option at this point would be to post here link to bug mentioned by Andy. If needed I can provide info about my environment to that bug and help to resolve it.

Thanks,
Boris

Hi Boris,

We should be able to run a command to correct the issue you're facing. When would be a good time to do that?

Andy

Hi Andy,

Not exactly sure what you mean. If you want to do something right on the gear, it is fine to do it any time.

Or if you want me to do something, then please provide steps that I have to execute.

Thanks,
Boris

Hi Boris,

Andy meant that we will launch one of our scritps(oo-admin-ctl-app), which will recreate gear-registry.json, that currently has an incomplete uuid for the child gear.

They wanted to ask you if it will be OK with you if your app will be down, because it will restart haproxy so there will be down.

Jakub

Sure, go ahead any time and please post update here as soon as you are done.

Hi Boris,

we tried to launch the script, but we ran into a problem, which should be fixed very soon (any time now), and as soon as we will deal with this problem, we'll re-launch the script to fix your problem and let you know that it's has been fixed.

Jakub

Boris, could you please try to git push at your convenience and let me know how it goes?

Hi Andy,

I'm happy to confirm that "git push origin" is working again for me:

---------- snip -----------
remote: Starting Ruby cartridge
remote: Executing 'post_start_ruby' hook...
remote: Starting 'check_queue'...
remote: Executing 'post_deploy' hook...
remote: /opt/rh/ruby193/root/usr/bin/ruby /var/lib/openshift/4fcf6f211aeb4279b098ba74635a1e3b/app-root/runtime/repo/vendor/bundle/ruby/1.9.1/bin/rake assets:precompile:all RAILS_ENV=production RAILS_GROUPS=assets
remote: Result: success
remote: Distribution status: success
remote: Activation status: success
remote: Deployment completed with status: success


Thanks,
Boris

Hi Boris,

happy to read that.
If you or Andy (or anyone) won't mind, I will close this issue.

Jakub

Hi Jakub,

I think it is perfectly fine to close this ticket.

Thanks again,
Boris