Bug 1126343

Created attachment 923801 [details]
Error thrown after accessing rails app on openshift

I can't think of any great ways for OpenShift to address this. I can think of some workarounds.

(In reply to Miheer Salunke from comment #0)
> Now, the Gemfile.lock will not contain the entry for it and thereby it won't
> get bundle installed on the remotely on the gear.

I want to clarify what (I think) happens here. Gemfile specifies what dependencies are directly required. Gemfile.lock is used to further lock down the versions of dependencies (direct and transitive) that are required. Bundler uses these two, if available, to determine the exact gems it needs to install in order to run the app.

Technically, you don't generally have to have Gemfile.lock, just Gemfile. OpenShift's ruby cart won't let you actually deploy without Gemfile.lock, but you could actually remove most of the contents and bundle install would still use whatever fulfilled the requirements.

> This will led to error as per the screenshot attached as it will not find
> the "Javascript runtime" on the gear.

ExecJS is a little special in that it requires a JavaScript runtime, but not a specific gem (several implementations are available, none are universal). There's no way to encode that flexibility in Gemfile or Gemfile.lock.

therubyracer is a specific implementation that's used on Linux but not available under Windows. So of course, when you use bundler to write out Gemfile.lock for a Rails app created on Windows, it's not going to include therubyracer. In fact, under Windows, I think the JS runtime is inferred without any reference in Gemfile.

> The code related to it checking whether Gemfile is modified or not.
> https://github.com/openshift/origin-server/blob/master/cartridges/openshift-
> origin-cartridge-ruby/lib/util#L65

That particular line just checks for existence of a Gemfile.lock, but in general, if Gemfile is changed, bundle install will be invoked, and if not, it won't. That makes sense.

> Now, this is an example of the 'therubyracer' gem but, there are several
> gems which are not compatible with windows but, Linux.

This would potentially be a problem when using any "native" gem that only exists for particular platforms. It could be a problem for any framework with dependencies that can vary between platforms. I don't think there's a general solution to that, so I'm leaning toward CANTFIX on this bug.

I think you could solve this specific issue simply by specifying "therubyracer" in the Gemfile after bundling everything else. You couldn't then bundle install on Windows without commenting it out temporarily, but the Gemfile pushed to OpenShift would still specify the dependency adequately without it being in Gemfile.lock.

More complicated scenarios could require ssh'ing into the gear to run bundle install on the target platform, perhaps doing surgery on the Gemfile/Gemfile.lock there and bringing it back to your git repo. You might need to keep separate Gemfile{,.lock} for Windows vs Linux if you want to develop the app in both places.

Can you try this out? We should provide guidance.

Hi Sir,

I did a bundle install of sqlite3 and the rubyracer gem on a Linux system and copied the contents of the Gemfile.lock present on Linux system to Gemfile.lock of application present on windows.
Then did a git push deployment was a success and the gems were installed. But, when I access the url of the application it throws 500. I have attached the screenshot accordingly.

I tried in another way also. Create an app and then first comment the "therubyracer" and "sqlite3" in the Gemfile. Then I did a bundle install on windows machine and accordingly a Gemfile.lock was created. Then I pushed these 2 files. The deployment was a success. Then I added "therubyracer" and "sqlite3" to the Gemfile and pushed it.
Then I sshed into the gear and did a bundle install still the same issue.

Thanks and regards,
Miheer.

Created attachment 925093 [details]
500 error

Thanks for checking this out. Sounds like it's a little trickier than I hoped.

A 500 error doesn't really tell you much other than something went wrong on the server side. Logs from the gear would help to determine if the nature of the problem is the same.

I will note that when you were having bundler issues initially, the passenger loader actually gave you a nice error telling you exactly what was missing. It didn't do that for these most recent examples - is it possible it's running into some other kind of error? Or is it just a difference in environment?

(In reply to Luke Meyer from comment #6)
> Thanks for checking this out. Sounds like it's a little trickier than I
> hoped.
> 
> A 500 error doesn't really tell you much other than something went wrong on
> the server side. Logs from the gear would help to determine if the nature of
> the problem is the same.
> 
> I will note that when you were having bundler issues initially, the
> passenger loader actually gave you a nice error telling you exactly what was
> missing. It didn't do that for these most recent examples - is it possible
> it's running into some other kind of error? Or is it just a difference in
> environment?

Luke,
I am the local SA working with the customer and our local partner (that opened this case)
You mention a difference in environment..do you think it would
be a good idea to communicate this workaround to the customer
and maybe try it on their environment?
Do you think this will have any value add here?
If so, supposing it will not work, what assets from their environment 
would you like to see for furthur investigating this issue?
Thanks, Amir

Amir, 
Under development environment, the browser gives you a lot of details about went wrong (as in first screenshot). Under production environment (openshift default), it doesn't and you have to look at the logs. It just wasn't clear to me whether the workaround failed completely, or if it actually worked and some other problem caused an error (e.g. missing DB config).

Just going to note that cross-platform deploy is a general problem under bundler:
https://github.com/bundler/bundler-features/issues/4
No implementation of that feature yet. That said, if I can get some time to look at it, there may be decent workarounds for users that we can document. There may even be something the PaaS can do, although the only clear workaround I heard was "Heroku deletes Gemfile.lock" which doesn't seem satisfactory to me and I'm not sure it would even work for this case.

It should be noted http://whyjava.wordpress.com/2012/12/26/openshift-rails-quickstart/ was written under the original (v1) cartridge format and needs some adjustment anyway. It's also not clear which version of Rails you would have wound up with at the time. At this time I think the default is 4.1. My guess would be 3.1 or 3.2 at the time. Although the problem of "cross-platform bundler" is generic, the specifics are likely to vary depending both on Rails versions as well as any other dependencies (e.g. pg and other drivers that wind up with native components).

I would recommend starting with https://github.com/openshift/rails-example (it's worth noting that at this time, if you do so, you get Rails 3.2, and use of mysql or pg is *required* unless you hack out a bunch of stuff from the action hooks etc.). 

When you say "create rails app for OpenShift" there are also a number of variations that need to be taken into account:
1. What's RAILS_ENV? It defaults to production, but could be changed with an env var to development. In production, execjs/therubyracer shouldn't even be needed; the assets should be compiled at deploy time.
2. OpenShift has some markers that affect how/when bundler is invoked, e.g. .openshift/markers/force_clean_build - depending on what procedure you follow, these may be set without your knowledge.
3. Same with action_hooks - the quickstart above puts quite a few in place.

There are also a number of frustrating factors about using OpenShift here:
1. Getting mysql2 or pg gems to install on Windows is... a bit challenging. And if you do, your Gemfile.lock will probably have a Windows-specific version that you need to edit manually for use on Linux.
2. OpenShift ruby will not let you deploy with just a Gemfile and no Gemfile.lock.
3. If you make changes to Gemfile *only* and not Gemfile.lock, it falsely reports
remote: NOTE: Skipping 'bundle install' because Gemfile is not modified.
... and so it won't update Gemfile.lock for you. (This seems like a bug, at least in that it's not reporting the actual criterion used to decide.)
4. If you make some change to Gemfile.lock so that it *will* run bundle install, it does so in --deployment mode, which apparently means it refuses to update Gemfile.lock for you.
5. If you ssh in to the gear to try to run bundle install, it apparently still defaults to --deployment mode.

So in the end, to recover from having run "bundle install" on Windows, I needed to do the following:
1. Push everything up to the gear.
2. ssh to the gear
> rhc ssh railsapp
3. Go to a copy of the repo:
> cd app-root/repo
4. Modify Gemfile/.lock as needed. In my example, to add mysql2 back to Gemfile; if something Windows-specific were in the Gemfile.lock, would have to remove it.
5. Run bundle install (repeat adjustments in 4 until satisfactory):
>  bundle install --no-deployment
6. Copy the updated Gemfile/.lock back to my repository
> rhc scp railsapp download  ./ app-root/repo/Gemfile 
> rhc scp railsapp download  ./ app-root/repo/Gemfile.lock  
7. Commit and push the changes


I didn't at any point see anything about therubyracer. I'm not quite sure why. But in any case, I think the above procedure will in general work out for getting Gemfile/.lock right. Basically, you just do it on Linux. You'd have to maintain separate Gemfile/.lock to get the app to actually run on Windows, which is a pain for sure :(

Thoughts?

I'm not sure what you mean by this: "OpenShift ruby will not let you deploy with just a Gemfile and no Gemfile.lock." It is possible to push without Gemfile.lock, and OpenShift will deploy the new code; the absence of Gemfile.lock just causes OpenShift to skip the build step.

However, it seems like the build step is exactly what we want in this situation: One develops on Windows with the Gemfile.lock that works on Windows but not Linux and pushes the code tree minus Gemfile.lock to OpenShift, and we want OpenShift then to run Bundler to generate a Gemfile.lock that will work on Linux.

Miheer and I tried using an action hook to perform the build step:

(1) Add "gem 'therubyracer' if PLATFORM.match('linux')" to Gemfile.

(2) Delete Gemfile.lock (`git rm Gemfile.lock`).

(3) Create a .openshift/action_hooks/build script that runs the `bundle install --no-deployment` command in app-root/repo, remember to `git update-index --chmod=+x` the new script, `git add` it, and commit.

(5) Push.

At one point, it looked like this was working, so Gemfile and Gemfile.lock on the gear were both good.  There was an error message "'<class: model>': uninitialize constant ActiveModel::Observing(NameError)" during the push (perhaps somehow related to bug 1105660?), but the application itself was working.  However, we started seeing "An error occurred executing 'gear prereceive' (exit code: 1)" errors.  That's where Miheer and I left off yesterday.

Sorry, that should be RUBY_PLATFORM, not PLATFORM (the latter is absent in Ruby 1.9.3, we discovered).

(In reply to Miciah Dashiel Butler Masters from comment #12)
> I'm not sure what you mean by this: "OpenShift ruby will not let you deploy
> with just a Gemfile and no Gemfile.lock."

I meant that I get this:

remote: Building Ruby cartridge
remote: Restoring previously bundled RubyGems
remote: NOTE: You can commit .openshift/markers/force_clean_build to force a clean bundle
remote: bundle install --deployment --path ./app-root/repo/vendor/bundle
remote: NOTE: You can prevent installing certain Gemfile group using: rhc env set BUNDLE_WITHOUT=groupname
remote: The --deployment flag requires a Gemfile.lock. Please make sure you have checked
remote: your Gemfile.lock into version control before deploying.
remote: An error occurred executing 'gear postreceive' (exit code: 16)
remote: Error message: CLIENT_ERROR: Failed to execute: 'control build' for /var/lib/openshift/53ec2cbc5004469621000480/ruby

This is because the --deployment flag is hardcoded in the build script.
https://github.com/openshift/origin-server/blob/370f6c244933c02a46ab54b831a096ef668daed1/cartridges/openshift-origin-cartridge-ruby/bin/control#L160-L169

Yes, you can push the code, but the build fails and nothing works. Unless you work around it as below.

> Miheer and I tried using an action hook to perform the build step:
> 
> (1) Add "gem 'therubyracer' if PLATFORM.match('linux')" to Gemfile.
> 
> (2) Delete Gemfile.lock (`git rm Gemfile.lock`).

You don't actually have to delete it, just delete the specifications that aren't Linux-compatible. Deleting it means it will be generated from scratch, but that means you lose all of information in Gemfile.lock and you're at the mercy of locally-installed gems and rubygems.org for what versions get installed.

> (3) Create a .openshift/action_hooks/build script that runs the `bundle
> install --no-deployment` command in app-root/repo, remember to `git
> update-index --chmod=+x` the new script, `git add` it, and commit.

As far as I can tell, this is not executed prior to the part of the build that fails above. The pre_build action hook is probably where you should do this. That ought to get the Gemfile.lock into a state where the build step can use it (or fail trying).

>  There was an error message "'<class: model>':
> uninitialize constant ActiveModel::Observing(NameError)" during the push
> (perhaps somehow related to bug 1105660?), but the application itself was
> working. 

sounds like an app error, or maybe because you weren't getting the versions of dependencies that the app expected.

> However, we started seeing "An error occurred executing 'gear
> prereceive' (exit code: 1)" errors.  That's where Miheer and I left off
> yesterday.

More specifics would be helpful here :)

I think fixing up Gemfile.lock in pre_build is a good strategy in general for this problem -- seemed to work when I tried. It would still require a little knowledge of the issue but wouldn't be a complete pain to work around (and may not require maintaining separate versions of the Gemfile/.lock for the two platforms).

Just noticed this:
remote: NOTE: To disable assets compilation use 'disable_asset_compilation' marker.
If that's in place, then therubyracer would be needed for runtime compiles; otherwise it's only needed during the build step that compiles assets (don't think bundler would hit it). So that would be significant to know too.

Hello Sir,

What I think over here is that we can add gems in the Gemfile checking  RUBY_PLATFORM value.

Eg- gem "therubyracer"  if RUBY_PLATFORM == "x86_64-linux"
    
so, it wont get installed on the windows machine and then the bundle install shall not be done based on Gemfile.lock instead we should do bundle install of gems from the Gemfile pushed when any changes are made to it OR we can tell the customers to add this step in the build hook so that it's done on their own responsibility.

I have given 2 workarounds-

1. https://access.redhat.com/support/cases/01143954/#a0aA000000CRfppIAD

2. https://access.redhat.com/support/cases/01143954/#a0aA000000CRcIZIA1

But, you get and observer issue if we use rails 4 which is because of https://bugzilla.redhat.com/show_bug.cgi?id=1105660

Thanks & Regards,
Miheer

Ideally we would put together a kbase on how to deal with this general problem of apps/bundler outside the platform having different results and needs than apps/bundler in the platform. I think it can be summarized like this:

1. In the Gemfile, use logic based on RUBY_PLATFORM to dynamically specify platform-specific gems as needed for each platform (OpenShift and local dev).
2. In the Gemfile.lock on your local development platform, remove any platform-specific entries that cause problems under OpenShift (or, if you don't care about locking in versions to match what you develop and test with, just remove Gemfile.lock / put it in .gitignore).
3. Use bundle install --no-deployment on OpenShift itself to create/update the Gemfile.lock based on Gemfile. This can be done manually and committed back to the repo, but IMHO it would be best to just do this in a hook before the actual bundle install --deployment for a smoother process.

Does that sound about right? I need to work out the exact details and some examples...

-------------

The ActiveModel::Observing issue is a separate one (not related to x-platform) and I don't quite understand how that could be happening. I did not observe the problem in bug 1105660 when I created a Rails app with Rails 4.1. Bundler installs everything under app-root/runtime/repo/vendor/bundle so I don't even see how it could be affecting gear/cartridge functionality. It certainly shouldn't be possible unless gear/cartridge functionality is being invoked in the context of the app or under "bundle exec". By any chance, do you have an example application and specific steps to produce the error you saw?

(In reply to Miciah Dashiel Butler Masters from comment #12)
> I'm not sure what you mean by this: "OpenShift ruby will not let you deploy
> with just a Gemfile and no Gemfile.lock." It is possible to push without
> Gemfile.lock, and OpenShift will deploy the new code; the absence of
> Gemfile.lock just causes OpenShift to skip the build step.
> 
[...]
> (1) Add "gem 'therubyracer' if PLATFORM.match('linux')" to Gemfile.
> 
> (2) Delete Gemfile.lock (`git rm Gemfile.lock`).
> 
> (3) Create a .openshift/action_hooks/build script that runs the `bundle
> install --no-deployment` command in app-root/repo, remember to `git
> update-index --chmod=+x` the new script, `git add` it, and commit.

I finally see what you meant here. With *no* Gemfile.lock, you'd have to substitute your own build step to do the bundle install.

I don't think it's fair to assume we can completely remove Gemfile.lock. It does serve a purpose (allow you to lock down gem versions to match what you tested against locally). Under the condition where Gemfile.lock is actually present, adding a pre_build script to do the `bundle install --no-deployment` would fix it up (if possible) before the normal build script did a `bundle install --deployment` with all the logic surrounding that.

I suspect the problems encountered after that had to do with manual bundle install (or gem install) installing gems into $GEM_HOME rather than under the app vendor/bundle. $GEM_HOME gems could be pulled in by gear functions and cause them to fail.

(In reply to Miheer Salunke from comment #15)
> Hello Sir,
> 
> What I think over here is that we can add gems in the Gemfile checking 
> RUBY_PLATFORM value.
> 
> Eg- gem "therubyracer"  if RUBY_PLATFORM == "x86_64-linux"
>     
> so, it wont get installed on the windows machine and then the bundle install
> shall not be done based on Gemfile.lock instead we should do bundle install
> of gems from the Gemfile pushed when any changes are made to it OR we can
> tell the customers to add this step in the build hook so that it's done on
> their own responsibility.
> 
> I have given 2 workarounds-
> 
> 1. https://access.redhat.com/support/cases/01143954/#a0aA000000CRfppIAD
> 
> 2. https://access.redhat.com/support/cases/01143954/#a0aA000000CRcIZIA1
> 
> But, you get and observer issue if we use rails 4 which is because of
> https://bugzilla.redhat.com/show_bug.cgi?id=1105660
> 
> Thanks & Regards,
> Miheer

Hello Miheer,
I ran into the same problem (working with openshift from windows), can you publish the workarounds sent to the customers?

Thank you,
Eran

I would wonder if something like this would work:

http://yopefonic.wordpress.com/2011/06/23/multi-platform-ruby-development-with-bundler/

In that site the author effectively has the default bundler command work for his deployment environment and then in development users have to pass in "--without production".

I don't know how long this feature has existed, but it does seem like bundler has some concept of "platforms":

http://bundler.io/v1.3/man/gemfile.5.html

That seems like it's made to handle this situation.

As there are known workaround for this and with the impending EOL for v2, we are closing this bug.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days