Bug 982054 - Node JS modules are always re installed from scratch for every git push
Node JS modules are always re installed from scratch for every git push
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Hiro Asari
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-07 22:40 EDT by pesobility
Modified: 2015-05-14 19:23 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-22 11:21:53 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description pesobility 2013-07-07 22:40:39 EDT
Description of problem:
I didn't change anything in my deployment etc, but something changed in the server implementation I think. Previously, every time I pushed my changes, Openshift didn't delete the node_modules directory, and only installs the new ones. Now, it installs all node modules from scratch. Is there any marker or parameters I can use to avoid this.

Maybe this was done to avoid version conflicts when we upgrade node js version, but I'm okay with manually deleting my node_modules folder if I need it (using ssh). I have many modules in pesobility.com and down time is now longer due to this issue, I can't use hot_deploy marker too because node_modules folder gets deleted leading to server errors if not stopped. Please advise on the matter. Thank you.

Version-Release number of selected component (if applicable):


How reproducible:

Steps to Reproduce:
1. Modify package.json to add modules
2. git push
3. Modify anything
4. git push

Actual results:
See that all modules are still reinstalled

Expected results:
Installed modules shouldn't be reinstalled again.

Additional info:
Comment 1 Hiro Asari 2013-07-08 14:34:54 EDT
Hello.

How did you come to the conclusion that the modules are getting re-installed?

Right now, we run 'npm install' with the '-d' flag (https://github.com/openshift/origin-server/blob/master/cartridges/openshift-origin-cartridge-nodejs/bin/control#L222). Can it be that this output is a bit more verbose than you are expecting?

Here's the output that I get when I add 'coffee-script' to package.json:

$ git push
Warning: Permanently added 'node-fooooooooooo.dev.rhcloud.com,50.16.114.133' (RSA) to the list of known hosts.
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 281 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Saving away previously installed Node modules
remote: npm info it worked if it ends with ok
remote: npm info using npm@1.1.37
remote: npm info using node@v0.6.20
remote: npm info preinstall OpenShift-Sample-App@1.0.0
remote: npm info retry registry request attempt 1 at 14:28:47
remote: npm http GET https://registry.npmjs.org/coffee-script
remote: npm http 304 https://registry.npmjs.org/coffee-script
remote: npm info install coffee-script@1.6.3 into /var/lib/openshift/206206877452268055035904/app-root/runtime/repo
remote: npm info installOne coffee-script@1.6.3
remote: npm WARN engine coffee-script@1.6.3: wanted: {"node":">=0.8.0"} (current: {"node":"0.6.20","npm":"1.1.37"})
remote: npm info /var/lib/openshift/206206877452268055035904/app-root/runtime/repo/node_modules/coffee-script unbuild
remote: npm info preinstall coffee-script@1.6.3
remote: npm info build /var/lib/openshift/206206877452268055035904/app-root/runtime/repo/node_modules/coffee-script
remote: npm info linkStuff coffee-script@1.6.3
remote: npm info install coffee-script@1.6.3
remote: npm info postinstall coffee-script@1.6.3
remote: npm info build /var/lib/openshift/206206877452268055035904/app-root/runtime/repo
remote: npm info linkStuff OpenShift-Sample-App@1.0.0
remote: npm info install OpenShift-Sample-App@1.0.0
remote: npm info postinstall OpenShift-Sample-App@1.0.0
remote: coffee-script@1.6.3 node_modules/coffee-script
remote: npm info ok 
To ssh://206206877452268055035904@node-fooooooooooo.dev.rhcloud.com/~/git/node.git/
   8cda48d..557e637  master -> master


If I remove the '-d' flag (you can't, but I am "cheating" a little bit), the output would be a lot more succinct, without the word "install":

$ git push              
Warning: Permanently added 'node-fooooooooooo.dev.rhcloud.com,50.16.114.133' (RSA) to the list of known hosts.
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 282 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Saving away previously installed Node modules
remote: npm http GET https://registry.npmjs.org/coffee-script
remote: npm http 304 https://registry.npmjs.org/coffee-script
remote: npm WARN engine coffee-script@1.6.3: wanted: {"node":">=0.8.0"} (current: {"node":"0.6.20","npm":"1.1.37"})
remote: coffee-script@1.6.3 node_modules/coffee-script
To ssh://206206877452268055035904@node-fooooooooooo.dev.rhcloud.com/~/git/node.git/
   557e637..56e52d3  master -> master
Comment 2 pesobility 2013-07-09 00:31:12 EDT
Hi Hiro,

Thank you for looking into this.

The reason why I think this is a bug because the difference between the logs of npm install -d in local and in openshift remote is large using the ff modules in package.json:
    "express": "3.x",
    "consolidate": "*",
    "swig": "*",
    "passport": "*",
    "passport-local" : "*",
    "mongodb" : "*",
    "mongoose" : "*",
    "connect-flash": "*",
    "numeral" : "*",
    "connect-mongodb" : "*",
    "request" : "*",
    "xml2js" : "*",
    "async" : "*",
    "oauth" : "*",
    "cron" : "*",
    "log4js" : "*",
    "moment" : "2.x",
    "string" : "*"

I think I have replicated the problem:

When I execute an npm install locally then also commit the contents of my node_modules, the bug is gone!

Why is this happening? I never committed/pushed my node_modules folder ever since (because I want the server to install its dependencies).  The issue was not there before. Now, if I don't commit a module, then it is reinstalled everytime.


Replication Steps:
1. rhc app create -a testapp -t nodejs-0.6
2. Add the modules above
3. commit and git push (npm takes a while = normal)
4. change anything (like add a space in readme file)
5. commit and git push (npm still takes a while = NOT normal = bug)
Workaround:
6. "npm install -d" in local environment
7. commit the modules and git push (npm is instant, problem is gone. = bug is confirmed?)
Comment 3 Hiro Asari 2013-07-09 12:03:32 EDT
I verified the problem here. Sorry about the delay in verifying.

We are currently using $OPENSHIFT_REPO_DIR to stash away the existing modules, but that directory is refreshed every time you push the code.

I'm working on a fix.

Thank you.
Comment 4 pesobility 2013-07-10 01:35:24 EDT
No problem, and thanks too! I'm glad that it is getting fixed.
Comment 5 pesobility 2013-07-10 04:04:16 EDT
Would like to add a special case for this scenario:

It is also possible that user commits a node module that is not part of package.json.

E.g. user adds node_module/test_module and pushes it to openshift. The final node modules should be package.json's node modules + user's committed node modules.

This scenario might affect the fix. Thank you.
Comment 6 Hiro Asari 2013-07-10 14:21:01 EDT
Do you mind opening a separate ticket for the new use case?
Comment 7 openshift-github-bot 2013-07-10 15:59:36 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/2c6d2576cd1fd7b74ca78c6793625e92a1c9f840
Bug 982054

$OPENSHIFT_REPO_DIR is cleaned up before pre-build() function is
invoked, so we were simply saving away an empty directory.

Move this logic to a correct phase of the build cycle, and we really
save away the installed modules.
Comment 8 pesobility 2013-07-10 22:23:17 EDT
Thank you for the fast turn around. For the new use case, maybe I'll just add it if it's a problem after the fix? Since I think based on your description, the case should be handled properly.

Btw, when will it be up in production? I'm not familiar with the workflow if the fixes are incorporated as one deployment to production (e.g. when there's a planned outage for server maintenance) or it is added incrementally and I can also test it now.

Regards,
Manny
Comment 9 Meng Bo 2013-07-11 05:46:26 EDT
Checked on devenv_3482,

With the dependency list which reporter provided.
In the second push, it will not install the packages again.

remote: Saving away previously installed Node modules
remote: npm info it worked if it ends with ok
remote: npm info using npm@1.1.37
remote: npm info using node@v0.6.20
remote: npm info preinstall OpenShift-Sample-App@1.0.0
remote: npm WARN package.json passport-local@0.1.6 No README.md file found!
remote: npm info build /var/lib/openshift/51de79ff36ff800705000004/app-root/runtime/repo
remote: npm info linkStuff OpenShift-Sample-App@1.0.0
remote: npm info install OpenShift-Sample-App@1.0.0
remote: npm info postinstall OpenShift-Sample-App@1.0.0
remote: npm info ok 
To ssh://51de79ff36ff800705000004@no1-bmeng.dev.rhcloud.com/~/git/no1.git/
   f74739f..cce87a3  master -> master


@pesobility 
The changes should be pushed to PROD on next Tuesday in a normal condition.

Note You need to log in before you can comment on or make changes to this bug.