Description of problem: I didn't change anything in my deployment etc, but something changed in the server implementation I think. Previously, every time I pushed my changes, Openshift didn't delete the node_modules directory, and only installs the new ones. Now, it installs all node modules from scratch. Is there any marker or parameters I can use to avoid this. Maybe this was done to avoid version conflicts when we upgrade node js version, but I'm okay with manually deleting my node_modules folder if I need it (using ssh). I have many modules in pesobility.com and down time is now longer due to this issue, I can't use hot_deploy marker too because node_modules folder gets deleted leading to server errors if not stopped. Please advise on the matter. Thank you. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Modify package.json to add modules 2. git push 3. Modify anything 4. git push Actual results: See that all modules are still reinstalled Expected results: Installed modules shouldn't be reinstalled again. Additional info:
Hello. How did you come to the conclusion that the modules are getting re-installed? Right now, we run 'npm install' with the '-d' flag (https://github.com/openshift/origin-server/blob/master/cartridges/openshift-origin-cartridge-nodejs/bin/control#L222). Can it be that this output is a bit more verbose than you are expecting? Here's the output that I get when I add 'coffee-script' to package.json: $ git push Warning: Permanently added 'node-fooooooooooo.dev.rhcloud.com,50.16.114.133' (RSA) to the list of known hosts. Counting objects: 5, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 281 bytes | 0 bytes/s, done. Total 3 (delta 2), reused 0 (delta 0) remote: Saving away previously installed Node modules remote: npm info it worked if it ends with ok remote: npm info using npm.37 remote: npm info using node.20 remote: npm info preinstall OpenShift-Sample-App.0 remote: npm info retry registry request attempt 1 at 14:28:47 remote: npm http GET https://registry.npmjs.org/coffee-script remote: npm http 304 https://registry.npmjs.org/coffee-script remote: npm info install coffee-script.3 into /var/lib/openshift/206206877452268055035904/app-root/runtime/repo remote: npm info installOne coffee-script.3 remote: npm WARN engine coffee-script.3: wanted: {"node":">=0.8.0"} (current: {"node":"0.6.20","npm":"1.1.37"}) remote: npm info /var/lib/openshift/206206877452268055035904/app-root/runtime/repo/node_modules/coffee-script unbuild remote: npm info preinstall coffee-script.3 remote: npm info build /var/lib/openshift/206206877452268055035904/app-root/runtime/repo/node_modules/coffee-script remote: npm info linkStuff coffee-script.3 remote: npm info install coffee-script.3 remote: npm info postinstall coffee-script.3 remote: npm info build /var/lib/openshift/206206877452268055035904/app-root/runtime/repo remote: npm info linkStuff OpenShift-Sample-App.0 remote: npm info install OpenShift-Sample-App.0 remote: npm info postinstall OpenShift-Sample-App.0 remote: coffee-script.3 node_modules/coffee-script remote: npm info ok To ssh://206206877452268055035904.rhcloud.com/~/git/node.git/ 8cda48d..557e637 master -> master If I remove the '-d' flag (you can't, but I am "cheating" a little bit), the output would be a lot more succinct, without the word "install": $ git push Warning: Permanently added 'node-fooooooooooo.dev.rhcloud.com,50.16.114.133' (RSA) to the list of known hosts. Counting objects: 5, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 282 bytes | 0 bytes/s, done. Total 3 (delta 2), reused 0 (delta 0) remote: Saving away previously installed Node modules remote: npm http GET https://registry.npmjs.org/coffee-script remote: npm http 304 https://registry.npmjs.org/coffee-script remote: npm WARN engine coffee-script.3: wanted: {"node":">=0.8.0"} (current: {"node":"0.6.20","npm":"1.1.37"}) remote: coffee-script.3 node_modules/coffee-script To ssh://206206877452268055035904.rhcloud.com/~/git/node.git/ 557e637..56e52d3 master -> master
Hi Hiro, Thank you for looking into this. The reason why I think this is a bug because the difference between the logs of npm install -d in local and in openshift remote is large using the ff modules in package.json: "express": "3.x", "consolidate": "*", "swig": "*", "passport": "*", "passport-local" : "*", "mongodb" : "*", "mongoose" : "*", "connect-flash": "*", "numeral" : "*", "connect-mongodb" : "*", "request" : "*", "xml2js" : "*", "async" : "*", "oauth" : "*", "cron" : "*", "log4js" : "*", "moment" : "2.x", "string" : "*" I think I have replicated the problem: When I execute an npm install locally then also commit the contents of my node_modules, the bug is gone! Why is this happening? I never committed/pushed my node_modules folder ever since (because I want the server to install its dependencies). The issue was not there before. Now, if I don't commit a module, then it is reinstalled everytime. Replication Steps: 1. rhc app create -a testapp -t nodejs-0.6 2. Add the modules above 3. commit and git push (npm takes a while = normal) 4. change anything (like add a space in readme file) 5. commit and git push (npm still takes a while = NOT normal = bug) Workaround: 6. "npm install -d" in local environment 7. commit the modules and git push (npm is instant, problem is gone. = bug is confirmed?)
I verified the problem here. Sorry about the delay in verifying. We are currently using $OPENSHIFT_REPO_DIR to stash away the existing modules, but that directory is refreshed every time you push the code. I'm working on a fix. Thank you.
No problem, and thanks too! I'm glad that it is getting fixed.
Would like to add a special case for this scenario: It is also possible that user commits a node module that is not part of package.json. E.g. user adds node_module/test_module and pushes it to openshift. The final node modules should be package.json's node modules + user's committed node modules. This scenario might affect the fix. Thank you.
Do you mind opening a separate ticket for the new use case?
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/2c6d2576cd1fd7b74ca78c6793625e92a1c9f840 Bug 982054 $OPENSHIFT_REPO_DIR is cleaned up before pre-build() function is invoked, so we were simply saving away an empty directory. Move this logic to a correct phase of the build cycle, and we really save away the installed modules.
Thank you for the fast turn around. For the new use case, maybe I'll just add it if it's a problem after the fix? Since I think based on your description, the case should be handled properly. Btw, when will it be up in production? I'm not familiar with the workflow if the fixes are incorporated as one deployment to production (e.g. when there's a planned outage for server maintenance) or it is added incrementally and I can also test it now. Regards, Manny
Checked on devenv_3482, With the dependency list which reporter provided. In the second push, it will not install the packages again. remote: Saving away previously installed Node modules remote: npm info it worked if it ends with ok remote: npm info using npm.37 remote: npm info using node.20 remote: npm info preinstall OpenShift-Sample-App.0 remote: npm WARN package.json passport-local.6 No README.md file found! remote: npm info build /var/lib/openshift/51de79ff36ff800705000004/app-root/runtime/repo remote: npm info linkStuff OpenShift-Sample-App.0 remote: npm info install OpenShift-Sample-App.0 remote: npm info postinstall OpenShift-Sample-App.0 remote: npm info ok To ssh://51de79ff36ff800705000004.rhcloud.com/~/git/no1.git/ f74739f..cce87a3 master -> master @pesobility The changes should be pushed to PROD on next Tuesday in a normal condition.