Bug 976112

Summary: Nodejs apps are failing to migrate
Product: OpenShift Online Reporter: Dan McPherson <dmcphers>
Component: ContainersAssignee: Hiro Asari <hasari>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.xCC: admiller, bmeng, jkeck, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-22 15:16:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dan McPherson 2013-06-20 00:08:45 UTC
With the following error:

Migrating app 'nodejs06s' gear '51be94bd6cec0e6e0a000238' with uuid '51be94bd6cec0e6e0a000238' on node 'ex-sml-node2.int.rhcloud.com' for user: jhou+1
Failed to migrate with cmd: '/usr/bin/rhc-admin-migrate --login 'jhou+1' --migrate-gear '51be94bd6cec0e6e0a000238' --app-name 'nodejs06s' --version '2.0.29' --ignore-cartridge-version' after 2 tries with exception: Failed migrating gear. Rerun with: /usr/bin/rhc-admin-migrate --login 'jhou+1' --migrate-gear '51be94bd6cec0e6e0a000238' --app-name 'nodejs06s' --version '2.0.29' --ignore-cartridge-version
["/usr/bin/rhc-admin-migrate:83:in `block in migrate_gear'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'", "/usr/bin/rhc-admin-migrate:51:in `migrate_gear'", "/usr/bin/rhc-admin-migrate:570:in `block in migrate_from_file'", "/usr/bin/rhc-admin-migrate:568:in `each'", "/usr/bin/rhc-admin-migrate:568:in `migrate_from_file'", "/usr/bin/rhc-admin-migrate:716:in `<main>'"]
Output:
Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'jhou+1' --migrate-gear '51be94bd6cec0e6e0a000238' --app-name 'nodejs06s' --version '2.0.29' --ignore-cartridge-version
Migrating on node...
***time_migrate_on_node_measured_from_broker=1807***
Migrate on node output:
 Beginning 2.0.29 migration for 51be94bd6cec0e6e0a000238
Inspecting gear at /var/lib/openshift/51be94bd6cec0e6e0a000238
Migrating gear at /var/lib/openshift/51be94bd6cec0e6e0a000238
Refreshing cartridge redhat:nodejs:0.6:0.0.1, ignoring cartridge version.
Creating migration instruction validate_gear
Creating migration instruction restart_gear
Incompatible migration of cartridge redhat:nodejs:0.6:0.0.1
Resetting quota blocks: 1048576  inodes: 40000
Shell command '/bin/cp -ad /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/LICENSE /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/hooks /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/logs /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/metadata /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/env /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/lib /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/COPYRIGHT /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/versions /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/README.md /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/bin /var/lib/openshift/51be94bd6cec0e6e0a000238/nodejs' returned an error. rc=1 stdout => 
  stderr => 
 /bin/cp: cannot overwrite directory `/var/lib/openshift/51be94bd6cec0e6e0a000238/nodejs/versions/0.6' with non-directory

Comment 1 Hiro Asari 2013-06-20 12:31:44 UTC
I believe the migration code (which is now removed) looks like this: https://github.com/openshift/li/blob/cefa2a61903e0d81bccc2de339375791ec56e2c9/rhc-node/mcollective/lib/migrate-v2-nodejs-0.6.rb

Comment 2 Hiro Asari 2013-06-20 12:47:37 UTC
(In reply to Hiro Asari from comment #1)
> I believe the migration code (which is now removed) looks like this:
> https://github.com/openshift/li/blob/
> cefa2a61903e0d81bccc2de339375791ec56e2c9/rhc-node/mcollective/lib/migrate-v2-
> nodejs-0.6.rb

Wrong. The error is coming from a shell command, so it's elsewhere.

Comment 3 Hiro Asari 2013-06-20 13:48:26 UTC
`versions/0.6` is a symbolic link pointing to `shared` in the same directory.

[root@ip-10-32-34-46 ~]# ls -l /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/versions/
total 4
lrwxrwxrwx. 1 root root    7 Jun 20 01:50 0.6 -> shared/
drwxr-xr-x. 4 root root 4096 Jun 20 01:50 shared

If the target gear is already on v2 and has nodejs/versions/0.6 already, the 'cp' command fails.

Comment 4 Xiaoli Tian 2013-06-20 14:17:44 UTC
FYI:
The following node.js apps should be created after INT resetting and before INT migration as well, but seems they work correctly now:
https://etherpad-bmeng1int.int.rhcloud.com/
http://no1s-bmeng1int.int.rhcloud.com/

Comment 5 Hiro Asari 2013-06-21 14:34:02 UTC
We should remove 'versions/**/*' before copying files over.

https://github.com/openshift/origin-server/pull/2917

Comment 6 openshift-github-bot 2013-06-25 02:18:46 UTC
Commits pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/966b9c68f22c6ee3c5e5a1728dde5db74c8be7e9
Bug 976112

Some node.js application migration may fail if the existing application
has a real directory `$OPENSHIFT_NODEJS_DIR/versions/0.6`.
Since we are copying the files over from the new version of the
cartridge, simply remove the files during migration so that `cp -ad`
does not fail.

https://github.com/openshift/origin-server/commit/3d225627268a83ea315df56855db073bfff8ad1f
Merge pull request #2917 from BanzaiMan/dev/hasari/bz976112

Bug 976112

Comment 7 Meng Bo 2013-06-25 10:14:17 UTC
Since the code has not been merged into latest ami, I do the testing with manually change the file.

Checked on devenv_3409,

1. Create nodejs app, make the .../versions/0.6 a real dir under the app home.
2. Modify the managed_file.yml under /usr/libexec/openshift/cartridges/v2/nodejs
3. oo-admin-install the nodejs cartridge
4. Do migration

Still failed due to the cp issue.

Shell command '/bin/cp -ad /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/versions /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/lib /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/bin /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/hooks /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/metadata /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/README.md /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/logs /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/COPYRIGHT /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/LICENSE /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/env /var/lib/openshift/451132768970074887290880/nodejs' returned an error. rc=1 stdout => 
  stderr => 
 /bin/cp: cannot overwrite directory `/var/lib/openshift/451132768970074887290880/nodejs/versions/0.6' with non-directory

Migrate on node exit code: 1


With the fail, the metadata/managed_file.yml has been updated to the latest version for my app.

Seems the fix not work as expected.

Comment 8 Hiro Asari 2013-06-25 19:10:57 UTC
"versions/**/*" doesn't work here, as it turns out.

Ultimately, we will be using Dir.glob to pick up files to process in setup_rewritten, but it doesn't see the symbolic link at all.


$ tree tmp     
tmp
├── symlink -> this
└── this

2 directories, 0 files
$ irb          
1.9.3p429 :001 > RUBY_DESCRIPTION
 => "ruby 1.9.3p429 (2013-05-15 revision 40747) [x86_64-darwin12.4.0]" 
1.9.3p429 :002 > Dir.glob("tmp")
 => ["tmp"]


We will need a different way to attack this problem.

Comment 9 Hiro Asari 2013-06-25 19:12:42 UTC
Ooops. That's wrong.

Comment 10 Hiro Asari 2013-06-25 20:20:35 UTC
OK, we are only processing files for 'setup_rewritten'.

Here's PR addresses this: https://github.com/openshift/origin-server/pull/2928

I'll be working on tests.

Comment 11 Dan McPherson 2013-06-25 20:50:27 UTC
This needs a stage PR.

Comment 12 Hiro Asari 2013-06-26 18:53:58 UTC
Stage PR: https://github.com/openshift/origin-server/pull/2935

Comment 13 openshift-github-bot 2013-06-26 20:17:19 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/6f5bf7cb4106492912f8df778b3173775af93fc5
Bug 976112

In setup_rewritten, we need to process all file types.

Comment 14 Meng Bo 2013-06-27 05:42:56 UTC
Checked on devenv_3419,

Create nodejs app, and modify the versions/ dir for the gear, to make the real dir 0.6 exist.

Do migration.
Migration got successfully.

After migration, the versions/0.6 change back to symlink.

Will move bug status after checked on devenv-stage ami

Comment 15 Meng Bo 2013-06-28 05:43:43 UTC
Tested on devenv-stage_382, issue fixed too.