Bug 976112 - Nodejs apps are failing to migrate
Nodejs apps are failing to migrate
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Hiro Asari
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-19 20:08 EDT by Dan McPherson
Modified: 2015-05-14 19:22 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-22 11:16:53 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dan McPherson 2013-06-19 20:08:45 EDT
With the following error:

Migrating app 'nodejs06s' gear '51be94bd6cec0e6e0a000238' with uuid '51be94bd6cec0e6e0a000238' on node 'ex-sml-node2.int.rhcloud.com' for user: jhou+1@redhat.com
Failed to migrate with cmd: '/usr/bin/rhc-admin-migrate --login 'jhou+1@redhat.com' --migrate-gear '51be94bd6cec0e6e0a000238' --app-name 'nodejs06s' --version '2.0.29' --ignore-cartridge-version' after 2 tries with exception: Failed migrating gear. Rerun with: /usr/bin/rhc-admin-migrate --login 'jhou+1@redhat.com' --migrate-gear '51be94bd6cec0e6e0a000238' --app-name 'nodejs06s' --version '2.0.29' --ignore-cartridge-version
["/usr/bin/rhc-admin-migrate:83:in `block in migrate_gear'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'", "/usr/bin/rhc-admin-migrate:51:in `migrate_gear'", "/usr/bin/rhc-admin-migrate:570:in `block in migrate_from_file'", "/usr/bin/rhc-admin-migrate:568:in `each'", "/usr/bin/rhc-admin-migrate:568:in `migrate_from_file'", "/usr/bin/rhc-admin-migrate:716:in `<main>'"]
Output:
Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'jhou+1@redhat.com' --migrate-gear '51be94bd6cec0e6e0a000238' --app-name 'nodejs06s' --version '2.0.29' --ignore-cartridge-version
Migrating on node...
***time_migrate_on_node_measured_from_broker=1807***
Migrate on node output:
 Beginning 2.0.29 migration for 51be94bd6cec0e6e0a000238
Inspecting gear at /var/lib/openshift/51be94bd6cec0e6e0a000238
Migrating gear at /var/lib/openshift/51be94bd6cec0e6e0a000238
Refreshing cartridge redhat:nodejs:0.6:0.0.1, ignoring cartridge version.
Creating migration instruction validate_gear
Creating migration instruction restart_gear
Incompatible migration of cartridge redhat:nodejs:0.6:0.0.1
Resetting quota blocks: 1048576  inodes: 40000
Shell command '/bin/cp -ad /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/LICENSE /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/hooks /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/logs /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/metadata /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/env /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/lib /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/COPYRIGHT /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/versions /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/README.md /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/bin /var/lib/openshift/51be94bd6cec0e6e0a000238/nodejs' returned an error. rc=1 stdout => 
  stderr => 
 /bin/cp: cannot overwrite directory `/var/lib/openshift/51be94bd6cec0e6e0a000238/nodejs/versions/0.6' with non-directory
Comment 1 Hiro Asari 2013-06-20 08:31:44 EDT
I believe the migration code (which is now removed) looks like this: https://github.com/openshift/li/blob/cefa2a61903e0d81bccc2de339375791ec56e2c9/rhc-node/mcollective/lib/migrate-v2-nodejs-0.6.rb
Comment 2 Hiro Asari 2013-06-20 08:47:37 EDT
(In reply to Hiro Asari from comment #1)
> I believe the migration code (which is now removed) looks like this:
> https://github.com/openshift/li/blob/
> cefa2a61903e0d81bccc2de339375791ec56e2c9/rhc-node/mcollective/lib/migrate-v2-
> nodejs-0.6.rb

Wrong. The error is coming from a shell command, so it's elsewhere.
Comment 3 Hiro Asari 2013-06-20 09:48:26 EDT
`versions/0.6` is a symbolic link pointing to `shared` in the same directory.

[root@ip-10-32-34-46 ~]# ls -l /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/versions/
total 4
lrwxrwxrwx. 1 root root    7 Jun 20 01:50 0.6 -> shared/
drwxr-xr-x. 4 root root 4096 Jun 20 01:50 shared

If the target gear is already on v2 and has nodejs/versions/0.6 already, the 'cp' command fails.
Comment 4 Xiaoli Tian 2013-06-20 10:17:44 EDT
FYI:
The following node.js apps should be created after INT resetting and before INT migration as well, but seems they work correctly now:
https://etherpad-bmeng1int.int.rhcloud.com/
http://no1s-bmeng1int.int.rhcloud.com/
Comment 5 Hiro Asari 2013-06-21 10:34:02 EDT
We should remove 'versions/**/*' before copying files over.

https://github.com/openshift/origin-server/pull/2917
Comment 6 openshift-github-bot 2013-06-24 22:18:46 EDT
Commits pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/966b9c68f22c6ee3c5e5a1728dde5db74c8be7e9
Bug 976112

Some node.js application migration may fail if the existing application
has a real directory `$OPENSHIFT_NODEJS_DIR/versions/0.6`.
Since we are copying the files over from the new version of the
cartridge, simply remove the files during migration so that `cp -ad`
does not fail.

https://github.com/openshift/origin-server/commit/3d225627268a83ea315df56855db073bfff8ad1f
Merge pull request #2917 from BanzaiMan/dev/hasari/bz976112

Bug 976112
Comment 7 Meng Bo 2013-06-25 06:14:17 EDT
Since the code has not been merged into latest ami, I do the testing with manually change the file.

Checked on devenv_3409,

1. Create nodejs app, make the .../versions/0.6 a real dir under the app home.
2. Modify the managed_file.yml under /usr/libexec/openshift/cartridges/v2/nodejs
3. oo-admin-install the nodejs cartridge
4. Do migration

Still failed due to the cp issue.

Shell command '/bin/cp -ad /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/versions /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/lib /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/bin /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/hooks /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/metadata /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/README.md /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/logs /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/COPYRIGHT /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/LICENSE /var/lib/openshift/.cartridge_repository/redhat-nodejs/0.0.1/env /var/lib/openshift/451132768970074887290880/nodejs' returned an error. rc=1 stdout => 
  stderr => 
 /bin/cp: cannot overwrite directory `/var/lib/openshift/451132768970074887290880/nodejs/versions/0.6' with non-directory

Migrate on node exit code: 1


With the fail, the metadata/managed_file.yml has been updated to the latest version for my app.

Seems the fix not work as expected.
Comment 8 Hiro Asari 2013-06-25 15:10:57 EDT
"versions/**/*" doesn't work here, as it turns out.

Ultimately, we will be using Dir.glob to pick up files to process in setup_rewritten, but it doesn't see the symbolic link at all.


$ tree tmp     
tmp
├── symlink -> this
└── this

2 directories, 0 files
$ irb          
1.9.3p429 :001 > RUBY_DESCRIPTION
 => "ruby 1.9.3p429 (2013-05-15 revision 40747) [x86_64-darwin12.4.0]" 
1.9.3p429 :002 > Dir.glob("tmp")
 => ["tmp"]


We will need a different way to attack this problem.
Comment 9 Hiro Asari 2013-06-25 15:12:42 EDT
Ooops. That's wrong.
Comment 10 Hiro Asari 2013-06-25 16:20:35 EDT
OK, we are only processing files for 'setup_rewritten'.

Here's PR addresses this: https://github.com/openshift/origin-server/pull/2928

I'll be working on tests.
Comment 11 Dan McPherson 2013-06-25 16:50:27 EDT
This needs a stage PR.
Comment 12 Hiro Asari 2013-06-26 14:53:58 EDT
Stage PR: https://github.com/openshift/origin-server/pull/2935
Comment 13 openshift-github-bot 2013-06-26 16:17:19 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/6f5bf7cb4106492912f8df778b3173775af93fc5
Bug 976112

In setup_rewritten, we need to process all file types.
Comment 14 Meng Bo 2013-06-27 01:42:56 EDT
Checked on devenv_3419,

Create nodejs app, and modify the versions/ dir for the gear, to make the real dir 0.6 exist.

Do migration.
Migration got successfully.

After migration, the versions/0.6 change back to symlink.

Will move bug status after checked on devenv-stage ami
Comment 15 Meng Bo 2013-06-28 01:43:43 EDT
Tested on devenv-stage_382, issue fixed too.

Note You need to log in before you can comment on or make changes to this bug.