During the V1 -> V2 migration in INT, 5 python 2.6 apps failed to migrate due to the following issue: Output: Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'xtian+int1' --migrate-gear '5195e2906cec0e33b30000b1' --app-name 'django' --version '2.0.28' Migrating on node... ***time_migrate_on_node_measured_from_broker=1126*** Migrate on node output: Beginning V1 -> V2 migration Creating V2 stop_lock Marking step migrate_stop_lock complete Caught an exception during internal migration steps: No such file or directory - /var/l ib/openshift/5195e2906cec0e33b30000b1/python-2.6/virtenv/lib64 /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:1515:in `stat' /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:1515:in `block in fu_each_src_dest' /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:1531:in `fu_each_src_dest0' /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:1513:in `fu_each_src_dest' /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:508:in `mv' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate-util.rb:203:in `block (2 levels) in move_directory_between_carts' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate-util.rb:200:in `each' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate-util.rb:200:in `block in move_directory_between_carts' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate-util.rb:190:in `each' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate-util.rb:190:in `move_directory_between_carts' /opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate-v2-python-2.6.rb:21:in `post_process'
Did not meet this issue when do migration from devenv-stage_353 to devenv_3262 with multi-node. There are about 9 python2.6 app, and 2 django quickstarts. Log: Migrating app 'django' gear 'django' with uuid '519daee05033da6c2e000001' on node 'ip-10-147-132-203' for user: bmeng Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'bmeng' --migrate-gear '519daee05033da6c2e000001' --app-name 'django' --version '2.0.28' Migrating on node... ***time_migrate_on_node_measured_from_broker=39160*** Migrate on node output: Beginning V1 -> V2 migration Inspecting gear at /var/lib/openshift/519daee05033da6c2e000001 Marking step inspect_gear_state complete V1 stop lock not detected Marking step migrate_stop_lock complete Stopping gear with uuid '519daee05033da6c2e000001' on node 'ip-10-147-132-203' Marking step stop_gear complete Marking step pam_nproc_soft complete Marking step typeless_translated_vars complete Marking step gear_env_cleanup complete Migrated TMP v1 value [export TMP='/tmp/'] to raw value [/tmp/] Migrated HOME v1 value [export HOME='/var/lib/openshift/519daee05033da6c2e000001/'] to raw value [/var/lib/openshift/519daee05033da6c2e000001/] Migrated OPENSHIFT_REPO_DIR v1 value [export OPENSHIFT_REPO_DIR='/var/lib/openshift/519daee05033da6c2e000001/app-root/runtime/repo/'] to raw value [/var/lib/openshift/519daee05033da6c2e000001/app-root/runtime/repo/] Migrated OPENSHIFT_INTERNAL_PORT v1 value [export OPENSHIFT_INTERNAL_PORT='8080'] to raw value [8080] Migrated JENKINS_URL v1 value [export JENKINS_URL='https://jenkins-bmengdev.dev.rhcloud.com/'] to raw value [https://jenkins-bmengdev.dev.rhcloud.com/] Migrated OPENSHIFT_APP_DNS v1 value [export OPENSHIFT_APP_DNS='django-bmengdev.dev.rhcloud.com'] to raw value [django-bmengdev.dev.rhcloud.com] Migrated OPENSHIFT_GEAR_DNS v1 value [export OPENSHIFT_GEAR_DNS='django-bmengdev.dev.rhcloud.com'] to raw value [django-bmengdev.dev.rhcloud.com] Migrated OPENSHIFT_HOMEDIR v1 value [export OPENSHIFT_HOMEDIR='/var/lib/openshift/519daee05033da6c2e000001/'] to raw value [/var/lib/openshift/519daee05033da6c2e000001/] Migrated TMPDIR v1 value [export TMPDIR='/tmp/'] to raw value [/tmp/] Migrated OPENSHIFT_APP_UUID v1 value [export OPENSHIFT_APP_UUID='519daee05033da6c2e000001'] to raw value [519daee05033da6c2e000001] Migrated OPENSHIFT_PYTHON_PORT v1 value [export OPENSHIFT_PYTHON_PORT='8080'] to raw value [8080] Migrated JENKINS_USERNAME v1 value [export JENKINS_USERNAME='system_builder'] to raw value [system_builder] Migrated HISTFILE v1 value [export HISTFILE='/var/lib/openshift/519daee05033da6c2e000001/app-root/data/.bash_history'] to raw value [/var/lib/openshift/519daee05033da6c2e000001/app-root/data/.bash_history] Migrated TMP_DIR v1 value [export TMP_DIR='/tmp/'] to raw value [/tmp/] Migrated OPENSHIFT_PYTHON_LOG_DIR v1 value [export OPENSHIFT_PYTHON_LOG_DIR='/var/lib/openshift/519daee05033da6c2e000001/python-2.6/logs/'] to raw value [/var/lib/openshift/519daee05033da6c2e000001/python-2.6/logs/] Migrated JENKINS_PASSWORD v1 value [export JENKINS_PASSWORD='QR_TH2vHhIkP'] to raw value [QR_TH2vHhIkP] Migrated OPENSHIFT_TMP_DIR v1 value [export OPENSHIFT_TMP_DIR='/tmp/'] to raw value [/tmp/] Migrated OPENSHIFT_APP_NAME v1 value [export OPENSHIFT_APP_NAME='django'] to raw value [django] Migrated OPENSHIFT_INTERNAL_IP v1 value [export OPENSHIFT_INTERNAL_IP='127.10.67.1'] to raw value [127.10.67.1] Migrated OPENSHIFT_GEAR_UUID v1 value [export OPENSHIFT_GEAR_UUID='519daee05033da6c2e000001'] to raw value [519daee05033da6c2e000001] Migrated OPENSHIFT_DATA_DIR v1 value [export OPENSHIFT_DATA_DIR='/var/lib/openshift/519daee05033da6c2e000001/app-root/data/'] to raw value [/var/lib/openshift/519daee05033da6c2e000001/app-root/data/] Migrated OPENSHIFT_GEAR_NAME v1 value [export OPENSHIFT_GEAR_NAME='django'] to raw value [django] Migrated PATH v1 value [export PATH=/usr/libexec/openshift/cartridges/python-2.6/info/bin/:/usr/libexec/openshift/cartridges/abstract-httpd/info/bin/:/usr/libexec/openshift/cartridges/abstract/info/bin/:/bin:/usr/bin:/sbin:/usr/sbin] to raw value [/usr/libexec/openshift/cartridges/python-2.6/info/bin/:/usr/libexec/openshift/cartridges/abstract-httpd/info/bin/:/usr/libexec/openshift/cartridges/abstract/info/bin/:/bin:/usr/bin:/sbin:/usr/sbin] Migrated OPENSHIFT_PYTHON_IP v1 value [export OPENSHIFT_PYTHON_IP='127.10.67.1'] to raw value [127.10.67.1] Marking step env_vars_to_raw complete Marking step relocate_uservars complete Marking step reconfigure_git_repo complete Carts to migrate: ["python-2.6"] Migrating cartridge python Marking step python_create_directory complete Marking step python_setup complete Marking step python_erb complete applying python-2.6 migration post-process Moving contents of python-2.6/logs to python/logs Moving /var/lib/openshift/519daee05033da6c2e000001/python-2.6/logs/error_log-20130523-000000-EST Moving contents of python-2.6/virtenv to python/virtenv Moving /var/lib/openshift/519daee05033da6c2e000001/python-2.6/virtenv/lib64 Moving /var/lib/openshift/519daee05033da6c2e000001/python-2.6/virtenv/lib Moving /var/lib/openshift/519daee05033da6c2e000001/python-2.6/virtenv/include Moving /var/lib/openshift/519daee05033da6c2e000001/python-2.6/virtenv/bin Moving /var/lib/openshift/519daee05033da6c2e000001/python-2.6/virtenv/YourAppName.egg-info Marking step python_hook complete Marking step python_ownership complete Marking step python_connect_frontend complete 0 Starting gear with uuid '519daee05033da6c2e000001' on node 'ip-10-147-132-203' Marking step start_gear complete Validating gear 519daee05033da6c2e000001 post-migration Pre-migration state: started Post-migration response code: 200 Marking step validate_gear complete Cleaning up after migration ***time_migrate_on_node_measured_from_node=39036*** ***time_redeploy_httpd_proxy=0*** ***time_restart=0*** ***time_total_migrate_gear_measured_from_broker=39174***
It's possible that you may not see this when upgrading from one particular AMI to another AMI, but the risk is still there. The issue is that the migration utility method to move directories between cartridge directories, which we use to move directories in the V1 cartridge dir to the V2 cartridge dir, does not skip symlinks. Since we are using this inside a loop of each entry in a glob, and there are no ordering guarantees with a glob, it's possible that you may hit the symlink entry first. In INT during the initial V1 -> V2 migration, it seems that the actual dir (virtualenv/lib) was moved first, breaking the symlink (virtualenv/lib64) which the move then failed for.
PR submitted. The eventual fix for this was to shell out for the mv command. Absolute links to V1 cartridge dirs should remain solvent because we add a symlink from the V2 directory to the V1 directory name.
Commit pushed to master at https://github.com/openshift/li https://github.com/openshift/li/commit/0d472b7b31f8052d4153d48c7e6c8c9b60124661 Fix bug 966224
QE- This will be fixed for your Monday testing. P
Verified on INT, the python-2.6 apps and quickstarts are available now.