First Last Prev Next    This bug is not in your last search results.
Bug 966224 - Python 2.6 migrations for V1 -> V2 in int sometimes fail
Python 2.6 migrations for V1 -> V2 in int sometimes fail
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Paul Morie
libra bugs
:
Depends On:
Blocks:
  Show dependency tree / graph
 
Reported: 2013-05-22 15:56 EDT by Paul Morie
Modified: 2015-05-14 19:19 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-11 00:08:47 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

  None (edit)
Description Paul Morie 2013-05-22 15:56:48 EDT 
During the V1 -> V2 migration in INT, 5 python 2.6 apps failed to migrate due to the following issue:

Output:
Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'xtian+int1@redhat.com'
 --migrate-gear '5195e2906cec0e33b30000b1' --app-name 'django' --version '2.0.28'
Migrating on node...
***time_migrate_on_node_measured_from_broker=1126***
Migrate on node output:
 Beginning V1 -> V2 migration
Creating V2 stop_lock
Marking step migrate_stop_lock complete
Caught an exception during internal migration steps: No such file or directory - /var/l
ib/openshift/5195e2906cec0e33b30000b1/python-2.6/virtenv/lib64
/opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:1515:in `stat'
/opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:1515:in `block in fu_each_src_dest'
/opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:1531:in `fu_each_src_dest0'
/opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:1513:in `fu_each_src_dest'
/opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:508:in `mv'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate-util.rb:203:in `block (2 levels) in move_directory_between_carts'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate-util.rb:200:in `each'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate-util.rb:200:in `block in move_directory_between_carts'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate-util.rb:190:in `each'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate-util.rb:190:in `move_directory_between_carts'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/lib/migrate-v2-python-2.6.rb:21:in `post_process'
Comment 1 Meng Bo 2013-05-23 05:42:03 EDT 
Did not meet this issue when do migration from devenv-stage_353 to devenv_3262 with multi-node.
There are about 9 python2.6 app, and 2 django quickstarts.

Log:
Migrating app 'django' gear 'django' with uuid '519daee05033da6c2e000001' on node 'ip-10-147-132-203' for user: bmeng@redhat.com
Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'bmeng@redhat.com' --migrate-gear '519daee05033da6c2e000001' --app-name 'django' --version '2.0.28'
Migrating on node...
***time_migrate_on_node_measured_from_broker=39160***
Migrate on node output:
 Beginning V1 -> V2 migration
Inspecting gear at /var/lib/openshift/519daee05033da6c2e000001
Marking step inspect_gear_state complete
V1 stop lock not detected
Marking step migrate_stop_lock complete
Stopping gear with uuid '519daee05033da6c2e000001' on node 'ip-10-147-132-203'
Marking step stop_gear complete
Marking step pam_nproc_soft complete
Marking step typeless_translated_vars complete
Marking step gear_env_cleanup complete
Migrated TMP v1 value [export TMP='/tmp/'] to raw value [/tmp/]
Migrated HOME v1 value [export HOME='/var/lib/openshift/519daee05033da6c2e000001/'] to raw value [/var/lib/openshift/519daee05033da6c2e000001/]
Migrated OPENSHIFT_REPO_DIR v1 value [export OPENSHIFT_REPO_DIR='/var/lib/openshift/519daee05033da6c2e000001/app-root/runtime/repo/'] to raw value [/var/lib/openshift/519daee05033da6c2e000001/app-root/runtime/repo/]
Migrated OPENSHIFT_INTERNAL_PORT v1 value [export OPENSHIFT_INTERNAL_PORT='8080'] to raw value [8080]
Migrated JENKINS_URL v1 value [export JENKINS_URL='https://jenkins-bmengdev.dev.rhcloud.com/'] to raw value [https://jenkins-bmengdev.dev.rhcloud.com/]
Migrated OPENSHIFT_APP_DNS v1 value [export OPENSHIFT_APP_DNS='django-bmengdev.dev.rhcloud.com'] to raw value [django-bmengdev.dev.rhcloud.com]
Migrated OPENSHIFT_GEAR_DNS v1 value [export OPENSHIFT_GEAR_DNS='django-bmengdev.dev.rhcloud.com'] to raw value [django-bmengdev.dev.rhcloud.com]
Migrated OPENSHIFT_HOMEDIR v1 value [export OPENSHIFT_HOMEDIR='/var/lib/openshift/519daee05033da6c2e000001/'] to raw value [/var/lib/openshift/519daee05033da6c2e000001/]
Migrated TMPDIR v1 value [export TMPDIR='/tmp/'] to raw value [/tmp/]
Migrated OPENSHIFT_APP_UUID v1 value [export OPENSHIFT_APP_UUID='519daee05033da6c2e000001'] to raw value [519daee05033da6c2e000001]
Migrated OPENSHIFT_PYTHON_PORT v1 value [export OPENSHIFT_PYTHON_PORT='8080'] to raw value [8080]
Migrated JENKINS_USERNAME v1 value [export JENKINS_USERNAME='system_builder'] to raw value [system_builder]
Migrated HISTFILE v1 value [export HISTFILE='/var/lib/openshift/519daee05033da6c2e000001/app-root/data/.bash_history'] to raw value [/var/lib/openshift/519daee05033da6c2e000001/app-root/data/.bash_history]
Migrated TMP_DIR v1 value [export TMP_DIR='/tmp/'] to raw value [/tmp/]
Migrated OPENSHIFT_PYTHON_LOG_DIR v1 value [export OPENSHIFT_PYTHON_LOG_DIR='/var/lib/openshift/519daee05033da6c2e000001/python-2.6/logs/'] to raw value [/var/lib/openshift/519daee05033da6c2e000001/python-2.6/logs/]
Migrated JENKINS_PASSWORD v1 value [export JENKINS_PASSWORD='QR_TH2vHhIkP'] to raw value [QR_TH2vHhIkP]
Migrated OPENSHIFT_TMP_DIR v1 value [export OPENSHIFT_TMP_DIR='/tmp/'] to raw value [/tmp/]
Migrated OPENSHIFT_APP_NAME v1 value [export OPENSHIFT_APP_NAME='django'] to raw value [django]
Migrated OPENSHIFT_INTERNAL_IP v1 value [export OPENSHIFT_INTERNAL_IP='127.10.67.1'] to raw value [127.10.67.1]
Migrated OPENSHIFT_GEAR_UUID v1 value [export OPENSHIFT_GEAR_UUID='519daee05033da6c2e000001'] to raw value [519daee05033da6c2e000001]
Migrated OPENSHIFT_DATA_DIR v1 value [export OPENSHIFT_DATA_DIR='/var/lib/openshift/519daee05033da6c2e000001/app-root/data/'] to raw value [/var/lib/openshift/519daee05033da6c2e000001/app-root/data/]
Migrated OPENSHIFT_GEAR_NAME v1 value [export OPENSHIFT_GEAR_NAME='django'] to raw value [django]
Migrated PATH v1 value [export PATH=/usr/libexec/openshift/cartridges/python-2.6/info/bin/:/usr/libexec/openshift/cartridges/abstract-httpd/info/bin/:/usr/libexec/openshift/cartridges/abstract/info/bin/:/bin:/usr/bin:/sbin:/usr/sbin] to raw value [/usr/libexec/openshift/cartridges/python-2.6/info/bin/:/usr/libexec/openshift/cartridges/abstract-httpd/info/bin/:/usr/libexec/openshift/cartridges/abstract/info/bin/:/bin:/usr/bin:/sbin:/usr/sbin]
Migrated OPENSHIFT_PYTHON_IP v1 value [export OPENSHIFT_PYTHON_IP='127.10.67.1'] to raw value [127.10.67.1]
Marking step env_vars_to_raw complete
Marking step relocate_uservars complete
Marking step reconfigure_git_repo complete
Carts to migrate: ["python-2.6"]
Migrating cartridge python
Marking step python_create_directory complete

Marking step python_setup complete
Marking step python_erb complete
applying python-2.6 migration post-process
Moving contents of python-2.6/logs to python/logs
 Moving /var/lib/openshift/519daee05033da6c2e000001/python-2.6/logs/error_log-20130523-000000-EST
Moving contents of python-2.6/virtenv to python/virtenv
 Moving /var/lib/openshift/519daee05033da6c2e000001/python-2.6/virtenv/lib64
 Moving /var/lib/openshift/519daee05033da6c2e000001/python-2.6/virtenv/lib
 Moving /var/lib/openshift/519daee05033da6c2e000001/python-2.6/virtenv/include
 Moving /var/lib/openshift/519daee05033da6c2e000001/python-2.6/virtenv/bin
 Moving /var/lib/openshift/519daee05033da6c2e000001/python-2.6/virtenv/YourAppName.egg-info

Marking step python_hook complete
Marking step python_ownership complete
Marking step python_connect_frontend complete
0
Starting gear with uuid '519daee05033da6c2e000001' on node 'ip-10-147-132-203'
Marking step start_gear complete
Validating gear 519daee05033da6c2e000001 post-migration
Pre-migration state: started
Post-migration response code: 200
Marking step validate_gear complete
Cleaning up after migration
***time_migrate_on_node_measured_from_node=39036***
***time_redeploy_httpd_proxy=0***
***time_restart=0***
***time_total_migrate_gear_measured_from_broker=39174***
Comment 2 Paul Morie 2013-05-23 07:45:31 EDT 
It's possible that you may not see this when upgrading from one particular AMI to another AMI, but the risk is still there.  The issue is that the migration utility method to move directories between cartridge directories, which we use to move directories in the V1 cartridge dir to the V2 cartridge dir, does not skip symlinks.  Since we are using this inside a loop of each entry in a glob, and there are no ordering guarantees with a glob, it's possible that you may hit the symlink entry first.  In INT during the initial V1 -> V2 migration, it seems that the actual dir (virtualenv/lib) was moved first, breaking the symlink (virtualenv/lib64) which the move then failed for.
Comment 3 Paul Morie 2013-05-23 13:54:10 EDT 
PR submitted.

The eventual fix for this was to shell out for the mv command.  Absolute links to V1 cartridge dirs should remain solvent because we add a symlink from the V2 directory to the V1 directory name.
Comment 5 Paul Morie 2013-05-23 17:44:14 EDT 
QE-

This will be fixed for your Monday testing.

P
Comment 6 Jianwei Hou 2013-05-27 06:27:06 EDT 
Verified on INT, the python-2.6 apps and quickstarts are available now.
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.