Bug 970840 - Gear fails to migrate if low on disk space
Gear fails to migrate if low on disk space
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Dan McPherson
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-05 00:10 EDT by Wesley Hearn
Modified: 2015-05-14 19:20 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-11 00:17:04 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Wesley Hearn 2013-06-05 00:10:54 EDT
Description of problem:
rhc-admin-migrate will fail if the user has no disk space left. 

Version-Release number of selected component (if applicable):
Latest

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'login' --migrate-gear '5134d41de0b8cd997d0003a2' --app-name 'v2' --version '2.0.28'
Migrating on node...
***time_migrate_on_node_measured_from_broker=3311***
Migrate on node output:
 Beginning V1 -> V2 migration
Inspecting gear at /var/lib/openshift/5134d41de0b8cd997d0003a2
Checking for V1 stop lock in ["php-5.3"]
V1 stop lock not detected
Marking step migrate_stop_lock complete
Stopping gear with uuid '5134d41de0b8cd997d0003a2' on node 'ex-std-node125'
Carts to migrate: {:framework_carts=>["php-5.3"], :leftover_carts=>["haproxy-1.4"]}
Migrating cartridge php
Shell command '/sbin/runuser -m -s /bin/sh 5134d41de0b8cd997d0003a2 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c1,c849' /bin/sh -c \"/var/lib/openshift/5134d41de0b8cd997d0003a2/ph
p/bin/setup --version 5.3\""' returned an error. rc=1 stdout => 
  stderr => 
 cp: cannot create directory `/var/lib/openshift/5134d41de0b8cd997d0003a2/php//configuration/etc/conf': Disk quota exceeded
cp: cannot create directory `/var/lib/openshift/5134d41de0b8cd997d0003a2/php//configuration/etc/conf.d': Disk quota exceeded
cp: writing `/var/lib/openshift/5134d41de0b8cd997d0003a2/php//configuration/etc/php.ini.erb': Disk quota exceeded

Migrate on node exit code: 1
        from /usr/bin/rhc-admin-migrate:38:in `migrate_gear'
        from /usr/bin/rhc-admin-migrate:713:in `<main>'


Expected results:
It to migrate the gear

Additional info:
Comment 1 Dan McPherson 2013-06-05 13:38:26 EDT
https://github.com/openshift/li/pull/1545
Comment 2 Meng Bo 2013-06-06 04:49:52 EDT
Checked on devenv-stage_360, issue fixed.

1. Prepare app with quota full filled on devenv-stage_353.
2. Do upgrade and migration.
3. App migration successfully.

Migrating app 'quota' gear 'quota' with uuid '689acd84ce6b11e28c1222000a8d945a' on node 'ip-10-141-148-90' for user: bmeng
Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'bmeng' --migrate-gear '689acd84ce6b11e28c1222000a8d945a' --app-name 'quota' --version '2.0.28'
Migrating on node...
***time_migrate_on_node_measured_from_broker=29946***
Migrate on node output:
 Beginning V1 -> V2 migration
Inspecting gear at /var/lib/openshift/689acd84ce6b11e28c1222000a8d945a
Pre-migration state: started
Marking step inspect_gear_state complete
Marking step detect_switchyard complete
Beginning quota blocks: 1048576  inodes: 40000
Checking for V1 stop lock in ["php-5.3"]
V1 stop lock not detected
Marking step migrate_stop_lock complete
Stopping gear with uuid '689acd84ce6b11e28c1222000a8d945a' on node 'ip-10-141-148-90'
Marking step stop_gear complete
Marking step pam_nproc_soft complete
Marking step gear_env_cleanup complete
Migrated TMP v1 value [export TMP='/tmp/'] to raw value [/tmp/]
Migrated HOME v1 value [export HOME='/var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/'] to raw value [/var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/]
Migrated OPENSHIFT_REPO_DIR v1 value [export OPENSHIFT_REPO_DIR='/var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/app-root/runtime/repo/'] to raw value [/var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/app-root/runtime/repo/]
Migrated OPENSHIFT_INTERNAL_PORT v1 value [export OPENSHIFT_INTERNAL_PORT='8080'] to raw value [8080]
Migrated OPENSHIFT_APP_DNS v1 value [export OPENSHIFT_APP_DNS='quota-bmeng.dev.rhcloud.com'] to raw value [quota-bmeng.dev.rhcloud.com]
Migrated OPENSHIFT_GEAR_DNS v1 value [export OPENSHIFT_GEAR_DNS='quota-bmeng.dev.rhcloud.com'] to raw value [quota-bmeng.dev.rhcloud.com]
Migrated OPENSHIFT_HOMEDIR v1 value [export OPENSHIFT_HOMEDIR='/var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/'] to raw value [/var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/]
Migrated TMPDIR v1 value [export TMPDIR='/tmp/'] to raw value [/tmp/]
Migrated OPENSHIFT_APP_UUID v1 value [export OPENSHIFT_APP_UUID='689acd84ce6b11e28c1222000a8d945a'] to raw value [689acd84ce6b11e28c1222000a8d945a]
Migrated OPENSHIFT_PHP_IP v1 value [export OPENSHIFT_PHP_IP='127.0.253.129'] to raw value [127.0.253.129]
Migrated HISTFILE v1 value [export HISTFILE='/var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/app-root/data/.bash_history'] to raw value [/var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/app-root/data/.bash_history]
Migrated TMP_DIR v1 value [export TMP_DIR='/tmp/'] to raw value [/tmp/]
Migrated OPENSHIFT_TMP_DIR v1 value [export OPENSHIFT_TMP_DIR='/tmp/'] to raw value [/tmp/]
Migrated OPENSHIFT_APP_NAME v1 value [export OPENSHIFT_APP_NAME='quota'] to raw value [quota]
Migrated OPENSHIFT_PHP_PORT v1 value [export OPENSHIFT_PHP_PORT='8080'] to raw value [8080]
Migrated OPENSHIFT_INTERNAL_IP v1 value [export OPENSHIFT_INTERNAL_IP='127.0.253.129'] to raw value [127.0.253.129]
Migrated OPENSHIFT_PHP_LOG_DIR v1 value [export OPENSHIFT_PHP_LOG_DIR='/var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/php-5.3/logs/'] to raw value [/var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/php-5.3/logs/]
Migrated OPENSHIFT_GEAR_UUID v1 value [export OPENSHIFT_GEAR_UUID='689acd84ce6b11e28c1222000a8d945a'] to raw value [689acd84ce6b11e28c1222000a8d945a]
Migrated OPENSHIFT_DATA_DIR v1 value [export OPENSHIFT_DATA_DIR='/var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/app-root/data/'] to raw value [/var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/app-root/data/]
Migrated OPENSHIFT_GEAR_NAME v1 value [export OPENSHIFT_GEAR_NAME='quota'] to raw value [quota]
Migrated PATH v1 value [export PATH=/usr/libexec/openshift/cartridges/php-5.3/info/bin/:/usr/libexec/openshift/cartridges/abstract-httpd/info/bin/:/usr/libexec/openshift/cartridges/abstract/info/bin/:/bin:/usr/bin:/sbin:/usr/sbin] to raw value [/usr/libexec/openshift/cartridges/php-5.3/info/bin/:/usr/libexec/openshift/cartridges/abstract-httpd/info/bin/:/usr/libexec/openshift/cartridges/abstract/info/bin/:/bin:/usr/bin:/sbin:/usr/sbin]
Marking step env_vars_to_raw complete
Migrating TYPELESS_TRANSLATED_VARS to discrete variables
Marking step typeless_translated_vars complete
Marking step relocate_uservars complete
Marking step reconfigure_git_repo complete
Carts to migrate: {:framework_carts=>["php-5.3"]}
Migrating cartridge php
Marking step php_create_directory complete

Marking step php_setup complete
Marking step php_erb complete
applying php-5.3 migration post-process
Moving contents of php-5.3/logs to php/logs
 Moving /var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/php-5.3/logs/error_log-20130606-000000-EST
Moving contents of php-5.3/sessions to php/sessions
Moving contents of php-5.3/phplib to php/phplib
 Moving /var/lib/openshift/689acd84ce6b11e28c1222000a8d945a/php-5.3/phplib/pear

Marking step php_hook complete
Marking step php_ownership complete
Marking step php_connect_frontend complete
0
Starting gear with uuid '689acd84ce6b11e28c1222000a8d945a' on node 'ip-10-141-148-90'
Start gear output: CLIENT_MESSAGE: Starting Apache+mod_php HTTPD server

Marking step start_gear complete
Validating gear 689acd84ce6b11e28c1222000a8d945a post-migration
Pre-migration state: started
Post-migration response code: 200
Marking step validate_gear complete
Cleaning up after migration
***time_migrate_on_node_measured_from_node=29694***
Resetting quota blocks: 1048576  inodes: 40000


After migration, check disk quota, still full

[quota-bmeng.dev.rhcloud.com 689acd84ce6b11e28c1222000a8d945a]\> quota
Disk quotas for user 689acd84ce6b11e28c1222000a8d945a (uid 507): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
     /dev/xvde2 1049016*      0 1048576             193       0   40000

Note You need to log in before you can comment on or make changes to this bug.