Created attachment 853763 [details] the thread-dump logs in development.log Description of problem: It is failed to thread-dump after quota exceeds 90%, will meet "An invalid exit code (nil) was returned from the server ip-10-180-231-119. This indicates an unexpected problem during the execution of your request." errors in development.log Version-Release number of selected component (if applicable): devenv_4257 How reproducible: always Steps to Reproduce: 1. Create an app which can be thread-dump like ruby-1.9 rhc app create crb19 ruby-1.9 --no-git 2. SSH into crb19 rhc ssh crb19 3. Consume 90+% space of the gear, e.g add a large file in app-root/data by "dd if=/dev/zero of=~/app-root/data/testfile bs=1M count=950" 4. Access the app webpage 5. Thread-dump this app rhc threaddump crb19 Actual results: return nothing Expected results: Thread-dump after quota exceeds 90% should be successfully. Additional info: 1. Please see the attachment for the detail logs. 2. Tried ruby-1.9 and jbossas-7, they are all failed.
When operating app like stop/start/restart after quota exceeds 90%, they are also failed, all met "A gear stop did not complete on 1 gear. Please try again and contact support if the issue persists." errors
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/06baec9aefd6f29316b7a2ca6484dad331440653 Bug 1056480 - Removed random character in code
It's fixed, verified on devenv_4261, please refer to the following results: 1. Create an app rhc app create crb19 ruby-1.9 --no-git 2. SSH into this app rhc ssh crb19 3. Consume 90+% space of the gear, e.g add a large file in app-root/data by "dd if=/dev/zero of=~/app-root/data/testfile bs=1M count=950" 4. Access the app webpage 5. Control this app 1) $ rhc threaddump crb19 Warning: Gear 52e07c61cd9bea0aa7000007 is using 92.9% of disk quota Success The thread dump file will be available via: rhc tail crb19 -g 52e07c61cd9bea0aa7000007 -f /var/lib/openshift/52e07c61cd9bea0aa7000007/ruby//logs//error_log-20140123-* -o '-n 250' 2) $ rhc app restart crb19 Warning: Gear 52e07c61cd9bea0aa7000007 is using 92.9% of disk quota RESULT: crb19 restarted 3) $ rhc cartridge add mysql-5.1 -a crb19 Adding mysql-5.1 to application 'crb19' ... Warning: Gear 52e07c61cd9bea0aa7000007 is using 93.0% of disk quota Warning: Gear 52e07c61cd9bea0aa7000007 is using 97.1% of disk quota done mysql-5.1 (MySQL 5.1) --------------------- Gears: Located with ruby-1.9 Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/ Database Name: crb19 Password: 7YCWP-uR27KI Username: adminiBUfasB MySQL 5.1 database added. Please make note of these credentials: Root User: adminiBUfasB Root Password: 7YCWP-uR27KI Database Name: crb19 Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/ You can manage your new MySQL database by also embedding phpmyadmin. The phpmyadmin username and password will be the same as the MySQL credentials above. 4) $ rhc cartridge restart mysql-5.1 -a crb19 Restarting mysql-5.1 ... Warning: Gear 52e07c61cd9bea0aa7000007 is using 97.1% of disk quota done