Bug 1056480
| Summary: | Failed to control app after quota exceeds 90% | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Online | Reporter: | chunchen <chunchen> | ||||
| Component: | Containers | Assignee: | Jhon Honce <jhonce> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | libra bugs <libra-bugs> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 2.x | CC: | wsun | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-02-26 19:09:59 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
When operating app like stop/start/restart after quota exceeds 90%, they are also failed, all met "A gear stop did not complete on 1 gear. Please try again and contact support if the issue persists." errors Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/06baec9aefd6f29316b7a2ca6484dad331440653 Bug 1056480 - Removed random character in code It's fixed, verified on devenv_4261, please refer to the following results:
1. Create an app
rhc app create crb19 ruby-1.9 --no-git
2. SSH into this app
rhc ssh crb19
3. Consume 90+% space of the gear, e.g add a large file in app-root/data by "dd if=/dev/zero of=~/app-root/data/testfile bs=1M count=950"
4. Access the app webpage
5. Control this app
1) $ rhc threaddump crb19
Warning: Gear 52e07c61cd9bea0aa7000007 is using 92.9% of disk quota
Success
The thread dump file will be available via: rhc tail crb19 -g 52e07c61cd9bea0aa7000007 -f /var/lib/openshift/52e07c61cd9bea0aa7000007/ruby//logs//error_log-20140123-* -o '-n 250'
2) $ rhc app restart crb19
Warning: Gear 52e07c61cd9bea0aa7000007 is using 92.9% of disk quota
RESULT:
crb19 restarted
3) $ rhc cartridge add mysql-5.1 -a crb19
Adding mysql-5.1 to application 'crb19' ...
Warning: Gear 52e07c61cd9bea0aa7000007 is using 93.0% of disk quota
Warning: Gear 52e07c61cd9bea0aa7000007 is using 97.1% of disk quota
done
mysql-5.1 (MySQL 5.1)
---------------------
Gears: Located with ruby-1.9
Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/
Database Name: crb19
Password: 7YCWP-uR27KI
Username: adminiBUfasB
MySQL 5.1 database added. Please make note of these credentials:
Root User: adminiBUfasB
Root Password: 7YCWP-uR27KI
Database Name: crb19
Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/
You can manage your new MySQL database by also embedding phpmyadmin.
The phpmyadmin username and password will be the same as the MySQL credentials above.
4) $ rhc cartridge restart mysql-5.1 -a crb19
Restarting mysql-5.1 ... Warning: Gear 52e07c61cd9bea0aa7000007 is using 97.1% of disk quota
done
|
Created attachment 853763 [details] the thread-dump logs in development.log Description of problem: It is failed to thread-dump after quota exceeds 90%, will meet "An invalid exit code (nil) was returned from the server ip-10-180-231-119. This indicates an unexpected problem during the execution of your request." errors in development.log Version-Release number of selected component (if applicable): devenv_4257 How reproducible: always Steps to Reproduce: 1. Create an app which can be thread-dump like ruby-1.9 rhc app create crb19 ruby-1.9 --no-git 2. SSH into crb19 rhc ssh crb19 3. Consume 90+% space of the gear, e.g add a large file in app-root/data by "dd if=/dev/zero of=~/app-root/data/testfile bs=1M count=950" 4. Access the app webpage 5. Thread-dump this app rhc threaddump crb19 Actual results: return nothing Expected results: Thread-dump after quota exceeds 90% should be successfully. Additional info: 1. Please see the attachment for the detail logs. 2. Tried ruby-1.9 and jbossas-7, they are all failed.