Bug 1281733
| Summary: | Not all pods get started when deploying 50 mysql concurrently | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | DeShuai Ma <dma> |
| Component: | Documentation | Assignee: | Thien-Thi Nguyen <tnguyen> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Vikram Goyal <vigoyal> |
| Severity: | low | Docs Contact: | Vikram Goyal <vigoyal> |
| Priority: | medium | ||
| Version: | 3.1.0 | CC: | amelicha, aos-bugs, bparees, dma, hhorak, jeder, jokerman, mmccomas, mnagy, tnguyen, xtian |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-05-11 13:39:47 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
DeShuai Ma
2015-11-13 10:41:59 UTC
Is this 50 independent pods, or is this a master/slave scenario? Since you say 50 projects I assume it's 50 independent ones. which volume type are you using for the pods? DeShuai, there appear to be two possible solutions, can you please try both of them, independently? 1) Increase fs.aio-max-nr kernel limit on the nodes. See the answer here: https://unix.stackexchange.com/questions/116520/mysql-server-wont-install-to-a-new-os-debian-ubuntu 2) Set the MYSQL_AIO environment variable to 0 to all MySQL pods that you create. This should disable native AIO and the limit should not be hit. (In reply to Ben Parees from comment #1) > Is this 50 independent pods, or is this a master/slave scenario? Since you > say 50 projects I assume it's 50 independent ones. > > which volume type are you using for the pods? 50 pods are independent. all pod is created using file:https://github.com/openshift/origin/blob/master/examples/db-templates/mysql-ephemeral-template.json (In reply to Martin Nagy from comment #2) > DeShuai, there appear to be two possible solutions, can you please try both > of them, independently? > > 1) Increase fs.aio-max-nr kernel limit on the nodes. See the answer here: > https://unix.stackexchange.com/questions/116520/mysql-server-wont-install-to- > a-new-os-debian-ubuntu > > 2) Set the MYSQL_AIO environment variable to 0 to all MySQL pods that you > create. This should disable native AIO and the limit should not be hit. After try both two method, pod can be running well.thanks. Now, what to do with this ... Doubt this is something we'd want to do globally on the kernel side for systems running openshift. My thought is that we should convert this to a documentation bugzilla and put a note in the openshift docs. I agree. Should we move this bug to the Documentation component? Martin, please document this in the openshift mysql image docs. Thanks. Martin, did you want the docs team to add this note? If yes, I will assign it to one of the writers. (In reply to Vikram Goyal from comment #8) > Martin, did you want the docs team to add this note? > If yes, I will assign it to one of the writers. Martin and i discussed this today. The plan is to add to the MySQL images section an explanation behind why MySQL fails w/ the above-mentioned error messages, as well as documentation on the two resolution paths: - increase the kernel resource limit on the node - set the env var The latter will mention that editing /etc/my.cfg directly is not the OpenShift way, and xref the "Managing Environment Variables" section of the Dev Guide. Martin, additions/corrections welcome. Assigning this to myself. Commented in the pull, looks great! Hi DeShuai, WDYT? Hi DeShuai, WDYT? The doc pr LGTM. When pr is merged will verify this bug. Hi DeShuai, PR merged: https://github.com/openshift/openshift-docs/pull/1701#event-593982568 WDYT? Thanks DeShuai. Moving status to RELEASE_PENDING. This is now live: https://access.redhat.com/documentation/en/openshift-enterprise/version-3.1/using-images/#troubleshooting Closing. |