Bug 1418359
| Summary: | [3.4] Image data stored in ETCD accounting for large DB. | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Michal Minar <miminar> | ||||
| Component: | Image Registry | Assignee: | Michal Minar <miminar> | ||||
| Status: | CLOSED ERRATA | QA Contact: | ge liu <geliu> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 3.4.1 | CC: | adellape, aos-bugs, bleanhar, byount, erich, geliu, haowang, jeder, jmorales, jokerman, mfojtik, miminar, mmccomas, pep, pweil, rhowe, sdodson, tdawson, tstclair, wsun | ||||
| Target Milestone: | --- | Keywords: | Performance | ||||
| Target Release: | 3.4.z | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: |
Cause: OpenShift cluster used to store manifests of all images in etcd database.
Consequence: The manifests occupied a lot of space in the database making it big and slow.
Fix: The integrated registry now stores manifests in its associated storage rather than in etcd. Also manifests of remote images are not stored at all. They are fetch from external registries when needed.
A migration script has been provided to move manifests from all existing images in the cluster into the integrated registry.
Result: Newly pushed images will not cause etcd database to grow so fast. By using the migration script, the admin is able to reduce etcd size considerably.
|
Story Points: | --- | ||||
| Clone Of: | 1378180 | Environment: | |||||
| Last Closed: | 2017-03-15 20:02:17 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1378180 | ||||||
| Bug Blocks: | 1267746, 1418358 | ||||||
| Attachments: |
|
||||||
|
Comment 2
ge liu
2017-02-04 05:19:45 UTC
PR https://github.com/openshift/ose/pull/582 [merge]d for 3.4. Migration script PR: https://github.com/openshift/ose/pull/607 @miminar, I need more info about the 3.4 fork ami or detailed version info that(lacking PR#582), and also the fork ami(having PR#582), thanks in advance. The fork_ami with 3.4 branch without the migration patch (#582) will be hopefully ready at: https://ci.openshift.redhat.com/jenkins/job/fork_ami/326/ The 3.4 fork_ami with #582 is at: https://ci.openshift.redhat.com/jenkins/job/fork_ami/315/parameters/ Slight correction. The fork_ami with 3.4 without the migration patch is at https://ci.openshift.redhat.com/jenkins/job/fork_ami/328/ The fork_ami referenced in comment 8 failed unfortunately. @geliu, please let me know if I can help in any way. @miminar, I done step 1-5 in the suggested scrnario, but regarding to "step 6: Update to the latest 3.4 cluster (having PR#582) - don't forget to update the registry image as well." my test execution plan is: start openshift cluster with form-aim(having PR#582), then copy openshift binary to replace the openshift cluster(lacking PR#582), then restart openshift service. the openshift binary is: # which openshift /data/src/github.com/openshift/origin/_output/local/bin/linux/amd64/openshift/* do you think it's ok for the update operation? and currently issue is： there is not fork_ami with: (The 3.4 fork_ami with #582 is at: https://ci.openshift.redhat.com/jenkins/job/fork_ami/315/parameters/), could u help to build a form_ami for me? thanks! Suggested scrnarios: Step 1-5: 1. swith to project/user: lgproj2/geliu, and do some builds: #oc process -f https://raw.githubusercontent.com/openshift/origin/master/examples/sample-app/application-template-stibuild.json |oc create -f - 2.swith to project/user: proj-lg/lg, and import images #oc import-image --confirm --from=centos/ruby-22-centos7 ruby-22-centos7:latest #oc new-build --image-stream=ruby-22-centos7 https://github.com/openshift/ruby-ex.git 3. check the images: dockerImageManifest # oc get image -o go-template=$'{{if.dockerImageManifest}}present{{else}}removed{{end}}\n' sha256:8426f95b35ff6484360565155b470903bcffc82f0d87d72a0a528bcea9d8e0b8 present # oc get image -o go-template=$'{{if.dockerImageManifest}}present{{else}}removed{{end}}\n' sha256:06f0fcff7cfbbf7aacd81d84efb0ce4d3c84be37bebbd35513f8f1fadab4b9a2 present it should works as expected in 3.4 cluster (lacking PR#582), and the next step is update to cluster (having PR#582) >and currently issue is： there is not fork_ami with: (The 3.4 fork_ami with #582 is at: https://ci.openshift.redhat.com/jenkins/job/fork_ami/315/parameters/), could u help to build a form_ami for me? thanks! @geliu I don't quite understand. You should be able to pull the fork_ami build from https://ci.openshift.redhat.com/jenkins/job/fork_ami/315/ the same way as from https://ci.openshift.redhat.com/jenkins/job/fork_ami/328/. How do you check whether there's a build or not? > do you think it's ok for the update operation? The registry needs to be re-deployed as well. Otherwise it looks good to me. As we talked from IRC, Michal will prepare two ami, and we will test like this: 1. Create all-in-one instance using the ami that don't have bug fix 2. prepare data 3. launch instance using the new ami with bug fix, and copy the openshift binary to old instance, also the registry image.(we can push the image to a external registry, then pull it in the old instance) 4. start the cluster in old instance using new binary, modify the dc/docker-registry to using the new registry image ..... Ok, I've just started a new fork_ami build at: https://ci.openshift.redhat.com/jenkins/job/fork_ami/331/ Let's see if it passes. For some reason, the build from comment 14 has been deleted. I've started a new one at https://ci.openshift.redhat.com/jenkins/job/fork_ami/335/. It has the #582 fix applied. The second fork_ami with the fix not applied is at https://ci.openshift.redhat.com/jenkins/job/fork_ami/328/. Created attachment 1256863 [details]
Verified test results
Verified and test steps in attached file.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0512 |