Bug 1322587
| Summary: | Image pruning fails with a secured registry. | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Wesley Hearn <whearn> |
| Component: | Image Registry | Assignee: | Maciej Szulik <maszulik> |
| Status: | CLOSED ERRATA | QA Contact: | Wei Sun <wsun> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.2.0 | CC: | aos-bugs, mfojtik, pweil, tdawson, whearn, yinzhou |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: |
We've only added more logging around this bug to debug the root cause. Unfortunately since then we haven't seen this bug. This doesn't deserve any documentation, imho.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-09-27 09:37:14 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1303130 | ||
|
Description
Wesley Hearn
2016-03-30 19:52:26 UTC
Maciej, It appears that this can be caused by a 404 returning on the https call to delete the blob on the secure registry which then causes an http call to be issued which receives the malformed response error since the registry does not support http. (deleteFromRegistry function). I seemed to be able to reproduce this 2 ways. First was bouncing docker for some reason. I was not clear why that was producing 404s. The other way was to manually curl a DELETE to one of the SHAs like the pruner would do before running pruning. Here is what the logs look like: https://gist.github.com/pweil-/008198f231f0057b40294426915e69bc Update: it appears that the container was using an old config that did not have delete: enabled. This appears to have fixed the problem for online but since I was able to reproduce this sporadically in vagrant I think this still needs a second look This looks to be regression after fixing bug 1310062, apparently we might get 404 for certain layers, which should be just ignored. It might be that the layer got removed with some previous delete action, but I'm not sure about that. I'll confirm that with Michal Minar, since he's a registry expert and I'll prepare a fix for it. I've created https://github.com/openshift/origin/pull/8318 which logs the error and continues pruning. @Wesley what I'd like to get from you is the reproducer which will allow me to investigate more what is the root cause of this problem, why we're trying to delete those layers in the first place. I'll lower the prio of this bug once this PR merges, but I'll leave it open until the root cause is identified and nailed down. Commits pushed to master at https://github.com/openshift/origin https://github.com/openshift/origin/commit/16375903659e3b820233ffe16e1528a27a31b0a6 Bug 1322587 - NotFound error (404) when deleting layers is logged but we'll be continuing the execution. https://github.com/openshift/origin/commit/9b043534a0baa15b051811bc32f9fed5891601c7 Merge pull request #8318 from soltysh/bug1322587 Merged by openshift-bot Since the fix landed I'm lowering the priority, as discussed. But I'll keep the bug open until the root cause is nailed down and fixed. Any information about the bug Wesley? Wesley bump. I am not sure how to make a reproducer. I have checked a few logs for our clusters and I have not seen it erroring any more. Based on the last comment I'll move this bug to ON_QA, maybe they will be able to find something. If not it'll get closed. Confirmed with ose3.2.1.10, the issue has fixed: openshift version openshift v3.2.1.10-1-g668ed0a kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 oadm prune images --keep-younger-than=0 --keep-tag-revisions=1 Dry run enabled - no modifications will be made. Add --confirm to remove images Deleting registry layer blobs ... BLOB sha256:5a3e2c75e860737f0df84c92839626de8eebcaf510bdf3140cb3d033bd9befaf sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 sha256:0653bff3c5cf23727e0ebceae7a28f7534ab64ed13966e080e4c9b035176c401 sha256:39a454e5d9a158bcd4954e06d9f022823f10396e9fdd85a28a3fcc42d75a7ce7 sha256:4aff2ecfba9fc0ee02a77211fb0f54299839271120aa3066bffb511e26c27a70 sha256:33077750a5c8d59f9f7bed1abf9e156fc8975edc69f837d77aa4e68844a42567 sha256:05a0da00a5f4b0fa32b64da2a3ee17a112532d283f74aa2b8d5aff3e97eac9e4 sha256:bc16d5c17c382cc12d7c428de7766d1b49c8eb6b45f7986282cb2cdf39f1eae6 sha256:ab185f66a5e98f07be14592bc801b64ede8e23eb7edcf1c131e4a33a7a06995b sha256:d409edbefba76d54916d1e2244246d979242be6e5c1e798634617c7951561319 Deleting images from server ... IMAGE sha256:b9fbd3ee64ef0a893d3281fc1ddb211d31e6643ebf660d3aa81118f83b693c03 [root@ip-172-18-1-224 ~]# oadm prune images --keep-younger-than=0 --keep-tag-revisions=1 --confirm Deleting registry layer blobs ... BLOB sha256:bc16d5c17c382cc12d7c428de7766d1b49c8eb6b45f7986282cb2cdf39f1eae6 sha256:05a0da00a5f4b0fa32b64da2a3ee17a112532d283f74aa2b8d5aff3e97eac9e4 sha256:d409edbefba76d54916d1e2244246d979242be6e5c1e798634617c7951561319 sha256:0653bff3c5cf23727e0ebceae7a28f7534ab64ed13966e080e4c9b035176c401 sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 sha256:ab185f66a5e98f07be14592bc801b64ede8e23eb7edcf1c131e4a33a7a06995b sha256:39a454e5d9a158bcd4954e06d9f022823f10396e9fdd85a28a3fcc42d75a7ce7 sha256:5a3e2c75e860737f0df84c92839626de8eebcaf510bdf3140cb3d033bd9befaf sha256:33077750a5c8d59f9f7bed1abf9e156fc8975edc69f837d77aa4e68844a42567 sha256:4aff2ecfba9fc0ee02a77211fb0f54299839271120aa3066bffb511e26c27a70 Deleting images from server ... IMAGE sha256:b9fbd3ee64ef0a893d3281fc1ddb211d31e6643ebf660d3aa81118f83b693c03 [root@ip-172-18-1-224 ~]# oadm prune images --keep-younger-than=0 --keep-tag-revisions=1 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1933 |