Description of problem: When attempting to prune images to restore storage capacity, one container image provides the following error: ERRO[0000] error reading image "cc7639456e4a4c922cb086cf35f297427b3e8c38c14716533f67c66d70dd052a" as image: error locating item named "manifest" for image with ID "cc7639456e4a4c922cb086cf35f297427b3e8c38c14716533f67c66d70dd052a": file does not exist This prevents all subsequent images from being pruned. This occurs with the command # podman --log-level debug image rm cc7639456e4a --force but all subsequent attempts to remove this image have failed with similar results. Attempts to delete the container storage related to that container id have also failed. Investigation and attempts combined with the no manifest error seem to indicate the data is gone, but prune cannot progress past this point as something is indicating it exists Actual results: Image fails to prune and prevents storage reclaimation, resulting in eventual capacity Expected results: Unused images should prune Additional info: Cluster mustgather is in flopbox associated with case 02757892 and can be retrieved via supportshell
Where are you trying to delete the image from?
(In reply to Oleg Bulatov from comment #1) > Where are you trying to delete the image from? It looks like the last attempt from the cu was from a worker node: [root@worker-4 ~]# podman image prune -a Error:failed to prune image: error reading image "cc7639456e4a4c922cb086cf35f297427b3e8c38c14716533f67c66d70dd052a" as image: error locating item named "manifest" for image with ID "cc7639456e4a4c922cb086cf35f297427b3e8c38c14716533f67c66d70dd052a": file does not exist
I'm trying to understand what registry this is about. Do they want to get more space on worker-4's /var?
Brandon, Can you provied the version of Podman on the machine please and also the RHEL version? `podman --version`. Also please provide the output from `podman info` Thanks
(In reply to Tom Sweeney from comment #5) > Brandon, > > Can you provied the version of Podman on the machine please and also the > RHEL version? `podman --version`. Also please provide the output from > `podman info` > > Thanks osImage: Red Hat Enterprise Linux CoreOS 43.81.202007010053.0 (Ootpa) I can provide the full node description if needed.
cluster must-gather is attached to related case 02757892 and accessible via supportshell.
Brandon, thanks for the info, the main thing I was looking for was Podman 1.6.4 which I found in the attached support case. Matt can I have you take a look at this please? I seem to remeber you having fixed this or have a workaround for it handy.
No, I'm not familiar with this issue. Looks like a potential c/storage bug to me - inability to remove an incomplete image. Swapping assignee to Nalin.
Setting needinfo for Nalin to make sure he sees this.
It would be helpful if someone could test a podman built with this patch, to see if this fixes the issue. https://github.com/containers/podman/pull/7945
Checked on 4.7.0-0.nightly-2021-10-18-191324, podman version is 3.2.3-0.11 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2021-10-18-191324 True False 167m Cluster version is 4.7.0-0.nightly-2021-10-18-191324 sh-4.4# cat /etc/os-release NAME="Red Hat Enterprise Linux CoreOS" VERSION="47.84.202110151631-0" VERSION_ID="4.7" OPENSHIFT_VERSION="4.7" RHEL_VERSION="8.4" PRETTY_NAME="Red Hat Enterprise Linux CoreOS 47.84.202110151631-0 (Ootpa)" ID="rhcos" ID_LIKE="rhel fedora" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform" REDHAT_BUGZILLA_PRODUCT_VERSION="4.7" REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform" REDHAT_SUPPORT_PRODUCT_VERSION="4.7" OSTREE_VERSION='47.84.202110151631-0' sh-4.4# rpm -q podman podman-3.2.3-0.11.module+el8.4.0+12050+ef972f71.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.7.36 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3931