Bug 1942608 - crictl cannot list the images with an error: error locating item named "manifest" for image with ID
Summary: crictl cannot list the images with an error: error locating item named "manif...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.5
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: 4.8.0
Assignee: Qi Wang
QA Contact: Weinan Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-24 15:54 UTC by Vinu K
Modified: 2023-03-15 18:27 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: When list the image, kubelet will try to get the satus of the image that hasn't done pulling. Consequence: Hit the error `error locating item named "manifest"`. Fix: Keep a list of imaegs that haven't finished pulling. When traverse throught the images, skip those images. Result: Should not get error when list images.
Clone Of:
Environment:
Last Closed: 2021-07-27 22:55:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:55:34 UTC

Description Vinu K 2021-03-24 15:54:35 UTC
Description of problem:
'crictl images -q' fails to list the images with an error: error locating item named "manifest" for image with ID

Version-Release number of selected component (if applicable):
v4u5

How reproducible:
cannot reproduce the error

Steps to Reproduce:
1.
2.
3.

Actual results:
$ crictl images -q
  FATA[0000] listing images: rpc error: code = Unknown desc = error locating item named "manifest" for image with ID "9e11cc7364c9791ebf68949c852bf121cb4ae5829c925a1da6546be6f63b3536": file does not exist

Expected results:
Should list the images

Additional info:
https://coreos.slack.com/archives/CBBJY9GSX/p1616599699018100

Comment 2 Vinu K 2021-03-29 17:07:38 UTC
Hello Team,

Any update on this?

Thanks,
Vinu K

Comment 3 Vinu K 2021-04-05 19:18:38 UTC
Hello Team,

Do we have any temporary workaround for this to stop the continuous error reporting?

Thanks,
Vinu K

Comment 5 Qi Wang 2021-04-07 20:39:22 UTC
crio pr https://github.com/cri-o/cri-o/pull/4726

Comment 6 Qi Wang 2021-04-09 19:56:12 UTC
Did not get completed this sprint.

Comment 7 Qi Wang 2021-04-21 02:21:37 UTC
Change to MODIFIED since https://github.com/cri-o/cri-o/pull/4726 merged to crio:master.

Comment 9 Vinu K 2021-04-26 18:21:29 UTC
Hello Team,

Thanks for the progress on this. Could you please confirm that if there is any workaround for the time being to stop the error alert or is it okay to ignore the error on the node with its missing manifest?

Thanks,
Vinu K

Comment 15 Vinu K 2021-05-17 16:06:53 UTC
Hello Team,

CU has upgraded the cluster to 4.6.26 and still, the problem persists in the same master node.

---
[core@master2 ~]$ sudo crictl images -q
FATA[0000] listing images: rpc error: code = Unknown desc = error locating item named "manifest" for image with ID "9e11cc7364c9791ebf68949c852bf121cb4ae5829c925a1da6546be6f63b3536": file does not exist
---

Should we bring the cluster to 4.8 to get rid of this error? The event log keeps reporting the error and CU expects a workaround.

Thanks,
Vinu K

Comment 16 Qi Wang 2021-05-17 22:54:20 UTC
Hi Vine K,

The fix is not included in 4.6.26. They can bring in the 4.8 clusters. And after re-pulling images, the data will be restored to the same place, as long as the ID of the image didn't change (i.e., a different image wasn't pushed to the same location that's specified using a tag). 

Thanks,
Qi

Comment 19 errata-xmlrpc 2021-07-27 22:55:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438

Comment 20 Vinu K 2023-03-09 13:42:51 UTC
Hello Team,

CU has observed the same issue even after upgrading to 4.10.

master2.ocpdev.stantec.com
failed to get image stats: rpc error: code = Unknown desc = error locating item named "manifest" for image with ID "9e11cc7364c9791ebf68949c852bf121cb4ae5829c925a1da6546be6f63b3536": file does not exist

Thanks,
Vinu K

Comment 21 Qi Wang 2023-03-13 18:25:25 UTC
Hi @vkochuku, thanks for the update. I will take a look later.

Comment 22 Vinu K 2023-03-15 15:22:33 UTC
Hello @qi wang, that you for your update. I have a question. Do we need to open a new Jira for this to track?

Comment 23 Qi Wang 2023-03-15 18:27:16 UTC
Yes. It'd be better to have a new Jira bug to track this. You can assign it to me when you create that bug. Thanks.


Note You need to log in before you can comment on or make changes to this bug.