Created attachment 1523054 [details]
/var/lib/containers/storage/overlay-images/images.json

Seems like same with https://bugzilla.redhat.com/show_bug.cgi?id=1546324

Is there a consistent reproducer? If it can be reproduced only sometimes, as the report says, would it be possible to give me access to the registry when it does happen?

*** Bug 1671178 has been marked as a duplicate of this bug. ***

This is a constant pain for my development workflow today.  I hit this while developing containers that run on the master.

What I've been doing to work around this is:

# To ensure we can kill pods referencing the corrupted images
# and we won't race with kublet to schedule them back
$ kubectl taint nodes osiris-master-0 walters=foo:NoSchedule
$ oc delete pods/xxx

(ssh to node)
# podman images -q | xargs podman rmi

# And remove the taint, allowing the pod to get rescheduled
$ kubectl taint nodes osiris-master-0 walters:NoSchedule-

Checked with 
# cat /etc/os-release 
NAME="Red Hat CoreOS"
VERSION="4.0"
ID="rhcos"
ID_LIKE="rhel fedora"
VERSION_ID="4.0"
PRETTY_NAME="Red Hat CoreOS 4.0"
ANSI_COLOR="0;31"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat 7"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.0"
REDHAT_SUPPORT_PRODUCT="Red Hat"
REDHAT_SUPPORT_PRODUCT_VERSION="4.0"
OSTREE_VERSION=47.315

# rpm -qa|grep -i cri-o
cri-o-1.12.5-6.rhaos4.0.git80d1487.el7.x86_64

And the issue has been fixed now.


# cat /var/lib/containers/storage/overlay-images/images.json |python -m json.tool
<------------snip--------->
    {
        "big-data-digests": {
            "manifest": "sha256:d68f85b5ca3adccdc2f4a4c5263f1792798ed44a9b1d63a96004b6e283dc338d",
            "manifest-sha256:d68f85b5ca3adccdc2f4a4c5263f1792798ed44a9b1d63a96004b6e283dc338d": "sha256:d68f85b5ca3adccdc2f4a4c5263f1792798ed44a9b1d63a96004b6e283dc338d"
        },
        "big-data-names": [
            "manifest-sha256:d68f85b5ca3adccdc2f4a4c5263f1792798ed44a9b1d63a96004b6e283dc338d",
            "manifest"
        ],
        "big-data-sizes": {
            "manifest": 3862,
            "manifest-sha256:d68f85b5ca3adccdc2f4a4c5263f1792798ed44a9b1d63a96004b6e283dc338d": 3862
        },
        "created": "2018-11-02T18:24:31.956261005Z",
        "digest": "sha256:d68f85b5ca3adccdc2f4a4c5263f1792798ed44a9b1d63a96004b6e283dc338d",
        "id": "b02de22ff740f0bfa7e5dde5aa1a8169051375a5f0c69c28fafefc9408f72b06",
        "layer": "b6f3704e8e7510f3cd5d5d2b439ecc72815f85b02c608bf370d82636cccab6ca",
        "metadata": "{}",
        "names": [
            "quay.io/coreos/kube-client-agent:36c62ccd7b16b522450c61e96fc556b217ee24f5"
        ]
    },
<------------snip--------->

# crictl -D inspecti quay.io/coreos/kube-client-agent:36c62ccd7b16b522450c61e96fc556b217ee24f5
DEBU[0000] ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:quay.io/coreos/kube-client-agent:36c62ccd7b16b522450c61e96fc556b217ee24f5,},Verbose:true,} 
DEBU[0000] ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:b02de22ff740f0bfa7e5dde5aa1a8169051375a5f0c69c28fafefc9408f72b06,RepoTags:[quay.io/coreos/kube-client-agent:36c62ccd7b16b522450c61e96fc556b217ee24f5],RepoDigests:[quay.io/coreos/kube-client-agent@sha256:d68f85b5ca3adccdc2f4a4c5263f1792798ed44a9b1d63a96004b6e283dc338d],Size_:33806892,Uid:nil,Username:,},Info:map[string]string{},} 
{
  "status": {
    "id": "b02de22ff740f0bfa7e5dde5aa1a8169051375a5f0c69c28fafefc9408f72b06",
    "repoTags": [
      "quay.io/coreos/kube-client-agent:36c62ccd7b16b522450c61e96fc556b217ee24f5"
    ],
    "repoDigests": [
      "quay.io/coreos/kube-client-agent@sha256:d68f85b5ca3adccdc2f4a4c5263f1792798ed44a9b1d63a96004b6e283dc338d"
    ],
    "size": "33806892",
    "uid": null,
    "username": ""
  }
}

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758