Bug 1546324 - Manifest does not match provided manifest digest
Summary: Manifest does not match provided manifest digest
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.9.0
Assignee: Lokesh Mandvekar
QA Contact: DeShuai Ma
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-02-16 20:19 UTC by Seth Jennings
Modified: 2018-06-27 18:02 UTC (History)
6 users (show)

Fixed In Version: cri-o-1.9.7-2.gita98f9c9.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 18:01:32 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2013 None None None 2018-06-27 18:02:05 UTC

Description Seth Jennings 2018-02-16 20:19:08 UTC
Description of problem:

Getting the follow error deploying the mysql pod of the cakephp+mysql ephemeral example application template 

Feb 16 15:09:49 node2.lab.variantweb.net atomic-openshift-node[17606]: E0216 15:09:49.863495   17606 remote_image.go:83] ImageStatus "registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e" from image service failed: rpc error: code = Unknown desc = Manifest does not match provided manifest digest sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e

Events from OCP:

$ oc describe mysql-1-rgpkd
the server doesn't have a resource type "mysql-1-rgpkd"
[sjennings@cerebellum bin]$ oc describe pod mysql-1-rgpkd
Name:           mysql-1-rgpkd
...
Containers:
  mysql:
    Container ID:   
    Image:          registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e
...
Events:
  Type     Reason                 Age               From                               Message
  ----     ------                 ----              ----                               -------
  Normal   Scheduled              33s               default-scheduler                  Successfully assigned mysql-1-rgpkd to node2.lab.variantweb.net
  Normal   SuccessfulMountVolume  33s               kubelet, node2.lab.variantweb.net  MountVolume.SetUp succeeded for volume "data"
  Normal   SuccessfulMountVolume  33s               kubelet, node2.lab.variantweb.net  MountVolume.SetUp succeeded for volume "default-token-jj24n"
  Warning  InspectFailed          3s (x5 over 32s)  kubelet, node2.lab.variantweb.net  Failed to inspect image "registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e": rpc error: code = Unknown desc = Manifest does not match provided manifest digest sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e
  Warning  Failed                 3s (x5 over 32s)  kubelet, node2.lab.variantweb.net  Error: ImageInspectError

The image is pulled successfully on the node:

# crictl images
IMAGE                                                    TAG                 IMAGE ID            SIZE
docker.io/kubernetes/pause                               latest              f9d5de0795395       251kB
registry.access.redhat.com/rhscl/mysql-57-rhel7          <none>              ae6638598a98d       433MB
registry.reg-aws.openshift.com/openshift3/ose-deployer   v3.9.0-0.45.0       95218503d9290       1.27GB


Version-Release number of selected component (if applicable):
cri-o 1.9.4 (in system conatiner)
OCP v3.9.0-0.45.0

How reproducible:
Always

Steps to Reproduce:
1. install cluster with openshift-ansible with following options

openshift_docker_additional_registries: registry.reg-aws.openshift.com
openshift_use_crio: true
openshift_crio_systemcontainer_image_override: registry.reg-aws.openshift.com/openshift3/cri-o:v3.9
oreg_url: 'registry.reg-aws.openshift.com/openshift3/ose-${component}:${version}'
oreg_auth_user: sjenning@redhat.com
oreg_auth_password: supersecret

2. start the cakephp+mysql sample application
3. mysql pod never deploys with above mentioned error

Actual results:

Above error

Expected results:

mysql pod starts without error

Additional info:

Comment 1 Seth Jennings 2018-02-16 20:59:28 UTC
Just to make sure, I tried to start the pod on a different node, same issue.

Started mongodb pod, same issue.

However, if I build the sample php application that pushes and pulls from the internal registry, that works.

Also, if I start a pod with a image from docker.io, it also works.

Seems to be an issue with registry.access.redhat.com.

Comment 2 Nalin Dahyabhai 2018-02-16 21:36:21 UTC
Hmm, near as I can figure (by setting --log-level=debug for cri-o and trying it myself), the registry is serving up that image's manifest with an indicated MIME type of "text/plain" instead of a more typical manifest type like "application/vnd.docker.distribution.manifest.v1+prettyjws".

Because "text/plain" is not one of the formats that the containers-storage transport claims to support in its list of acceptable manifest MIME types, the image's manifest is being converted, on the fly, to a "application/vnd.docker.distribution.manifest.v2+json" in order to be acceptable, and as a result it no longer matches the digest when any subsequent attempt is made to access the image and the digest check is performed.

Comment 3 Seth Jennings 2018-02-19 04:00:15 UTC
container/images fix (or workaround?):
https://github.com/containers/image/pull/417

Comment 4 Seth Jennings 2018-02-19 19:45:51 UTC
New container/images PR:
https://github.com/containers/image/pull/418

and the pick and version bump for cri-o:
https://github.com/kubernetes-incubator/cri-o/pull/1351

Comment 8 DeShuai Ma 2018-02-26 09:09:12 UTC
Verify on openshift v3.9.0-0.51.0 with cri-o

# openshift version
openshift v3.9.0-0.51.0
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.8
# pwd
/var/lib/containers/atomic/cri-o.0/rootfs/bin
# ./crio --version
crio version 1.9.7

Steps to verify:
1. Trigger cakephp+mysql ephemeral in console when env is ready

2. Check the mysql app is running.
# oc get po -n dma1
NAME                            READY     STATUS    RESTARTS   AGE
cakephp-mysql-example-1-build   1/1       Running   0          6m
mysql-1-bs9bn                   1/1       Running   0          6m

# oc get po -n dma1
NAME                            READY     STATUS    RESTARTS   AGE
cakephp-mysql-example-1-build   1/1       Running   0          6m
mysql-1-bs9bn                   1/1       Running   0          6m
# oc describe pod mysql-1-bs9bn -n dma1
Name:           mysql-1-bs9bn
Namespace:      dma1
Node:           ip-172-18-14-130.ec2.internal/172.18.14.130
Start Time:     Mon, 26 Feb 2018 03:57:13 -0500
Labels:         deployment=mysql-1
                deploymentconfig=mysql
                name=mysql
Annotations:    openshift.io/deployment-config.latest-version=1
                openshift.io/deployment-config.name=mysqla
                openshift.io/deployment.name=mysql-1
                openshift.io/scc=restricted
Status:         Running
IP:             10.128.0.11
Controlled By:  ReplicationController/mysql-1
Containers:
  mysql:
    Container ID:   cri-o://c9842048503ee30b9eefd89512e6a9a697682c93271d57bca8954b977046a0de
    Image:          registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e
    Image ID:       registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e
    Port:           3306/TCP
    State:          Running
      Started:      Mon, 26 Feb 2018 03:58:20 -0500
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  512Mi
    Requests:
      memory:   512Mi
    Liveness:   tcp-socket :3306 delay=30s timeout=1s period=10s #success=1 #failure=3
    Readiness:  exec [/bin/sh -i -c MYSQL_PWD='YlOSByUhBxOYYK70' mysql -h 127.0.0.1 -u cakephp -D default -e 'SELECT 1'] delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      MYSQL_USER:      <set to the key 'database-user' in secret 'cakephp-mysql-example'>      Optional: false
      MYSQL_PASSWORD:  <set to the key 'database-password' in secret 'cakephp-mysql-example'>  Optional: false
      MYSQL_DATABASE:  default
    Mounts:
      /var/lib/mysql/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kr7l4 (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  data:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  default-token-kr7l4:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kr7l4
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
Events:
  Type     Reason                 Age   From                                    Message
  ----     ------                 ----  ----                                    -------
  Normal   Scheduled              7m    default-scheduler                       Successfully assigned mysql-1-bs9bn to ip-172-18-14-130.ec2.internal
  Normal   SuccessfulMountVolume  7m    kubelet, ip-172-18-14-130.ec2.internal  MountVolume.SetUp succeeded for volume "data"
  Normal   SuccessfulMountVolume  7m    kubelet, ip-172-18-14-130.ec2.internal  MountVolume.SetUp succeeded for volume "default-token-kr7l4"
  Normal   Pulling                7m    kubelet, ip-172-18-14-130.ec2.internal  pulling image "registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e"
  Normal   Pulled                 6m    kubelet, ip-172-18-14-130.ec2.internal  Successfully pulled image "registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e"
  Normal   Created                6m    kubelet, ip-172-18-14-130.ec2.internal  Created container
  Normal   Started                6m    kubelet, ip-172-18-14-130.ec2.internal  Started container
  Warning  Unhealthy              5m    kubelet, ip-172-18-14-130.ec2.internal  Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)

Comment 9 DeShuai Ma 2018-02-26 09:21:23 UTC
Though I have verify the bug with latest cri-o, But still have one question:
Now "./crio --version" only shows version is "1.9.7", how to make sure the cri-o in my system is cri-o-1.9.7-2.gita98f9c9.el7 not cri-o-1.9.7-1.gita98f9c9.el7 ?

Comment 10 weiwei jiang 2018-02-28 08:18:17 UTC
Currently starter-ca-central-1 is deployed with docker nodes, so need wait for crio node to give a check  on it for https://trello.com/c/9DGSzqg8/164-starter-ca-central-1-cri-o-issue-pulling-images-from-red-hat-registry.

Comment 11 Lokesh Mandvekar 2018-05-04 13:58:24 UTC
(In reply to DeShuai Ma from comment #9)
> Though I have verify the bug with latest cri-o, But still have one question:
> Now "./crio --version" only shows version is "1.9.7", how to make sure the
> cri-o in my system is cri-o-1.9.7-2.gita98f9c9.el7 not
> cri-o-1.9.7-1.gita98f9c9.el7 ?

can you try rpm -q cri0 ?

Comment 12 Lokesh Mandvekar 2018-05-04 13:58:47 UTC
rpm -q cri-o, sorry

Comment 14 errata-xmlrpc 2018-06-27 18:01:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2013


Note You need to log in before you can comment on or make changes to this bug.