Hide Forgot
Description of problem: Getting the follow error deploying the mysql pod of the cakephp+mysql ephemeral example application template Feb 16 15:09:49 node2.lab.variantweb.net atomic-openshift-node[17606]: E0216 15:09:49.863495 17606 remote_image.go:83] ImageStatus "registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e" from image service failed: rpc error: code = Unknown desc = Manifest does not match provided manifest digest sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e Events from OCP: $ oc describe mysql-1-rgpkd the server doesn't have a resource type "mysql-1-rgpkd" [sjennings@cerebellum bin]$ oc describe pod mysql-1-rgpkd Name: mysql-1-rgpkd ... Containers: mysql: Container ID: Image: registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 33s default-scheduler Successfully assigned mysql-1-rgpkd to node2.lab.variantweb.net Normal SuccessfulMountVolume 33s kubelet, node2.lab.variantweb.net MountVolume.SetUp succeeded for volume "data" Normal SuccessfulMountVolume 33s kubelet, node2.lab.variantweb.net MountVolume.SetUp succeeded for volume "default-token-jj24n" Warning InspectFailed 3s (x5 over 32s) kubelet, node2.lab.variantweb.net Failed to inspect image "registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e": rpc error: code = Unknown desc = Manifest does not match provided manifest digest sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e Warning Failed 3s (x5 over 32s) kubelet, node2.lab.variantweb.net Error: ImageInspectError The image is pulled successfully on the node: # crictl images IMAGE TAG IMAGE ID SIZE docker.io/kubernetes/pause latest f9d5de0795395 251kB registry.access.redhat.com/rhscl/mysql-57-rhel7 <none> ae6638598a98d 433MB registry.reg-aws.openshift.com/openshift3/ose-deployer v3.9.0-0.45.0 95218503d9290 1.27GB Version-Release number of selected component (if applicable): cri-o 1.9.4 (in system conatiner) OCP v3.9.0-0.45.0 How reproducible: Always Steps to Reproduce: 1. install cluster with openshift-ansible with following options openshift_docker_additional_registries: registry.reg-aws.openshift.com openshift_use_crio: true openshift_crio_systemcontainer_image_override: registry.reg-aws.openshift.com/openshift3/cri-o:v3.9 oreg_url: 'registry.reg-aws.openshift.com/openshift3/ose-${component}:${version}' oreg_auth_user: sjenning oreg_auth_password: supersecret 2. start the cakephp+mysql sample application 3. mysql pod never deploys with above mentioned error Actual results: Above error Expected results: mysql pod starts without error Additional info:
Just to make sure, I tried to start the pod on a different node, same issue. Started mongodb pod, same issue. However, if I build the sample php application that pushes and pulls from the internal registry, that works. Also, if I start a pod with a image from docker.io, it also works. Seems to be an issue with registry.access.redhat.com.
Hmm, near as I can figure (by setting --log-level=debug for cri-o and trying it myself), the registry is serving up that image's manifest with an indicated MIME type of "text/plain" instead of a more typical manifest type like "application/vnd.docker.distribution.manifest.v1+prettyjws". Because "text/plain" is not one of the formats that the containers-storage transport claims to support in its list of acceptable manifest MIME types, the image's manifest is being converted, on the fly, to a "application/vnd.docker.distribution.manifest.v2+json" in order to be acceptable, and as a result it no longer matches the digest when any subsequent attempt is made to access the image and the digest check is performed.
container/images fix (or workaround?): https://github.com/containers/image/pull/417
New container/images PR: https://github.com/containers/image/pull/418 and the pick and version bump for cri-o: https://github.com/kubernetes-incubator/cri-o/pull/1351
Verify on openshift v3.9.0-0.51.0 with cri-o # openshift version openshift v3.9.0-0.51.0 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.8 # pwd /var/lib/containers/atomic/cri-o.0/rootfs/bin # ./crio --version crio version 1.9.7 Steps to verify: 1. Trigger cakephp+mysql ephemeral in console when env is ready 2. Check the mysql app is running. # oc get po -n dma1 NAME READY STATUS RESTARTS AGE cakephp-mysql-example-1-build 1/1 Running 0 6m mysql-1-bs9bn 1/1 Running 0 6m # oc get po -n dma1 NAME READY STATUS RESTARTS AGE cakephp-mysql-example-1-build 1/1 Running 0 6m mysql-1-bs9bn 1/1 Running 0 6m # oc describe pod mysql-1-bs9bn -n dma1 Name: mysql-1-bs9bn Namespace: dma1 Node: ip-172-18-14-130.ec2.internal/172.18.14.130 Start Time: Mon, 26 Feb 2018 03:57:13 -0500 Labels: deployment=mysql-1 deploymentconfig=mysql name=mysql Annotations: openshift.io/deployment-config.latest-version=1 openshift.io/deployment-config.name=mysqla openshift.io/deployment.name=mysql-1 openshift.io/scc=restricted Status: Running IP: 10.128.0.11 Controlled By: ReplicationController/mysql-1 Containers: mysql: Container ID: cri-o://c9842048503ee30b9eefd89512e6a9a697682c93271d57bca8954b977046a0de Image: registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e Image ID: registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e Port: 3306/TCP State: Running Started: Mon, 26 Feb 2018 03:58:20 -0500 Ready: True Restart Count: 0 Limits: memory: 512Mi Requests: memory: 512Mi Liveness: tcp-socket :3306 delay=30s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [/bin/sh -i -c MYSQL_PWD='YlOSByUhBxOYYK70' mysql -h 127.0.0.1 -u cakephp -D default -e 'SELECT 1'] delay=5s timeout=1s period=10s #success=1 #failure=3 Environment: MYSQL_USER: <set to the key 'database-user' in secret 'cakephp-mysql-example'> Optional: false MYSQL_PASSWORD: <set to the key 'database-password' in secret 'cakephp-mysql-example'> Optional: false MYSQL_DATABASE: default Mounts: /var/lib/mysql/data from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-kr7l4 (ro) Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: data: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: default-token-kr7l4: Type: Secret (a volume populated by a Secret) SecretName: default-token-kr7l4 Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 7m default-scheduler Successfully assigned mysql-1-bs9bn to ip-172-18-14-130.ec2.internal Normal SuccessfulMountVolume 7m kubelet, ip-172-18-14-130.ec2.internal MountVolume.SetUp succeeded for volume "data" Normal SuccessfulMountVolume 7m kubelet, ip-172-18-14-130.ec2.internal MountVolume.SetUp succeeded for volume "default-token-kr7l4" Normal Pulling 7m kubelet, ip-172-18-14-130.ec2.internal pulling image "registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e" Normal Pulled 6m kubelet, ip-172-18-14-130.ec2.internal Successfully pulled image "registry.access.redhat.com/rhscl/mysql-57-rhel7@sha256:7638d886370ca1eb3cefcfecf483f8dc4dc1ca05559dd521d1585ae7c4ed668e" Normal Created 6m kubelet, ip-172-18-14-130.ec2.internal Created container Normal Started 6m kubelet, ip-172-18-14-130.ec2.internal Started container Warning Unhealthy 5m kubelet, ip-172-18-14-130.ec2.internal Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device sh: no job control in this shell ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
Though I have verify the bug with latest cri-o, But still have one question: Now "./crio --version" only shows version is "1.9.7", how to make sure the cri-o in my system is cri-o-1.9.7-2.gita98f9c9.el7 not cri-o-1.9.7-1.gita98f9c9.el7 ?
Currently starter-ca-central-1 is deployed with docker nodes, so need wait for crio node to give a check on it for https://trello.com/c/9DGSzqg8/164-starter-ca-central-1-cri-o-issue-pulling-images-from-red-hat-registry.
(In reply to DeShuai Ma from comment #9) > Though I have verify the bug with latest cri-o, But still have one question: > Now "./crio --version" only shows version is "1.9.7", how to make sure the > cri-o in my system is cri-o-1.9.7-2.gita98f9c9.el7 not > cri-o-1.9.7-1.gita98f9c9.el7 ? can you try rpm -q cri0 ?
rpm -q cri-o, sorry
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2013