RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1322762 - sha256 Conflict while pull images after upgrade
Summary: sha256 Conflict while pull images after upgrade
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.2
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: rc
: ---
Assignee: Antonio Murdaca
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-03-31 09:39 UTC by Anping Li
Modified: 2019-03-06 02:36 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-05-12 15:17:36 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
reproduce-procedure (6.71 KB, text/plain)
2016-04-05 17:11 UTC, Seth Jennings 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:1034 0 normal SHIPPED_LIVE Moderate: docker security, bug fix, and enhancement update 2016-05-12 19:15:01 UTC

Description Anping Li 2016-03-31 09:39:49 UTC 
Description of problem:

The pod cakephp-mysql-example report sha256 Conflict error after upgrade. only cakephp-mysql-example hit this issue, the other pod works well, for example cakephp-example. 

Pull this images by docker directly get same error. after delete the Conflict images , it can be pulled.

No sure if it is for upgrade. please feel free to change the component type.


Version-Release number of selected component (if applicable):
openshift3/ose-deployer:v3.2.0.8
atomic-openshift-utils-3.0.68

How reproducible:
Always in my environment.

Steps to Reproduce:
1. setup containerlized env and customized the network_cidr
   osm_cluster_network_cidr="10.2.0.0/16"
   openshift_master_portal_net="172.31.0.0/16"
2. create cakephp-mysql-example applications
3. upgrade to ose3.2
4. check the pod status
5. scale up pod and check the pod status

Actual results:

4.1) #oc get pods
NAME                              READY     STATUS             RESTARTS   AGE
cakephp-mysql-example-1-prehook   0/1       Error              0          3h
cakephp-mysql-example-2-gsuwd     0/1       ImagePullBackOff   0          33m
mysql-1-hag2n                     1/1       Running            0          33m

4.2) #oc describe pod cakephp-mysql-exsummary: sha256 Conflict when pull images after upgrade 

upgrade
M
M

Description:
The pod cakephp-mysql-example report sha256 Conflict error after upgrade. only cakephp-mysql-example hit this issue, the other pod works well, for example cakephp-example. 


Steps:
1. setup containerlized env and customized the network_cidr
   osm_cluster_network_cidr="10.2.0.0/16"
   openshift_master_portal_net="172.31.0.0/16"
2. create cakephp-mysql-example applications
3. upgrade to ose3.2
4. check the pod status
5. scale up pod and check the pod status

Actual Result:
4.1) #oc get pods
NAME                              READY     STATUS             RESTARTS   AGE
cakephp-mysql-example-1-prehook   0/1       Error              0          3h
cakephp-mysql-example-2-gsuwd     0/1       ImagePullBackOff   0          33m
mysql-1-hag2n                     1/1       Running            0          33m

4.2) #oc describe pod cakephp-mysql-example-2-gsuwd
Name:        cakephp-mysql-example-2-gsuwd
Namespace:    cakephpmysql
Node:        osecontain-master1.example.com/10.66.78.90
Start Time:    Thu, 31 Mar 2016 15:37:58 +0800
Labels:        app=cakephp-mysql-example,deployment=cakephp-mysql-example-2,deploymentconfig=cakephp-mysql-example,name=cakephp-mysql-example
Status:        Pending
IP:        10.2.0.4
Controllers:    ReplicationController/cakephp-mysql-example-2
Containers:
  cakephp-mysql-example:
    Container ID:    
    Image:        172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d
    Image ID:        
    Port:        8080/TCP
    QoS Tier:
      cpu:        BestEffort
      memory:        BestEffort
    State:        Waiting
      Reason:        ImagePullBackOff
    Ready:        False
    Restart Count:    0
    Environment Variables:
      DATABASE_SERVICE_NAME:        mysql
      DATABASE_ENGINE:            mysql
      DATABASE_NAME:            default
      DATABASE_USER:            cakephp
      DATABASE_PASSWORD:        5jR1QM2co1LxGp7A
      CAKEPHP_SECRET_TOKEN:        hpLFfj37B2PXlkyxBcxp5W7v5ZTrG4j7k2MFDJCxasYD4OjDHA
      CAKEPHP_SECURITY_SALT:        DtYNAwI4JiOxxbKfnKQHVlwDGs3aVEciEmof4RKC
      CAKEPHP_SECURITY_CIPHER_SEED:    083731226737423233151050783143
      OPCACHE_REVALIDATE_FREQ:        2
Conditions:
  Type        Status
  Ready     False 
Volumes:
  default-token-zkwja:
    Type:    Secret (a volume populated by a Secret)
    SecretName:    default-token-zkwja
Events:
  FirstSeen    LastSeen    Count    From                        SubobjectPath                Type        Reason            Message
  ---------    --------    -----    ----                        -------------                --------    ------            -------
  35m        35m        6    {scheduler }                                                FailedScheduling    Failed for reason Region and possibly others
  34m        34m        1    {scheduler }                                                Scheduled        Successfully assigned cakephp-mysql-example-2-gsuwd to osecontain-master1.example.com
  34m        34m        1    {kubelet osecontain-master1.example.com}    implicitly required container POD            Pulled            Container image "openshift3/ose-pod:v3.1.1.6" already present on machine
  34m        34m        1    {kubelet osecontain-master1.example.com}    implicitly required container POD            Created            Created with docker id 678a813fee80
  34m        34m        1    {kubelet osecontain-master1.example.com}    implicitly required container POD            Started            Started with docker id 678a813fee80
  34m        34m        1    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}            Pulling            pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d"
  34m        34m        1    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}            Pulled            Successfully pulled image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d"
  34m        34m        1    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}            Created            Created with docker id 08efb8d3d396
  34m        34m        1    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}            Started            Started with docker id 08efb8d3d396
  8m        8m        1    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Warning        Failed            Failed to pull image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d": image pull failed for 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d, this may be because there are no credentials on this request.  details: (Get http://172.31.78.12:5000/v2/cakephpmysql/cakephp-mysql-example/manifests/sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d: no basic auth credentials)
  8m        8m        1    {kubelet osecontain-master1.example.com}                        Warning        FailedSync        Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ErrImagePull: "image pull failed for 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d, this may be because there are no credentials on this request.  details: (Get http://172.31.78.12:5000/v2/cakephpmysql/cakephp-mysql-example/manifests/sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d: no basic auth credentials)"

  10m    2m    6    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Normal    Pulling        pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d"
  8m    2m    5    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Warning    Failed        Failed to pull image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d": Conflict: Digest sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d is already set to image 4ceef947c3ceb2e8eb502520ddcc0104608a54b60725ba4cd32a37b27116f780
  8m    2m    5    {kubelet osecontain-master1.example.com}                        Warning    FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ErrImagePull: "Conflict: Digest sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d is already set to image 4ceef947c3ceb2e8eb502520ddcc0104608a54b60725ba4cd32a37b27116f780"

  8m    3s    34    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Normal    BackOff        Back-off pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d"
  8m    3s    34    {kubelet osecontain-master1.example.com}                        Warning    FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ImagePullBackOff: "Back-off pulling image \"172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d\""


5.1) 
# oc get pods
NAME                              READY     STATUS             RESTARTS   AGE
cakephp-mysql-example-1-05h21     0/1       ErrImagePull       0          1m
cakephp-mysql-example-1-dhsfp     0/1       ImagePullBackOff   0          1m
cakephp-mysql-example-1-prehook   0/1       Error              0          3h
cakephp-mysql-example-2-gsuwd     0/1       ImagePullBackOff   0          35m
mysql-1-hag2n                     1/1       Running            0          35

5.2) #oc describe pod cakephp-mysql-example-1-dhsfp
Name:        cakephp-mysql-example-1-dhsfp
Namespace:    cakephpmysql
Node:        osecontain-master1.example.com/10.66.78.90
Start Time:    Thu, 31 Mar 2016 16:11:06 +0800
Labels:        app=cakephp-mysql-example,deployment=cakephp-mysql-example-1,deploymentconfig=cakephp-mysql-example,name=cakephp-mysql-example
Status:        Pending
IP:        10.2.0.6
Controllers:    ReplicationController/cakephp-mysql-example-1
Containers:
  cakephp-mysql-example:
    Container ID:    
    Image:        172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16
    Image ID:        
    Port:        8080/TCP
    QoS Tier:
      cpu:        BestEffort
      memory:        BestEffort
    State:        Waiting
      Reason:        ImagePullBackOff
    Ready:        False
    Restart Count:    0
    Environment Variables:
      DATABASE_SERVICE_NAME:        mysql
      DATABASE_ENGINE:            mysql
      DATABASE_NAME:            default
      DATABASE_USER:            cakephp
      DATABASE_PASSWORD:        5jR1QM2co1LxGp7A
      CAKEPHP_SECRET_TOKEN:        hpLFfj37B2PXlkyxBcxp5W7v5ZTrG4j7k2MFDJCxasYD4OjDHA
      CAKEPHP_SECURITY_SALT:        DtYNAwI4JiOxxbKfnKQHVlwDGs3aVEciEmof4RKC
      CAKEPHP_SECURITY_CIPHER_SEED:    083731226737423233151050783143
      OPCACHE_REVALIDATE_FREQ:        2
Conditions:
  Type        Status
  Ready     False 
Volumes:
  default-token-zkwja:
    Type:    Secret (a volume populated by a Secret)
    SecretName:    default-token-zkwja
Events:
  FirstSeen    LastSeen    Count    From                        SubobjectPath                Type        Reason        Message
  ---------    --------    -----    ----                        -------------                --------    ------        -------
  36s        36s        1    {default-scheduler }                                    Normal        Scheduled    Successfully assigned cakephp-mysql-example-1-dhsfp to osecontain-master1.example.com
  24s        24s        1    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Normal        BackOff        Back-off pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16"
  24s        24s        1    {kubelet osecontain-master1.example.com}                        Warning        FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ImagePullBackOff: "Back-off pulling image \"172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16\""

  30s    9s    2    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Normal    Pulling        pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16"
  25s    9s    2    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Warning    Failed        Failed to pull image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16": Conflict: Digest sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16 is already set to image a909de35caf4069c702e18f470b22590dd5a6fdb7c8ef1d699fcd6d14011bf4c
  25s    9s    2    {kubelet osecontain-master1.example.com}                        Warning    FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ErrImagePull: "Conflict: Digest sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16 is already set to image a909de35caf4069c702e18f470b22590dd5a6fdb7c8ef1d699fcd6d14011bf4c"





ample-2-gsuwd
Name:        cakephp-mysql-example-2-gsuwd
Namespace:    cakephpmysql
Node:        osecontain-master1.example.com/10.66.78.90
Start Time:    Thu, 31 Mar 2016 15:37:58 +0800
Labels:        app=cakephp-mysql-example,deployment=cakephp-mysql-example-2,deploymentconfig=cakephp-mysql-example,name=cakephp-mysql-example
Status:        Pending
IP:        10.2.0.4
Controllers:    ReplicationController/cakephp-mysql-example-2
Containers:
  cakephp-mysql-example:
    Container ID:    
    Image:        172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d
    Image ID:        
    Port:        8080/TCP
    QoS Tier:
      cpu:        BestEffort
      memory:        BestEffort
    State:        Waiting
      Reason:        ImagePullBackOff
    Ready:        False
    Restart Count:    0
    Environment Variables:
      DATABASE_SERVICE_NAME:        mysql
      DATABASE_ENGINE:            mysql
      DATABASE_NAME:            default
      DATABASE_USER:            cakephp
      DATABASE_PASSWORD:        5jR1QM2co1LxGp7A
      CAKEPHP_SECRET_TOKEN:        hpLFfj37B2PXlkyxBcxp5W7v5ZTrG4j7k2MFDJCxasYD4OjDHA
      CAKEPHP_SECURITY_SALT:        DtYNAwI4JiOxxbKfnKQHVlwDGs3aVEciEmof4RKC
      CAKEPHP_SECURITY_CIPHER_SEED:    083731226737423233151050783143
      OPCACHE_REVALIDATE_FREQ:        2
Conditions:
  Type        Status
  Ready     False 
Volumes:
  default-token-zkwja:
    Type:    Secret (a volume populated by a Secret)
    SecretName:    default-token-zkwja
Events:
  FirstSeen    LastSeen    Count    From                        SubobjectPath                Type        Reason            Message
  ---------    --------    -----    ----                        -------------                --------    ------            -------
  35m        35m        6    {scheduler }                                                FailedScheduling    Failed for reason Region and possibly others
  34m        34m        1    {scheduler }                                                Scheduled        Successfully assigned cakephp-mysql-example-2-gsuwd to osecontain-master1.example.com
  34m        34m        1    {kubelet osecontain-master1.example.com}    implicitly required container POD            Pulled            Container image "openshift3/ose-pod:v3.1.1.6" already present on machine
  34m        34m        1    {kubelet osecontain-master1.example.com}    implicitly required container POD            Created            Created with docker id 678a813fee80
  34m        34m        1    {kubelet osecontain-master1.example.com}    implicitly required container POD            Started            Started with docker id 678a813fee80
  34m        34m        1    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}            Pulling            pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d"
  34m        34m        1    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}            Pulled            Successfully pulled image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d"
  34m        34m        1    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}            Created            Created with docker id 08efb8d3d396
  34m        34m        1    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}            Started            Started with docker id 08efb8d3d396
  8m        8m        1    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Warning        Failed            Failed to pull image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d": image pull failed for 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d, this may be because there are no credentials on this request.  details: (Get http://172.31.78.12:5000/v2/cakephpmysql/cakephp-mysql-example/manifests/sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d: no basic auth credentials)
  8m        8m        1    {kubelet osecontain-master1.example.com}                        Warning        FailedSync        Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ErrImagePull: "image pull failed for 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d, this may be because there are no credentials on this request.  details: (Get http://172.31.78.12:5000/v2/cakephpmysql/cakephp-mysql-example/manifests/sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d: no basic auth credentials)"

  10m    2m    6    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Normal    Pulling        pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d"
  8m    2m    5    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Warning    Failed        Failed to pull image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d": Conflict: Digest sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d is already set to image 4ceef947c3ceb2e8eb502520ddcc0104608a54b60725ba4cd32a37b27116f780
  8m    2m    5    {kubelet osecontain-master1.example.com}                        Warning    FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ErrImagePull: "Conflict: Digest sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d is already set to image 4ceef947c3ceb2e8eb502520ddcc0104608a54b60725ba4cd32a37b27116f780"

  8m    3s    34    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Normal    BackOff        Back-off pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d"
  8m    3s    34    {kubelet osecontain-master1.example.com}                        Warning    FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ImagePullBackOff: "Back-off pulling image \"172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:84a42be9b8fd29569b48e3f61175c7521565b30a1db8ed8fbf485dff8f12549d\""


5.1) 
# oc get pods
NAME                              READY     STATUS             RESTARTS   AGE
cakephp-mysql-example-1-05h21     0/1       ErrImagePull       0          1m
cakephp-mysql-example-1-dhsfp     0/1       ImagePullBackOff   0          1m
cakephp-mysql-example-1-prehook   0/1       Error              0          3h
cakephp-mysql-example-2-gsuwd     0/1       ImagePullBackOff   0          35m
mysql-1-hag2n                     1/1       Running            0          35

5.2) #oc describe pod cakephp-mysql-example-1-dhsfp
Name:        cakephp-mysql-example-1-dhsfp
Namespace:    cakephpmysql
Node:        osecontain-master1.example.com/10.66.78.90
Start Time:    Thu, 31 Mar 2016 16:11:06 +0800
Labels:        app=cakephp-mysql-example,deployment=cakephp-mysql-example-1,deploymentconfig=cakephp-mysql-example,name=cakephp-mysql-example
Status:        Pending
IP:        10.2.0.6
Controllers:    ReplicationController/cakephp-mysql-example-1
Containers:
  cakephp-mysql-example:
    Container ID:    
    Image:        172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16
    Image ID:        
    Port:        8080/TCP
    QoS Tier:
      cpu:        BestEffort
      memory:        BestEffort
    State:        Waiting
      Reason:        ImagePullBackOff
    Ready:        False
    Restart Count:    0
    Environment Variables:
      DATABASE_SERVICE_NAME:        mysql
      DATABASE_ENGINE:            mysql
      DATABASE_NAME:            default
      DATABASE_USER:            cakephp
      DATABASE_PASSWORD:        5jR1QM2co1LxGp7A
      CAKEPHP_SECRET_TOKEN:        hpLFfj37B2PXlkyxBcxp5W7v5ZTrG4j7k2MFDJCxasYD4OjDHA
      CAKEPHP_SECURITY_SALT:        DtYNAwI4JiOxxbKfnKQHVlwDGs3aVEciEmof4RKC
      CAKEPHP_SECURITY_CIPHER_SEED:    083731226737423233151050783143
      OPCACHE_REVALIDATE_FREQ:        2
Conditions:
  Type        Status
  Ready     False 
Volumes:
  default-token-zkwja:
    Type:    Secret (a volume populated by a Secret)
    SecretName:    default-token-zkwja
Events:
  FirstSeen    LastSeen    Count    From                        SubobjectPath                Type        Reason        Message
  ---------    --------    -----    ----                        -------------                --------    ------        -------
  36s        36s        1    {default-scheduler }                                    Normal        Scheduled    Successfully assigned cakephp-mysql-example-1-dhsfp to osecontain-master1.example.com
  24s        24s        1    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Normal        BackOff        Back-off pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16"
  24s        24s        1    {kubelet osecontain-master1.example.com}                        Warning        FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ImagePullBackOff: "Back-off pulling image \"172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16\""

  30s    9s    2    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Normal    Pulling        pulling image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16"
  25s    9s    2    {kubelet osecontain-master1.example.com}    spec.containers{cakephp-mysql-example}    Warning    Failed        Failed to pull image "172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16": Conflict: Digest sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16 is already set to image a909de35caf4069c702e18f470b22590dd5a6fdb7c8ef1d699fcd6d14011bf4c
  25s    9s    2    {kubelet osecontain-master1.example.com}                        Warning    FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "cakephp-mysql-example" with ErrImagePull: "Conflict: Digest sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16 is already set to image a909de35caf4069c702e18f470b22590dd5a6fdb7c8ef1d699fcd6d14011bf4c"

Expected results:


Additional info:
Comment 1 Anping Li 2016-03-31 10:39:13 UTC 
Hit same issue on another environment. But this time the bad pod is epa64mysqls2i, and the cakephp-mysql-example pod work well.

So prompt the severity to high. which will make some pod failed to started.

 14m	48s	7	{kubelet osecontain-master1.example.com}	spec.containers{eap-app}	Normal	Pulling		pulling image "172.30.21.73:5000/epa64mysqls2i/eap-app@sha256:a51088b4e25343882520aef6f983fb4cdd063546b5a11331789e530cf42a148c"
  11m	47s	6	{kubelet osecontain-master1.example.com}	spec.containers{eap-app}	Warning	Failed		Failed to pull image "172.30.21.73:5000/epa64mysqls2i/eap-app@sha256:a51088b4e25343882520aef6f983fb4cdd063546b5a11331789e530cf42a148c": Conflict: Digest sha256:a51088b4e25343882520aef6f983fb4cdd063546b5a11331789e530cf42a148c is already set to image a65a5a9f8576e7e281d1e92ef8161376614160886af5659617216a19ac4f18f0
Comment 3 Andy Goldstein 2016-03-31 17:35:50 UTC 
The error is coming from Docker. It means that the node previously pulled 172.31.78.12:5000/cakephpmysql/cakephp-mysql-example@sha256:298e1216450a9c1b646cf9221c886aac7ff76eef5b59032d9551a1c6bcc7bb16 and recorded that it was associated with Docker v1 image ID a909de35caf4069c702e18f470b22590dd5a6fdb7c8ef1d699fcd6d14011bf4c. On the pull that's not working, it's receiving the image manifest and for some reason it thinks that the same image manifest should be associated with a different v1 image ID. I don't have any theories yet as to why this is happening.
Comment 5 Anping Li 2016-04-01 05:32:50 UTC 
Hit this issue again at the third Env. cakephpmysql is bad this time while epa64mysqls2i works well.

  1h	4m	18	{kubelet minimaster.example.com}	spec.containers{cakephp-mysql-example}	Warning	Failed		Failed to pull image "172.30.220.99:5000/cakephpmysql/cakephp-mysql-example@sha256:ad065f2bbb70fe15a5d48d594f7dcf270d4d94453a0ca7d639fa6bd467560b56": Conflict: Digest sha256:ad065f2bbb70fe15a5d48d594f7dcf270d4d94453a0ca7d639fa6bd467560b56 is already set to image f00d7d7174c35f8fe373c1074cd79320ca4647448782dd8c3c85471e1ee743ec
Comment 7 Andy Goldstein 2016-04-01 21:05:53 UTC 
Something about https://github.com/projectatomic/docker/commit/504e67b867865a2835e8002c01087a2cfd7bfd0e is resulting in this error
Comment 8 Brenton Leanhardt 2016-04-04 13:47:13 UTC 
Andy, who should take this bug?
Comment 9 Seth Jennings 2016-04-04 18:16:08 UTC 
I'm looking at it right now.
Comment 10 Seth Jennings 2016-04-04 20:23:25 UTC 
A commit here that is in 1.9.1 that is not in 1.8.3 that refactors (possibly regresses) the digest tag code.  Maybe the new code inverts the layerIDs?

https://github.com/projectatomic/docker/commit/9b2426c606b92fa37b7484d9d884edd43352b69c
Comment 11 Andy Goldstein 2016-04-05 13:41:21 UTC 
When you pull by digest using 1.8, an entry is created in /var/lib/docker/repositories-devicemapper for the image repo -> digest -> v1 ID.

After the upgrade to 1.9, when you pull the image by digest again (which should be a no-op), you get this error. It looks like the "strong ID" logic that computes deterministic IDs for the layers is coming up with a different v1 ID and then this error.

I'm leaning toward this being an issue in 1.9, but there is a chance that the upgraded OSE registry coming from the upgrade from OSE 3.1 to 3.2 plays some part.
Comment 13 Seth Jennings 2016-04-05 17:10:18 UTC 
This can be reproduced fairly easy with just docker.

install docker 1.8.2 (yum downgrade docker, if needed)
start local registry:2 w/ persistent storage
docker pull busybox
docker push localhost:5000/busybox (prints sha256 digest after push)
docker pull localhost:5000/busybox@<digest>
yum update docker (to 1.9.1)
systemctl restart docker
docker start registry
docker pull localhost:5000/busybox@<digest>
Trying to pull repository localhost:5000/busybox ... sha256:<digest>: Pulling from busybox
failed
Conflict: Digest sha256:<digest> is already set to image <v1 weak ID>

See attachment for a my particular session.
Comment 14 Seth Jennings 2016-04-05 17:11:07 UTC 
Created attachment 1143899 [details]
reproduce-procedure
Comment 15 Antonio Murdaca 2016-04-20 16:51:29 UTC 
just curious while I setup an env to reproduce this bz, is this happening with the upstream 1.9.1 docker as well? Asking because we did some edits in the past to the affected code.
Comment 16 Andy Goldstein 2016-04-20 16:52:43 UTC 
We didn't test upstream - only our RPMs
Comment 17 Antonio Murdaca 2016-04-20 19:24:30 UTC 
alright, I see this in upstream 1.9.1 as well - will need to test more and see what's the cause.
Comment 18 Antonio Murdaca 2016-04-21 10:48:11 UTC 
Found the root cause of this in 1.9.1 - basically the new 1.9.1 code compute the id (not the strong one) based on the blobSum(s) hashes (among other things) that are different between the ones we get from the localhost registry and the one we originally got from docker.io (and later tagged) so we get the wrong (weak) ID (this doesn't happen using docker.io because the busybox you pull it's *derived* by that but the busybox we use in our registry comes from the docker.io manifest). In 1.8.3 this check isn't performed of course. I'm not sure how to fix this, still trying to play around with the code.
Comment 19 Antonio Murdaca 2016-04-21 11:12:50 UTC 
What's really interesting is that using upstream 1.8.3 -> projectatomic 1.9.1 doesn't trigger this issue - I was able to repull the image from my registry with the digest.
So, what is really broken is probably our projectatomic 1.8.2 because it appears that projectatomic 1.9.1 is working fine when upgrading from upstream 1.8.3 (but again, it's broken after upgrading from projectatomic 1.8.2)

We probably backported too much into projectatomic 1.8.2 or missed something from upstream 1.8.3.

I'm still playing around with this though. I'll try playing with our patches in our 1.8.2 (and upstream 1.8.2 also).
Comment 20 Antonio Murdaca 2016-04-21 12:23:27 UTC 
Just retested (note that we don't have a projectatomic docker 1.8.3):

upstream 1.8.2      -> projectatomic 1.9.1 -> broken
upstream 1.8.2      -> upstream 1.9.1      -> broken
upstream 1.8.3      -> projectatomic 1.9.1 -> works
upstream 1.8.3      -> upstream 1.9.1      -> works
projectatomic 1.8.2 -> projectatomic 1.9.1 -> broken
projectatomic 1.8.2 -> upstream 1.9.1      -> broken

So I think we're clearly missing some patches from our projectatomic 1.8.2 - specifically from upstream 1.8.3. I'll look around to see if it's feasible to patch (again) our docker 1.9.1 to handle this correctly but I don't think so since it would require removing patches (or adjusting code in a way I don't feel comfortable at all). To me, it looks like https://github.com/docker/docker/pull/16953 is missing from our 1.8.2 backports (which belongs to upstream 1.8.3 though) and which seems to be the discriminating factor here (still have to test this out, but it requires a backport to 1.8.2 which we don't provide rpm anymore)
Comment 21 Antonio Murdaca 2016-04-22 15:50:37 UTC 
Given we can't patch 1.9.1 to remove this check the only workaround I see here is to "docker rmi image@digest" the affected image and repull it again.
Comment 22 Daniel Walsh 2016-04-22 15:59:24 UTC 
So this is Release note issue then?
Comment 23 Brenton Leanhardt 2016-04-25 12:12:12 UTC 
Yes, we'll have to make this part of the OSE release notes.  If there was a know what to detect affected images we could remove them during the upgrade automatically.
Comment 25 Luwen Su 2016-05-02 08:47:52 UTC 
From docker-1.8.2-3.el7.x86_64.rpm to docker-1.9.1-38.el7.x86_64.rpm

In docker-1.8.2-3.el7.x86_64.rpm
# docker pull busybox
Using default tag: latest
Trying to pull repository registry.access.redhat.com/busybox ... not found
Trying to pull repository docker.io/library/busybox ... latest: Pulling from library/busybox
56ed16bd6310: Pull complete 
bc744c4ab376: Pull complete 
Digest: sha256:4a887a2326ec9e0fa90cce7b4764b0e627b5d6afcb81a3f73c85dc29cea00048
Status: Downloaded newer image for docker.io/busybox:latest

# docker tag busybox myregistrydomain.com:5000/shatest
# docker push myregistrydomain.com:5000/shatest
The push refers to a repository [myregistrydomain.com:5000/shatest] (len: 1)
bc744c4ab376: Pushed 
56ed16bd6310: Pushed 
latest: digest: sha256:a054d9525f4f48b18d7c2810ad3d570c8fd9118feba106a5363ee92f7edc42c0 size: 3202
# docker pull myregistrydomain.com:5000/shatest
Using default tag: latest
Trying to pull repository myregistrydomain.com:5000/shatest ... latest: Pulling from shatest
56ed16bd6310: Already exists 
bc744c4ab376: Already exists 
Digest: sha256:a054d9525f4f48b18d7c2810ad3d570c8fd9118feba106a5363ee92f7edc42c0
Status: Image is up to date for myregistrydomain.com:5000/shatest:latest

Upgrade to 1.9
#rpm -Uvh *.rpm
# docker pull myregistrydomain.com:5000/shatest
Using default tag: latest
Trying to pull repository myregistrydomain.com:5000/shatest ... latest: Pulling from shatest
3cbb862e15c2: Pull complete 
5d5461996ca8: Pull complete 
Digest: sha256:a054d9525f4f48b18d7c2810ad3d570c8fd9118feba106a5363ee92f7edc42c0
Status: Downloaded newer image for myregistrydomain.com:5000/shatest:latest


Move to verified
Comment 26 Antonio Murdaca 2016-05-02 08:59:13 UTC 
(In reply to Luwen Su from comment #25)
> From docker-1.8.2-3.el7.x86_64.rpm to docker-1.9.1-38.el7.x86_64.rpm
> 
> In docker-1.8.2-3.el7.x86_64.rpm
> # docker pull busybox
> Using default tag: latest
> Trying to pull repository registry.access.redhat.com/busybox ... not found
> Trying to pull repository docker.io/library/busybox ... latest: Pulling from
> library/busybox
> 56ed16bd6310: Pull complete 
> bc744c4ab376: Pull complete 
> Digest:
> sha256:4a887a2326ec9e0fa90cce7b4764b0e627b5d6afcb81a3f73c85dc29cea00048
> Status: Downloaded newer image for docker.io/busybox:latest
> 
> # docker tag busybox myregistrydomain.com:5000/shatest
> # docker push myregistrydomain.com:5000/shatest
> The push refers to a repository [myregistrydomain.com:5000/shatest] (len: 1)
> bc744c4ab376: Pushed 
> 56ed16bd6310: Pushed 
> latest: digest:
> sha256:a054d9525f4f48b18d7c2810ad3d570c8fd9118feba106a5363ee92f7edc42c0
> size: 3202
> # docker pull myregistrydomain.com:5000/shatest
> Using default tag: latest
> Trying to pull repository myregistrydomain.com:5000/shatest ... latest:
> Pulling from shatest
> 56ed16bd6310: Already exists 
> bc744c4ab376: Already exists 
> Digest:
> sha256:a054d9525f4f48b18d7c2810ad3d570c8fd9118feba106a5363ee92f7edc42c0
> Status: Image is up to date for myregistrydomain.com:5000/shatest:latest
> 
> Upgrade to 1.9
> #rpm -Uvh *.rpm
> # docker pull myregistrydomain.com:5000/shatest
> Using default tag: latest
> Trying to pull repository myregistrydomain.com:5000/shatest ... latest:
> Pulling from shatest
> 3cbb862e15c2: Pull complete 
> 5d5461996ca8: Pull complete 
> Digest:
> sha256:a054d9525f4f48b18d7c2810ad3d570c8fd9118feba106a5363ee92f7edc42c0
> Status: Downloaded newer image for myregistrydomain.com:5000/shatest:latest
> 
> 
> Move to verified

Sorry, the exact step to repro and verify are https://bugzilla.redhat.com/attachment.cgi?id=1143899 (specifically you aren't pulling by digest in your test)
We still have a pending PR to fix this BZ https://github.com/projectatomic/docker/pull/141
Comment 28 Antonio Murdaca 2016-05-12 06:46:22 UTC 
We didn't merge the pull request, did we?
Comment 29 Daniel Walsh 2016-05-12 12:04:37 UTC 
I don't know.
Comment 30 errata-xmlrpc 2016-05-12 15:17:36 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-1034.html
Note You need to log in before you can comment on or make changes to this bug.