Bug 1428320 - Duplicate ContainerImage records with same digest
Summary: Duplicate ContainerImage records with same digest
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.7.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: GA
: 5.8.0
Assignee: Erez Freiberger
QA Contact: Einat Pacifici
URL:
Whiteboard: container
Depends On:
Blocks: 1433500
TreeView+ depends on / blocked
 
Reported: 2017-03-02 10:08 UTC by Beni Paskin-Cherniavsky
Modified: 2018-05-03 12:33 UTC (History)
9 users (show)

Fixed In Version: 5.8.0.7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1433500 (view as bug list)
Environment:
Last Closed: 2017-06-12 16:22:14 UTC
Category: Bug
Cloudforms Team: Container Management
Target Upstream Version:
Embargoed:
epacific: automate_bug+


Attachments (Terms of Use)

Description Beni Paskin-Cherniavsky 2017-03-02 10:08:55 UTC
Description of problem:
After refreshing the openshift provider, I see same digest in 2 Container Image entities, with different data subsets, and different image_ref:
    image_ref: "docker://sha256:d608e58937e253f830bd7cd0207f7ff17ca06d08b6e774b7bf88cf82951162ef",
    image_ref: "docker-pullable://172.30.185.88:5000/red-dragon/nodejs-ex@sha256:2cf6ecb530537586db8c5558783762bfe6bd2f1c5528cdeb5c756a19f821706c",

Version-Release number of selected component (if applicable):
ManageIQ current master (2017-03-01, was already seen on 2017-02-12)
openshift origin v1.3.3

How reproducible:
Always with this openshift cluster, though with only 1 image; also saw it with another image on another openshift (also v1.3.3).

The `oc get` jsons of the only image and pod containing these hash(es) are here: https://gist.github.com/cben/8438aecc5f84b5c3c23421f208b9fac0

Actual results:

UI shows the image twice, but AFAICT details screen don't show the digest, so it's impossible in UI to confirm it's actually same image:

+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->>
|                  »Name«                   | »Provider« | »Tag«  |                                                                    »Id«                                                                     |>>
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->>
| red-dragon/nodejs-ex                      |            |        | docker://sha256:d608e58937e253f830bd7cd0207f7ff17ca06d08b6e774b7bf88cf82951162ef                                                            |>>
| red-dragon/nodejs-ex                      |            | latest | docker-pullable://172.30.185.88:5000/red-dragon/nodejs-ex@sha256:2cf6ecb530537586db8c5558783762bfe6bd2f1c5528cdeb5c756a19f821706c           |>>

rails console:

pry> ContainerImage.all.group_by(&:digest).except(nil).select{|k,vs| vs.count > 1}

=> {"sha256:2cf6ecb530537586db8c5558783762bfe6bd2f1c5528cdeb5c756a19f821706c"=>
  [#<ContainerImage:0x0055f4edb62410
    id: 50,
    tag: nil,
    name: "red-dragon/nodejs-ex",
    image_ref: "docker://sha256:d608e58937e253f830bd7cd0207f7ff17ca06d08b6e774b7bf88cf82951162ef",
    container_image_registry_id: 2,
    ems_id: 3,
    last_sync_on: nil,
    last_scan_attempt_on: nil,
    digest: "sha256:2cf6ecb530537586db8c5558783762bfe6bd2f1c5528cdeb5c756a19f821706c",
    registered_on: Wed, 01 Mar 2017 16:40:14 UTC +00:00,
    architecture: nil,
    author: nil,
    command: [],
    entrypoint: [],
    docker_version: nil,
    exposed_ports: {},
    environment_variables: {},
    size: nil,
    created_on: Wed, 01 Mar 2017 12:14:23 UTC +00:00,
    old_ems_id: nil,
    deleted_on: nil>,
   #<ContainerImage:0x0055f4edb61650
    id: 63,
    tag: "latest",
    name: "red-dragon/nodejs-ex",
    image_ref: "docker-pullable://172.30.185.88:5000/red-dragon/nodejs-ex@sha256:2cf6ecb530537586db8c5558783762bfe6bd2f1c5528cdeb5c756a19f821706c",
    container_image_registry_id: 2,
    ems_id: 3,
    last_sync_on: nil,
    last_scan_attempt_on: nil,
    digest: "sha256:2cf6ecb530537586db8c5558783762bfe6bd2f1c5528cdeb5c756a19f821706c",
    registered_on: Wed, 01 Mar 2017 16:40:14 UTC +00:00,
    architecture: "amd64",
    author: nil,
    command: ["/usr/libexec/s2i/run"],
    entrypoint: ["container-entrypoint"],
    docker_version: "1.12.5",
    exposed_ports: {"tcp"=>"8080"},
    environment_variables:
     {"OPENSHIFT_BUILD_NAME"=>"nodejs-ex-1",
      "OPENSHIFT_BUILD_NAMESPACE"=>"red-dragon",
      "OPENSHIFT_BUILD_SOURCE"=>"https://github.com/openshift/nodejs-ex.git",
      "OPENSHIFT_BUILD_REFERENCE"=>"master",
      "PATH"=>"/opt/app-root/src/node_modules/.bin/:/opt/app-root/src/.npm-global/bin/:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "STI_SCRIPTS_URL"=>"image:///usr/libexec/s2i",
      "STI_SCRIPTS_PATH"=>"/usr/libexec/s2i",
      "HOME"=>"/opt/app-root/src",
      "BASH_ENV"=>"/opt/app-root/etc/scl_enable",
      "ENV"=>"/opt/app-root/etc/scl_enable",
      "PROMPT_COMMAND"=>". /opt/app-root/etc/scl_enable",
      "NODEJS_VERSION"=>"4",
      "NPM_RUN"=>"start",
      "NPM_CONFIG_PREFIX"=>"/opt/app-root/src/.npm-global"},
    size: 167791498,
    created_on: Wed, 01 Mar 2017 12:14:24 UTC +00:00,
    old_ems_id: nil,
    deleted_on: nil>]}


Expected results:
Images with same digest have the same bytes, we should represent them as one Container Image.

Comment 2 Beni Paskin-Cherniavsky 2017-03-02 10:12:16 UTC
Quoting Solly Ross <sross> from mail discussion:

> any identifier prefixed with `docker://` should be
> disregarded, since it's not really useful to identify images with those
> on the registry.  Since we backported the Docker patch that makes the
> canoncial image pull specs always available, you should not ever see
> the `docker://` prefix on RH systems running Docker 1.10+, IIRC.
>
> The `docker-pullable://` format is actually a canonical pull spec, so
> it may be used to identify an image against the one stored in a registry,
> and may also be used to `docker pull`, etc.
>
> Note that the `docker-pullable://` ID follows the format
> `docker-pullable://IMAGE_NAME@SHA`.  This causes a small wrinkle -- if you
> have one image named two different ways (i.e. from two different repositories),
> there can be two different values of IMAGE_NAME (both of which are valid).
> Kubernetes will always use the first one that Docker reports, so if
> you want to compare, the following is true:
>
>     Two images with the same `docker-pullable://` ID are identical.
>     Two images with different `docker-pullable://` IDs may be identical.
>     Two images with different SHAs in the `docker-pullable://` ID must be different.
>     (and similarly, two images with the same SHA must be identical)
>
> While the case of two identical images from different repositories is slim, it's
> something to be aware of.

Comment 4 Mooli Tayer 2017-03-02 10:17:40 UTC
BTW at the time I did this and saw that one image is a complete subset of the other(except for image ref):

https://www.diffchecker.com/OtnPx4LS

Comment 5 Beni Paskin-Cherniavsky 2017-03-02 10:20:02 UTC
[root@vm-48-131 ~]# docker --version
Docker version 1.12.5, build 047e51b/1.12.5

Comment 6 Erez Freiberger 2017-03-05 14:09:02 UTC
We are currently using the images image-ref (imageID) and registry to id the images. This is problematic in these cases where the imageID from the running pod (docker) is not equal to the one we get from Openshift.

Proposed fix: https://github.com/ManageIQ/manageiq/pull/14185

Comment 9 Pavel Zagalsky 2017-04-24 08:14:10 UTC
We haven't been able to recreate this bug
Closing

Comment 11 Einat Pacifici 2017-04-24 10:15:40 UTC
Verified. The duplicates were not seen. However, since this was very difficult to reproduce, if this does happen again, we will need full information regarding docker version, ocp version etc.


Note You need to log in before you can comment on or make changes to this bug.