Bug 1820250 - Pods using internal images cannot pull the images after the migration
Summary: Pods using internal images cannot pull the images after the migration
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Migration Tooling
Version: 4.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.4.0
Assignee: Scott Seago
QA Contact: Xin jiang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-02 15:23 UTC by Sergio
Modified: 2020-05-28 11:10 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-28 11:09:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
logs (4.17 MB, application/zip)
2020-04-02 15:23 UTC, Sergio
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2020:2326 0 None None None 2020-05-28 11:10:21 UTC

Description Sergio 2020-04-02 15:23:24 UTC
Created attachment 1675803 [details]
logs

Description of problem:
When we migrate pods that are not owned by any deployment and use internal images, the pods in the target cluster cannot pull the images after the migration.

Version-Release number of selected component (if applicable):
CAM 1.1.2 stage
Target cluster: OCP 4.3
Source cluster: OCP 3.9

How reproducible:
Always

Steps to Reproduce:
1. Create the project
oc new-project bztest
 
2. Mirror a image to a imagestream in this project
 
$ oc process -f  https://github.com/sergiordlr/temp-testfiles/blob/master/helpers/mirror_helper_template.yml?raw=true -p NAMESPACE=bztest -p INT_REGISTRY=docker-registry.default.svc:5000 --namespace=bztest -p TGT_IMAGE=bztest/test-mirror:latest  | oc create -f -
 
3. Create a pod using this image
 
apiVersion: v1
kind: Pod
metadata:
  generateName: bztest-
  namespace: bztest
  labels:
    app: bztest
spec:
  containers:
  - args:
    - echo 'Hello world! I can load the image. docker-registry.default.svc:5000/bztest/test-mirror:latest';
      while true; do sleep 30; done;
    command:
    - /bin/sh
    - -c
    - --
    image: docker-registry.default.svc:5000/bztest/test-mirror:latest
    imagePullPolicy: Always
    name: podtest
    resources: {}
  restartPolicy: OnFailure

 
4. Migrate


Actual results:
In target cluster the pod cannot pull the image (it has configured the dockercfg secret from the source cluster)
 
$ oc get pods
NAME           READY   STATUS         RESTARTS   AGE
bztest-2mvbg   0/1     ErrImagePull   0          100s

describe:

Events:
  Type     Reason     Age                         From                                                Message
  ----     ------     ----                        ----                                                -------
  Normal   Scheduled  <unknown>                   default-scheduler                                   Successfully assigned bztest/bztest-2mvbg to ip-10-0-76-171.us-east-2.compute.internal
  Normal   Pulling    7m3s (x4 over 8m22s)        kubelet, ip-10-0-76-171.us-east-2.compute.internal  Pulling image "image-registry.openshift-image-registry.svc:5000/bztest/test-mirror:latest"
  Warning  Failed     7m3s (x4 over 8m22s)        kubelet, ip-10-0-76-171.us-east-2.compute.internal  Failed to pull image "image-registry.openshift-image-registry.svc:5000/bztest/test-mirror:latest": rpc error: code = Unknown desc = Error reading manifest latest in image-registry.openshift-image-registry.svc:5000/bztest/test-mirror: unauthorized: authentication required
  Warning  Failed     7m3s (x4 over 8m22s)        kubelet, ip-10-0-76-171.us-east-2.compute.internal  Error: ErrImagePull
  Warning  Failed     6m48s (x6 over 8m21s)       kubelet, ip-10-0-76-171.us-east-2.compute.internal  Error: ImagePullBackOff
  Normal   BackOff    <invalid> (x42 over 8m21s)  kubelet, ip-10-0-76-171.us-east-2.compute.internal  Back-off pulling image "image-registry.openshift-image-registry.svc:5000/bztest/test-mirror:latest"


Expected results:
The pod should be able to pull the internal image after the migration

Additional info:
The dockerconfig secret configured in the pod after the migration is the secret used in the source cluster, not the secret corresponding to the target cluster.

Comment 5 Sergio 2020-05-08 15:12:02 UTC
Verified using CAM 1.2
4.2 -> 4.3

Following the steps to reproduce the issue we were able to run the migration successfully, and the pods were running without problems in the target migration after the migration.

Comment 7 errata-xmlrpc 2020-05-28 11:09:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:2326


Note You need to log in before you can comment on or make changes to this bug.