Bug 1910518

Summary:	OCP3.x operator.yaml misses rsync-transfer image definition
Product:	Migration Toolkit for Containers	Reporter:	Xin jiang <xjiang>
Component:	General	Assignee:	Jason Montleon <jmontleo>
Status:	CLOSED ERRATA	QA Contact:	Xin jiang <xjiang>
Severity:	urgent	Docs Contact:	Avital Pinnick <apinnick>
Priority:	urgent
Version:	1.4.0	CC:	chezhang, ernelson, jmontleo, sregidor, whu
Target Milestone:	---
Target Release:	1.4.0
Hardware:	Unspecified
OS:	Unspecified
URL:	https://github.com/konveyor/mig-operator/pull/534
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-02-11 12:55:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Xin jiang 2020-12-24 09:18:41 UTC

Description of problem:
It's missing rsync-transfer image definition in OCP 3.x operator.yml, it causes ImagePullBackOff issue.

    - name: RSYNC_TRANSFER_REPO
      value: openshift-migration-rsync-transfer-rhel8@sha256
    - name: RSYNC_TRANSFER_TAG
      value: 715d6869c6d23d572808d64c511b94c6af07a6bce08b4f62a78542b61f3af02a

Version-Release number of selected component (if applicable):
MTC 1.4.0

openshift-migration-rhel7-operator@sha256:236d5f964061938f4e845e6d9e62926da68435c0f0304329a0473e060cdb53f1

How reproducible:
Always

Steps to Reproduce:
1. Download operator.yml
$ podman cp $(podman create brew.registry.redhat.io/rhmtc/openshift-migration-rhel7-operator@sha256:236d5f964061938f4e845e6d9e62926da68435c0f0304329a0473e060cdb53f1):/operator.yml ./

2. check operator.yml


Actual results:
It's missing rsync-transfer image

        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: REGISTRY
          value: registry.redhat.io
        - name: PROJECT
          value: rhmtc
        - name: HOOK_RUNNER_REPO
          value: openshift-migration-hook-runner-rhel7@sha256
        - name: HOOK_RUNNER_TAG
          value: 69b322784c2fc4227062daed570010add395734b8b2358b1dcb41fb97a209d5a
        - name: MIG_CONTROLLER_REPO
          value: openshift-migration-controller-rhel8@sha256
        - name: MIG_UI_REPO
          value: openshift-migration-ui-rhel8@sha256
        - name: MIG_LOG_READER_REPO
          value: openshift-migration-log-reader-rhel8@sha256
        - name: MIGRATION_REGISTRY_REPO
          value: openshift-migration-registry-rhel8@sha256
        - name: MIGRATION_REGISTRY_TAG
          value: ce78f94658169f04e67d2385998c42aad61d53fd12c45aa6911ee28f709fe2e9
        - name: VELERO_REPO
          value: openshift-migration-velero-rhel8@sha256
        - name: VELERO_PLUGIN_REPO
          value: openshift-velero-plugin-rhel8@sha256
        - name: VELERO_RESTIC_RESTORE_HELPER_REPO
          value: openshift-migration-velero-restic-restore-helper-rhel8@sha256
        - name: VELERO_AWS_PLUGIN_REPO
          value: openshift-migration-velero-plugin-for-aws-rhel8@sha256
        - name: VELERO_GCP_PLUGIN_REPO
          value: openshift-migration-velero-plugin-for-gcp-rhel8@sha256
        - name: VELERO_AZURE_PLUGIN_REPO
          value: openshift-migration-velero-plugin-for-microsoft-azure-rhel8@sha256
        - name: VELERO_TAG
          value: 9c065c17a02f7f7cd67eac37dabe6af0845002a77839823f23fef2b03ff6bbed
        - name: VELERO_RESTIC_RESTORE_HELPER_TAG
          value: 42188e1b4b1901096e5af2dbb98c9eb01f8eb5f5501bc78033167dd3a2229131
        - name: VELERO_PLUGIN_TAG
          value: 84d17dba4c0e46246e24255287b1e16e124a8eeccb4d14c5d1c0eda24c1c0410
        - name: VELERO_AWS_PLUGIN_TAG
          value: b88f43762a26aa53c621cecf7611fb6be13fd73f31bf6423ad5e456e22fa0854
        - name: VELERO_GCP_PLUGIN_TAG
          value: 74c6295ddad4daf584f45aa41dd0411ebaac806a35713f9b5a5c607cc81c4ff6
        - name: VELERO_AZURE_PLUGIN_TAG
          value: 9a83890dcc3836c34f45c927fe7a887f58af8c86aef5708829936527627b44cf
        - name: MIG_UI_TAG
          value: 6aabdf05a9b795cde3a224be3ad917d4246e17db66fba9910080eeb460aed7d8
        - name: MIG_CONTROLLER_TAG
          value: 5671667a75e12cd538a34c208bfb076534296cdf91f7e97ef48fc59f8ce3c172
        - name: MIG_LOG_READER_TAG
          value: a1cfa2f7f96e5deaf86c07c784d20debb13fa95b006bb735ffd2e08043d1cce5
      volumes:
        - name: runner
          emptyDir: {}

Expected results:
rsync-transfer image should be defined in operator.yml
    - name: RSYNC_TRANSFER_REPO
      value: openshift-migration-rsync-transfer-rhel8@sha256
    - name: RSYNC_TRANSFER_TAG
      value: 715d6869c6d23d572808d64c511b94c6af07a6bce08b4f62a78542b61f3af02a 

Additional info:

Comment 1 Xin jiang 2021-01-04 13:09:04 UTC

Jason,

Even I manually modified operator.yaml by adding rsync-transfer image and pointed image registry  to "brew.registry.redhat.io".  it still pull down image from "registry.redhat.io" (This is OCP 3.11 cluster) .


operator.yml:
......
    spec:
      serviceAccountName: migration-operator
      containers:
      - name: operator
        image: brew.registry.redhat.io/rhmtc/openshift-migration-rhel7-operator@sha256:236d5f964061938f4e845e6d9e62926da68435c0f0304329a0473e060cdb53f1
        imagePullPolicy: Always
......

        - name: REGISTRY
          value: brew.registry.redhat.io
        - name: PROJECT
          value: rhmtc
        - name: RSYNC_TRANSFER_REPO
          value: openshift-migration-rsync-transfer-rhel8@sha256
        - name: RSYNC_TRANSFER_TAG
          value: 715d6869c6d23d572808d64c511b94c6af07a6bce08b4f62a78542b61f3af02a


$ oc get pod -n openshift-migration migration-operator-58fbc99b59-k4gzt -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    openshift.io/scc: anyuid
  creationTimestamp: 2021-01-04T07:59:33Z
  generateName: migration-operator-58fbc99b59-
  labels:
    app: migration
    pod-template-hash: "1496755615"
  name: migration-operator-58fbc99b59-k4gzt
  namespace: openshift-migration
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: migration-operator-58fbc99b59
    uid: c822f5a0-4e62-11eb-9616-0e5a72214cc1
  resourceVersion: "63899"
  selfLink: /api/v1/namespaces/openshift-migration/pods/migration-operator-58fbc99b59-k4gzt
  uid: c829fe52-4e62-11eb-9616-0e5a72214cc1
spec:
  containers:
  - env:
    - name: OPERATOR_NAME
      value: migration-operator
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: WATCH_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: REGISTRY
      value: brew.registry.redhat.io
    - name: PROJECT
      value: rhmtc
    - name: RSYNC_TRANSFER_REPO
      value: openshift-migration-rsync-transfer-rhel8@sha256
    - name: RSYNC_TRANSFER_TAG
      value: 715d6869c6d23d572808d64c511b94c6af07a6bce08b4f62a78542b61f3af02a

$ oc  -n ocp-24769-cakephp  describe pod directvolumemigration-stunnel-transfer
.....
Events:
  Type     Reason          Age              From                                  Message
  ----     ------          ----             ----                                  -------
  Normal   Scheduled       4m               default-scheduler                     Successfully assigned ocp-24769-cakephp/directvolumemigration-stunnel-transfer to ip-172-18-3-10.ec2.internal
  Normal   Pulling         3m (x2 over 4m)  kubelet, ip-172-18-3-10.ec2.internal  pulling image "registry.redhat.io/rhmtc/openshift-migration-rsync-transfer-rhel8@sha256:715d6869c6d23d572808d64c511b94c6af07a6bce08b4f62a78542b61f3af02a"
  Warning  Failed          3m (x2 over 3m)  kubelet, ip-172-18-3-10.ec2.internal  Failed to pull image "registry.redhat.io/rhmtc/openshift-migration-rsync-transfer-rhel8@sha256:715d6869c6d23d572808d64c511b94c6af07a6bce08b4f62a78542b61f3af02a": rpc error: code = Unknown desc = unknown: Not Found
  Warning  Failed          3m (x2 over 3m)  kubelet, ip-172-18-3-10.ec2.internal  Error: ErrImagePull
  Normal   SandboxChanged  3m (x7 over 3m)  kubelet, ip-172-18-3-10.ec2.internal  Pod sandbox changed, it will be killed and re-created.
  Normal   BackOff         3m (x6 over 3m)  kubelet, ip-172-18-3-10.ec2.internal  Back-off pulling image "registry.redhat.io/rhmtc/openshift-migration-rsync-transfer-rhel8@sha256:715d6869c6d23d572808d64c511b94c6af07a6bce08b4f62a78542b61f3af02a"
  Warning  Failed          3m (x6 over 3m)  kubelet, ip-172-18-3-10.ec2.internal  Error: ImagePullBackOff

Comment 2 Jason Montleon 2021-01-04 13:50:19 UTC

Please post the contents of the mig-cluster-config configmap.

Comment 3 Xin jiang 2021-01-04 14:12:19 UTC

$ oc get configmap -n openshift-migration migration-cluster-config -o yaml
apiVersion: v1
data:
  REGISTRY_IMAGE: brew.registry.redhat.io/rhmtc/openshift-migration-registry-rhel8@sha256:ce78f94658169f04e67d2385998c42aad61d53fd12c45aa6911ee28f709fe2e9
  RSYNC_TRANSFER_IMAGE: brew.registry.redhat.io/rhmtc/openshift-migration-rsync-transfer-rhel8@sha256:715d6869c6d23d572808d64c511b94c6af07a6bce08b4f62a78542b61f3af02a
  STAGE_IMAGE: brew.registry.redhat.io/rhmtc/openshift-migration-velero-restic-restore-helper-rhel8@sha256:42188e1b4b1901096e5af2dbb98c9eb01f8eb5f5501bc78033167dd3a2229131
kind: ConfigMap
metadata:
  creationTimestamp: 2021-01-04T08:01:39Z
  labels:
    control-plane: controller-manager
    controller-tools.k8s.io: "1.0"
  name: migration-cluster-config
  namespace: openshift-migration
  ownerReferences:
  - apiVersion: migration.openshift.io/v1alpha1
    kind: MigrationController
    name: migration-controller
    uid: 09bf136f-4e63-11eb-9616-0e5a72214cc1
  resourceVersion: "64133"
  selfLink: /api/v1/namespaces/openshift-migration/configmaps/migration-cluster-config
  uid: 12f174b7-4e63-11eb-9616-0e5a72214cc1

Comment 4 Jason Montleon 2021-01-05 13:44:19 UTC

controller PR: https://github.com/konveyor/mig-controller/pull/871

Comment 9 Xin jiang 2021-01-06 09:42:06 UTC

verified that it can pull down rsync-transfer image from expected registry on 3.x. 

 
directvolumemigration-stunnel-transfer   0/1       Pending   0         0s
directvolumemigration-stunnel-transfer   0/1       Pending   0         0s
directvolumemigration-stunnel-transfer   0/1       ContainerCreating   0         0s
^@directvolumemigration-stunnel-transfer   1/1       Running   0         10s
directvolumemigration-rsync-transfer-mysql   0/1       Pending   0         0s
directvolumemigration-rsync-transfer-mysql   0/1       Pending   0         0s
directvolumemigration-rsync-transfer-mysql   0/1       ContainerCreating   0         0s
directvolumemigration-rsync-transfer-mysql   1/1       Running   0         19s

Comment 11 errata-xmlrpc 2021-02-11 12:55:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Migration Toolkit for Containers (MTC) tool image release advisory 1.4.0), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5329