Bug 1752883

Summary: Problems with push secret when using external images
Product: OpenShift Container Platform Reporter: Sergio <sregidor>
Component: Migration ToolingAssignee: Scott Seago <sseago>
Status: CLOSED ERRATA QA Contact: Sergio <sregidor>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.2.0CC: chezhang, rpattath, xjiang
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 06:41:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
original build yaml, velero logs and template
none
buildconfig yaml none

Description Sergio 2019-09-17 13:30:59 UTC
Created attachment 1615850 [details]
original build yaml, velero logs and template

Description of problem:
When a build is migrated and this build references an externa image, the builder secret is not handled properly by the velero plugin


Version-Release number of selected component (if applicable):

OCP4:
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.2.0-0.nightly-2019-09-15-221449   True        False         12h     Error while reconciling 4.2.0-0.nightly-2019-09-15-221449: the cluster operator ingress is degraded

OCP3:
$ oc version
oc v3.11.126
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO
Server https://
openshift v3.9.99
kubernetes v1.9.1+a0ce1bc657

Controller:
    image: quay.io/ocpmigrate/mig-controller:release-1.0
    imageID: quay.io/ocpmigrate/mig-controller@sha256:0f74db7171712ffc440b3d7b0f02a775ccd71238827ec856b7d090f90f2feffb
Velero:
    image: quay.io/ocpmigrate/velero:fusor-1.1
    imageID: quay.io/ocpmigrate/velero@sha256:6c16a1288bf6aca74afbb0184fa987506839c5193ae8bb2be05cb6aa0a9f3dc5
    image: quay.io/ocpmigrate/migration-plugin:release-1.0
    imageID: quay.io/ocpmigrate/migration-plugin@sha256:eb9b82c3f26bcd876bc501e18dde7cffe7e451c8c8a231959ed4d9f1127b91a6


How reproducible:
Always

Steps to Reproduce:
1. oc new-project django
2. oc create -f https://raw.githubusercontent.com/sergiordlr/temp-testfiles/master/app_migration/django-3.9/django_template.yml
3. oc new-app --template django-psql-persistent-modified

  the build is ok:
  $ oc get build -n django
NAME                       TYPE      FROM          STATUS     STARTED        DURATION
django-psql-persistent-1   Source    Git@0905223   Complete   23 hours ago   34s


4. Migrate the namespace

Actual results:
After the migration ends, in OCP4 the build complains about the push secret.
$ oc get build
NAME                       TYPE     FROM          STATUS                        STARTED   DURATION
django-psql-persistent-1   Source   Git@0905223   Pending (MissingPushSecret)        


Expected results:
There should be no problem after the migration.


Additional info:

Velero logs are attached, as well as the template and the original build yaml in ocp3.

We can read this in velero logs:
time="2019-09-16T14:48:32Z" level=error msg="[build-restore] build is not from internal image, skipping" cmd=/plugins/velero-plugins logSource="/go/src/github.com/fusor/openshift-migration-plugin/vendor/github.com/fusor/openshift-velero-plugin/velero-plugins/build/restore.go:60" pluginName=velero-plugins restore=openshift-migration/tpl-migration-cf1jq0jk-xclvf

The build is still pointing to the old ocp secret, as it can be seen in the following commands executed in OCP4.

-bash-4.1$ oc project django
Now using project "django" on server "https://XXXXXXXXXXXX:YYY".
-bash-4.1$ oc get secret
NAME                       TYPE                                  DATA   AGE
builder-dockercfg-qnhl4    kubernetes.io/dockercfg               1      27m
builder-token-grv5c        kubernetes.io/service-account-token   4      27m
builder-token-v76wq        kubernetes.io/service-account-token   4      27m
default-dockercfg-mhfkm    kubernetes.io/dockercfg               1      27m
default-token-d4jqv        kubernetes.io/service-account-token   4      27m
default-token-k8j6b        kubernetes.io/service-account-token   4      27m
deployer-dockercfg-jmp4m   kubernetes.io/dockercfg               1      27m
deployer-token-6xtnp       kubernetes.io/service-account-token   4      27m
deployer-token-hjnd2       kubernetes.io/service-account-token   4      27m
django-psql-persistent     Opaque                                3      27m

-bash-4.1$ oc get sa builder -o yaml | grep secret -A 4
secrets:
- name: builder-dockercfg-qnhl4
- name: builder-token-v76wq
-bash-4.1$ oc get build
NAME                       TYPE     FROM          STATUS                        STARTED   DURATION
django-psql-persistent-1   Source   Git@0905223   Pending (MissingPushSecret)             
-bash-4.1$ oc describe build | grep -i secret
Status:                Pending (Missing push secret.)
Push Secret:                builder-dockercfg-rjhnc
  Warning        FailedMount        91s (x21 over 28m)        kubelet, compute-0        MountVolume.SetUp failed for volume "builder-dockercfg-rjhnc-push" : secrets "builder-dockercfg-rjhnc" not found

Comment 1 Sergio 2019-09-17 14:41:10 UTC
Created attachment 1615879 [details]
buildconfig yaml

Comment 2 Scott Seago 2019-09-23 20:14:13 UTC
Fix is in this PR: https://github.com/fusor/openshift-migration-plugin/pull/27

This fix is also in the most recent migration-plugin stable build

Comment 4 Sergio 2019-09-25 07:56:23 UTC
Verified in:

Controller:
      image: quay.io/ocpmigrate/mig-controller:release-1.0
      imageID: quay.io/ocpmigrate/mig-controller@sha256:b9e78beef9f9c9d36dacb84d552ec0c7ce09fea556293d6fbec8c90c11f70cb7
Velero:
      image: quay.io/ocpmigrate/velero:fusor-1.1
      imageID: quay.io/ocpmigrate/velero@sha256:6c16a1288bf6aca74afbb0184fa987506839c5193ae8bb2be05cb6aa0a9f3dc5
      image: quay.io/ocpmigrate/migration-plugin:release-1.0
      imageID: quay.io/ocpmigrate/migration-plugin@sha256:f917ce774c08b3ea4e4d5c422521230d8fb8e03bc13ac0b9a323cbd4599cd341

OCP3
# oc version
oc v3.9.97
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://
openshift v3.9.97
kubernetes v1.9.1+a0ce1bc657

OCP4
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.2.0-0.nightly-2019-09-24-025718   True        False         10h     Cluster version is 4.2.0-0.nightly-2019-09-24-025718



In source OCP3 cluster, before migration this was the secret in the build:
# oc get build -o yaml | grep -i pushSecret -A 1
      pushSecret:
        name: builder-dockercfg-mnwtw

In target OCP4, after the migration, the secret was properly handled and updated and the build could run with no failures:
$ oc get build
NAME                       TYPE     FROM          STATUS     STARTED       DURATION
django-psql-persistent-1   Source   Git@0905223   Complete   2 hours ago   1m14s

$ oc get build -o yaml | grep -i pushSecret -A 1
      pushSecret:
        name: builder-dockercfg-9mgdx

Comment 5 errata-xmlrpc 2019-10-16 06:41:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922