Bug 1774492

Summary: Incremental build fails during image pull when forcePull is "true"
Product: OpenShift Container Platform Reporter: David Simansky <dsimansk>
Component: BuildAssignee: Gabe Montero <gmontero>
Status: CLOSED ERRATA QA Contact: wewang <wewang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: adam.kaplan, aos-bugs, gmontero, wzheng
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: setting both incremental and forcePull flags to true on a build strategy could result in builds using push image credentials to pull images Consequence: image pulls from private registries would fail Fix: the build image now properly manages registry push and pull credentials when both incremental and forcePull are set to true Result: both pushing and pulling image during a build works when both incremental and forcePull are set to true
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-04 11:16:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Failed build log none

Description David Simansky 2019-11-20 12:10:52 UTC 
Created attachment 1638119 [details]
Failed build log

Description of problem:

This might be related to fix needed for https://bugzilla.redhat.com/show_bug.cgi?id=1771335.

In addition this is likely a corner case of running incremental S2I build.

We have "incrementalRebuild" test, that defines it's own BuildConfig with "forcePull:true" property and from as direct DockerImage reference. 

The test is passing on 4.2.x.
On 4.3.0 nighly it started to fail on image pull from internal registry when base image from *authenticated registry* is used.

I was also able to reproduce this behavior also on ImageStream source when "forcePull:true" is added, see the steps below.

From my tests so far it's *only* reproducible with "forcePull:true" property.


```
apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  creationTimestamp: 2019-11-20T11:34:52Z
  name: sti
  namespace: dsimansk
  resourceVersion: "67319"
  selfLink: /apis/build.openshift.io/v1/namespaces/dsimansk/buildconfigs/sti
  uid: 71e8a5ff-15da-4091-a01a-741a680441af
spec:
  failedBuildsHistoryLimit: 5
  nodeSelector: {}
  output:
    to:
      kind: ImageStreamTag
      name: sti:latest
  postCommit: {}
  resources: {}
  runPolicy: Serial
  source:
    binary: {}
    type: Binary
  strategy:
    sourceStrategy:
      forcePull: true
      from:
        kind: DockerImage
        name: registry.redhat.io/openjdk/openjdk-11-rhel8:latest
      incremental: true
    type: Source
  successfulBuildsHistoryLimit: 5
  triggers:
  - generic:
      allowEnv: true
      secret: secret101
    type: Generic
status:
  lastVersion: 2

```


Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-11-19-053808

How reproducible:
Always

Steps to Reproduce:
1. Create pull secret for registry.redhat.io
2. Link them to builder and deployer SAs: 
https://docs.openshift.com/container-platform/4.2/openshift_images/managing-images/using-image-pull-secrets.html#images-allow-pods-to-reference-images-from-secure-registries_using-image-pull-secrets
3. oc new-app registry.redhat.io/openjdk/openjdk-11-rhel8:latest~https://github.com/openshift-vertx-examples/vertx-http-example.git
  - image from authenticated registry must be used
4. Add "forcePull: true" to BC vertx-http-example
  - property must be set to true
5. oc start-build --incremental -e BUILD_LOGLEVEL=8 vertx-http-example

Actual results:
Build fails.

Expected results:
Build is successfully completed.

Additional info:
Comment 1 Adam Kaplan 2019-11-20 16:38:47 UTC 
Think I see the bug here. Using `from` with an incremental s2i build is not necessary - incremental s2i builds pull from the output image.

However it appears that with `forcePull: true`, we are not setting the authentication for the true "base" image (the internal registry).
Comment 2 Adam Kaplan 2019-11-21 14:37:08 UTC 
Moving target to 4.4.0, as the proposed fix warrants more discussion.
Comment 3 Gabe Montero 2019-11-21 16:12:01 UTC 
10-4 re: 4.4

fwiw I was able to execute the test case running a version of the openshift builder image with the changes from https://github.com/openshift/builder/pull/116

1) used BC of 

apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  name: sti
spec:
  output:
    to:
      kind: ImageStreamTag
      name: sti:latest
  postCommit: {}
  resources: {}
  runPolicy: Serial
  source:
    binary: {}
    type: Binary
  strategy:
    sourceStrategy:
      forcePull: true
      from:
        kind: DockerImage
        name: registry.redhat.io/openjdk/openjdk-11-rhel8:latest
      incremental: true
    type: Source

2) created a secret for my registry.redhat.io credentials
3) linked the secret to the default and builder SAs
4) ran the build from my local clone of the openshift jenkins plugin maven/java based repo (as the binary build input)
Comment 5 wewang 2020-02-03 07:18:06 UTC 
Verified in version:
4.4.0-0.nightly-2020-02-02-225006

Steps:
1. Create a bc with follow content:
      sourceStrategy:
        forcePull: true
        from:
          kind: DockerImage
          name: registry.redhat.io/openjdk/openjdk-11-rhel8:latest
        incremental: true

2. Create pull secret for registry.redhat.io
3. Link builder to secret
4. Set pull build-secret 
5. Start a build
vertx-http-example-3   Source   Git@71ef489   Complete                      8 minutes ago    4m10s
6. Check build log
[root@wangwen ~]# oc logs -f build/vertx-http-example-3
Cloning "https://github.com/openshift-vertx-examples/vertx-http-example.git" ...
	Commit:	71ef48965c062a04a5bdaac9b1641ccc9e6034a0 (Merge pull request #91 from openshift-vertx-examples/dependabot/maven/io.fabri..)
	Author:	Rodney Russ <rdruss>
	Date:	Fri Sep 20 11:00:47 2019 -0600
Caching blobs under "/var/cache/blobs".
Getting image source signatures
Copying blob sha256:0e8ea260d0262eac3725175d3d499ead6fd77cb1fa8272b3e665e8f64044fb89
Copying blob sha256:340ff6d7f58c908c438ce89bb845caee3649de828a81569317ed0fe169a97de2
Copying blob sha256:9839973fd99c42bc282e1814bcbbca41770f977a30e44fcb29fb09e13a67ce8a
Copying config sha256:1ccd33f066a61debc231fd56f3cfa0b5a096f02d3273f0e60ed711728012607a
Writing manifest to image destination
Storing signatures
Getting image source signatures
Copying blob sha256:0e8ea260d0262eac3725175d3d499ead6fd77cb1fa8272b3e665e8f64044fb89
Copying blob sha256:340ff6d7f58c908c438ce89bb845caee3649de828a81569317ed0fe169a97de2
Copying blob sha256:9839973fd99c42bc282e1814bcbbca41770f977a30e44fcb29fb09e13a67ce8a
Copying blob sha256:d649e8c0ed19c70ab4b642ff46bccdba048dac3cb9ac9f3a868d4cd68a72082d
Copying config sha256:4321e1a2bfb3ee7bf52ac6826cb8cec4151192c14810f80baff47076813fcf40
Writing manifest to image destination
Storing signatures
Generating dockerfile with builder image registry.redhat.io/openjdk/openjdk-11-rhel8:latest
STEP 1: FROM image-registry.openshift-image-registry.svc:5000/wewang/vertx-http-example:latest AS cached
Comment 7 errata-xmlrpc 2020-05-04 11:16:09 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581