Bug 1767218

Summary: Builds Without BuildConfigs Cause Build Controller to Have Errant Behavior
Product: OpenShift Container Platform Reporter: Steve Kuznetsov <skuznets>
Component: BuildAssignee: Gabe Montero <gmontero>
Status: CLOSED ERRATA QA Contact: wewang <wewang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: aos-bugs, gmontero, wzheng
Target Milestone: ---   
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The build controller sometimes incorrectly assumed a build was instantiated from the build config endpoint when in fact it was instantiated directly from the build endpoint. Consequence: Confusing logging about non-existent build configs could appear in the build controller logs if a user instantiated a openshift build diretly (vs. initiating a build request off of the build config api endpoint) Fix: The build controller was update to better check whether a build was instantiated from the build config endpoint and refrain from loggin unnecessary error messages. Result: The build controller logs no longer have these confusing error messages for build instantiated directly vs. from the build config endpoint.
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-23 11:10:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Steve Kuznetsov 2019-10-30 22:39:48 UTC
Description of problem:

If you create a Build without a BuildConfig in a Namespace, the build controller will emit the following logs into the controller-manager log stream:

I1030 18:41:13.262823       1 build_controller.go:1399] Giving up retrying ci-op-vyzlc7c2/: build config ci-op-vyzlc7c2/ has no builds to run next


This is incorrectly trying to process a BuildConfig with name ""

Comment 1 Adam Kaplan 2019-11-01 20:40:14 UTC
Per discussion with Steve on slack, this isn't a high priority bug, but did interfere with debugging recent incidents on our CI cluster.

Targeting 4.4 as this is an existing 3.11 issue that shouldn't block the 4.3 release.

Comment 3 Gabe Montero 2019-11-20 20:58:51 UTC
Testing for this would include
- creating a valid build object directly in a namespace vs. creating a build config and running oc start-build
- see the build complete
- look at the logs for the 3 pods in the openshift-controller-manager namespace and make sure there are no build_controller logs with "Giving up retrying"

Comment 4 wewang 2019-11-21 07:48:36 UTC
It can be reproduced, wait latest payload which is include fixed pr, will verified it

[root@Desktop ~]# oc logs -f pod/controller-manager-w6drh -n openshift-controller-manager |grep -i "Giving up retrying"
I1121 05:54:11.598633       1 build_controller.go:1754] Giving up retrying wewang/pullsecret-ruby: build config wewang/pullsecret-ruby has no builds to run next
I1121 06:32:12.619971       1 build_controller.go:1754] Giving up retrying default/pullsecret-ruby: build config default/pullsecret-ruby has no builds to run next
I1121 06:39:49.952976       1 build_controller.go:1754] Giving up retrying default/pullsecret-ruby: build config default/pullsecret-ruby has no builds to run next
I1121 07:06:34.628087       1 build_controller.go:1754] Giving up retrying default/ruby-ex: build config default/ruby-ex has no builds to run next
I1121 07:29:19.976816       1 build_controller.go:1754] Giving up retrying default/ruby-ex: build config default/ruby-ex has no builds to run next

Comment 5 wewang 2019-11-21 08:21:16 UTC
@gabe I have a question, how do we create bc without in any namespace, in my experience, create a bc which should be in a namespace, for example default namespace or projects I created?

Comment 6 Gabe Montero 2019-11-21 14:38:25 UTC
@wen you don't have to create a BC without a namespace to verify this bug

you create builds without buildconfigs

Comment 7 wewang 2019-11-22 08:34:45 UTC
@Gabe Thanks, I should improve my english, haha~~, verified in latest payload which is include the pr.

Version:
4.3.0-0.nightly-2019-11-21-122827

Steps:
 1. Create a build with buildconfig in my project
  $oc create -f build.yaml
  ----------build.yaml---------------
  apiVersion: v1
  kind: Build
  metadata:
    labels:
    name: build-without-bc-1
  spec:
    resources: {}
    source:
      git:
        uri: git://github.com/openshift/ruby-hello-world
      type: Git
    strategy:
      sourceStrategy:
        from:
          kind: ImageStreamTag
          name: ruby:2.5
          namespace: openshift
      type: Source
   -------------------------------------

2. Create 6 builds 
   $oc start-build --from-build=build-without-bc-1
   build-without-bc-1              Source   Git@7e2b285   Complete                       5 minutes ago        59s
   build-without-bc-1-7514139911   Source   Git@7e2b285   Complete                       4 minutes ago        1m18s
   build-without-bc-1-2939714793   Source   Git@7e2b285   Complete                       4 minutes ago        1m4s
   build-without-bc-1-6971280066   Source   Git@7e2b285   Complete                       4 minutes ago        1m7s
   build-without-bc-1-8970521568   Source   Git@7e2b285   Complete                       4 minutes ago        1m16s
   build-without-bc-1-0382288048   Source   Git@7e2b285   Complete                       3 minutes ago        54s

3. Check the the 3 pods log in openshift-controller-manager, no info about "Giving up retrying"

Comment 9 errata-xmlrpc 2020-01-23 11:10:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062