Bug 1767218 - Builds Without BuildConfigs Cause Build Controller to Have Errant Behavior
Summary: Builds Without BuildConfigs Cause Build Controller to Have Errant Behavior
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.3.0
Assignee: Gabe Montero
QA Contact: wewang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-10-30 22:39 UTC by Steve Kuznetsov
Modified: 2020-01-23 11:10 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The build controller sometimes incorrectly assumed a build was instantiated from the build config endpoint when in fact it was instantiated directly from the build endpoint. Consequence: Confusing logging about non-existent build configs could appear in the build controller logs if a user instantiated a openshift build diretly (vs. initiating a build request off of the build config api endpoint) Fix: The build controller was update to better check whether a build was instantiated from the build config endpoint and refrain from loggin unnecessary error messages. Result: The build controller logs no longer have these confusing error messages for build instantiated directly vs. from the build config endpoint.
Clone Of:
Environment:
Last Closed: 2020-01-23 11:10:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openshift-controller-manager pull 48 0 'None' 'open' 'Bug 1767218: Bug 1767219: don''t enqueue non-existent buildconfigs for builds' 2019-11-20 15:11:23 UTC
Red Hat Product Errata RHBA-2020:0062 0 None None None 2020-01-23 11:10:30 UTC

Description Steve Kuznetsov 2019-10-30 22:39:48 UTC
Description of problem:

If you create a Build without a BuildConfig in a Namespace, the build controller will emit the following logs into the controller-manager log stream:

I1030 18:41:13.262823       1 build_controller.go:1399] Giving up retrying ci-op-vyzlc7c2/: build config ci-op-vyzlc7c2/ has no builds to run next


This is incorrectly trying to process a BuildConfig with name ""

Comment 1 Adam Kaplan 2019-11-01 20:40:14 UTC
Per discussion with Steve on slack, this isn't a high priority bug, but did interfere with debugging recent incidents on our CI cluster.

Targeting 4.4 as this is an existing 3.11 issue that shouldn't block the 4.3 release.

Comment 3 Gabe Montero 2019-11-20 20:58:51 UTC
Testing for this would include
- creating a valid build object directly in a namespace vs. creating a build config and running oc start-build
- see the build complete
- look at the logs for the 3 pods in the openshift-controller-manager namespace and make sure there are no build_controller logs with "Giving up retrying"

Comment 4 wewang 2019-11-21 07:48:36 UTC
It can be reproduced, wait latest payload which is include fixed pr, will verified it

[root@Desktop ~]# oc logs -f pod/controller-manager-w6drh -n openshift-controller-manager |grep -i "Giving up retrying"
I1121 05:54:11.598633       1 build_controller.go:1754] Giving up retrying wewang/pullsecret-ruby: build config wewang/pullsecret-ruby has no builds to run next
I1121 06:32:12.619971       1 build_controller.go:1754] Giving up retrying default/pullsecret-ruby: build config default/pullsecret-ruby has no builds to run next
I1121 06:39:49.952976       1 build_controller.go:1754] Giving up retrying default/pullsecret-ruby: build config default/pullsecret-ruby has no builds to run next
I1121 07:06:34.628087       1 build_controller.go:1754] Giving up retrying default/ruby-ex: build config default/ruby-ex has no builds to run next
I1121 07:29:19.976816       1 build_controller.go:1754] Giving up retrying default/ruby-ex: build config default/ruby-ex has no builds to run next

Comment 5 wewang 2019-11-21 08:21:16 UTC
@gabe I have a question, how do we create bc without in any namespace, in my experience, create a bc which should be in a namespace, for example default namespace or projects I created?

Comment 6 Gabe Montero 2019-11-21 14:38:25 UTC
@wen you don't have to create a BC without a namespace to verify this bug

you create builds without buildconfigs

Comment 7 wewang 2019-11-22 08:34:45 UTC
@Gabe Thanks, I should improve my english, haha~~, verified in latest payload which is include the pr.

Version:
4.3.0-0.nightly-2019-11-21-122827

Steps:
 1. Create a build with buildconfig in my project
  $oc create -f build.yaml
  ----------build.yaml---------------
  apiVersion: v1
  kind: Build
  metadata:
    labels:
    name: build-without-bc-1
  spec:
    resources: {}
    source:
      git:
        uri: git://github.com/openshift/ruby-hello-world
      type: Git
    strategy:
      sourceStrategy:
        from:
          kind: ImageStreamTag
          name: ruby:2.5
          namespace: openshift
      type: Source
   -------------------------------------

2. Create 6 builds 
   $oc start-build --from-build=build-without-bc-1
   build-without-bc-1              Source   Git@7e2b285   Complete                       5 minutes ago        59s
   build-without-bc-1-7514139911   Source   Git@7e2b285   Complete                       4 minutes ago        1m18s
   build-without-bc-1-2939714793   Source   Git@7e2b285   Complete                       4 minutes ago        1m4s
   build-without-bc-1-6971280066   Source   Git@7e2b285   Complete                       4 minutes ago        1m7s
   build-without-bc-1-8970521568   Source   Git@7e2b285   Complete                       4 minutes ago        1m16s
   build-without-bc-1-0382288048   Source   Git@7e2b285   Complete                       3 minutes ago        54s

3. Check the the 3 pods log in openshift-controller-manager, no info about "Giving up retrying"

Comment 9 errata-xmlrpc 2020-01-23 11:10:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062


Note You need to log in before you can comment on or make changes to this bug.