Bug 1418906 - More than 1000+ complete builds makes new build start slow with New status or hang
Summary: More than 1000+ complete builds makes new build start slow with New status or...
Keywords:
Status: CLOSED DUPLICATE of bug 1400132
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.3.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Cesar Wong
QA Contact: Wang Haoran
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-03 05:14 UTC by Kenjiro Nakayama
Modified: 2020-03-11 15:43 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-06 13:01:08 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1400132 0 unspecified CLOSED Node reports as Ready, but it takes about 15 mins to schedule a build 2021-02-22 00:41:40 UTC
Red Hat Knowledge Base (Solution) 2902871 0 None None None 2017-02-03 05:31:36 UTC

Internal Links: 1400132

Description Kenjiro Nakayama 2017-02-03 05:14:10 UTC
Description of problem:
---
- If there are more than 1000+ "Complete" status builds, new build starts slow or sometimes hang.

Version-Release number of selected component (if applicable):
---
- OCP 3.3.1

How reproducible:
---
- Create "Complete" status builds more than 1000+ and start another new build.

Steps to Reproduce:
---
1. Create Completed build more than 1000+. Please see test.sh[1] and sti-php.yaml[2] below.

  $ ls
  sti-php.yaml  test.sh

sti-php.yaml ... "complete" status build resource
test.sh      ... create 1000 "complete" status build.

2. Create 1000 builds

  $ bash test.sh

3. Check the number of builds

  $ oc get build | wc -l
  1005

4. start another new build (e.g. ruby sti)

  $ oc new-app https://github.com/openshift/sti-ruby.git --context-dir=2.0/test/puma-test-app -l app=ruby --strategy=source

5. Check build status

  $ oc get build -w
  ... snip ...
  sti-php-1000             Source    Git@8c03432   Complete   7 days ago   2m37s
  sti-php-999              Source    Git@8c03432   Complete   7 days ago   2m37s
  sti-ruby-1               Source    Git           New

Actual results:
---
- In step-5, new build sti-ruby-1 doesn't start (or taking long time to start) with New status.

Expected results:
---
- New build start.

Additional info:
---
- After cleaned up*1 the builds, new build start working fine.

*1. for example...
  # oadm prune builds --orphans --keep-complete=5 --keep-failed=1     --keep-younger-than=60m --confirm
or
  $ oc delete build <BUILD>


[1] test.sh
~~~
#!/bin/bash

mkdir ./tmp
for i in {1..1000}; do
  cat ./sti-php.yaml | sed -e "s/sti-php-N/sti-php-${i}/g" > ./tmp/sti-php-${i}.yaml
  oc create -f ./tmp/sti-php-${i}.yaml
done
~~~

[2] sti-php.yaml
~~~
apiVersion: v1
kind: Build
metadata:
  annotations:
    openshift.io/build-config.name: sti-php
    openshift.io/build.number: "1"
    openshift.io/build.pod-name: sti-php-1-build
  creationTimestamp: 2017-01-26T07:52:28Z
  labels:
    test: bug
    app: php
    buildconfig: sti-php
    openshift.io/build-config.name: sti-php
    openshift.io/build.start-policy: Serial
  name: sti-php-N
  namespace: s1
  resourceVersion: "713512"
  selfLink: /oapi/v1/namespaces/s1/builds/sti-php-1
  uid: 622be909-e39c-11e6-af24-001a4a40dc69
spec:
  output:
    pushSecret:
      name: builder-dockercfg-jm2ls
    to:
      kind: ImageStreamTag
      name: sti-php:latest
  postCommit: {}
  resources: {}
  revision:
    git:
      author:
        email: bparees@users.noreply.github.com
        name: Ben Parees
      commit: 8c034322d850cbed99a60db1926fba49444f732a
      committer:
        email: noreply@github.com
        name: GitHub
      message: 'Merge pull request #146 from andrewklau/patch-1'
    type: Git
  serviceAccount: builder
  source:
    contextDir: 5.5/test/test-app
    git:
      uri: https://github.com/openshift/sti-php.git
    type: Git
  strategy:
    sourceStrategy:
      from:
        kind: DockerImage
        name: registry.access.redhat.com/rhscl/php-56-rhel7@sha256:9cfa0ccfa3560cee94959508f250ad12edddea15ecc2422b9687bf6d645d8d21
    type: Source
  triggeredBy:
  - message: Build configuration change
status:
  completionTimestamp: 2017-01-26T07:55:15Z
  config:
    kind: BuildConfig
    name: sti-php
    namespace: s1
  duration: 157000000000
  phase: Complete
  startTimestamp: 2017-01-26T07:52:38Z
~~~

Comment 5 Ryan Howe 2017-02-07 14:55:40 UTC

*** This bug has been marked as a duplicate of bug 1400132 ***

Comment 6 Ryan Howe 2017-02-07 14:59:16 UTC
Fixed via errata: 

https://access.redhat.com/errata/RHBA-2017:0218


Note You need to log in before you can comment on or make changes to this bug.