Bug 1274652 - Build doesn't fail if pending time exceeds completionDeadlineSeconds limitation
Build doesn't fail if pending time exceeds completionDeadlineSeconds limitation
Status: CLOSED CURRENTRELEASE
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build (Show other bugs)
3.1.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Maciej Szulik
Wenjing Zheng
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-23 05:33 EDT by Qixuan Wang
Modified: 2015-11-23 09:24 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-23 09:24:34 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Qixuan Wang 2015-10-23 05:33:05 EDT
Description of problem:
Set completionDeadlineSeconds to 5s in buildconfig part and use an invalid image as builder image, then start build with this buildconfig, build doesn't fail if pending time exceeds completionDeadlineSeconds limitation.


Version-Release number of selected component (if applicable):
OSE

How reproducible:
Always

Steps to Reproduce:
1. Set completionDeadlineSeconds to 5s in buildconfig part and use an invalid image as builder image
# vi sourcebuildconfig.json
{
  "kind": "List",
  "apiVersion": "v1",
  "metadata": {},
  "items": [
    {
      "kind": "ImageStream",
      "apiVersion": "v1",
      "metadata": {
        "name": "origin-php-sample"
      },
      "spec": {},
      "status": {
        "dockerImageRepository": ""
      }
    },
    {
      "kind": "BuildConfig",
      "apiVersion": "v1",
      "metadata": {
        "name": "source-build"
      },
      "spec": {
        "completionDeadlineSeconds": 5,
        "triggers": [],
        "source": {
          "type": "Git",
          "git": {
            "uri": "https://github.com/wzheng1/pending-pod-php-rhel7"
          }
        },
        "strategy": {
          "type": "Source",
          "sourceStrategy": {
            "from": {
              "kind": "ImageStreamTag",
              "name": "php:5.5",
              "namespace": "openshift"
            },
          "pullSecret": {"name" : "pull"}
          }
        }
      }
    }
  ]
}

# oc create -f sourcebuildconfig.json

2. Start build with the buildconfig
# oc start-build source-build

3. Describe buildconfig
# oc describe bc source-build

4. Check build/pod status
# oc get pod
# oc get build


Actual results:
3. [root@localhost ~]# oc describe bc source-build
Name:			source-build
Created:		20 minutes ago
Labels:			<none>
Latest Version:		1
Strategy:		Source
Source Type:		Git
URL:			https://github.com/wzheng1/pending-pod-php-rhel7
From Image:		ImageStreamTag openshift/php:5.5
Pull Secret Name:	pull
Fail Build After:	5s
Triggered by:		<none>

Build		Status		Duration		Creation Time
source-build-1 	pending 	waiting for 17m57s 	2015-10-23 17:04:56 +0800 CST

4. [root@localhost ~]# oc get pod -o wide
NAME                             READY     STATUS             RESTARTS   AGE       NODE
source-build-1-build             0/1       Pending            0          17m       openshift-159.lab.eng.nay.redhat.com


Expected results:
Since completionDeadlineSeconds set to 5s, build should fail if pending time exceeds 5s


Additional info:
Comment 2 Qixuan Wang 2015-10-23 05:40:05 EDT
Sorry, Version-Release number should be:
openshift v3.0.2.902
kubernetes v1.2.0-alpha.1-1107-g4c8e6f4
etcd 2.1.2
Comment 3 Maciej Szulik 2015-10-26 10:33:50 EDT
Tested on both origin master and aforementioned ose version. In both cases the build failed for me after ~9-10 seconds. Please re-test.

# oc describe bc/source-build
Name:			source-build
Created:		10 minutes ago
Labels:			<none>
Latest Version:		1
Strategy:		Source
Source Type:		Git
URL:			https://github.com/wzheng1/pending-pod-php-rhel7
From Image:		ImageStreamTag openshift/php:5.5
Pull Secret Name:	pull
Fail Build After:	5s
Triggered by:		<none>

Build		Status	Duration	Creation Time
source-build-1 	failed 	waited for 10s 	2015-10-26 10:27:54 -0400 EDT
Comment 4 Maciej Szulik 2015-10-26 10:44:16 EDT
Sorry, one slight difference, it was tested against latest ose master.
Comment 5 Qixuan Wang 2015-10-27 05:41:52 EDT
Tested on the latest OSE(openshift v3.0.2.903-29-g49953d6, kubernetes v1.2.0-alpha.1-1107-g4c8e6f4, etcd 2.1.2), the bug can't be reproduced now.

Here is the test result:
[root@localhost testv3]# oc describe bc source-build
Name:			source-build
Created:		38 seconds ago
Labels:			<none>
Latest Version:		1
Strategy:		Source
Source Type:		Git
URL:			https://github.com/wzheng1/pending-pod-php-rhel7
From Image:		ImageStreamTag openshift/php:5.5
Pull Secret Name:	pull
Fail Build After:	5s
Triggered by:		<none>

Build		Status	Duration	Creation Time
source-build-1 	failed 	waited for 10s 	2015-10-27 17:31:37 +0800 CST


[root@localhost testv3]# oc get pod
NAME                   READY     STATUS             RESTARTS   AGE
source-build-1-build   0/1       DeadlineExceeded   0          39s
[root@localhost testv3]# oc get build
NAME             TYPE      FROM      STATUS    STARTED   DURATION
source-build-1   Source    Git       Failed
Comment 6 chaoyang 2015-11-04 00:19:52 EST
this is happened on the 
oc v3.0.2.903-114-g2849767
kubernetes v1.2.0-alpha.1-1107-g4c8e6f4

[root@dhcp-128-8 ~]# oc describe bc source-build
Name:			source-build
Created:		15 hours ago
Labels:			<none>
Latest Version:		1
Strategy:		Source
Image Reference:	ImageStreamTag openshift/php:5.5
Pull Secret Name:	pull
Source Type:		Git
URL:			https://github.com/wzheng1/pending-pod-php-rhel7
Builds:
  Name			Status		Duration		Creation Time
  source-build-1 	pending 	waiting for 15h26m27s 	2015-11-03 18:12:02 +0800 CST
Comment 7 Maciej Szulik 2015-11-04 07:44:54 EST
I don't see the timeout being set, iow. this line:

Fail Build After:	5s

without it, your build will never fail. I've just tested it against latest origin master and got following result:

$ oc version
oc v1.0.7-257-g9a1d442
kubernetes v1.2.0-alpha.1-1107-g4c8e6f4


$ oc describe bc/source-build
Name:			source-build
Created:		About a minute ago
Labels:			<none>
Latest Version:		1
Strategy:		Source
Source Type:		Git
URL:			https://github.com/wzheng1/pending-pod-php-rhel7
From Image:		ImageStreamTag openshift/php:5.5
Pull Secret Name:	pull
Fail Build After:	5s
Triggered by:		<none>

Build		Status	Duration	Creation Time
source-build-1 	failed 	waited for 10s 	2015-11-04 12:43:20 +0000 UTC
Comment 8 chaoyang 2015-11-05 05:41:21 EST
I re-run this on the 
oc v3.1.0.0
kubernetes v1.2.0-alpha.1-1107-g4c8e6f4

it is ok.
Name:			source-build
Created:		25 seconds ago
Labels:			<none>
Latest Version:		1
Strategy:		Source
Source Type:		Git
URL:			https://github.com/wzheng1/pending-pod-php-rhel7
From Image:		ImageStreamTag openshift/php:5.5
Pull Secret Name:	pull
Fail Build After:	5s
Triggered by:		<none>

Build		Status	Duration	Creation Time
source-build-1 	failed 	waited for 8s 	2015-11-05 18:37:58 +0800 CST
Comment 9 Brenton Leanhardt 2015-11-23 09:24:34 EST
This fix is available in OpenShift Enterprise 3.1.

Note You need to log in before you can comment on or make changes to this bug.