Bug 1549832

Summary:

CRI-O runtime - cannot delete builds.build.openshift.io in the namespace

Product:

OpenShift Container Platform

Reporter:

Vikas Laad <vlaad>

Component:

Build

Assignee:

Corey Daley <cdaley>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Vikas Laad <vlaad>

Severity:

medium

Docs Contact:

Priority:

unspecified

Version:

3.9.0

CC:

aos-bugs, bparees, smunilla, xtian

Target Milestone:

---

Target Release:

3.9.0

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

openshift v3.9.3

Doc Type:

Bug Fix

Doc Text:

Cause: Service account did not have permission to delete builds Consequence: Builds were not being pruned correctly Fix: Added correct permission to service account Result: Builds are now pruned correctly

Story Points:

---

Clone Of:

Environment:

Last Closed:

2018-06-18 17:44:56 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
master logs	none

Description Vikas Laad 2018-02-27 20:49:29 UTC

Description of problem:
I see lot of these errors in master logs. I am noticing it as part of Reliability tests run for 3.9. Reliability tests use quickstart apps to create/delete/build/scale apps over a period of time.

Feb 27 17:10:24 ip-172-31-58-173.us-west-2.compute.internal atomic-openshift-master-controllers[15884]: E0227 17:10:24.673624   15884 buildconfig_controller.go:103] builds.build.openshift.io "dancer-mysql-example-2" is forbidden: User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in the namespace "dancer-mysql-u-201-72-335-124": User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in project "dancer-mysql-u-201-72-335-124"
Feb 27 17:12:32 ip-172-31-58-173.us-west-2.compute.internal atomic-openshift-master-controllers[15884]: E0227 17:12:32.715105   15884 buildconfig_controller.go:103] builds.build.openshift.io "dancer-mysql-example-2" is forbidden: User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in the namespace "dancer-mysql-u-201-72-335-124": User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in project "dancer-mysql-u-201-72-335-124"
Feb 27 17:48:45 ip-172-31-58-173.us-west-2.compute.internal atomic-openshift-master-controllers[15884]: E0227 17:48:45.812548   15884 buildconfig_controller.go:103] builds.build.openshift.io "django-psql-example-1" is forbidden: User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in the namespace "django-psql-u-234-83-368-135": User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in project "django-psql-u-234-83-368-135"
Feb 27 17:48:45 ip-172-31-58-173.us-west-2.compute.internal atomic-openshift-master-controllers[15884]: E0227 17:48:45.812643   15884 buildconfig_controller.go:103] builds.build.openshift.io "dancer-mysql-example-2" is forbidden: User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in the namespace "dancer-mysql-u-49-18-351-130": User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in project "dancer-mysql-u-49-18-351-130"
Feb 27 17:58:45 ip-172-31-58-173.us-west-2.compute.internal atomic-openshift-master-controllers[15884]: E0227 17:58:45.812275   15884 buildconfig_controller.go:103] builds.build.openshift.io "dancer-mysql-example-2" is forbidden: User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in the namespace "dancer-mysql-u-49-18-351-130": User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in project "dancer-mysql-u-49-18-351-130"
Feb 27 17:58:45 ip-172-31-58-173.us-west-2.compute.internal atomic-openshift-master-controllers[15884]: E0227 17:58:45.812324   15884 buildconfig_controller.go:103] builds.build.openshift.io "django-psql-example-1" is forbidden: User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in the namespace "django-psql-u-234-83-368-135": User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in project "django-psql-u-234-83-368-135"
Feb 27 18:08:45 ip-172-31-58-173.us-west-2.compute.internal atomic-openshift-master-controllers[15884]: E0227 18:08:45.812832   15884 buildconfig_controller.go:103] builds.build.openshift.io "django-psql-example-1" is forbidden: User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in the namespace "django-psql-u-234-83-368-135": User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in project "django-psql-u-234-83-368-135"
Feb 27 18:08:45 ip-172-31-58-173.us-west-2.compute.internal atomic-openshift-master-controllers[15884]: E0227 18:08:45.812880   15884 buildconfig_controller.go:103] builds.build.openshift.io "dancer-mysql-example-2" is forbidden: User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in the namespace "dancer-mysql-u-49-18-351-130": User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in project "dancer-mysql-u-49-18-351-130"
Feb 27 18:18:45 ip-172-31-58-173.us-west-2.compute.internal atomic-openshift-master-controllers[15884]: E0227 18:18:45.812790   15884 buildconfig_controller.go:103] builds.build.openshift.io "django-psql-example-1" is forbidden: User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in the namespace "django-psql-u-234-83-368-135": User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in project "django-psql-u-234-83-368-135"
Feb 27 18:18:45 ip-172-31-58-173.us-west-2.compute.internal atomic-openshift-master-controllers[15884]: E0227 18:18:45.812844   15884 buildconfig_controller.go:103] builds.build.openshift.io "dancer-mysql-example-2" is forbidden: User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in the namespace "dancer-mysql-u-49-18-351-130": User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in project "dancer-mysql-u-49-18-351-130"
Feb 27 18:28:45 ip-172-31-58-173.us-west-2.compute.internal atomic-openshift-master-controllers[15884]: E0227 18:28:45.814070   15884 buildconfig_controller.go:103] builds.build.openshift.io "dancer-mysql-example-2" is forbidden: User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in the namespace "dancer-mysql-u-49-18-351-130": User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in project "dancer-mysql-u-49-18-351-130"
Feb 27 18:28:45 ip-172-31-58-173.us-west-2.compute.internal atomic-openshift-master-controllers[15884]: E0227 18:28:45.814070   15884 buildconfig_controller.go:103] builds.build.openshift.io "django-psql-example-1" is forbidden: User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in the namespace "django-psql-u-234-83-368-135": User "system:serviceaccount:openshift-infra:build-config-change-controller" cannot delete builds.build.openshift.io in project "django-psql-u-234-83-368-135"

Version-Release number of selected component (if applicable):
openshift v3.9.0-0.53.0
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.8


How reproducible:
I see it the error 209 times in 1 day

Steps to Reproduce:
1. create/build/delete/modify lot of quickstart apps
2. watch master logs

Actual results:
Errors in master logs.

Expected results:
No errors in logs.

Additional info:
Master logs attached.

Comment 1 Vikas Laad 2018-02-27 20:54:04 UTC

Created attachment 1401531 [details]
master logs

Comment 2 Corey Daley 2018-02-28 20:37:51 UTC

Submitted pull request: https://github.com/openshift/origin/pull/18781

Comment 3 Ben Parees 2018-03-01 14:06:20 UTC

PR for 3.9: https://github.com/openshift/origin/pull/18782

Comment 4 Ben Parees 2018-03-06 15:00:18 UTC

Corey please don't forget to move your bugs to modified when your PRs merge.

Comment 6 Vikas Laad 2018-03-08 22:38:58 UTC

Verified on following version, did not see these errors

openshift v3.9.3
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.16