Bug 1465325

Summary:	[3.5][Backport] Concurrent build registry push hangs - baremetal cluster with CNS Gluster registry storage
Product:	OpenShift Container Platform	Reporter:	Michal Minar <miminar>
Component:	Image Registry	Assignee:	Michal Minar <miminar>
Status:	CLOSED ERRATA	QA Contact:	Hongkai Liu <hongkliu>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.5.1	CC:	aos-bugs, erich, haowang, hchiramm, hongkliu, jeder, mfojtik, mifiedle, miminar, pprakash, pweil, vlaad, xtian
Target Milestone:	---
Target Release:	3.5.z
Hardware:	x86_64
OS:	Linux
Whiteboard:	aos-scalability-35
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: A bug in a regulator of concurrent filesystem access could cause a routine to hang. Consequence: When the registry configured with filesystem storage driver had been under heavy load, some could have hang forever. Fix: The regulator has been fixed. Result: The concurrent pushes no longer hang.	Story Points:	---
Clone Of:	1436841	Environment:
Last Closed:	2017-08-31 17:00:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1436841
Bug Blocks:

Comment 1 Michal Minar 2017-06-27 08:47:11 UTC

PR with a back-port: https://github.com/openshift/ose/pull/798

Comment 2 Michal Minar 2017-07-21 06:22:44 UTC

Merged.

Comment 3 Mike Fiedler 2017-07-21 14:50:41 UTC

Should not go ON_QA until it is in a 3.5 puddle.  Moving back to MODIFIED

Comment 7 Hongkai Liu 2017-08-09 17:13:01 UTC

Verify with following setting:

REGISTRY_STORAGE_FILESYSTEM_MAXTHREADS=100
300 concurrent builds for several hours which consistently push to images to docker-reg.

Some build failed but no hanging occurred.

Comment 8 Mike Fiedler 2017-08-09 18:45:51 UTC

Verified on 3.5.5.31.12

Comment 9 Mike Fiedler 2017-08-09 18:47:08 UTC

Also, verified with the registry configured to use the filesystem storage driver.

Comment 11 errata-xmlrpc 2017-08-31 17:00:23 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1828