Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1620272 - Binary builds hang the openshift api server in cluster up
Binary builds hang the openshift api server in cluster up
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master (Show other bugs)
3.10.0
Unspecified Unspecified
low Severity low
: ---
: 3.11.0
Assigned To: David Eads
zhou ying
:
Depends On:
Blocks: 1624197
  Show dependency treegraph
 
Reported: 2018-08-22 16:46 EDT by Ben Parees
Modified: 2018-10-11 03:25 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1624197 (view as bug list)
Environment:
Last Closed: 2018-10-11 03:25:25 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
binary build payload (15.49 MB, application/x-tar)
2018-08-22 16:46 EDT, Ben Parees
no flags Details
pprof output after a binarybuild is started (1.39 MB, text/plain)
2018-08-22 16:51 EDT, Ben Parees
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:2652 None None None 2018-10-11 03:25 EDT

  None (edit)
Description Ben Parees 2018-08-22 16:46:46 EDT
Created attachment 1477965 [details]
binary build payload

Description of problem:
After starting a binary build w/ a sufficiently large payload, the build hangs and all subsequent write operations to openshift resources also hang.


Version-Release number of selected component (if applicable):
oc v3.10.0+a5e4ac9-10
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://127.0.0.1:8443
openshift v3.10.0+a5e4ac9-10
kubernetes v1.10.0+b81c8f8

(this is openshift/origin's master level currently).


How reproducible:
always

Steps to Reproduce:
1. oc cluster up --tag v3.10
2. oc new-build nodejs --binary=true --name=testlarge
3. oc start-build testlarge --from-archive source.tar 
(see attachment for source.tar)
4. oc edit bc testlarge
5. change some minor field like the historylimit
6. save/exit.  The attempt to update the BC will hang.
7. attempts to create other openshift resources (e.g. imagestreams) also hang


Actual results:
Editing openshift resources hangs.
The build pod is never created


Expected results:
editing openshift resources succeeds


Additional info:
restarting the openshift api server allows edits to proceed, but starting another binary build will hang it again.
Comment 1 Ben Parees 2018-08-22 16:51 EDT
Created attachment 1477966 [details]
pprof output after a binarybuild is started
Comment 2 Ben Parees 2018-08-22 17:00:54 EDT
Earlier i thought this was not affecting 3.11 but it looks like it is hanging there also.
Comment 10 zhou ying 2018-09-10 05:22:26 EDT
Blocked by Bug 1627005
Comment 11 zhou ying 2018-09-11 02:21:09 EDT
Verified with ocp:
[root@Master-cluster-up ~]$ oc version
oc v3.11.0-0.32.0
kubernetes v1.11.0+d4cacc0

Can't reproduce it.
Comment 12 wewang 2018-09-12 03:52:06 EDT
@zhou ying verified in ocp:v3.11.0-0.28.0 too 

@Ben Parees What's the special of your source.tar? if we create tar file from https://github.com/sclorg/nodejs-ex/, it cannot met the issue, right? I just want to know how to make it payload,just a curiosity, thanks.
Comment 13 Ben Parees 2018-09-12 10:11:38 EDT
Any sufficiently large source.tar file will trigger the issue.  Make one that's about 10megs.
Comment 15 errata-xmlrpc 2018-10-11 03:25:25 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652

Note You need to log in before you can comment on or make changes to this bug.