Bug 1707447

Summary: jenkins-slave produce process defunct [ Jenkins "SLAVE" ]
Product: OpenShift Container Platform Reporter: Gabe Montero <gmontero>
Component: ImageStreamsAssignee: Gabe Montero <gmontero>
Status: CLOSED ERRATA QA Contact: XiuJuan Wang <xiuwang>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.10.0CC: aos-bugs, fgrosjea, gmontero, jokerman, maupadhy, mmccomas, vbobade, wzheng, xiuwang
Target Milestone: ---   
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Long running jenkins agent/slave pods can experience the defect process phenomenon that we previously observed with the jenkins master Consequence: A lot of defect processes show up in process listings until the pod is terminated. Fix: Employ `dumb-init` as with the openshift/jenkins master image to clean up these defect processes which occur during jenkins job processing. Result: Process listings within agent/slave pods and on the hosts those pods reside no longer include the defunct processes.
Story Points: ---
Clone Of: 1700314 Environment:
Last Closed: 2019-06-27 16:41:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1700314, 1707448, 1718379    
Bug Blocks: 1705123    

Comment 3 Gabe Montero 2019-05-13 14:51:49 UTC
Turns out we will need https://github.com/openshift/ocp-build-data/pull/121 to merge before we can start getting slave builds at osbs/brew with dumb-init

Comment 4 Gabe Montero 2019-06-10 14:18:53 UTC
looks like we have new images on brew for this

Comment 6 XiuJuan Wang 2019-06-12 07:47:13 UTC
verified with 
openshift3/jenkins-agent-nodejs-8-rhel7:v3.10 (v3.10.149)
openshift3/jenkins-agent-maven-35-rhel7:v3.10 (v3.10.149)
openshift3/jenkins-slave-nodejs-rhel7:v3.10 (v3.10.149)
openshift3/jenkins-slave-maven-rhel7:v3.10 (v3.10.149)

Steps:
1. Create jenkins server and maven| nodejs pipeline buildconfigs.
2.Login to jenkins console to set the maven/nodejs pod idle 30 mins
3.Trigger maven and nodejs pipeline builds.
4.Rsh into slave pod when time is almost out.
dumb-init process has cleaned defunct processes, no defunct processes exist.

maven-nc44t                           1/1       Running     0          30m
nodejs-3nxf6                          1/1       Running     0          30m


#  oc rsh  -n xiuwang  nodejs-3nxf6 
sh-4.2$ ps -ef 
UID         PID   PPID  C STIME TTY          TIME CMD
default       1      0  0 07:06 ?        00:00:00 /usr/bin/dumb-init -- /usr/local/bin/run-jnlp-client b4b4586fb94d09888b23ffe144cd18271ae9eb9075483b37a933a77ac5a69786 nodejs-3nxf6
default       7      1  1 07:06 ?        00:00:17 java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -cp /home/jenkins/remoting.jar hudson.re
default     251      0  0 07:34 ?        00:00:00 /bin/sh
default     260    251  0 07:34 ?        00:00:00 ps -ef

#  oc rsh  -n xiuwang maven-nc44t 
sh-4.2$ ps -ef
UID         PID   PPID  C STIME TTY          TIME CMD
default       1      0  0 07:07 ?        00:00:00 /usr/bin/dumb-init -- /usr/local/bin/run-jnlp-client ce6250144ae1685757b511ca3be8555faeebf1451574103da5e4fcea6cfddbc3 maven-nc44t
default       7      1  1 07:07 ?        00:00:18 java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -cp /home/jenkins/remoting.jar hudson.re
default     265      0  0 07:37 ?        00:00:00 /bin/sh
default     274    265  0 07:37 ?        00:00:00 ps -ef

Comment 8 errata-xmlrpc 2019-06-27 16:41:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1607