1878164 – Cluster consumes up to 10 CPUs while idling. CPU consumption on nodes is higher than the CPU count of the node.

Bug 1878164 - Cluster consumes up to 10 CPUs while idling. CPU consumption on nodes is higher than the CPU count of the node.

Summary: Cluster consumes up to 10 CPUs while idling. CPU consumption on nodes is high...

Keywords:
Status:	CLOSED DUPLICATE of bug 1878766
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Monitoring
Sub Component:
Version:	4.6
Hardware:	s390x
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	4.6.0
Assignee:	Sergiusz Urbaniak
QA Contact:	Junqi Zhao
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-09-11 14:03 UTC by wvoesch
Modified:	2020-09-14 13:55 UTC (History)
CC List:	16 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-09-14 13:55:09 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
output of "ps -ef --forest". (101.76 KB, text/plain) 2020-09-11 15:38 UTC, wvoesch	no flags	Details
View All

Description wvoesch 2020-09-11 14:03:16 UTC

Description of problem:

There are three problems described which might be caused by the same root problem. If not I will report them separately, of course. 

1.	In several clusters the CPU consumption is between 9 and 11 cores while idling. The clusters are a few days old, on some, some test have been executed, others were kept as they are and were idling since creation. 
2.	On these cluster there are nodes which consume a lot more CPUs than what has been configured. For example, according to the OCP dashboard a master consumes 6 CPUs while only 4 are configured. According to the dashboard -2,000.00m are available. The hypervisor however shows much lower utilization. 
3.	On several nodes I have observed zombie processes. Usually there are 2 or 3, but once 547 are counted. 


Observed on:
Z14: Version: 4.6.0-0.nightly-s390x-2020-09-05-222506 RHCOS: 46.82.202009042339-0
Z13: Version: 4.6.0-0.nightly-s390x-2020-09-05-222506 RHCOS: 46.82.202009042339-0
Z13: Version: 4.6.0-0.nightly-s390x-2020-09-07-065507 RHCOS: 46.82.202009042339-0
Z13: Version: 4.6.0-0.nightly-s390x-2020-08-27-080214 RHCOS: 46.82.202008261939-0

The clusters include nodes with ECKD and ZCP disks, direct attached OSA, RoCE and Hipersockes. Typically the clusters have three masters and 2 workers, of size 2 CPUs and 8 GB. But one cluster has 3 workers with 10 CPUs and 32 GB each. 

Because of the size, I just added some basic information below. Please let me know what information you need for further debugging and I shall provide it happily. 
Thank you. 


Additional info:
###############################################################################################################################################
Cluster-identifier: m3559001, Version: 4.6.0-0.nightly-s390x-2020-09-02-081215 RHCOS: 46.82.202009011939-0
Environment: Z14, 3 masters [16 GB, 4 CPUs], 2 workers [8 GB, 1x 2 CPUs; 1x 4 CPUs], 5x fcp 120 GB, RoCE, age: 2d23h
Resource consumption stated by the dashboard
CPU: ~5.17 of 18
Breakdown:
    openshift-kube-apiserver
    1.37
    openshift-etcd
    740.4m
    openshift-monitoring
    634.2m
    openshift-apiserver
    382.1m
    openshift-kube-controller-manager
    43.87m

###############################################################################################################################################
Cluster-identifier: s8343008, Version: 4.6.0-0.nightly-s390x-2020-09-05-222506 RHCOS: 46.82.202009042339-0
Environment: Z13, 3 masters [16 GB, 4 CPUs], 2 workers [8 GB, 2 CPUs], 4x fcp 120 GB, 1x 50 GB dasd, direct attached OSA, age: 3d21h
Resource consumption stated by the dashboard
CPU: ~10.01 of 16
Breakdown:
    openshift-kube-apiserver
    1.99
    openshift-etcd
    1.22
    openshift-monitoring
    1.21
    openshift-apiserver
    623m
    openshift-multus
    97.05m

master 3: configured 4 CPUs
OCP dashboard: CPU consumption ~6.09 of 4. -2,052.037m available 

on the node:
top - 08:43:24 up 3 days, 22:14,  1 user,  load average: 47.41, 40.55, 36.88
Tasks: 322 total,   5 running, 314 sleeping,   0 stopped,   3 zombie
%Cpu(s): 27.6 us,  9.7 sy,  0.0 ni, 14.9 id,  0.1 wa,  1.6 hi,  1.7 si, 44.3 st
MiB Mem :  16097.1 total,    101.4 free,   9331.9 used,   6663.7 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   9982.2 avail Mem

note the 3 zombies, on the other masters there are 3 and 2 respectively. 

# sudo hyptop
system   #cpu    cpu   Cpu+   online memuse memmax wcur
(str)     (#)    (%)   (hm)    (dhm)  (GiB)  (GiB)  (#)
S8343012    4 272.16 485:02 21:18:53  15.76  16.00  100
=:V:N       4 272.16 485:02 21:18:53  15.76  16.00  100

###############################################################################################################################################
Cluster-identifier: s8343001, Version:     4.6.0-0.nightly-s390x-2020-09-07-065507 RHCOS: 46.82.202009042339-0
Environment: Z13, 3 masters [16 GB, 4 CPUs], 3 workers [32 GB, 10 CPUs], 6x fcp 120 GB, Hipersockets, age: 3d23h
Resource consumption stated by the dashboard
CPU: ~10.2 of 42
    openshift-kube-apiserver
    2.04
    openshift-etcd
    1.27
    openshift-monitoring
    1.2
    openshift-apiserver
    598.2m
    openshift-multus
    103.8m

master 1: configured 4 CPUs
OCP dashboard: CPU consumption ~4.95  of 4. -947.756m available  

top - 12:52:43 up 3 days,  5:53,  1 user,  load average: 30.24, 25.30, 24.83
Tasks: 308 total,   4 running, 303 sleeping,   0 stopped,   1 zombie
%Cpu(s): 23.0 us,  7.5 sy,  0.0 ni,  7.7 id,  0.1 wa,  1.0 hi,  1.4 si, 59.2 st
MiB Mem :  15962.1 total,    711.8 free,   8834.4 used,   6415.9 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  11069.8 avail Mem

hyptop:
system   #cpu    cpu   Cpu+   online memuse memmax wcur
(str)     (#)    (%)   (hm)    (dhm)  (GiB)  (GiB)  (#)
S8343003    4 268.16 494:21 52:21:10  15.82  15.87  100
=:V:N       4 268.16 494:21 52:21:10  15.82  15.87  100

###############################################################################################################################################
Cluster-identifier: s8343022, Version: 4.6.0-0.nightly-s390x-2020-08-27-080214 RHCOS: 46.82.202008261939-0
Environment: Z13, 3 masters [16 GB, 4 CPUs], 2 workers [8 GB, 2 CPUs], 5x fcp 120 GB, direct attached OSA, age: 14d
Resource consumption stated by the dashboard
CPU: ~11.33 of 16
Breakdown:
    openshift-kube-apiserver
    2.8
    openshift-etcd
    1.36
    openshift-monitoring
    1.11
    openshift-apiserver
    771.9m
    openshift-multus
    214.9m

master 2: configured 4 CPUs
OCP dashboard: CPU consumption ~5.034  of 4. -1,043.793m available  

top - 08:56:39 up 14 days, 17:38,  1 user,  load average: 22.24, 26.02, 25.87
Tasks: 332 total,   1 running, 329 sleeping,   0 stopped,   2 zombie
%Cpu(s): 19.7 us,  7.5 sy,  0.0 ni, 41.4 id,  0.5 wa,  1.2 hi,  1.3 si, 28.3 st
MiB Mem :  16097.1 total,    140.4 free,  11375.0 used,   4581.6 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   9145.1 avail Mem

hyptop:
system   #cpu    cpu    Cpu+   online memuse memmax wcur
(str)     (#)    (%)    (hm)    (dhm)  (GiB)  (GiB)  (#)
S8343025    4 200.61 1164:01 65:00:21  15.77  16.00  100
=:V:N       4 200.61 1164:01 65:00:21  15.77  16.00  100


worker 1: configured 2 CPUs
OC dashboard: COU consumption ~3.2 of 2. -1,113.494m available  
top - 12:13:42 up 14 days, 20:46,  1 user,  load average: 31.80, 27.86, 29.22
Tasks: 833 total,   7 running, 279 sleeping,   0 stopped, 547 zombie
%Cpu(s): 26.0 us, 12.4 sy,  0.0 ni,  0.6 id,  0.3 wa,  0.5 hi,  1.3 si, 58.9 st
MiB Mem :   8022.1 total,     57.8 free,   5968.1 used,   1996.2 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   1915.7 avail Mem

system   #cpu    cpu   Cpu+   online memuse memmax wcur
(str)     (#)    (%)   (hm)    (dhm)  (GiB)  (GiB)  (#)
S8343027    2 158.48 578:13 65:03:38   7.68   7.99  100
=:V:N       2 158.48 578:13 65:03:38   7.68   7.99  100

note the 547 zombie


$ ps -aux |grep "Z "
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1000340+   31731  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+   38356  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+   95582  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  194542  0.0  0.0      0     0 ?        Z    Sep01   0:00 [curl] <defunct>
1000340+  334576  0.0  0.0      0     0 ?        Z    Sep07   0:00 [curl] <defunct>
1000340+  385309  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  385516  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  391215  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  400364  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  401028  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  402968  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  403531  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  404268  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  409293  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  414634  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  414940  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  415269  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  418154  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  418213  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  421242  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  431113  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  431971  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  454671  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  469455  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  476518  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  487905  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  488819  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  497364  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  515319  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  521968  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  528580  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  541824  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  560762  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  569324  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  574989  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  584752  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  597721  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  604738  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  606140  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  610346  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  610989  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  613453  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  619830  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  642716  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  643112  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  647855  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  650703  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  658801  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  663885  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  665837  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  706449  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  712136  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  712236  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  715608  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  728020  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  733351  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  734851  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  745572  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  745684  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  747644  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  751390  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  751748  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  752053  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  753973  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  760610  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  764431  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  768192  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  777785  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  778668  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  785517  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  794485  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  796783  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  923880  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  923936  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+  931043  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1071229  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1109789  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1111032  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1114539  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1114726  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1116390  0.0  0.0      0     0 ?        Z    Sep10   0:00 [sh] <defunct>
1000340+ 1116680  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1122429  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1123373  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1124293  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1125219  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1126780  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1127052  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1127360  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1129032  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1130212  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1130331  0.0  0.0      0     0 ?        Z    Sep10   0:00 [sh] <defunct>
1000340+ 1132482  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1134741  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1134943  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1135066  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1137593  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1142358  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1142909  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1147026  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1147628  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1149208  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1150136  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1151100  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1151790  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1152311  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1157462  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1158385  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1159343  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1162135  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1162391  0.0  0.0      0     0 ?        Z    Sep10   0:00 [sh] <defunct>
1000340+ 1162471  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1163634  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1164347  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1164982  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1173551  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1187710  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1189576  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1190439  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1199359  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1202169  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1206330  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1207458  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1207544  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1210317  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1223358  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1223873  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1226558  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1363663  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1411022  0.0  0.0      0     0 ?        Z    Sep06   0:00 [curl] <defunct>
1000340+ 1454753  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1515382  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1534583  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1536841  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1542122  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1543120  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1553543  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1576833  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1578159  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1592289  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1626312  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1657436  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1686800  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1731474  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1738876  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1746596  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 1754991  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1755883  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1758656  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1758696  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1774815  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1787823  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1788796  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1798548  0.0  0.0      0     0 ?        Z    00:19   0:00 [curl] <defunct>
1000340+ 1805227  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1811850  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1813632  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1814915  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1839847  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1843748  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1847970  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1859562  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1886089  0.0  0.0      0     0 ?        Z    Sep09   0:00 [sh] <defunct>
1000340+ 1887618  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1933354  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 1939874  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2027513  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2064077  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2101309  0.0  0.0      0     0 ?        Z    Sep09   0:00 [sh] <defunct>
1000340+ 2113210  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2114407  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2119036  0.0  0.0      0     0 ?        Z    03:08   0:00 [curl] <defunct>
1000340+ 2124864  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2127092  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2127488  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2128665  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2143864  0.0  0.0      0     0 ?        Z    03:21   0:00 [curl] <defunct>
1000340+ 2173066  0.0  0.0      0     0 ?        Z    03:36   0:00 [curl] <defunct>
1000340+ 2190787  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2192041  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2282162  0.0  0.0      0     0 ?        Z    04:34   0:00 [curl] <defunct>
1000340+ 2338276  0.0  0.0      0     0 ?        Z    05:04   0:00 [curl] <defunct>
1000340+ 2346869  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2364818  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2386373  0.0  0.0      0     0 ?        Z    05:29   0:00 [curl] <defunct>
1000340+ 2397473  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2404640  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2421266  0.0  0.0      0     0 ?        Z    05:47   0:00 [curl] <defunct>
1000340+ 2462873  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2491257  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2495055  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2501967  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2518863  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2530253  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2564006  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2576617  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2578581  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2578765  0.0  0.0      0     0 ?        Z    07:11   0:00 [curl] <defunct>
1000340+ 2579400  0.0  0.0      0     0 ?        Z    07:11   0:00 [curl] <defunct>
1000340+ 2586677  0.0  0.0      0     0 ?        Z    07:15   0:00 [curl] <defunct>
1000340+ 2586881  0.0  0.0      0     0 ?        Z    07:15   0:00 [curl] <defunct>
1000340+ 2586963  0.0  0.0      0     0 ?        Z    07:15   0:00 [curl] <defunct>
1000340+ 2589848  0.0  0.0      0     0 ?        Z    07:16   0:00 [curl] <defunct>
1000340+ 2592766  0.0  0.0      0     0 ?        Z    07:18   0:00 [curl] <defunct>
1000340+ 2593722  0.0  0.0      0     0 ?        Z    07:18   0:00 [curl] <defunct>
1000340+ 2595028  0.0  0.0      0     0 ?        Z    07:19   0:00 [curl] <defunct>
1000340+ 2595682  0.0  0.0      0     0 ?        Z    07:20   0:00 [curl] <defunct>
1000340+ 2596297  0.0  0.0      0     0 ?        Z    07:20   0:00 [curl] <defunct>
1000340+ 2596461  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2597247  0.0  0.0      0     0 ?        Z    07:20   0:00 [curl] <defunct>
1000340+ 2598671  0.0  0.0      0     0 ?        Z    07:21   0:00 [curl] <defunct>
1000340+ 2599631  0.0  0.0      0     0 ?        Z    07:22   0:00 [curl] <defunct>
1000340+ 2601299  0.0  0.0      0     0 ?        Z    07:23   0:00 [curl] <defunct>
1000340+ 2603156  0.0  0.0      0     0 ?        Z    07:23   0:00 [curl] <defunct>
1000340+ 2609757  0.0  0.0      0     0 ?        Z    07:27   0:00 [curl] <defunct>
1000340+ 2609798  0.0  0.0      0     0 ?        Z    07:27   0:00 [curl] <defunct>
1000340+ 2613290  0.0  0.0      0     0 ?        Z    07:29   0:00 [curl] <defunct>
1000340+ 2614580  0.0  0.0      0     0 ?        Z    07:29   0:00 [curl] <defunct>
1000340+ 2620981  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2627513  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2628415  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2629558  0.0  0.0      0     0 ?        Z    07:37   0:00 [curl] <defunct>
1000340+ 2646505  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2648607  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2658490  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2672958  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2680287  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2680369  0.0  0.0      0     0 ?        Z    Sep09   0:00 [sh] <defunct>
1000340+ 2682093  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2686426  0.0  0.0      0     0 ?        Z    08:07   0:00 [curl] <defunct>
1000340+ 2695175  0.0  0.0      0     0 ?        Z    08:12   0:00 [curl] <defunct>
1000340+ 2695248  0.0  0.0      0     0 ?        Z    08:12   0:00 [curl] <defunct>
1000340+ 2696431  0.0  0.0      0     0 ?        Z    08:13   0:00 [curl] <defunct>
1000340+ 2698738  0.0  0.0      0     0 ?        Z    08:14   0:00 [curl] <defunct>
1000340+ 2701090  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2707590  0.0  0.0      0     0 ?        Z    08:19   0:00 [curl] <defunct>
1000340+ 2715290  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2716280  0.0  0.0      0     0 ?        Z    08:23   0:00 [curl] <defunct>
1000340+ 2721695  0.0  0.0      0     0 ?        Z    08:26   0:00 [curl] <defunct>
1000340+ 2724009  0.0  0.0      0     0 ?        Z    08:27   0:00 [curl] <defunct>
1000340+ 2725227  0.0  0.0      0     0 ?        Z    08:28   0:00 [curl] <defunct>
1000340+ 2731532  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2731769  0.0  0.0      0     0 ?        Z    08:31   0:00 [curl] <defunct>
1000340+ 2733422  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2734268  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2738065  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2750999  0.0  0.0      0     0 ?        Z    08:42   0:00 [curl] <defunct>
1000340+ 2752326  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2753945  0.0  0.0      0     0 ?        Z    08:43   0:00 [curl] <defunct>
1000340+ 2755911  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2768696  0.0  0.0      0     0 ?        Z    08:51   0:00 [curl] <defunct>
1000340+ 2772889  0.0  0.0      0     0 ?        Z    08:53   0:00 [curl] <defunct>
1000340+ 2773399  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2774683  0.0  0.0      0     0 ?        Z    08:54   0:00 [curl] <defunct>
1000340+ 2776338  0.0  0.0      0     0 ?        Z    08:55   0:00 [curl] <defunct>
1000340+ 2776635  0.0  0.0      0     0 ?        Z    08:55   0:00 [curl] <defunct>
1000340+ 2776985  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2778515  0.0  0.0      0     0 ?        Z    08:56   0:00 [curl] <defunct>
1000340+ 2779763  0.0  0.0      0     0 ?        Z    08:57   0:00 [curl] <defunct>
1000340+ 2782986  0.0  0.0      0     0 ?        Z    08:58   0:00 [curl] <defunct>
1000340+ 2786391  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2786764  0.0  0.0      0     0 ?        Z    09:00   0:00 [curl] <defunct>
1000340+ 2789887  0.0  0.0      0     0 ?        Z    09:02   0:00 [curl] <defunct>
1000340+ 2790877  0.0  0.0      0     0 ?        Z    09:03   0:00 [curl] <defunct>
1000340+ 2795906  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2797820  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2798961  0.0  0.0      0     0 ?        Z    09:07   0:00 [curl] <defunct>
1000340+ 2800888  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2801586  0.0  0.0      0     0 ?        Z    09:08   0:00 [curl] <defunct>
1000340+ 2802791  0.0  0.0      0     0 ?        Z    09:09   0:00 [curl] <defunct>
1000340+ 2804939  0.0  0.0      0     0 ?        Z    09:10   0:00 [curl] <defunct>
1000340+ 2810995  0.0  0.0      0     0 ?        Z    09:13   0:00 [curl] <defunct>
1000340+ 2813161  0.0  0.0      0     0 ?        Z    09:14   0:00 [curl] <defunct>
1000340+ 2813621  0.0  0.0      0     0 ?        Z    09:15   0:00 [curl] <defunct>
1000340+ 2814470  0.0  0.0      0     0 ?        Z    09:15   0:00 [curl] <defunct>
1000340+ 2816399  0.0  0.0      0     0 ?        Z    09:16   0:00 [curl] <defunct>
1000340+ 2817006  0.0  0.0      0     0 ?        Z    09:16   0:00 [curl] <defunct>
1000340+ 2820170  0.0  0.0      0     0 ?        Z    09:18   0:00 [curl] <defunct>
1000340+ 2821037  0.0  0.0      0     0 ?        Z    09:19   0:00 [curl] <defunct>
1000340+ 2823986  0.0  0.0      0     0 ?        Z    09:20   0:00 [curl] <defunct>
1000340+ 2824811  0.0  0.0      0     0 ?        Z    09:21   0:00 [curl] <defunct>
1000340+ 2828021  0.0  0.0      0     0 ?        Z    09:22   0:00 [curl] <defunct>
1000340+ 2828260  0.0  0.0      0     0 ?        Z    09:22   0:00 [curl] <defunct>
1000340+ 2832218  0.0  0.0      0     0 ?        Z    09:24   0:00 [curl] <defunct>
1000340+ 2832527  0.0  0.0      0     0 ?        Z    09:25   0:00 [curl] <defunct>
1000340+ 2834828  0.0  0.0      0     0 ?        Z    09:26   0:00 [curl] <defunct>
1000340+ 2837294  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2837392  0.0  0.0      0     0 ?        Z    09:27   0:00 [curl] <defunct>
1000340+ 2838017  0.0  0.0      0     0 ?        Z    09:27   0:00 [curl] <defunct>
1000340+ 2838804  0.0  0.0      0     0 ?        Z    09:28   0:00 [curl] <defunct>
1000340+ 2841519  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2844960  0.0  0.0      0     0 ?        Z    09:31   0:00 [curl] <defunct>
1000340+ 2845614  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2845775  0.0  0.0      0     0 ?        Z    09:31   0:00 [curl] <defunct>
1000340+ 2846236  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2846857  0.0  0.0      0     0 ?        Z    09:32   0:00 [curl] <defunct>
1000340+ 2847877  0.0  0.0      0     0 ?        Z    09:32   0:00 [curl] <defunct>
1000340+ 2848750  0.0  0.0      0     0 ?        Z    09:33   0:00 [curl] <defunct>
1000340+ 2848793  0.0  0.0      0     0 ?        Z    09:33   0:00 [curl] <defunct>
1000340+ 2852526  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2853871  0.0  0.0      0     0 ?        Z    09:35   0:00 [curl] <defunct>
1000340+ 2853914  0.0  0.0      0     0 ?        Z    09:35   0:00 [curl] <defunct>
1000340+ 2855887  0.0  0.0      0     0 ?        Z    09:36   0:00 [curl] <defunct>
1000340+ 2858494  0.0  0.0      0     0 ?        Z    09:38   0:00 [curl] <defunct>
1000340+ 2866403  0.0  0.0      0     0 ?        Z    09:42   0:00 [curl] <defunct>
1000340+ 2868300  0.0  0.0      0     0 ?        Z    09:43   0:00 [curl] <defunct>
1000340+ 2868697  0.0  0.0      0     0 ?        Z    09:43   0:00 [curl] <defunct>
1000340+ 2871178  0.0  0.0      0     0 ?        Z    09:44   0:00 [curl] <defunct>
1000340+ 2872049  0.0  0.0      0     0 ?        Z    09:45   0:00 [curl] <defunct>
1000340+ 2873217  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2874246  0.0  0.0      0     0 ?        Z    09:46   0:00 [curl] <defunct>
1000340+ 2881896  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2882544  0.0  0.0      0     0 ?        Z    09:50   0:00 [curl] <defunct>
1000340+ 2886050  0.0  0.0      0     0 ?        Z    09:52   0:00 [curl] <defunct>
1000340+ 2896792  0.0  0.0      0     0 ?        Z    09:58   0:00 [curl] <defunct>
1000340+ 2900019  0.0  0.0      0     0 ?        Z    10:00   0:00 [curl] <defunct>
1000340+ 2901126  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2901245  0.0  0.0      0     0 ?        Z    10:00   0:00 [curl] <defunct>
1000340+ 2902738  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2904436  0.0  0.0      0     0 ?        Z    10:02   0:00 [sh] <defunct>
1000340+ 2904613  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2904723  0.0  0.0      0     0 ?        Z    10:02   0:00 [curl] <defunct>
1000340+ 2905385  0.0  0.0      0     0 ?        Z    10:02   0:00 [curl] <defunct>
1000340+ 2909370  0.0  0.0      0     0 ?        Z    10:05   0:00 [curl] <defunct>
1000340+ 2913302  0.0  0.0      0     0 ?        Z    10:07   0:00 [sh] <defunct>
1000340+ 2916692  0.0  0.0      0     0 ?        Z    10:08   0:00 [curl] <defunct>
1000340+ 2917657  0.0  0.0      0     0 ?        Z    10:09   0:00 [curl] <defunct>
1000340+ 2919730  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2921130  0.0  0.0      0     0 ?        Z    10:11   0:00 [curl] <defunct>
1000340+ 2921739  0.0  0.0      0     0 ?        Z    10:11   0:00 [sh] <defunct>
1000340+ 2922917  0.0  0.0      0     0 ?        Z    10:12   0:00 [curl] <defunct>
1000340+ 2925140  0.0  0.0      0     0 ?        Z    10:13   0:00 [curl] <defunct>
1000340+ 2925178  0.0  0.0      0     0 ?        Z    10:13   0:00 [curl] <defunct>
1000340+ 2929621  0.0  0.0      0     0 ?        Z    10:15   0:00 [curl] <defunct>
1000340+ 2929899  0.0  0.0      0     0 ?        Z    10:15   0:00 [curl] <defunct>
1000340+ 2930858  0.0  0.0      0     0 ?        Z    10:16   0:00 [curl] <defunct>
1000340+ 2931252  0.0  0.0      0     0 ?        Z    10:16   0:00 [curl] <defunct>
1000340+ 2938019  0.0  0.0      0     0 ?        Z    10:20   0:00 [curl] <defunct>
1000340+ 2938132  0.0  0.0      0     0 ?        Z    10:20   0:00 [curl] <defunct>
1000340+ 2939050  0.0  0.0      0     0 ?        Z    10:20   0:00 [curl] <defunct>
1000340+ 2940261  0.0  0.0      0     0 ?        Z    10:21   0:00 [curl] <defunct>
1000340+ 2943069  0.0  0.0      0     0 ?        Z    10:22   0:00 [curl] <defunct>
1000340+ 2944113  0.0  0.0      0     0 ?        Z    10:23   0:00 [curl] <defunct>
1000340+ 2950454  0.0  0.0      0     0 ?        Z    10:26   0:00 [curl] <defunct>
1000340+ 2951357  0.0  0.0      0     0 ?        Z    10:27   0:00 [sh] <defunct>
1000340+ 2953794  0.0  0.0      0     0 ?        Z    10:28   0:00 [curl] <defunct>
1000340+ 2954166  0.0  0.0      0     0 ?        Z    10:28   0:00 [curl] <defunct>
1000340+ 2955423  0.0  0.0      0     0 ?        Z    10:29   0:00 [curl] <defunct>
1000340+ 2956329  0.0  0.0      0     0 ?        Z    10:29   0:00 [curl] <defunct>
1000340+ 2956615  0.0  0.0      0     0 ?        Z    10:30   0:00 [curl] <defunct>
1000340+ 2956983  0.0  0.0      0     0 ?        Z    10:30   0:00 [sh] <defunct>
1000340+ 2957997  0.0  0.0      0     0 ?        Z    10:30   0:00 [curl] <defunct>
1000340+ 2961157  0.0  0.0      0     0 ?        Z    10:32   0:00 [curl] <defunct>
1000340+ 2964069  0.0  0.0      0     0 ?        Z    10:33   0:00 [curl] <defunct>
1000340+ 2964400  0.0  0.0      0     0 ?        Z    10:34   0:00 [curl] <defunct>
1000340+ 2964481  0.0  0.0      0     0 ?        Z    10:34   0:00 [curl] <defunct>
1000340+ 2966274  0.0  0.0      0     0 ?        Z    10:35   0:00 [curl] <defunct>
1000340+ 2967865  0.0  0.0      0     0 ?        Z    10:35   0:00 [curl] <defunct>
1000340+ 2968226  0.0  0.0      0     0 ?        Z    10:36   0:00 [curl] <defunct>
1000340+ 2968272  0.0  0.0      0     0 ?        Z    10:36   0:00 [curl] <defunct>
1000340+ 2971737  0.0  0.0      0     0 ?        Z    10:37   0:00 [sh] <defunct>
1000340+ 2972241  0.0  0.0      0     0 ?        Z    10:38   0:00 [sh] <defunct>
1000340+ 2972478  0.0  0.0      0     0 ?        Z    10:38   0:00 [curl] <defunct>
1000340+ 2972634  0.0  0.0      0     0 ?        Z    10:38   0:00 [curl] <defunct>
1000340+ 2977340  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2977601  0.0  0.0      0     0 ?        Z    10:41   0:00 [curl] <defunct>
1000340+ 2978313  0.0  0.0      0     0 ?        Z    10:41   0:00 [sh] <defunct>
1000340+ 2978429  0.0  0.0      0     0 ?        Z    10:41   0:00 [curl] <defunct>
1000340+ 2979835  0.0  0.0      0     0 ?        Z    10:42   0:00 [curl] <defunct>
1000340+ 2984853  0.0  0.0      0     0 ?        Z    10:44   0:00 [curl] <defunct>
1000340+ 2985887  0.0  0.0      0     0 ?        Z    10:45   0:00 [curl] <defunct>
1000340+ 2987107  0.0  0.0      0     0 ?        Z    10:46   0:00 [curl] <defunct>
1000340+ 2987238  0.0  0.0      0     0 ?        Z    10:46   0:00 [curl] <defunct>
1000340+ 2988413  0.0  0.0      0     0 ?        Z    10:46   0:00 [sh] <defunct>
1000340+ 2989624  0.0  0.0      0     0 ?        Z    10:47   0:00 [curl] <defunct>
1000340+ 2989653  0.0  0.0      0     0 ?        Z    10:47   0:00 [curl] <defunct>
1000340+ 2990520  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 2990598  0.0  0.0      0     0 ?        Z    10:47   0:00 [curl] <defunct>
1000340+ 2991524  0.0  0.0      0     0 ?        Z    10:48   0:00 [curl] <defunct>
1000340+ 2993736  0.0  0.0      0     0 ?        Z    10:49   0:00 [curl] <defunct>
1000340+ 2995472  0.0  0.0      0     0 ?        Z    10:50   0:00 [curl] <defunct>
1000340+ 2996302  0.0  0.0      0     0 ?        Z    10:50   0:00 [curl] <defunct>
1000340+ 2997000  0.0  0.0      0     0 ?        Z    10:51   0:00 [curl] <defunct>
1000340+ 2999412  0.0  0.0      0     0 ?        Z    10:52   0:00 [curl] <defunct>
1000340+ 2999413  0.0  0.0      0     0 ?        Z    10:52   0:00 [curl] <defunct>
1000340+ 3001055  0.0  0.0      0     0 ?        Z    10:53   0:00 [curl] <defunct>
1000340+ 3001071  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3003223  0.0  0.0      0     0 ?        Z    10:54   0:00 [curl] <defunct>
1000340+ 3004752  0.0  0.0      0     0 ?        Z    10:55   0:00 [curl] <defunct>
1000340+ 3004755  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3004809  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3005795  0.0  0.0      0     0 ?        Z    10:55   0:00 [curl] <defunct>
1000340+ 3005900  0.0  0.0      0     0 ?        Z    10:56   0:00 [curl] <defunct>
1000340+ 3012948  0.0  0.0      0     0 ?        Z    10:59   0:00 [curl] <defunct>
1000340+ 3014018  0.0  0.0      0     0 ?        Z    11:00   0:00 [curl] <defunct>
1000340+ 3014414  0.0  0.0      0     0 ?        Z    11:00   0:00 [curl] <defunct>
1000340+ 3014638  0.0  0.0      0     0 ?        Z    11:00   0:00 [curl] <defunct>
1000340+ 3014738  0.0  0.0      0     0 ?        Z    11:00   0:00 [curl] <defunct>
1000340+ 3018001  0.0  0.0      0     0 ?        Z    11:02   0:00 [curl] <defunct>
1000340+ 3018498  0.0  0.0      0     0 ?        Z    11:02   0:00 [curl] <defunct>
1000340+ 3019643  0.0  0.0      0     0 ?        Z    11:03   0:00 [curl] <defunct>
1000340+ 3025657  0.0  0.0      0     0 ?        Z    11:06   0:00 [curl] <defunct>
1000340+ 3027203  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3031340  0.0  0.0      0     0 ?        Z    11:09   0:00 [curl] <defunct>
1000340+ 3033221  0.0  0.0      0     0 ?        Z    11:10   0:00 [curl] <defunct>
1000340+ 3040767  0.0  0.0      0     0 ?        Z    11:14   0:00 [curl] <defunct>
1000340+ 3042042  0.0  0.0      0     0 ?        Z    11:15   0:00 [curl] <defunct>
1000340+ 3043280  0.0  0.0      0     0 ?        Z    11:15   0:00 [curl] <defunct>
1000340+ 3043882  0.0  0.0      0     0 ?        Z    11:16   0:00 [curl] <defunct>
1000340+ 3043901  0.0  0.0      0     0 ?        Z    11:16   0:00 [curl] <defunct>
1000340+ 3044556  0.0  0.0      0     0 ?        Z    11:16   0:00 [curl] <defunct>
1000340+ 3045593  0.0  0.0      0     0 ?        Z    11:17   0:00 [curl] <defunct>
1000340+ 3047072  0.0  0.0      0     0 ?        Z    11:17   0:00 [curl] <defunct>
1000340+ 3048649  0.0  0.0      0     0 ?        Z    11:18   0:00 [curl] <defunct>
1000340+ 3049647  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3051583  0.0  0.0      0     0 ?        Z    11:20   0:00 [curl] <defunct>
1000340+ 3052470  0.0  0.0      0     0 ?        Z    11:20   0:00 [curl] <defunct>
1000340+ 3053651  0.0  0.0      0     0 ?        Z    11:21   0:00 [curl] <defunct>
1000340+ 3055254  0.0  0.0      0     0 ?        Z    11:22   0:00 [curl] <defunct>
1000340+ 3055599  0.0  0.0      0     0 ?        Z    11:22   0:00 [curl] <defunct>
1000340+ 3055600  0.0  0.0      0     0 ?        Z    11:22   0:00 [curl] <defunct>
1000340+ 3056840  0.0  0.0      0     0 ?        Z    11:22   0:00 [curl] <defunct>
1000340+ 3057905  0.0  0.0      0     0 ?        Z    11:23   0:00 [curl] <defunct>
1000340+ 3058713  0.0  0.0      0     0 ?        Z    11:23   0:00 [curl] <defunct>
1000340+ 3059635  0.0  0.0      0     0 ?        Z    11:24   0:00 [curl] <defunct>
1000340+ 3062555  0.0  0.0      0     0 ?        Z    11:26   0:00 [curl] <defunct>
1000340+ 3062826  0.0  0.0      0     0 ?        Z    11:26   0:00 [curl] <defunct>
1000340+ 3065320  0.0  0.0      0     0 ?        Z    11:27   0:00 [curl] <defunct>
1000340+ 3066420  0.0  0.0      0     0 ?        Z    11:28   0:00 [curl] <defunct>
1000340+ 3066598  0.0  0.0      0     0 ?        Z    11:28   0:00 [curl] <defunct>
1000340+ 3067556  0.0  0.0      0     0 ?        Z    11:28   0:00 [curl] <defunct>
1000340+ 3069091  0.0  0.0      0     0 ?        Z    11:29   0:00 [sh] <defunct>
1000340+ 3069092  0.0  0.0      0     0 ?        Z    11:29   0:00 [curl] <defunct>
1000340+ 3069982  0.0  0.0      0     0 ?        Z    11:29   0:00 [curl] <defunct>
1000340+ 3070444  0.0  0.0      0     0 ?        Z    11:30   0:00 [curl] <defunct>
1000340+ 3070814  0.0  0.0      0     0 ?        Z    11:30   0:00 [curl] <defunct>
1000340+ 3070984  0.0  0.0      0     0 ?        Z    11:30   0:00 [curl] <defunct>
1000340+ 3071147  0.0  0.0      0     0 ?        Z    11:30   0:00 [curl] <defunct>
1000340+ 3072396  0.0  0.0      0     0 ?        Z    Sep08   0:00 [curl] <defunct>
1000340+ 3073012  0.0  0.0      0     0 ?        Z    11:31   0:00 [curl] <defunct>
1000340+ 3073658  0.0  0.0      0     0 ?        Z    11:31   0:00 [curl] <defunct>
1000340+ 3076034  0.0  0.0      0     0 ?        Z    11:33   0:00 [curl] <defunct>
1000340+ 3077896  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3081136  0.0  0.0      0     0 ?        Z    11:35   0:00 [curl] <defunct>
1000340+ 3083231  0.0  0.0      0     0 ?        Z    11:36   0:00 [curl] <defunct>
1000340+ 3085184  0.0  0.0      0     0 ?        Z    11:37   0:00 [curl] <defunct>
1000340+ 3085228  0.0  0.0      0     0 ?        Z    11:37   0:00 [curl] <defunct>
1000340+ 3087389  0.0  0.0      0     0 ?        Z    11:39   0:00 [curl] <defunct>
1000340+ 3088983  0.0  0.0      0     0 ?        Z    11:39   0:00 [curl] <defunct>
1000340+ 3092691  0.0  0.0      0     0 ?        Z    11:41   0:00 [curl] <defunct>
1000340+ 3093094  0.0  0.0      0     0 ?        Z    11:42   0:00 [curl] <defunct>
1000340+ 3094650  0.0  0.0      0     0 ?        Z    11:42   0:00 [curl] <defunct>
1000340+ 3094994  0.0  0.0      0     0 ?        Z    11:43   0:00 [curl] <defunct>
1000340+ 3095583  0.0  0.0      0     0 ?        Z    11:43   0:00 [curl] <defunct>
1000340+ 3097114  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3097492  0.0  0.0      0     0 ?        Z    11:44   0:00 [curl] <defunct>
1000340+ 3098739  0.0  0.0      0     0 ?        Z    11:45   0:00 [curl] <defunct>
1000340+ 3101486  0.0  0.0      0     0 ?        Z    11:46   0:00 [curl] <defunct>
1000340+ 3102643  0.0  0.0      0     0 ?        Z    11:47   0:00 [curl] <defunct>
1000340+ 3103721  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3104350  0.0  0.0      0     0 ?        Z    11:48   0:00 [curl] <defunct>
1000340+ 3105354  0.0  0.0      0     0 ?        Z    11:48   0:00 [curl] <defunct>
1000340+ 3105405  0.0  0.0      0     0 ?        Z    11:48   0:00 [curl] <defunct>
1000340+ 3106018  0.0  0.0      0     0 ?        Z    11:48   0:00 [curl] <defunct>
1000340+ 3109802  0.0  0.0      0     0 ?        Z    11:50   0:00 [curl] <defunct>
1000340+ 3110127  0.0  0.0      0     0 ?        Z    11:51   0:00 [curl] <defunct>
1000340+ 3110252  0.0  0.0      0     0 ?        Z    11:51   0:00 [sh] <defunct>
1000340+ 3111992  0.0  0.0      0     0 ?        Z    11:52   0:00 [curl] <defunct>
1000340+ 3113570  0.0  0.0      0     0 ?        Z    11:52   0:00 [curl] <defunct>
1000340+ 3114600  0.0  0.0      0     0 ?        Z    11:53   0:00 [curl] <defunct>
1000340+ 3123029  0.0  0.0      0     0 ?        Z    11:57   0:00 [curl] <defunct>
1000340+ 3125331  0.0  0.0      0     0 ?        Z    11:59   0:00 [sh] <defunct>
1000340+ 3130939  0.0  0.0      0     0 ?        Z    12:02   0:00 [curl] <defunct>
1000340+ 3131859  0.0  0.0      0     0 ?        Z    12:02   0:00 [curl] <defunct>
1000340+ 3133437  0.0  0.0      0     0 ?        Z    12:03   0:00 [curl] <defunct>
1000340+ 3134442  0.0  0.0      0     0 ?        Z    12:03   0:00 [curl] <defunct>
1000340+ 3134443  0.0  0.0      0     0 ?        Z    12:03   0:00 [sh] <defunct>
1000340+ 3134689  0.0  0.0      0     0 ?        Z    12:04   0:00 [curl] <defunct>
1000340+ 3135493  0.0  0.0      0     0 ?        Z    12:04   0:00 [curl] <defunct>
1000340+ 3137154  0.0  0.0      0     0 ?        Z    12:05   0:00 [curl] <defunct>
1000340+ 3137292  0.0  0.0      0     0 ?        Z    12:05   0:00 [curl] <defunct>
1000340+ 3142051  0.0  0.0      0     0 ?        Z    12:07   0:00 [curl] <defunct>
1000340+ 3142085  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3142178  0.0  0.0      0     0 ?        Z    12:08   0:00 [sh] <defunct>
1000340+ 3143324  0.0  0.0      0     0 ?        Z    12:08   0:00 [curl] <defunct>
1000340+ 3144905  0.0  0.0      0     0 ?        Z    12:09   0:00 [curl] <defunct>
1000340+ 3145614  0.0  0.0      0     0 ?        Z    12:09   0:00 [curl] <defunct>
1000340+ 3146407  0.0  0.0      0     0 ?        Z    12:10   0:00 [sh] <defunct>
1000340+ 3147112  0.0  0.0      0     0 ?        Z    12:10   0:00 [curl] <defunct>
1000340+ 3151288  0.0  0.0      0     0 ?        Z    12:12   0:00 [curl] <defunct>
1000340+ 3152419  0.0  0.0      0     0 ?        Z    12:13   0:00 [curl] <defunct>
1000340+ 3152577  0.0  0.0      0     0 ?        Z    12:13   0:00 [curl] <defunct>
1000340+ 3154679  0.0  0.0      0     0 ?        Z    12:14   0:00 [curl] <defunct>
1000340+ 3161703  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3166421  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3173599  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3185898  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3207091  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3211584  0.0  0.0      0     0 ?        Z    Sep09   0:00 [curl] <defunct>
1000340+ 3236475  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3337940  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3393441  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3402687  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3414930  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3434802  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3444274  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3537372  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3559933  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3560408  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3653831  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3675941  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3683439  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3689436  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3717164  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3718037  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3753593  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3787341  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3796486  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3812560  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3814069  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3817865  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3821902  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3826710  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3833854  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3835630  0.0  0.0      0     0 ?        Z    Sep04   0:00 [curl] <defunct>
1000340+ 3838694  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3845584  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3846187  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3849149  0.0  0.0      0     0 ?        Z    Sep07   0:00 [curl] <defunct>
1000340+ 3854972  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3890790  0.0  0.0      0     0 ?        Z    Sep04   0:00 [curl] <defunct>
1000340+ 3897828  0.0  0.0      0     0 ?        Z    Sep01   0:00 [curl] <defunct>
1000340+ 3908913  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3929566  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3932676  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3935466  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3955345  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3967316  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3976391  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3984901  0.0  0.0      0     0 ?        Z    Sep08   0:00 [curl] <defunct>
1000340+ 3986495  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3997775  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 3999316  0.0  0.0      0     0 ?        Z    Sep07   0:00 [curl] <defunct>
1000340+ 4048654  0.0  0.0      0     0 ?        Z    Sep07   0:00 [curl] <defunct>
1000340+ 4074459  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 4109087  0.0  0.0      0     0 ?        Z    Sep10   0:00 [curl] <defunct>
1000340+ 4178712  0.0  0.0      0     0 ?        Z    Sep01   0:00 [curl] <defunct>

Comment 1 Lili Cosic 2020-09-11 14:13:01 UTC

1.	In several clusters the CPU consumption is between 9 and 11 cores while idling. The clusters are a few days old, on some, some test have been executed, others were kept as they are and were idling since creation. 


Is this only monitoring related? Or other components as well. When you mean idle, you mean nothing is running, but cluster is not paused or scaled down?

2. Might be a problem with metrics or rules used to display those. So I would say that one might be ours or console team.

3.	On several nodes I have observed zombie processes. Usually there are 2 or 3, but once 547 are counted. 

Are these monitoring related?

I can't see how all of these are monitoring component problem, can you explain a bit more, thanks!

Comment 2 wvoesch 2020-09-11 14:25:14 UTC

Thank you for the quick response. 

My apologies, if I picked the wrong component, from my perspective monitoring seemed to be a good fit, but perhaps I was wrong about that. 
As I stated, I found all these issues and they might be connected, maybe not. 

1. With idling I mean that all cluster nodes are switched on, but no workload is running on the cluster, other than the one generated by OCP itself. Do you have any suggestion which component might be a better fit?
2. okay. 
3. Maybe zombies disturb the monitoring system? 

How do we proceed from here? Shall we continue point 2 in this bug and I shall open two more bugs for point 1 and 3 against another component? Any suggestions? Or can we move this bug to a different component? 
Thank you.

Comment 3 Lili Cosic 2020-09-11 14:32:41 UTC

1. Its normal for openshift out of the box to use resources, we run a lot of workload, for example on monitoring side Prometheus is still ingesting metrics for all of the platform related things like k8s and openshift components, so its normal to do that.

2. We have a related bug I believe, might be a different one, can you have a look at this one, otherwise feel free to open a new one for that. https://bugzilla.redhat.com/show_bug.cgi?id=1845710

3. Are there any alerts firing to indicate this? Is the stack up and running and healthy, otherwise I would say no.

Comment 4 wvoesch 2020-09-11 14:40:50 UTC

1. This is certainly not normal. In previous releases the "idle" CPU consumption on the exact same hardware is about 1.7-1.9 CPUs. Here the utilization is around 10 CPUs and the cluster only has 16 CPUs. 

2. The mentioned bug specifically states after a reboot. This does not apply here. 

3. The stack is up and running, but there are no alerts concerning zombies.

Comment 5 Ben Parees 2020-09-11 14:43:52 UTC

The zombies are likely caused by this:
https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/

I'm not sure how to figure out which container is leaking them and i don't think they would cause high cpu usage, but they are problematic.  Probably should be a separate bug.

Comment 6 wvoesch 2020-09-11 14:46:48 UTC

Thank you Ben. Any suggestions against which component I should open this bug?

Comment 7 Lili Cosic 2020-09-11 14:47:36 UTC

1. Got it thanks for the info, do you have any more info if idle CPU consumption was for monitoring in earlier versions? Is this reproducable or a one of?
2. Do you mind opening a new bugzilla for this specific problem, thanks!
3. Yes we do not have any those kind of alerts, but we do have alerts that disrupt the stack, so none of these disrupt the monitoring stack.

Comment 8 Ben Parees 2020-09-11 14:54:51 UTC

Can we analyze the prometheus api-request metrics?  This likely indicates a hot-looping controller/operator and would show up as a high rate of api-requests to the apiserver (kube or openshift).

Comment 9 Ben Parees 2020-09-11 14:57:38 UTC

> Thank you Ben. Any suggestions against which component I should open this bug?

unfortunately not, but i'm kinda doubtful any of our product pods are spawning "curl" processes directly.  Are we sure this isn't the result of a user workload or admin script on the cluster?

but here's some guidance on trying to find the cause of zombies:
https://unix.stackexchange.com/questions/33668/is-there-a-way-to-identify-which-process-turns-into-zombie-process

Comment 10 Simon Pasquier 2020-09-11 15:03:13 UTC

(In reply to Ben Parees from comment #9)
> > Thank you Ben. Any suggestions against which component I should open this bug?
> 
> unfortunately not, but i'm kinda doubtful any of our product pods are
> spawning "curl" processes directly.  Are we sure this isn't the result of a
> user workload or admin script on the cluster?

The Prometheus container has liveness/readiness probes that use curl/wget.

Comment 11 wvoesch 2020-09-11 15:37:24 UTC

(In reply to Lili Cosic from comment #7)
> 1. Got it thanks for the info, do you have any more info if idle CPU
> consumption was for monitoring in earlier versions? Is this reproducable or
> a one of?
> 2. Do you mind opening a new bugzilla for this specific problem, thanks!
> 3. Yes we do not have any those kind of alerts, but we do have alerts that
> disrupt the stack, so none of these disrupt the monitoring stack.

1. The idle consumption is discussed in detail in this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1798224 
The situation here is reproducible I would say, as I see this on 4 different clusters, that run on two different CECs (Z13, Z14). These 4 clusters are running 3 different versions of 4.6. I have installed a 5th cluster on a Z15 and until now, the CPU consumption is only slightly higher than expected. I will report again on Monday. 

2. I will do that. Thank you. 

3. Correct. The stack is up and is informing me about the usual things, like it cannot receive an update, which is expected, as it is a nightly build.


(In reply to Ben Parees from comment #9)

Thank you for the link. I will have a closer look. 
This is certainly not caused by any admin script. This cluster was build for reference and has seen no workload. 


(In reply to Ben Parees from comment #9) and (In reply to Simon Pasquier from comment #10)

By now the system counts 677 zombies. I followed the link's descriptions and at first glance it looks as if Thanos and Prometheus are responsible, correct? 

$ ps -ef --forest # because of the lengthy output I removed some parts. Full list is added as attachment.  
UID          PID    PPID  C STIME TTY          TIME CMD
root        7137       1  0 Aug27 ?        00:00:00 /usr/libexec/crio/conmon -b /var/run/containers/storage/overlay-containers/7e73bfbb7e581ff9198d3b1ef1df1a9d103ec1a873a35e4f2f9f75800deadcce/userdata -c 7e73bfbb7e581ff9198d3b1ef1df1a9d103ec1a873a35e4f2f9f75800deadcce -
1000340+    7199    7137  0 Aug27 ?        00:42:14  \_ /bin/thanos query --grpc-address=127.0.0.1:10901 --http-address=127.0.0.1:9090 --query.replica-label=prometheus_replica --query.replica-label=thanos_ruler_replica --store=dnssrv+_grpc._tcp.prometheus-operated.opens
1000340+ 3897828    7199  0 Sep01 ?        00:00:00      \_ [curl] <defunct>
...
1000340+ 1859562    7199  0 Sep09 ?        00:00:00      \_ [curl] <defunct>
1000340+ 1886089    7199  0 Sep09 ?        00:00:00      \_ [sh] <defunct>
1000340+ 1887618    7199  0 Sep09 ?        00:00:00      \_ [curl] <defunct>
...
1000340+ 3497600    7199  0 15:15 ?        00:00:00      \_ [curl] <defunct>

root        9084       1  0 Aug27 ?        00:00:00 /usr/libexec/crio/conmon -b /var/run/containers/storage/overlay-containers/f573dd497990651a4fbe47b6165854bf7d8af0dc1cdb8f240e6baec31efce51e/userdata -c f573dd497990651a4fbe47b6165854bf7d8af0dc1cdb8f240e6baec31efce51e -
1000340+    9152    9084 27 Aug27 ?        4-04:29:31  \_ /bin/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --config.file=/etc/prometheus/config_out/prometheus.env.yaml --storage.tsdb.path=/prometh
1000340+ 1843748    9152  0 Sep09 ?        00:00:00      \_ [curl] <defunct>
root        9085       1  0 Aug27 ?        00:00:00 /usr/libexec/crio/conmon -b /var/run/containers/storage/overlay-containers/c8aab67010bec144cab5835cfc58e4ccce22e85f002a1bf723d9f112cfb8cba3/userdata -c c8aab67010bec144cab5835cfc58e4ccce22e85f002a1bf723d9f112cfb8cba3 -
1000340+    9137    9085 27 Aug27 ?        4-04:14:48  \_ /bin/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --config.file=/etc/prometheus/config_out/prometheus.env.yaml --storage.tsdb.path=/prometh
1000340+ 1787823    9137  0 Sep09 ?        00:00:00      \_ [curl] <defunct>
1000340+ 1813632    9137  0 Sep09 ?        00:00:00      \_ [curl] <defunct>
1000340+  931043    9137  0 Sep10 ?        00:00:00      \_ [curl] <defunct>

root     3293814       1  0 13:27 ?        00:00:00 /usr/libexec/crio/conmon -b /var/run/containers/storage/overlay-containers/52f89083fd2fecbcc9d9e02d5cb956d67928d0fe5cd25e15459aca5fabbf0924/userdata -c 52f89083fd2fecbcc9d9e02d5cb956d67928d0fe5cd25e15459aca5fabbf0924 -
1000340+ 3293840 3293814  3 13:27 ?        00:03:27  \_ /bin/thanos query --grpc-address=127.0.0.1:10901 --http-address=127.0.0.1:9090 --query.replica-label=prometheus_replica --query.replica-label=thanos_ruler_replica --store=dnssrv+_grpc._tcp.prometheus-operated.opens
1000340+ 3300653 3293840  0 13:31 ?        00:00:00      \_ [curl] <defunct>
...
1000340+ 3343560 3293840  0 13:54 ?        00:00:00      \_ [curl] <defunct>
1000340+ 3344894 3293840  0 13:54 ?        00:00:00      \_ [sh] <defunct>
1000340+ 3345400 3293840  0 13:55 ?        00:00:00      \_ [curl] <defunct>
...
1000340+ 3501803 3293840  0 15:17 ?        00:00:00      \_ [curl] <defunct>

Comment 12 wvoesch 2020-09-11 15:38:39 UTC

Created attachment 1714581 [details]
output of "ps -ef --forest".

Comment 13 Ben Parees 2020-09-11 15:40:12 UTC

> By now the system counts 677 zombies. I followed the link's descriptions and at first glance it looks as if Thanos and Prometheus are responsible, correct? 

looks like it(plus Simon's confirmation that curl is being launched), please open that as a separate bug for the Monitoring component for this.

Comment 14 wvoesch 2020-09-11 15:44:58 UTC

Thank you for the confirmation.  

I will open separate bugs for the original problems #2 (problems with the display of CPU utilization) and #3 (zombies). 
How shall we handle problem #1, the surprisingly high CPU utilization? 

Thank you again for your quick responses and the fruitful discussion.

Comment 15 wvoesch 2020-09-14 13:55:09 UTC

The bugs in this bugzilla are opened separately.

https://bugzilla.redhat.com/show_bug.cgi?id=1878766
https://bugzilla.redhat.com/show_bug.cgi?id=1878770
https://bugzilla.redhat.com/show_bug.cgi?id=1878772
https://bugzilla.redhat.com/show_bug.cgi?id=1878774
https://bugzilla.redhat.com/show_bug.cgi?id=1878780

*** This bug has been marked as a duplicate of bug 1878766 ***

Note You need to log in before you can comment on or make changes to this bug.