Bug 680849

Summary: The CPU occupancy rate is around 100% on rhev-h 5.6-9
Product: Red Hat Enterprise Linux 5 Reporter: Guohua Ouyang <gouyang>
Component: vdsm22Assignee: Dan Kenigsberg <danken>
Status: CLOSED NEXTRELEASE QA Contact: yeylon <yeylon>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.6CC: abaron, apevec, bazulay, cpelland, danken, iheim, lilu, mburns, moli, ovirt-maint, qwan, srevivo, vbian, ycui, ykaul
Target Milestone: rcKeywords: Regression, TestBlocker, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-03-02 15:51:30 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 681148    
Attachments:
Description Flags
ps.log
none
top.output none

Description Guohua Ouyang 2011-02-28 07:10:42 UTC 
Description of problem:
The CPU occupancy rate can reach 100% on a clean new installed build of rhev-h 5.6-9.

#top
Tasks: 410 total,  83 running, 309 sleeping,   0 stopped,  18 zombie
Cpu(s): 97.0%us,  3.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3848572k total,  1549424k used,  2299148k free,    56696k buffers
Swap:     8184k total,        0k used,     8184k free,   462944k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 6813 vdsm      16  -5 85808 6172 2800 R 31.6  0.2   0:06.15 python             
 6926 vdsm      16  -5 85808 6180 2800 R 31.6  0.2   0:07.15 python             
 8312 vdsm      17  -5 85808 6172 2800 R 31.6  0.2   0:01.04 python             
 7036 vdsm      20  -5 85812 6180 2800 R 16.1  0.2   0:07.79 python             
 7984 vdsm      20  -5 85812 6176 2800 R 16.1  0.2   0:07.15 python             
 8397 vdsm      15  -5 85812 6180 2800 R 16.1  0.2   0:00.55 python             
 6575 vdsm      20  -5 85812 6184 2800 R 15.8  0.2   0:09.54 python             
 6600 vdsm      20  -5 85812 6180 2800 R 15.8  0.2   0:04.65 python             
 6645 vdsm      20  -5 85816 6184 2800 R 15.8  0.2   0:10.07 python             
 6672 vdsm      20  -5 85816 6176 2800 R 15.8  0.2   0:04.58 python             
 6724 vdsm      20  -5 85816 6176 2800 R 15.8  0.2   0:06.15 python             
 6770 vdsm      20  -5 85816 6188 2800 R 15.8  0.2   0:05.58 python             
 6822 vdsm      18  -5 85808 6176 2800 R 15.8  0.2   0:01.54 python             
 6833 vdsm      18  -5 85780 6172 2800 R 15.8  0.2   0:06.65 python             
 6944 vdsm      20  -5 85808 6180 2800 R 15.8  0.2   0:05.04 python             
 7009 vdsm      20  -5 85812 6180 2800 R 15.8  0.2   0:03.66 python             
 8388 vdsm      20  -5 85812 6176 2800 R 15.8  0.2   0:02.04 python 

#free
total      used       free     shared    buffers     cached       
Mem:       3848572    1705556    2143016          0      56748     463080       
-/+ buffers/cache:    1185728    2662844                                        
Swap:         8184          0       8184


Version-Release number of selected component (if applicable):
rhev-hypervisor-5.6-9

vdsm version:
vdsm22-4.5-63.20.el5_6

How reproducible:
Always.

Steps to Reproduce:
1. Install RHEVH via PXE/USB/CDROM.
2. Login by root, check CPU status by command "top".

Actual results:
The cpu occupancy rate is about 100%.

Expected results:
The cpu occupancy rate should lower than 5%.

Additional info: 
Attaching a log from 'ps'. Seems a plenty of 'storage/remoteFileHandler.pyc' with strange paths were running at the same time.
Comment 1 Guohua Ouyang 2011-02-28 07:13:03 UTC 
Created attachment 481316 [details]
ps.log
Comment 2 Guohua Ouyang 2011-02-28 07:17:27 UTC 
Created attachment 481321 [details]
top.output
Comment 3 Dan Kenigsberg 2011-02-28 20:35:04 UTC 
The problem here is that when vdsm imports storage.misc it checks if it is running on rhev-h by firing the new out-of-process mechanism, which in its turn imports misc again. It's all turtles from their to the abyss.

There was no need for such a convoluted check in the first place. And there was certainly no need to do it out-of-process.

The following patch takes ovirtness check from RHEL6's vdsm and avoids the curse of endless recursion:

http://post-office.corp.redhat.com/archives/rhev-patches/2011-February/msg01944.html
Comment 5 Dan Kenigsberg 2011-03-02 15:51:30 UTC 
grrr. closing this annoying clone.