Bug 768995

Summary: [ovirt] [vdsm] vmStatsThread failing on keyError running on Fedora host as it miss some params in dict
Product: [Retired] oVirt Reporter: Haim <hateya>
Component: vdsmAssignee: Barak <bazulay>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: abaron, acathrow, bazulay, danken, fsimonce, hateya, iheim, ilvovsky, jdenemar, lvroyce, mgoldboi, michal.skrivanek, yeylon, ykaul
Target Milestone: ---   
Target Release: 3.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-15 06:47:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Haim 2011-12-19 16:03:39 UTC
Description of problem:

error looks as follows:

Traceback (most recent call last):
  File "/usr/share/vdsm/utils.py", line 389, in collect
    statsFunction()
  File "/usr/share/vdsm/utils.py", line 266, in __call__
    retValue = self._function(*args, **kwargs)
  File "/usr/share/vdsm/libvirtvm.py", line 157, in _sampleDiskLatency
    stats = _blockstatsParses(out)
  File "/usr/share/vdsm/libvirtvm.py", line 144, in _blockstatsParses
    'flush_op':devStats['flush_operations'],
KeyError: 'flush_operations'

problem is in the following function:
------------------------------------------------------------------------------
def _sampleDiskLatency(self):
        if not self._vm._volumesPrepared:
            # Avoid queries from storage during recovery process
            return

        def _blockstatsParses(devList):
            # The json output looks like:
            # {u'return': [{u'device': u'drive-ide0-0-0',
            #               u'stats': {u'rd_operations': 0, u'flush_total_time_ns': 0, u'wr_highest_offset': 0, u'rd_total_time_ns': 0,
            #                          u'rd_bytes': 0, u'wr_total_time_ns': 0, u'flush_operations': 0, u'wr_operations': 0, u'wr_bytes':0},
            #               u'parent': {u'stats': {u'rd_operations': 0, u'flush_total_time_ns': 0, u'wr_highest_offset': 0,
            #                                      u'rd_total_time_ns': 0, u'rd_bytes': 0, u'wr_total_time_ns': 0, u'flush_operations': 0,
            #                                      u'wr_operations': 0, u'wr_bytes': 0}
            #                          }
            #               },
            #               {u'device': u'drive-ide0-1-0',
            #                u'stats': {u'rd_operations': 0, u'flush_total_time_ns': 0, u'wr_highest_offset': 0, u'rd_total_time_ns': 0,
            #                           u'rd_bytes': 0, u'wr_total_time_ns': 0, u'flush_operations': 0, u'wr_operations': 0, u'wr_bytes': 0}
            #               }],
            #  u'id': u'libvirt-9'}
            stats = {}
            self._log.debug('#### print device list: %s', devList)
            for item in devList['return']:
                fullDevName = item['device']
                alias = fullDevName[len('drive-'):].strip()
                devStats = item['stats']
                stats[alias] = {'rd_op':devStats['rd_operations'],
                                'wr_op':devStats['wr_operations'],
                                'flush_op':devStats['flush_operations'],
                                'rd_total_time_ns':devStats['rd_total_time_ns'],
                                'wr_total_time_ns':devStats['wr_total_time_ns'],
                                'flush_total_time_ns':devStats['flush_total_time_ns']}

------------------------------------------------------------------------------

from printing I added, i noticed that devList is missing some params function relies on; hence failing on keyError:

devList['return'] printing: 

{u'wr_highest_offset': 3183537664, u'rd_bytes': 70507520, u'wr_operations': 189, u'rd_operations': 3402, u'wr_bytes': 1369088}

git commit: 82969b3b5ca2f9487ea56485fafd843b21d4f447

Comment 1 Dan Kenigsberg 2011-12-19 16:19:50 UTC
Haim, which libvirt and qemu version are you using?

Is there a sane libvirt API to obtain this data?

Comment 2 Haim 2011-12-19 19:36:57 UTC
(In reply to comment #1)
> Haim, which libvirt and qemu version are you using?
> 
> Is there a sane libvirt API to obtain this data?

well, for libvirt i use latest upstream git; 98b942c6f51f0b383f9438fe1ec678b6e8c4984e from Dec 13. 
for qemu, qemu-kvm-0.15.1-3.fc16.x86_64 (latest from koji). 
what kind of API are you considering ?

Comment 3 Jiri Denemark 2012-01-17 15:54:03 UTC
I think the API you're looking for is called virDomainBlockStatsFlags.

Comment 4 Igor Lvovsky 2012-07-10 13:51:00 UTC
Please check it with
http://gerrit.ovirt.org/#/c/4304/