Bug 591609

Summary: Migrating guests in RHEVM generates exception traceback emails from Satellite
Product: Red Hat Satellite 5 Reporter: Martin Osvald 🛹 <mosvald>
Component: VirtualizationAssignee: Milan Zázrivec <mzazrivec>
Status: CLOSED ERRATA QA Contact: Dimitar Yordanov <dyordano>
Severity: medium Docs Contact:
Priority: medium    
Version: 530CC: apevec, brian, cperry, dyordano, hchiramm, jhutar, rjones
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, after migrating a Xen guest domain to KVM using Red Hat Enterprise Virtualization Manager, the following error may have occurred: While running 'registration.virt_notify': caught exceptions.KeyError : 'Unknown' This was caused by the use of invalid machine state, "Unknown". With this update, the incorrect state has been changed to "nostate".
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-11 14:34:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 518253    
Attachments:
Description Flags
email with exception none

Description Martin Osvald 🛹 2010-05-12 17:12:40 UTC
Created attachment 413493 [details]
email with exception

Description of problem:

If vdsmd doesn't return state of the machine in domain info, poller.py sends "Unknown" machine state to Satellite which causes exception, because Satellite doesn't know about "Unknown" state.


Version-Release number of selected component (if applicable):

Satellite 5.3 - spacewalk-backend-0.5.28-54
rhn-virtualization-host-5.3.0-17


How reproducible:

always


Steps to Reproduce:

1. Migrate xen to kvm guest through RHEVM (vdsmd has to be enabled at host)

  
Actual results:

see attached email (sensitive data removed)


Expected results:

no exception emails should be sent by Satellite


Additional info:

related client side source code (related code is concatenated with '<<<---'):

===< snip from client/tools/rhn-virtualization/virtualization/poller.py ===
def poll_through_vdsm():
   """
    This method polls all the virt guests running on a VDSM enabled Host.
    Libvirt is disabled by default on RHEV-M managed clients.
    * Imports the localvdsm client that talks to the localhost
      and fetches the list of vms and their info.
    * Extract the data and construct the state to pass it to the 
      execution plan for guest polling.
    * The server should account for business rules similar to
      xen/kvm.
   """
   import localvdsm
   try:
       server = localvdsm.connect()
   except:
       # VDSM raised an exception we're done here
       return {}
   # Extract list of vm's. True returns full list
   try:
       domains = server.list(True)
   except:
       # Something went wrong in vdsm, exit
       return {}

   if not len(domains['vmList']):
       # No domains, exit.
       return

   state = {}
   for domain in domains['vmList']:
       #trim uuid
       uuid = domain['vmId'].lower().replace('-', '')
       # Map the VDSM status to libvirt for server compatibility
       status = "Unknown"          <<<---
       if VIRT_VDSM_STATUS_MAP.has_key(domain['status']):  <<<---
           status = VIRT_VDSM_STATUS_MAP[domain['status']]
       # This is gonna be fully virt as its managed by VDSM
       virt_type = VirtualizationType.FULLY

       #Memory
       memory = int(domain['memSize']) * 1024

       # vcpus
       if domain.has_key('smp'):
           vcpus = domain['smp']
       else:
           vcpus = '1'

       properties = {
           PropertyType.NAME   : domain['vmName'],
           PropertyType.UUID   : uuid,
           PropertyType.TYPE   : virt_type,
           PropertyType.MEMORY : memory, # current memory
           PropertyType.VCPUS  : vcpus,
           PropertyType.STATE  : status}

       state[uuid] = properties

   if state: _log_debug("Polled state: %s" % repr(state))

   return state
=== snip >===

There is also probably a bug in routine which migrate host from xen to kvm with not updating domain status correctly through vdsm during migration, but the fact that the above code sends/could send unknown state for Satellite is a bug. It shouldn't send such state or should send one of the known states:

NOSTATE     = 'nostate'
RUNNING     = 'running'
BLOCKED     = 'blocked'
PAUSED      = 'paused'
SHUTDOWN    = 'shutdown'
SHUTOFF     = 'shutoff'
CRASHED     = 'crashed'

or there should be implemented another state named "Unknown".

Traceback from Satellite side (see attached email for complete info):

Exception Handler Information
Traceback (most recent call last):
  File "/usr/share/rhn/server/apacheRequest.py", line 120, in call_function
    response = apply(func, params)
  File "/usr/share/rhn/server/handlers/xmlrpc/registration.py", line 957, in virt_notify
    rhnVirtualization._virt_notify(server_id, actions)
  File "/usr/share/rhn/server/rhnVirtualization.py", line 844, in _virt_notify
    handler.handle(server_id, action)
  File "/usr/share/rhn/server/rhnVirtualization.py", line 214, in handle
    self.__convert_properties(properties)
  File "/usr/share/rhn/server/rhnVirtualization.py", line 773, in __convert_properties
    properties[PropertyType.STATE] = CLIENT_SERVER_STATE_MAP[state]
KeyError: 'Unknown'

Comment 3 Matthew Booth 2010-06-03 10:10:21 UTC
There's a vocab nit which becomes important in this kind of discussion: we talk about 'converting' a guest from Xen to KVM, and 'migrating' a guest from hypervisor node A to hypervisor node B.

If I'm reading this correctly, a Xen guest has been converted to run on RHEV. It subsequently can't be migrated. Is this correct? How was it converted?

Comment 5 Alan Pevec 2010-06-07 15:55:26 UTC
Do you have a VM in this state? If yes, please run on the Hypervisor: vdsClient -s 0 list

Comment 8 Milan Zázrivec 2010-07-15 16:18:53 UTC
spacewalk.git master:

commit a4be5dcb417d1ad218c205f5cae6adc278f76bf6
Author: Milan Zazrivec <mzazrivec>
Date:   Thu Jul 15 18:14:35 2010 +0200

    591609 - 'Unknown' is not a valid virt. guest state
    
    'nostate' is the valid fallback state value

Comment 9 Milan Zázrivec 2010-07-15 16:21:57 UTC
satellite.git SATELLITE-5.3: 192a1ff8f74dab587d3fc0d21cd8b363d6dbe9c3

Comment 12 Dimitar Yordanov 2010-09-01 08:55:06 UTC
Hi Martin,

 I have just checked that the patch is included in  package rhn-virtualization-5.3.0-18.el5sat.src.rpm

+- 591609 - 'Unknown' is not a valid virt. guest state
-        status = "Unknown"
+        status = 'nostate'

 Unfortunately I faced a problem. I have no access to RHEV-M.
 I would like to ask you to if you can install  rhn-virtualization-common-5.3.0-18.el5sat.noarch.rpm  and  rhn-virtualization-common-5.3.0-18.el5sat.noarch.rpm  on the target machine and let us know if the issue is fixed correctly.
 Thank you in advance.

Regards
Dimitar Yordanov

Comment 13 Jaromir Hradilek 2010-09-03 15:53:15 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, after migrating a Xen guest domain to KVM using Red Hat Enterprise Virtualization Manager (RHEV-M), the following error may have occurred:

  While running 'registration.virt_notify': caught exceptions.KeyError : 'Unknown'

This was caused by the use of invalid machine state, "Unknown". With this update, the incorrect state has been changed to "nostate".

Comment 14 Jaromir Hradilek 2010-09-06 08:20:26 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,4 +1,4 @@
-Previously, after migrating a Xen guest domain to KVM using Red Hat Enterprise Virtualization Manager (RHEV-M), the following error may have occurred:
+Previously, after migrating a Xen guest domain to KVM using Red Hat Enterprise Virtualization Manager, the following error may have occurred:
 
   While running 'registration.virt_notify': caught exceptions.KeyError : 'Unknown'

Comment 21 errata-xmlrpc 2010-10-11 14:34:50 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0761.html