Bug 1305338 - Issue with vdsm-hook-vmfex-dev-4.16.33-1 - "InvalidatedWeakRef"
Summary: Issue with vdsm-hook-vmfex-dev-4.16.33-1 - "InvalidatedWeakRef"
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.5.7
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ovirt-4.0.0-alpha
: ---
Assignee: Dan Kenigsberg
QA Contact: Michael Burman
URL:
Whiteboard:
Depends On: 1308839
Blocks: 1305586
TreeView+ depends on / blocked
 
Reported: 2016-02-07 09:53 UTC by Michael Burman
Modified: 2016-05-01 09:20 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1305586 (view as bug list)
Environment:
Last Closed: 2016-04-25 14:26:41 UTC
oVirt Team: Network
Target Upstream Version:


Attachments (Terms of Use)
vdsm logs (394.46 KB, application/x-gzip)
2016-02-07 09:54 UTC, Michael Burman
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:1671 normal SHIPPED_LIVE VDSM 4.0 GA bug fix and enhancement update 2016-09-02 21:32:03 UTC
oVirt gerrit 53160 master MERGED vmfex hook: hold a ref for a virDomain object 2016-02-08 09:51:26 UTC
oVirt gerrit 53200 ovirt-3.6 MERGED vmfex hook: hold a ref for a virDomain object 2016-02-08 13:23:20 UTC
oVirt gerrit 53202 ovirt-3.6.3 MERGED vmfex hook: hold a ref for a virDomain object 2016-02-08 13:23:50 UTC
oVirt gerrit 53203 ovirt-3.5 MERGED vmfex hook: hold a ref for a virDomain object 2016-02-08 13:32:20 UTC

Description Michael Burman 2016-02-07 09:53:49 UTC
Description of problem:
Issue with vdsm-hook-vmfex-dev-4.16.33-1 - "InvalidatedWeakRef"

It's not possible to use the latest vdsm-hook-vmfex-dev-4.16.33-1(3.5.8) for migration/hotplug/hotunplug 

vdsm.log migration error -->

Thread-965::DEBUG::2016-02-07 11:40:47,698::utils::776::root::(execCmd) FAILED: <err> = 'vmfex_dev hook: [unexpected error]: Traceback (most recent call last):\n  File "/usr/libexec/vdsm/hooks/before_device_migrat
e_destination/50_vmfex", line 179, in <module>\n    main()\n  File "/usr/libexec/vdsm/hooks/before_device_migrate_destination/50_vmfex", line 166, in main\n    handleDirectPool(libvirtconnection.get())\n  File "/u
sr/libexec/vdsm/hooks/before_device_migrate_destination/50_vmfex", line 110, in handleDirectPool\n    elif not qbhInUse(conn) and not isDirectPoolUpToDate(conn):\n  File "/usr/libexec/vdsm/hooks/before_device_migr
ate_destination/50_vmfex", line 84, in qbhInUse\n    domxml = minidom.parseString(conn.lookupByID(vm).XMLDesc(0))\n  File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 119, in wrapper\n    ret
 = f(*args, **kwargs)\n  File "/usr/lib/python2.6/site-packages/vdsm/utils.py", line 1412, in wrapper\n    raise InvalidatedWeakRef()\nInvalidatedWeakRef\n\n\n'; <rc> = 2
Thread-965::INFO::2016-02-07 11:40:47,700::hooks::100::root::(_runHooksDir) vmfex_dev hook: [unexpected error]: Traceback (most recent call last):
  File "/usr/libexec/vdsm/hooks/before_device_migrate_destination/50_vmfex", line 179, in <module>
    main()
  File "/usr/libexec/vdsm/hooks/before_device_migrate_destination/50_vmfex", line 166, in main
    handleDirectPool(libvirtconnection.get())
  File "/usr/libexec/vdsm/hooks/before_device_migrate_destination/50_vmfex", line 110, in handleDirectPool
    elif not qbhInUse(conn) and not isDirectPoolUpToDate(conn):
  File "/usr/libexec/vdsm/hooks/before_device_migrate_destination/50_vmfex", line 84, in qbhInUse
    domxml = minidom.parseString(conn.lookupByID(vm).XMLDesc(0))
  File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 119, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/vdsm/utils.py", line 1412, in wrapper
    raise InvalidatedWeakRef()
InvalidatedWeakRef



Thread-965::ERROR::2016-02-07 11:40:47,701::BindingXMLRPC::1152::vds::(wrapper) unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1136, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 495, in vmMigrationCreate
    return vm.migrationCreate(params)
  File "/usr/share/vdsm/API.py", line 555, in migrationCreate
    if not v.waitForMigrationDestinationPrepare():
  File "/usr/share/vdsm/virt/vm.py", line 5812, in waitForMigrationDestinationPrepare
    dev._deviceXML, self.conf, dev.custom)
  File "/usr/share/vdsm/hooks.py", line 182, in before_device_migrate_destination
    vmconf=vmconf, params=customProperties)
  File "/usr/share/vdsm/hooks.py", line 110, in _runHooksDir
    raise HookError()
HookError

vdsm.log hotplug error -->

jsonrpc.Executor-worker-5::DEBUG::2016-02-07 11:44:01,728::utils::776::root::(execCmd) FAILED: <err> = 'vmfex_dev hook: [unexpected error]: Traceback (most recent call last):\n  File "/usr/libexec/vdsm/hooks/befor
e_nic_hotplug/50_vmfex", line 179, in <module>\n    main()\n  File "/usr/libexec/vdsm/hooks/before_nic_hotplug/50_vmfex", line 166, in main\n    handleDirectPool(libvirtconnection.get())\n  File "/usr/libexec/vdsm
/hooks/before_nic_hotplug/50_vmfex", line 110, in handleDirectPool\n    elif not qbhInUse(conn) and not isDirectPoolUpToDate(conn):\n  File "/usr/libexec/vdsm/hooks/before_nic_hotplug/50_vmfex", line 84, in qbhInU
se\n    domxml = minidom.parseString(conn.lookupByID(vm).XMLDesc(0))\n  File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 119, in wrapper\n    ret = f(*args, **kwargs)\n  File "/usr/lib/pytho
n2.6/site-packages/vdsm/utils.py", line 1412, in wrapper\n    raise InvalidatedWeakRef()\nInvalidatedWeakRef\n\n\n'; <rc> = 2
jsonrpc.Executor-worker-5::INFO::2016-02-07 11:44:01,729::hooks::100::root::(_runHooksDir) vmfex_dev hook: [unexpected error]: Traceback (most recent call last):
  File "/usr/libexec/vdsm/hooks/before_nic_hotplug/50_vmfex", line 179, in <module>
    main()
  File "/usr/libexec/vdsm/hooks/before_nic_hotplug/50_vmfex", line 166, in main
    handleDirectPool(libvirtconnection.get())
  File "/usr/libexec/vdsm/hooks/before_nic_hotplug/50_vmfex", line 110, in handleDirectPool
    elif not qbhInUse(conn) and not isDirectPoolUpToDate(conn):
  File "/usr/libexec/vdsm/hooks/before_nic_hotplug/50_vmfex", line 84, in qbhInUse
    domxml = minidom.parseString(conn.lookupByID(vm).XMLDesc(0))
  File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 119, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/vdsm/utils.py", line 1412, in wrapper
    raise InvalidatedWeakRef()
InvalidatedWeakRef



jsonrpc.Executor-worker-5::ERROR::2016-02-07 11:44:01,729::__init__::506::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/yajsonrpc/__init__.py", line 501, in _serveRequest
    res = method(**params)
  File "/usr/share/vdsm/rpc/Bridge.py", line 271, in _dynamicMethod
    result = fn(*methodArgs)
  File "/usr/share/vdsm/API.py", line 419, in hotplugNic
    return curVm.hotplugNic(params)
  File "/usr/share/vdsm/virt/vm.py", line 3492, in hotplugNic
    params=nic.custom)
  File "/usr/share/vdsm/hooks.py", line 266, in before_nic_hotplug
    params=params)
  File "/usr/share/vdsm/hooks.py", line 110, in _runHooksDir
    raise HookError()
HookError

Version-Release number of selected component (if applicable):
vdsm-4.16.33-1.el6ev.x86_64
3.5.7-0.1.el6ev


Steps to Reproduce:
1. Tr to use the latest vdsm-hook-vmfex-dev-4.16.33-1 hook for migration/hotplug


Actual results:
Failed with error "Unexpected exception"

Expected results:
Should work as expected

Comment 1 Michael Burman 2016-02-07 09:54:16 UTC
Created attachment 1121866 [details]
vdsm logs

Comment 3 Michael Burman 2016-03-17 11:04:55 UTC
Verified on - 3.6.4-0.1.el6 and vdsm-4.17.23.1-0.el7ev.noarch

Comment 4 Eyal Edri 2016-04-25 14:26:41 UTC
3.6.5 released

Comment 5 Dan Kenigsberg 2016-05-01 08:58:36 UTC
bygone


Note You need to log in before you can comment on or make changes to this bug.