Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1305338

Summary:

Issue with vdsm-hook-vmfex-dev-4.16.33-1 - "InvalidatedWeakRef"

Product:

Red Hat Enterprise Virtualization Manager

Reporter:

Michael Burman <mburman>

Component:

vdsm

Assignee:

Dan Kenigsberg <danken>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Michael Burman <mburman>

Severity:

urgent

Docs Contact:

Priority:

unspecified

Version:

3.5.7

CC:

bazulay, eedri, gklein, lsurette, myakove, snaftaly, srevivo, ycui, ylavi

Target Milestone:

ovirt-4.0.0-alpha

Keywords:

Regression

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Clones:

1305586 (view as bug list)

Environment:

Last Closed:

2016-04-25 14:26:41 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

Network

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

1308839

Bug Blocks:

1305586

Attachments:

Description	Flags
vdsm logs	none

Description Michael Burman 2016-02-07 09:53:49 UTC

Description of problem:
Issue with vdsm-hook-vmfex-dev-4.16.33-1 - "InvalidatedWeakRef"

It's not possible to use the latest vdsm-hook-vmfex-dev-4.16.33-1(3.5.8) for migration/hotplug/hotunplug 

vdsm.log migration error -->

Thread-965::DEBUG::2016-02-07 11:40:47,698::utils::776::root::(execCmd) FAILED: <err> = 'vmfex_dev hook: [unexpected error]: Traceback (most recent call last):\n  File "/usr/libexec/vdsm/hooks/before_device_migrat
e_destination/50_vmfex", line 179, in <module>\n    main()\n  File "/usr/libexec/vdsm/hooks/before_device_migrate_destination/50_vmfex", line 166, in main\n    handleDirectPool(libvirtconnection.get())\n  File "/u
sr/libexec/vdsm/hooks/before_device_migrate_destination/50_vmfex", line 110, in handleDirectPool\n    elif not qbhInUse(conn) and not isDirectPoolUpToDate(conn):\n  File "/usr/libexec/vdsm/hooks/before_device_migr
ate_destination/50_vmfex", line 84, in qbhInUse\n    domxml = minidom.parseString(conn.lookupByID(vm).XMLDesc(0))\n  File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 119, in wrapper\n    ret
 = f(*args, **kwargs)\n  File "/usr/lib/python2.6/site-packages/vdsm/utils.py", line 1412, in wrapper\n    raise InvalidatedWeakRef()\nInvalidatedWeakRef\n\n\n'; <rc> = 2
Thread-965::INFO::2016-02-07 11:40:47,700::hooks::100::root::(_runHooksDir) vmfex_dev hook: [unexpected error]: Traceback (most recent call last):
  File "/usr/libexec/vdsm/hooks/before_device_migrate_destination/50_vmfex", line 179, in <module>
    main()
  File "/usr/libexec/vdsm/hooks/before_device_migrate_destination/50_vmfex", line 166, in main
    handleDirectPool(libvirtconnection.get())
  File "/usr/libexec/vdsm/hooks/before_device_migrate_destination/50_vmfex", line 110, in handleDirectPool
    elif not qbhInUse(conn) and not isDirectPoolUpToDate(conn):
  File "/usr/libexec/vdsm/hooks/before_device_migrate_destination/50_vmfex", line 84, in qbhInUse
    domxml = minidom.parseString(conn.lookupByID(vm).XMLDesc(0))
  File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 119, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/vdsm/utils.py", line 1412, in wrapper
    raise InvalidatedWeakRef()
InvalidatedWeakRef



Thread-965::ERROR::2016-02-07 11:40:47,701::BindingXMLRPC::1152::vds::(wrapper) unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1136, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 495, in vmMigrationCreate
    return vm.migrationCreate(params)
  File "/usr/share/vdsm/API.py", line 555, in migrationCreate
    if not v.waitForMigrationDestinationPrepare():
  File "/usr/share/vdsm/virt/vm.py", line 5812, in waitForMigrationDestinationPrepare
    dev._deviceXML, self.conf, dev.custom)
  File "/usr/share/vdsm/hooks.py", line 182, in before_device_migrate_destination
    vmconf=vmconf, params=customProperties)
  File "/usr/share/vdsm/hooks.py", line 110, in _runHooksDir
    raise HookError()
HookError

vdsm.log hotplug error -->

jsonrpc.Executor-worker-5::DEBUG::2016-02-07 11:44:01,728::utils::776::root::(execCmd) FAILED: <err> = 'vmfex_dev hook: [unexpected error]: Traceback (most recent call last):\n  File "/usr/libexec/vdsm/hooks/befor
e_nic_hotplug/50_vmfex", line 179, in <module>\n    main()\n  File "/usr/libexec/vdsm/hooks/before_nic_hotplug/50_vmfex", line 166, in main\n    handleDirectPool(libvirtconnection.get())\n  File "/usr/libexec/vdsm
/hooks/before_nic_hotplug/50_vmfex", line 110, in handleDirectPool\n    elif not qbhInUse(conn) and not isDirectPoolUpToDate(conn):\n  File "/usr/libexec/vdsm/hooks/before_nic_hotplug/50_vmfex", line 84, in qbhInU
se\n    domxml = minidom.parseString(conn.lookupByID(vm).XMLDesc(0))\n  File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 119, in wrapper\n    ret = f(*args, **kwargs)\n  File "/usr/lib/pytho
n2.6/site-packages/vdsm/utils.py", line 1412, in wrapper\n    raise InvalidatedWeakRef()\nInvalidatedWeakRef\n\n\n'; <rc> = 2
jsonrpc.Executor-worker-5::INFO::2016-02-07 11:44:01,729::hooks::100::root::(_runHooksDir) vmfex_dev hook: [unexpected error]: Traceback (most recent call last):
  File "/usr/libexec/vdsm/hooks/before_nic_hotplug/50_vmfex", line 179, in <module>
    main()
  File "/usr/libexec/vdsm/hooks/before_nic_hotplug/50_vmfex", line 166, in main
    handleDirectPool(libvirtconnection.get())
  File "/usr/libexec/vdsm/hooks/before_nic_hotplug/50_vmfex", line 110, in handleDirectPool
    elif not qbhInUse(conn) and not isDirectPoolUpToDate(conn):
  File "/usr/libexec/vdsm/hooks/before_nic_hotplug/50_vmfex", line 84, in qbhInUse
    domxml = minidom.parseString(conn.lookupByID(vm).XMLDesc(0))
  File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 119, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/vdsm/utils.py", line 1412, in wrapper
    raise InvalidatedWeakRef()
InvalidatedWeakRef



jsonrpc.Executor-worker-5::ERROR::2016-02-07 11:44:01,729::__init__::506::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/yajsonrpc/__init__.py", line 501, in _serveRequest
    res = method(**params)
  File "/usr/share/vdsm/rpc/Bridge.py", line 271, in _dynamicMethod
    result = fn(*methodArgs)
  File "/usr/share/vdsm/API.py", line 419, in hotplugNic
    return curVm.hotplugNic(params)
  File "/usr/share/vdsm/virt/vm.py", line 3492, in hotplugNic
    params=nic.custom)
  File "/usr/share/vdsm/hooks.py", line 266, in before_nic_hotplug
    params=params)
  File "/usr/share/vdsm/hooks.py", line 110, in _runHooksDir
    raise HookError()
HookError

Version-Release number of selected component (if applicable):
vdsm-4.16.33-1.el6ev.x86_64
3.5.7-0.1.el6ev


Steps to Reproduce:
1. Tr to use the latest vdsm-hook-vmfex-dev-4.16.33-1 hook for migration/hotplug


Actual results:
Failed with error "Unexpected exception"

Expected results:
Should work as expected

Comment 1 Michael Burman 2016-02-07 09:54:16 UTC

Created attachment 1121866 [details]
vdsm logs

Comment 3 Michael Burman 2016-03-17 11:04:55 UTC

Verified on - 3.6.4-0.1.el6 and vdsm-4.17.23.1-0.el7ev.noarch

Comment 4 Eyal Edri 2016-04-25 14:26:41 UTC

3.6.5 released

Comment 5 Dan Kenigsberg 2016-05-01 08:58:36 UTC

bygone