Bug 756380

Summary: virt-who loses reading guest info upon libvirtd restarts
Product: Red Hat Enterprise Linux 5 Reporter: Keqin Hong <khong>
Component: virt-whoAssignee: Radek Novacek <rnovacek>
Status: CLOSED ERRATA QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 5.8CC: apevec, cshao, dallan, gouyang, huiwang, jsefler, leiwang, mkhusid, moli, ovasik, rvokal, ycui
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: virt-who-0.5-2.el5 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 746163 Environment:
Last Closed: 2012-02-21 06:06:46 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 746163    
Bug Blocks: 758797, 769266    

Comment 1 Radek Novacek 2011-11-30 12:59:17 UTC 
I have done suggested changes, see upstream commit:

https://fedorahosted.org/virt-who/changeset/15d839c36e9aedd4d653833d6a713acd6a2f60ba

Virt-who should now handle various errors (like network problem, wrong configuration etc.) more correctly.
Comment 2 Keqin Hong 2011-11-30 13:36:20 UTC 
Hi Radek,
Hosted QE will try it soon.
In the mean time, would you like to make a new brew build?

Thanks,
Keqin
Comment 3 Radek Novacek 2011-12-02 08:17:18 UTC 
Brew build done:

https://brewweb.devel.redhat.com/taskinfo?taskID=3865459
Comment 6 Keqin Hong 2011-12-09 11:38:55 UTC 
Notice Comment 4 and Comment 5 were tested with virt-who-0.4-1.el5.
Comment 7 Radek Novacek 2011-12-09 14:58:17 UTC 
It looks that when xen is used, the events about guest change (add/remove) arrives before the list of domains is updated. I have done workaround for that. It manually adds/removes domain to/from the list when new events is received. That should fix the issue. This change will be in the next build of virt-who.
Comment 8 Radek Novacek 2011-12-13 14:35:15 UTC 
This bug should be fixed in virt-who-0.5-1.el5

https://brewweb.devel.redhat.com/taskinfo?taskID=3889752
Comment 10 Hui Wang 2011-12-14 07:00:04 UTC 
Add/Delete a VM on RHEL-5.8-Server-beta-1.0 (Xen-HVM) ,it
can update the guests' info.

Steps to Reproduce:
1. ensure there are vms installed by virt-manager/libvirt
2. enable virt-who debug
# vi /etc/sysconfig/virt-who
VIRTWHO_BACKGROUND=1
VIRTWHO_DEBUG=1
3. start virt-who daemon on terminal1
# service virt-who restart
4. restart libvirtd service on terminal2
# service libvirtd restart
5. delete/add a VM
6. check the output of terminal1:

[root@localhost home]# service virt-who restart
................

^^^^^^^^^^^^After restarting libvirtd and then adding new guest rhel6.2-64-hvm
EVENT: Domain rhel6.2-64-hvm(12) Started Booted
Virtual machine found: Domain-0: 00000000-0000-0000-0000-000000000000
Virtual machine found: rhel6.2-64-hvm: bb61ed8f-6d86-5658-25d0-d6a5ba75db82
Virtual machine found: win: 05e1f656-3ff2-9cc5-0b3e-27c3de9ab811
Virtual machine found: rhel6.2-32-pv: 6edb91bc-ec4c-943d-2130-44da54044bb1
Virtual machine found: rhel6-64-pv: 68a0b766-0567-8e2e-646a-5631d03c45ff
Sending update to updateConsumer: ['00000000-0000-0000-0000-000000000000', '05e1f656-3ff2-9cc5-0b3e-27c3de9ab811', '68a0b766-0567-8e2e-646a-5631d03c45ff', '6edb91bc-ec4c-943d-2130-44da54044bb1', 'bb61ed8f-6d86-5658-25d0-d6a5ba75db82']
EVENT: Domain rhel6.2-64-hvm(-1) Defined Added
Virtual machine found: Domain-0: 00000000-0000-0000-0000-000000000000
Virtual machine found: rhel6.2-64-hvm: bb61ed8f-6d86-5658-25d0-d6a5ba75db82
Virtual machine found: win: 05e1f656-3ff2-9cc5-0b3e-27c3de9ab811
Virtual machine found: rhel6.2-32-pv: 6edb91bc-ec4c-943d-2130-44da54044bb1
Virtual machine found: rhel6-64-pv: 68a0b766-0567-8e2e-646a-5631d03c45ff
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/libvirt.py", line 1941, in dispatchDomainEventCallbacks
    cb(self,dom,event,detail,opaque)
  File "/usr/share/virt-who/virt.py", line 77, in changed
    if d.UUIDString() == uuid:
UnboundLocalError: local variable 'uuid' referenced before assignment
EVENT: Domain rhel6.2-64-hvm(-1) Defined Added
Virtual machine found: Domain-0: 00000000-0000-0000-0000-000000000000
Virtual machine found: rhel6.2-64-hvm: bb61ed8f-6d86-5658-25d0-d6a5ba75db82
Virtual machine found: win: 05e1f656-3ff2-9cc5-0b3e-27c3de9ab811
Virtual machine found: rhel6.2-32-pv: 6edb91bc-ec4c-943d-2130-44da54044bb1
Virtual machine found: rhel6-64-pv: 68a0b766-0567-8e2e-646a-5631d03c45ff
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/libvirt.py", line 1941, in dispatchDomainEventCallbacks
    cb(self,dom,event,detail,opaque)
  File "/usr/share/virt-who/virt.py", line 77, in changed
    if d.UUIDString() == uuid:
UnboundLocalError: local variable 'uuid' referenced before assignment
Virtual machine found: Domain-0: 00000000-0000-0000-0000-000000000000
Virtual machine found: rhel6.2-64-hvm: bb61ed8f-6d86-5658-25d0-d6a5ba75db82
Virtual machine found: win: 05e1f656-3ff2-9cc5-0b3e-27c3de9ab811
Virtual machine found: rhel6.2-32-pv: 6edb91bc-ec4c-943d-2130-44da54044bb1
Virtual machine found: rhel6-64-pv: 68a0b766-0567-8e2e-646a-5631d03c45ff
Sending update to updateConsumer: ['00000000-0000-0000-0000-000000000000', '05e1f656-3ff2-9cc5-0b3e-27c3de9ab811', '68a0b766-0567-8e2e-646a-5631d03c45ff', '6edb91bc-ec4c-943d-2130-44da54044bb1', 'bb61ed8f-6d86-5658-25d0-d6a5ba75db82']
EVENT: Domain rhel6.2-64-hvm(-1) Stopped Shutdown
Virtual machine found: Domain-0: 00000000-0000-0000-0000-000000000000
Virtual machine found: win: 05e1f656-3ff2-9cc5-0b3e-27c3de9ab811
Virtual machine found: rhel6.2-64-hvm: bb61ed8f-6d86-5658-25d0-d6a5ba75db82
Virtual machine found: rhel6.2-32-pv: 6edb91bc-ec4c-943d-2130-44da54044bb1
Virtual machine found: rhel6-64-pv: 68a0b766-0567-8e2e-646a-5631d03c45ff
Sending update to updateConsumer: ['00000000-0000-0000-0000-000000000000', '05e1f656-3ff2-9cc5-0b3e-27c3de9ab811', '68a0b766-0567-8e2e-646a-5631d03c45ff', '6edb91bc-ec4c-943d-2130-44da54044bb1', 'bb61ed8f-6d86-5658-25d0-d6a5ba75db82']

^^^^^^^^^^^^After restarting libvirtd and then deleting new guest rhel6.2-64-hvm

EVENT: Domain rhel6.2-64-hvm(-1) Undefined R
Virtual machine found: Domain-0: 00000000-0000-0000-0000-000000000000
Virtual machine found: win: 05e1f656-3ff2-9cc5-0b3e-27c3de9ab811
Virtual machine found: rhel6.2-64-hvm: bb61ed8f-6d86-5658-25d0-d6a5ba75db82
Virtual machine found: rhel6.2-32-pv: 6edb91bc-ec4c-943d-2130-44da54044bb1
Virtual machine found: rhel6-64-pv: 68a0b766-0567-8e2e-646a-5631d03c45ff
Sending update to updateConsumer: ['00000000-0000-0000-0000-000000000000', '05e1f656-3ff2-9cc5-0b3e-27c3de9ab811', '68a0b766-0567-8e2e-646a-5631d03c45ff', '6edb91bc-ec4c-943d-2130-44da54044bb1']
Comment 11 Radek Novacek 2011-12-14 07:58:43 UTC 
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/libvirt.py", line 1941, in
dispatchDomainEventCallbacks
    cb(self,dom,event,detail,opaque)
  File "/usr/share/virt-who/virt.py", line 77, in changed
    if d.UUIDString() == uuid:
UnboundLocalError: local variable 'uuid' referenced before assignment

Huh, sorry, I forgot to commit the fix before making the release. Fixed in virt-who-0.5-2.el5.
Comment 14 Hui Wang 2011-12-22 07:36:01 UTC 
I verified the issue,and the result is pass.

Verifying rpms Version:
virt-who-0.5-3.el5
subscription-manager-0.98.9-1.el5
subscription-manager-firstboot-0.98.9-1.el5
subscription-manager-gnome-0.98.9-1.el5   
python-rhsm-0.98.8-1.el5
libvirt-0.8.2-24.el5
libvirt-python-0.8.2-24.el5
Comment 15 Hui Wang 2011-12-26 05:02:53 UTC 
I verified the issue,and the result is pass.

Verifying rpms Version:
virt-who-0.5-3.el5
subscription-manager-0.98.9-1.el5
subscription-manager-firstboot-0.98.9-1.el5
subscription-manager-gnome-0.98.9-1.el5   
python-rhsm-0.98.8-1.el5
libvirt-0.8.2-24.el5
libvirt-python-0.8.2-24.el5

Steps to verify:
1. ensure there are vms installed by virt-manager/libvirt on xen (pv and xen hvm)
2. enable virt-who debug
# vi /etc/sysconfig/virt-who
VIRTWHO_BACKGROUND=1
VIRTWHO_DEBUG=1
3. start virt-who daemon on terminal1
# service virt-who restart
4. restart libvirtd service on terminal2
# service libvirtd restart
5.restart the virt-manager
6. add/delete a VM
7. check the output of terminal1:
Stopping virt-who:                                         [  OK  ]
Starting virt-who: Virt-who is running in libvirt mode
Starting infinite loop with 3600 seconds interval and event handling
                                                           [  OK  ]
[root@localhost ~]# Unable to obtain status from server, UEPConnection is likely not usable:
Traceback (most recent call last):
  File "/usr/share/virt-who/subscriptionmanager.py", line 60, in connect
    if not self.connection.ping()['result']:
  File "/usr/lib/python2.4/site-packages/rhsm/connection.py", line 411, in ping
    return self.conn.request_get("/status/")
  File "/usr/lib/python2.4/site-packages/rhsm/connection.py", line 267, in request_get
    return self._request("GET", method)
  File "/usr/lib/python2.4/site-packages/rhsm/connection.py", line 234, in _request
    self.validateResponse(result)
  File "/usr/lib/python2.4/site-packages/rhsm/connection.py", line 249, in validateResponse
    raise RemoteServerException(response['status'])
RemoteServerException: <unprintable instance object>
Virtual machine found: rhel5.8snaphsot2.0-server-32: ea898b80-5200-6ba1-6bb6-5dcd3ab07369
Virtual machine found: rhel4: 39b783f1-d199-0725-c4b1-50b14ca6dea3
Sending update to updateConsumer: ['39b783f1-d199-0725-c4b1-50b14ca6dea3', 'ea898b80-5200-6ba1-6bb6-5dcd3ab07369'] 

^^^^^^^^^^^Add guest rheladdpv(xen pv),the report info as following:
EVENT: Domain rheladdpv(5) Started Booted
Virtual machine found: rhel5.8snaphsot2.0-server-32: ea898b80-5200-6ba1-6bb6-5dcd3ab07369
Virtual machine found: rheladdpv: 1eebed71-934f-9169-a81b-ad6b0fb65019
Virtual machine found: rhel4: 39b783f1-d199-0725-c4b1-50b14ca6dea3
Sending update to updateConsumer: ['1eebed71-934f-9169-a81b-ad6b0fb65019', '39b783f1-d199-0725-c4b1-50b14ca6dea3', 'ea898b80-5200-6ba1-6bb6-5dcd3ab07369']
EVENT: Domain rheladdpv(-1) Defined Added
Virtual machine found: rhel5.8snaphsot2.0-server-32: ea898b80-5200-6ba1-6bb6-5dcd3ab07369
Virtual machine found: rheladdpv: 1eebed71-934f-9169-a81b-ad6b0fb65019
Virtual machine found: rhel4: 39b783f1-d199-0725-c4b1-50b14ca6dea3
Sending update to updateConsumer: ['1eebed71-934f-9169-a81b-ad6b0fb65019', '39b783f1-d199-0725-c4b1-50b14ca6dea3', 'ea898b80-5200-6ba1-6bb6-5dcd3ab07369']
EVENT: Domain rheladdpv(-1) Defined Added
Virtual machine found: rhel5.8snaphsot2.0-server-32: ea898b80-5200-6ba1-6bb6-5dcd3ab07369
Virtual machine found: rheladdpv: 1eebed71-934f-9169-a81b-ad6b0fb65019
Virtual machine found: rhel4: 39b783f1-d199-0725-c4b1-50b14ca6dea3
Sending update to updateConsumer: ['1eebed71-934f-9169-a81b-ad6b0fb65019', '39b783f1-d199-0725-c4b1-50b14ca6dea3', 'ea898b80-5200-6ba1-6bb6-5dcd3ab07369']

^^^^^^^^^^^^^^Add guest rheladdhvm(xen hvm),the report info as following:
EVENT: Domain rheladdhvm(6) Started Booted
Virtual machine found: rhel5.8snaphsot2.0-server-32: ea898b80-5200-6ba1-6bb6-5dcd3ab07369
Virtual machine found: rheladdhvm: 84b77d39-e4ed-af26-2312-083a5a92e667
Virtual machine found: rhel4: 39b783f1-d199-0725-c4b1-50b14ca6dea3
Virtual machine found: rheladdpv: 1eebed71-934f-9169-a81b-ad6b0fb65019
Sending update to updateConsumer: ['1eebed71-934f-9169-a81b-ad6b0fb65019', '39b783f1-d199-0725-c4b1-50b14ca6dea3', '84b77d39-e4ed-af26-2312-083a5a92e667', 'ea898b80-5200-6ba1-6bb6-5dcd3ab07369']
EVENT: Domain rheladdhvm(-1) Defined Added
Virtual machine found: rhel5.8snaphsot2.0-server-32: ea898b80-5200-6ba1-6bb6-5dcd3ab07369
Virtual machine found: rheladdhvm: 84b77d39-e4ed-af26-2312-083a5a92e667
Virtual machine found: rhel4: 39b783f1-d199-0725-c4b1-50b14ca6dea3
Virtual machine found: rheladdpv: 1eebed71-934f-9169-a81b-ad6b0fb65019
Sending update to updateConsumer: ['1eebed71-934f-9169-a81b-ad6b0fb65019', '39b783f1-d199-0725-c4b1-50b14ca6dea3', '84b77d39-e4ed-af26-2312-083a5a92e667', 'ea898b80-5200-6ba1-6bb6-5dcd3ab07369']
EVENT: Domain rheladdhvm(-1) Defined Added
Virtual machine found: rhel5.8snaphsot2.0-server-32: ea898b80-5200-6ba1-6bb6-5dcd3ab07369
Virtual machine found: rheladdhvm: 84b77d39-e4ed-af26-2312-083a5a92e667
Virtual machine found: rhel4: 39b783f1-d199-0725-c4b1-50b14ca6dea3
Virtual machine found: rheladdpv: 1eebed71-934f-9169-a81b-ad6b0fb65019
Sending update to updateConsumer: ['1eebed71-934f-9169-a81b-ad6b0fb65019', '39b783f1-d199-0725-c4b1-50b14ca6dea3', '84b77d39-e4ed-af26-2312-083a5a92e667', 'ea898b80-5200-6ba1-6bb6-5dcd3ab07369']

^^^^^^^^^Delete guest rheladdpv(xen pv),the report info as following:
EVENT: Domain rheladdpv(-1) Undefined R
Virtual machine found: rhel5.8snaphsot2.0-server-32: ea898b80-5200-6ba1-6bb6-5dcd3ab07369
Virtual machine found: rheladdhvm: 84b77d39-e4ed-af26-2312-083a5a92e667
Virtual machine found: rhel4: 39b783f1-d199-0725-c4b1-50b14ca6dea3
Virtual machine found: rheladdpv: 1eebed71-934f-9169-a81b-ad6b0fb65019
Sending update to updateConsumer: ['39b783f1-d199-0725-c4b1-50b14ca6dea3', '84b77d39-e4ed-af26-2312-083a5a92e667', 'ea898b80-5200-6ba1-6bb6-5dcd3ab07369']


^^^^^^^^Delete guest rheladdhvm(xen pv),the report info as following:
EVENT: Domain rheladdhvm(-1) Undefined R
Virtual machine found: rhel5.8snaphsot2.0-server-32: ea898b80-5200-6ba1-6bb6-5dcd3ab07369
Virtual machine found: rhel4: 39b783f1-d199-0725-c4b1-50b14ca6dea3
Virtual machine found: rheladdhvm: 84b77d39-e4ed-af26-2312-083a5a92e667
Sending update to updateConsumer: ['39b783f1-d199-0725-c4b1-50b14ca6dea3', 'ea898b80-5200-6ba1-6bb6-5dcd3ab07369']
Comment 16 errata-xmlrpc 2012-02-21 06:06:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-0225.html