Bug 1153405

Summary: virt-who can't work in the VDSM mode
Product: Red Hat Enterprise Linux 7 Reporter: Liushihui <shihliu>
Component: virt-whoAssignee: Radek Novacek <rnovacek>
Status: CLOSED ERRATA QA Contact: Li Bin Liu <liliu>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.0CC: bmcclain, fdeutsch, gxing, jherrman, ldai, liliu, ovasik, sgao, shihliu
Target Milestone: rcKeywords: TestBlocker, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: virt-who-0.11-3.el7 Doc Type: Bug Fix
Doc Text:
Prior to this update, the virt-who agent failed to read the list of virtual guests provided by the VDSM daemon. As a consequence, when in VDSM mode, virt-who was not able to send updates about virtual guests to the Subscription Asset Manager (SAM) and Red Hat Satellite. With this update, virt-who reads the list of guests when in VDSM mode correctly, and reports to SAM and Satellite as expected.
Story Points: ---
Clone Of:
: 1158441 (view as bug list) Environment:
Last Closed: 2015-03-05 10:23:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1158441    

Description Liushihui 2014-10-16 02:11:22 UTC
Description of problem:
During test the Rhevh7.0 + Rhevm3.5(vdsm mode) against SAM1.4.1, after set the virt-who work under the VDSM Mode, virt-who cannot work.

Version-Release number of selected component (if applicable):
subscription-manager-1.10.14-8.el7_0.x86_64
python-rhsm-1.10.12-2.el7.x86_64
virt-who-0.8-12.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. In the rhevh host, Set the virt-who parameters in file /etc/sysconfig/virt-who to make virt-who working in VDSM mode:
   VIRTWHO_DEBUG=1
   VIRTWHO_VDSM=1
   VIRTWHO_INTERVAL=10
2. Restart service virt-who
   # service virt-who restart
   Stopping virt-who:                                         [  OK  ]
   Starting virt-who:                                         [  OK  ]
3. Restart vdsmd service
   # service vdsmd restart
   Shutting down vdsm daemon: 
   vdsm watchdog stop                                         [  OK  ]
   vdsm stop                                                  [  OK  ]
   vdsm: libvirt already configured for vdsm                  [  OK  ]
   Starting iscsid: 
   Starting up vdsm daemon: 
   vdsm start                                                 [  OK  ]
4. check the log file:
   # tail -f /var/log/rhsm/rhsm.log

Actual results:
2014-10-16 02:08:22,661 [WARNING]  @virt-who.py:515 - Listening for events is not available in VDSM, ESX, RHEV-M or Hyper-V mode
2014-10-16 02:08:22,723 [DEBUG]  @virt-who.py:528 - Virt-who is running in vdsm mode
2014-10-16 02:08:22,723 [DEBUG]  @virt-who.py:535 - Starting infinite loop with 5 seconds interval and event handling
2014-10-16 02:08:22,723 [ERROR]  @virt-who.py:181 - Unable to create connection:
Traceback (most recent call last):
  File "/usr/share/virt-who/virt-who.py", line 178, in _send
  File "/usr/share/virt-who/virt-who.py", line 157, in checkConnections
  File "/usr/share/virt-who/virt-who.py", line 96, in initVirt
  File "/usr/share/virt-who/vdsm.py", line 34, in __init__
  File "/usr/share/virt-who/vdsm.py", line 43, in _readConfig
  File "/usr/lib64/python2.7/ConfigParser.py", line 618, in get
NoOptionError: No option 'trust_store_path' in section: 'vars'

It hasn't sent the host/guest association to SAM server

Expected results:
virt-who should run normally under the VDSM mode, it should send the correct host/guest association to SAM server.

Additional info:

Comment 1 Liushihui 2014-10-16 02:17:17 UTC
After adding "trust_store_path = /etc/pki/vdsm" to /etc/vdsm/vdsm.conf as bug 1017056 comment 2, it still can't send host/guest associate to SAM server, and it will show another error:
2014-10-16 02:14:44,052 [WARNING]  @virt-who.py:515 - Listening for events is not available in VDSM, ESX, RHEV-M or Hyper-V mode
2014-10-16 02:14:44,135 [DEBUG]  @virt-who.py:528 - Virt-who is running in vdsm mode
2014-10-16 02:14:44,135 [DEBUG]  @virt-who.py:535 - Starting infinite loop with 5 seconds interval and event handling
2014-10-16 02:14:44,187 [ERROR]  @virt-who.py:223 - Error in communication with subscription manager, trying to recover:
Traceback (most recent call last):
  File "/usr/share/virt-who/virt-who.py", line 205, in _send
  File "/usr/share/virt-who/subscriptionmanager.py", line 96, in sendVirtGuests
TypeError: 'str' object is not callable

Comment 3 Radek Novacek 2014-10-23 08:45:23 UTC
From the backtrace it seems that you're using an old version of virt-who.

The current up-to-date version of virt-who in RHEL-7.1 should be virt-who-0.11-2.el7.noarch.

Can you please make sure you have the correct version of virt-who installed and post the log messages from that version if you can still reproduce the issue.

Comment 4 Liushihui 2014-10-23 09:38:34 UTC
Radek, Because Rhev-hypervisor7-7.0-20141006.0.el7ev with the virt-who-0.8-12.el7.noarch by default.  I'm not sure if it would be possible to update virt-who in rhev-h. 

(In reply to Radek Novacek from comment #3)
> From the backtrace it seems that you're using an old version of virt-who.
> 
> The current up-to-date version of virt-who in RHEL-7.1 should be
> virt-who-0.11-2.el7.noarch.
> 
> Can you please make sure you have the correct version of virt-who installed
> and post the log messages from that version if you can still reproduce the
> issue.

Comment 7 Radek Novacek 2014-10-29 11:00:26 UTC
This issue does not require fixing in the y-stream, there is rebased virt-who package that doesn't have this issue.

Comment 9 xingge 2014-10-30 10:19:29 UTC
We now want to verify this bug, so does anyone can tell me where to download the newest rhevh7.0.z build with the newest virt-who package so we can do our test?

Comment 10 Fabian Deutsch 2014-10-31 15:21:34 UTC
(In reply to xingge from comment #9)
> We now want to verify this bug, so does anyone can tell me where to download
> the newest rhevh7.0.z build with the newest virt-who package so we can do
> our test?

The easiest way is if this bug could be verified without using rhevh? Is this possible?

Comment 11 Liushihui 2014-11-04 07:39:04 UTC
The easiest way should be use rhel7.0 instead of rhevh7.0. so we tried to add RHEL-7.0-20140507.0 to rhevm3.5, then upgrade virt-who to the latest one (virt-who-0.8-13.el7_0). But it was failed to add rhel7.0 to rhevm3.5 as "Network error during communication with the host.". please see the ovirt log error as the following: 
This problem block us to verify this bug in this way. does anyone can help us to resolve this problem first? or give us a new rhevh7.0.z build ? 

rhevm ip:10.66.78.3
rhel7.0 host:10.66.100.108 

Version-Release number of selected component (if applicable):
subscription-manager-1.10.14-8.el7_0.x86_64
python-rhsm-1.10.12-2.el7.x86_64
virt-who-0.8-12.el7.noarch
vdsm-4.13.2-4.el7.x86_64
rhevm-3.5.0-0.14.beta.el6ev.noarch

2014-11-04 01:20:11,328 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = 10.66.100.108, HostId = 6e0d8027-ccfa-4489-a895-19355f82bc30, vds=Host[10.66.100.108,6e0d8027-ccfa-4489-a895-19355f82bc30]) execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused
2014-11-04 01:20:11,332 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host 10.66.100.108 is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued.
2014-11-04 01:20:11,346 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:]
	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
	at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) [:1.7.0_09-icedtea]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
	at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_09-icedtea]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692) [rt.jar:1.7.0_09-icedtea]
	at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:]
	at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:]
	at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea]
	at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea]
	at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:]
	at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:](In reply to Fabian Deutsch from comment #10)

> (In reply to xingge from comment #9)
> > We now want to verify this bug, so does anyone can tell me where to download
> > the newest rhevh7.0.z build with the newest virt-who package so we can do
> > our test?
> 
> The easiest way is if this bug could be verified without using rhevh? Is
> this possible?

Comment 13 Liushihui 2014-11-24 09:14:22 UTC
As this bug need to check on rhel7.1, failed to register rhel7.1 to rhevm3.5-vt5,  as failed to restart vdsmd service, I will track it again with the latest rhevm3.5-vt11.

[root@hp-z220-05 yum.repos.d]# service vdsmd restart
Redirecting to /bin/systemctl restart  vdsmd.service
Job for vdsmd.service failed. See 'systemctl status vdsmd.service' and 'journalctl -xn' for details.

Comment 14 Liushihui 2014-11-26 05:56:55 UTC
As virt-who can send host/guest associate to SAM server on virt-who-0.11-3.el7, Verified it. 

Verify process:

Version-Release number of selected component (if applicable):
subscription-manager-1.13.8-1.el7.x86_64
python-rhsm-1.13.7-1.el7.x86_64
virt-who-0.11-3.el7.noarch
vdsm-4.16.7.5-1.el7ev.x86_64

Preconditon: Register rhel7.1 to rhevm-3.5.0
1. In the rhevh host, Set the virt-who parameters in file /etc/sysconfig/virt-who to make virt-who working in VDSM mode:
   VIRTWHO_DEBUG=1
   VIRTWHO_VDSM=1
   VIRTWHO_INTERVAL=5
2. Restart service virt-who
   # service virt-who restart
   Stopping virt-who:                                         [  OK  ]
   Starting virt-who:                                         [  OK  ]
3. Restart vdsmd service
   # service vdsmd restart
   Shutting down vdsm daemon: 
   vdsm watchdog stop                                         [  OK  ]
   vdsm stop                                                  [  OK  ]
   vdsm: libvirt already configured for vdsm                  [  OK  ]
   Starting iscsid: 
   Starting up vdsm daemon: 
   vdsm start                                                 [  OK  ]
4. check the log file:
   # tail -f /var/log/rhsm/rhsm.log

Actual results: virt-who send host/guest associate the SAM server
2014-11-26 13:55:11,380 [INFO]  @virtwho.py:458 - Using commandline or sysconfig configuration ("vdsm" mode)
2014-11-26 13:55:11,380 [DEBUG]  @virtwho.py:170 - Starting infinite loop with 5 seconds interval
2014-11-26 13:55:11,591 [INFO]  @subscriptionmanager.py:109 - Sending list of uuids: []
2014-11-26 13:55:17,234 [INFO]  @subscriptionmanager.py:109 - Sending list of uuids: []

Comment 17 errata-xmlrpc 2015-03-05 10:23:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0430.html