Bug 1247866

1. start reboot vcenter and virt-who stop to create message in rhsm.log
============The Time is 15:52:22 ==============
2015-09-01 15:52:22,880 [INFO]  @subscriptionmanager.py:165 - Sending update in hosts-to-guests mapping:



2. vcenter is restarting now, virt-who will create many timeout message
============The Time is 15:52:22 ==============
2015-09-01 15:54:22,366 [DEBUG]  @esx.py:105 - Wait for ESX event finished, timeout

3. after restart, the windows Server Manager is still not ready, but virt-who will feedback error message as following:
============The Time is 15:54:38 ==============
2015-09-01 15:54:38,861 [DEBUG]  @esx.py:55 - Log into ESX
2015-09-01 15:54:41,137 [ERROR]  @virt.py:304 - Virt backend 'env/cmdline' fails with exception:
Traceback (most recent call last):
  File "/usr/share/virt-who/virt/virt.py", line 300, in run
    self._run()
  File "/usr/share/virt-who/virt/esx/esx.py", line 68, in _run
    self._prepare()
  File "/usr/share/virt-who/virt/esx/esx.py", line 56, in _prepare
    self.login()
  File "/usr/share/virt-who/virt/esx/esx.py", line 229, in login
    self.sc = self.client.service.RetrieveServiceContent(_this=self.moRef)
  File "/usr/lib/python2.7/site-packages/suds/client.py", line 542, in __call__
    return client.invoke(args, kwargs)
  File "/usr/lib/python2.7/site-packages/suds/client.py", line 602, in invoke
    result = self.send(soapenv)
  File "/usr/lib/python2.7/site-packages/suds/client.py", line 653, in send
    result = self.failed(binding, e)
  File "/usr/lib/python2.7/site-packages/suds/client.py", line 714, in failed
    raise Exception((status, reason))
Exception: (503, u'Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::LocalServiceSpec:0000000000EA7300] _serverNamespace = /sdk _isRedirect = false _port = 8085)')
2015-09-01 15:54:41,137 [INFO]  @virt.py:313 - Waiting 5 seconds before retrying backend 'env/cmdline'

4. waiting for a moment, Server Manager is ready, virt-who can feedback the right mapping info
============The Time is 15:59:50 ==============
2015-09-01 15:59:50,564 [DEBUG]  @esx.py:55 - Log into ESX
2015-09-01 15:59:51,691 [DEBUG]  @esx.py:58 - Creating ESX event filter
2015-09-01 15:59:56,559 [DEBUG]  @esx.py:142 - Waiting for ESX changes
2015-09-01 15:59:56,561 [DEBUG]  @subscriptionmanager.py:112 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2015-09-01 15:59:58,156 [DEBUG]  @subscriptionmanager.py:146 - Checking if server has capability 'hypervisor_async'
2015-09-01 16:00:00,205 [DEBUG]  @subscriptionmanager.py:158 - Server does not have 'hypervisors_async' capability
2015-09-01 16:00:00,206 [INFO]  @subscriptionmanager.py:165 - Sending update in hosts-to-guests mapping: {
    "aee4ff00-8c33-11e2-994a-6c3be51d959a": [
        {
            "guestId": "564d3964-c083-deba-55f9-000279dd82f9", 
            "state": 5, 
            "attributes": {
                "active": 0, 
                "virtWhoType": "esx", 
                "hypervisorType": "vmware"
            }
        }
    ]
}



In general, there is no long delay after reboot the vcenter host, but I don't know the error message of step(3) is expected or not

It seems that the error in step 3 is some kind of internal failure in vCenter (some internal service is not yet started after the reboot).

I don't think there is much that virt-who can do about it, apart for hiding the error (that would be bad idea because it can hide other problems).

It looks like the bug is now fixed. Can we close this report now?

Radek, According to the comment 1,this problem has been resolved in the latest virt-who. I think we can close this bug on this version. Therefore, close it as worksforme. Thanks.

Checked version:
virt-who-0.14-4.el7.noarch
subscription-manager-1.15.9-6.el7.x86_64
python-rhsm-1.15.4-2.el7.x86_64
Satellite6.1.0-20150820.0

It has reproduced on virt-who-0.14-7.el7.noarch, Therefore, reopen it.

Reproduced version:
virt-who-0.14-7.el7.noarch
subscription-manager-1.15.9-7.el7.x86_64
python-rhsm-1.15.4-4.el7.x86_64
SAM-1.4.1-RHEL-6-20141113.0

Reproduced process:
1. Register system to SAM
2. Configure virt-who run at esx mode, and the refresh interval is 5s.
3. Restart virt-who service, virt-who send the host/guest mapping to SAM
4. Stop vcenter, then start vcenter, check the virt-who's log

Result:
virt-who can't connect to vcenter until 30 min later.

2015-09-16 17:13:22,820 [DEBUG]  @esx.py:142 - Waiting for ESX changes
2015-09-16 17:13:22,824 [DEBUG]  @subscriptionmanager.py:112 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2015-09-16 17:13:22,846 [DEBUG]  @subscriptionmanager.py:146 - Checking if server has capability 'hypervisor_async'
2015-09-16 17:13:22,867 [DEBUG]  @subscriptionmanager.py:158 - Server does not have 'hypervisors_async' capability
2015-09-16 17:13:22,868 [INFO]  @subscriptionmanager.py:165 - Sending update in hosts-to-guests mapping: {
    "93b45c80-28ec-11e2-87c4-b4b52fe0f08d": [
        {
            "guestId": "422c92cb-14b2-e7a5-77fe-a530c7311de7", 
            "state": 1, 
            "attributes": {
                "active": 1, 
                "virtWhoType": "esx", 
                "hypervisorType": "vmware"
            }
        }
    ], 
    "826c4f80-28ee-11e2-b149-b4b52fe0ce88": []
}

******************After 30min, virt-who send h/g mapping to SAM again********

2015-09-16 17:43:22,927 [DEBUG]  @esx.py:105 - Wait for ESX event finished, timeout
2015-09-16 17:43:22,968 [DEBUG]  @esx.py:55 - Log into ESX
2015-09-16 17:43:23,352 [DEBUG]  @esx.py:58 - Creating ESX event filter
2015-09-16 17:43:23,410 [DEBUG]  @esx.py:142 - Waiting for ESX changes
2015-09-16 17:43:23,412 [DEBUG]  @subscriptionmanager.py:112 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2015-09-16 17:43:23,443 [DEBUG]  @subscriptionmanager.py:146 - Checking if server has capability 'hypervisor_async'
2015-09-16 17:43:23,473 [DEBUG]  @subscriptionmanager.py:158 - Server does not have 'hypervisors_async' capability
2015-09-16 17:43:23,473 [INFO]  @subscriptionmanager.py:165 - Sending update in hosts-to-guests mapping: {
    "93b45c80-28ec-11e2-87c4-b4b52fe0f08d": [
        {
            "guestId": "422c92cb-14b2-e7a5-77fe-a530c7311de7", 
            "state": 1, 
            "attributes": {
                "active": 1, 
                "virtWhoType": "esx", 
                "hypervisorType": "vmware"
            }
        }
    ], 
    "826c4f80-28ee-11e2-b149-b4b52fe0ce88": []
}


Expected result:
virt-who should report host/guest to SAM immediate after reboot vcenter

This bug seems to be similar to bug 1247866. Moving to RHEL-7.3.

Since this bug is cloned from bug #1231602 that is already fixed in RHEL-6.8 and the code base for 7.3 rebase is the same, the upcoming rebase of virt-who should fix this issue.

Fixed in virt-who-0.17-1.el7.

Verified it on virt-who-0.17-2.el7.noarch since virt-who can connect vcenter every refresh interval, virt-who also can send  h/g mapping info to server immediately after reboot vcenter successfully. Therefore, verify it.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2387.html