This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1295644 - virt-who can't send two hypervisors' host/guests info at the same time, need to wait for 60s
virt-who can't send two hypervisors' host/guests info at the same time, need ...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: virt-who (Show other bugs)
6.8
x86_64 Linux
medium Severity medium
: rc
: ---
Assigned To: Radek Novacek
Eko
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-05 00:37 EST by Eko
Modified: 2016-11-30 19:35 EST (History)
5 users (show)

See Also:
Fixed In Version: virt-who-0.16-6.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-10 19:57:12 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Eko 2016-01-05 00:37:10 EST
Description of problem:
If I configured two hypervisors modes, such as libvirt and esx, after virt-who restart, the mapping info can't be sent at the same time.
It will send libvirt host/guests info firstly
after 60s, will send the esx host/guests info 

Version-Release number of selected component (if applicable):
virt-who-0.16-1.el6.noarch

How reproducible:
always

Steps to Reproduce:
1. prepare two different hypervisors mode in /etc/virt-who.d, such as:
# vi /etc/virt-who.d/libvirt.conf
[test-libvirt]
type=libvirt
server=10.66.144.8
username=root
password=redhat
owner=ACME_Corporation
env=Library

# vi /etc/virt-who.d/esx.conf
[test-esx]
type=esx
server=10.73.2.15
username=Administrator@vsphere.local
password=Welcome1!
owner=ACME_Corporation
env=Library


2. restart virt-who service
# /etc/init.d/virt-who restart

3. check rhsm.log
2016-01-05 13:31:48,335 [virtwho.init DEBUG] MainProcess(21368):MainThread @virtwho.py:__init__:144 - Using config named 'test-esx'
2016-01-05 13:31:48,335 [virtwho.init DEBUG] MainProcess(21368):MainThread @virtwho.py:__init__:144 - Using config named 'test-libvirt'
2016-01-05 13:31:48,335 [virtwho.init INFO] MainProcess(21368):MainThread @virtwho.py:main:811 - Using configuration "test-esx" ("esx" mode)
2016-01-05 13:31:48,335 [virtwho.init INFO] MainProcess(21368):MainThread @virtwho.py:main:811 - Using configuration "test-libvirt" ("libvirt" mode)
2016-01-05 13:31:48,348 [virtwho.main DEBUG] MainProcess(21370):MainThread @virtwho.py:run:287 - Starting infinite loop with 60 seconds interval
2016-01-05 13:31:48,444 [virtwho.test-esx DEBUG] Esx-1(21378):MainThread @virt.py:run:338 - Virt backend 'test-esx' started
2016-01-05 13:31:48,444 [virtwho.test-esx DEBUG] Esx-1(21378):MainThread @esx.py:_prepare:55 - Log into ESX
2016-01-05 13:31:48,446 [virtwho.test-libvirt DEBUG] Libvirtd-2(21380):MainThread @virt.py:run:338 - Virt backend 'test-libvirt' started
2016-01-05 13:31:48,447 [virtwho.test-libvirt INFO] Libvirtd-2(21380):MainThread @libvirtd.py:_get_url:124 - Protocol is not specified in libvirt url, using qemu+ssh://
2016-01-05 13:31:48,447 [virtwho.test-libvirt INFO] Libvirtd-2(21380):MainThread @libvirtd.py:_get_url:135 - Libvirt path is not specified in the url, using /system
2016-01-05 13:31:48,447 [virtwho.test-libvirt INFO] Libvirtd-2(21380):MainThread @libvirtd.py:_connect:156 - Using libvirt url: qemu+ssh://root@10.66.144.8/system?no_tty=1
2016-01-05 13:31:48,790 [virtwho.test-libvirt DEBUG] Libvirtd-2(21380):MainThread @libvirtd.py:_listDomains:228 - Virtual machine found: rhel7.2-satellite: 8d3f130d-b8f5-4b5e-3a3b-5d07483a615c
2016-01-05 13:31:48,796 [virtwho.test-libvirt DEBUG] Libvirtd-2(21380):MainThread @libvirtd.py:_listDomains:234 - Virtual machine found: rhel7.2: cb33ddce-fd1e-0e6a-7e7f-d3c6fca67ede
2016-01-05 13:31:48,796 [virtwho.test-libvirt DEBUG] Libvirtd-2(21380):MainThread @libvirtd.py:_listDomains:238 - Libvirt domains found: 8d3f130d-b8f5-4b5e-3a3b-5d07483a615c, cb33ddce-fd1e-0e6a-7e7f-d3c6fca67ede
2016-01-05 13:31:48,796 [virtwho.test-libvirt DEBUG] Libvirtd-2(21380):MainThread @virt.py:enqueue:331 - Report gathered, putting to queue for sending
2016-01-05 13:31:48,799 [virtwho.main DEBUG] MainProcess(21370):MainThread @virtwho.py:update_report_to_send:274 - Report for config "test-libvirt" updated
2016-01-05 13:31:48,801 [virtwho.main DEBUG] MainProcess(21370):MainThread @subscriptionmanager.py:_connect:112 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2016-01-05 13:31:48,794 [virtwho.test-esx DEBUG] Esx-1(21378):MainThread @esx.py:_prepare:58 - Creating ESX event filter
2016-01-05 13:31:49,001 [virtwho.test-esx DEBUG] Esx-1(21378):MainThread @virt.py:enqueue:331 - Report gathered, putting to queue for sending
2016-01-05 13:31:49,002 [virtwho.test-esx DEBUG] Esx-1(21378):MainThread @esx.py:_run:150 - Waiting for ESX changes
2016-01-05 13:31:49,031 [virtwho.main DEBUG] MainProcess(21370):MainThread @subscriptionmanager.py:hypervisorCheckIn:149 - Checking if server has capability 'hypervisor_async'
2016-01-05 13:31:49,513 [virtwho.main DEBUG] MainProcess(21370):MainThread @subscriptionmanager.py:hypervisorCheckIn:161 - Server does not have 'hypervisors_async' capability
2016-01-05 13:31:49,513 [virtwho.main INFO] MainProcess(21370):MainThread @subscriptionmanager.py:hypervisorCheckIn:170 - Sending update in hosts-to-guests mapping: 1 hypervisors and 2 guests found
2016-01-05 13:31:49,514 [virtwho.main DEBUG] MainProcess(21370):MainThread @subscriptionmanager.py:hypervisorCheckIn:171 - Host-to-guest mapping: {
    "80804c56-82fb-e111-a260-b4b52fcb471e": [
        {
            "guestId": "8d3f130d-b8f5-4b5e-3a3b-5d07483a615c", 
            "state": 1, 
            "attributes": {
                "active": 1, 
                "hypervisorVersion": "0.12.1", 
                "virtWhoType": "libvirt", 
                "hypervisorType": "QEMU"
            }
        }, 
        {
            "guestId": "cb33ddce-fd1e-0e6a-7e7f-d3c6fca67ede", 
            "state": 5, 
            "attributes": {
                "active": 0, 
                "hypervisorVersion": "0.12.1", 
                "virtWhoType": "libvirt", 
                "hypervisorType": "QEMU"
            }
        }
    ]
}
2016-01-05 13:31:49,827 [virtwho.main DEBUG] MainProcess(21370):MainThread @virtwho.py:send_current_report:206 - Report for config "test-libvirt" sent
2016-01-05 13:31:49,829 [virtwho.main DEBUG] MainProcess(21370):MainThread @virtwho.py:update_report_to_send:274 - Report for config "test-esx" updated

========== only send libvirt host/guests info as above ==========

========== after 60s, will send esx host/guest info as below ==========
2016-01-05 13:32:48,853 [virtwho.main DEBUG] MainProcess(21370):MainThread @subscriptionmanager.py:_connect:112 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2016-01-05 13:32:49,067 [virtwho.test-esx DEBUG] Esx-1(21378):MainThread @esx.py:_run:150 - Waiting for ESX changes
2016-01-05 13:32:49,089 [virtwho.main DEBUG] MainProcess(21370):MainThread @subscriptionmanager.py:hypervisorCheckIn:149 - Checking if server has capability 'hypervisor_async'
2016-01-05 13:32:49,317 [virtwho.main DEBUG] MainProcess(21370):MainThread @subscriptionmanager.py:hypervisorCheckIn:161 - Server does not have 'hypervisors_async' capability
2016-01-05 13:32:49,317 [virtwho.main INFO] MainProcess(21370):MainThread @subscriptionmanager.py:hypervisorCheckIn:170 - Sending update in hosts-to-guests mapping: 2 hypervisors and 3 guests found
2016-01-05 13:32:49,318 [virtwho.main DEBUG] MainProcess(21370):MainThread @subscriptionmanager.py:hypervisorCheckIn:171 - Host-to-guest mapping: {
    "4c4c4544-0048-5a10-8043-b5c04f393358": [
        {
            "guestId": "42293e7d-ba7e-1cea-d897-d8a373b26925", 
            "state": 5, 
            "attributes": {
                "active": 0, 
                "hypervisorVersion": "6.0.0", 
                "virtWhoType": "esx", 
                "hypervisorType": "VMware ESXi"
            }
        }
    ], 
    "86b2bd00-8bad-11e2-87f4-6c3be514699d": [
        {
            "guestId": "42098426-0d49-7dc6-0cb7-2331f149b8aa", 
            "state": 3, 
            "attributes": {
                "active": 1, 
                "hypervisorVersion": "6.0.0", 
                "virtWhoType": "esx", 
                "hypervisorType": "VMware ESXi"
            }
        }, 
        {
            "guestId": "4239b745-4369-40fd-569c-ba0d396654e9", 
            "state": 5, 
            "attributes": {
                "active": 0, 
                "hypervisorVersion": "6.0.0", 
                "virtWhoType": "esx", 
                "hypervisorType": "VMware ESXi"
            }
        }
    ]
}
2016-01-05 13:32:49,616 [virtwho.main DEBUG] MainProcess(21370):MainThread @virtwho.py:send_current_report:206 - Report for config "test-esx" sent



Actual results:
virt-who can't send two hypervisors' host/guests info at the same time, need to wait for 60s

Expected results:
after restart, virt-who should send all the hypervisors' host/guests info at the same time

Additional info:
Comment 2 Radek Novacek 2016-01-05 03:30:55 EST
This is now expected behaviour. In order to prevent flooding the candlepin server, virt-who now limits how often it sends reports. By default reports are not sent more often than once per minute.

For details please see: https://github.com/virt-who/virt-who/wiki/Interval-changes
Comment 3 Eko 2016-01-07 02:30:37 EST
Hi radek, 
I agree with you, if the virt-who is running now, it's ok to check the reports update per minute.

but when restart virt-who service, virt-who should send all the hypervisors json at the same time. 

Configure two hypervisors as example, such as:
hypervisor_1 (kvm): host/guests json
hypervisor_2 (esx): host/guests json

after restart virt-who service, hypervisor_1 (kvm) will be sent immediately, but hypervisor_2 (esx) will be delaied 60 seconds.
Comment 4 Radek Novacek 2016-02-11 06:53:49 EST
Sending all the hypervisor reports at the same time creates a lot of load in the candlepin and that is something we generally don't want to do. Although I agree that from user perspective it would be better to send both reports asap, that's something that candlepin doesn't want.
Comment 5 Eko 2016-02-16 02:57:45 EST
if the customer create 100 config files in /etc/virt-who.d/, it will take 100 minutes to open all the config files and send the host/guests json.
if change the Interval value to 120 seconds, it will take 200 minutes to polling one time.
Comment 6 Radek Novacek 2016-02-16 03:25:14 EST
Chris, what do you think would be correct behavior in this case? Let the customer wait for initial run too long doesn't seem right. OTOH the load for candlepin can be quite high, given how big environments customers have.
Comment 8 Radek Novacek 2016-02-23 08:25:59 EST
I don't know how many configs our users have. I don't think I saw more than 2 configs on one virt-who instance, but I'm not in touch with our customers. Eko, do you have an idea?
Comment 9 Radek Novacek 2016-02-24 03:21:13 EST
Chris, how about sending the first batch immediately when virt-who is starting and apply the rate limit after one report for each config is sent. It would be better from user perspective and candlepin can still reply with 429 - then virt-who will delay all future reports (even if the first batch isn't finished). Is this solution acceptable?
Comment 10 Chris Snyder 2016-02-24 09:16:10 EST
Radek,

That sounds reasonable to me.

Thanks,

Chris
Comment 11 Radek Novacek 2016-02-25 10:39:57 EST
Fixed in virt-who-0.16-6.el6.
Comment 13 Liushihui 2016-03-16 02:25:00 EDT
Verified it on virt-who-0.16-7.el6.noarch since virt-who can get multi esx/rhevm/hyperv/remote libvirt mapping info and send it to server side immediately. Therefore, verify it.

Verified version:
virt-who-0.16-7.el6.noarch
python-rhsm-1.16.6-1.el6.x86_64
subscription-manager-1.16.8-4.el6.x86_64

Verified process:
1. Register system to satellite6.1
2. Configure two type of mode in /etc/virt-who.d/
[root@intel-piketon-01 ~]# cat /etc/virt-who.d/virt 
[test-rhevm1]
type=rhevm
server=https://10.73.2.65:443
username=admin@internal
password=redhat
owner=ACME_Corporation
env=Library
[root@intel-piketon-01 ~]# cat /etc/virt-who.d/esx 
[test-esx1]
type=esx
server=10.73.2.95
username=Administrator@vsphere.local
password=Welcome1!
owner=ACME_Corporation
env=Library
3. Restart virt-who service and check virt-who's log
[root@intel-piketon-01 ~]# service virt-who restart && tail -f /var/log/rhsm/rhsm.log 
Stopping virt-who: [  OK  ]
Starting virt-who: [  OK  ]
..............
2016-03-16 02:21:04,047 [virtwho.init INFO] MainProcess(27419):MainThread @virtwho.py:parseOptions:637 - Using reporter_id='intel-piketon-01.lab.bos.redhat.com'
2016-03-16 02:21:04,051 [virtwho.init DEBUG] MainProcess(27419):MainThread @virtwho.py:__init__:125 - Using config named 'test-esx1'
2016-03-16 02:21:04,051 [virtwho.init DEBUG] MainProcess(27419):MainThread @virtwho.py:__init__:125 - Using config named 'test-rhevm1'
2016-03-16 02:21:04,051 [virtwho.init INFO] MainProcess(27419):MainThread @virtwho.py:main:729 - Using configuration "test-esx1" ("esx" mode)
2016-03-16 02:21:04,051 [virtwho.init INFO] MainProcess(27419):MainThread @virtwho.py:main:729 - Using configuration "test-rhevm1" ("rhevm" mode)
2016-03-16 02:21:04,067 [virtwho.main DEBUG] MainProcess(27421):MainThread @virtwho.py:run:231 - Starting infinite loop with 60 seconds interval
2016-03-16 02:21:04,186 [virtwho.test-esx1 DEBUG] Esx-1(27430):MainThread @virt.py:run:358 - Virt backend 'test-esx1' started
2016-03-16 02:21:04,187 [virtwho.test-esx1 DEBUG] Esx-1(27430):MainThread @esx.py:_prepare:55 - Log into ESX
2016-03-16 02:21:04,190 [virtwho.test-rhevm1 DEBUG] RhevM-2(27432):MainThread @virt.py:run:358 - Virt backend 'test-rhevm1' started
2016-03-16 02:21:07,735 [virtwho.test-esx1 DEBUG] Esx-1(27430):MainThread @esx.py:_prepare:58 - Creating ESX event filter
2016-03-16 02:21:08,435 [virtwho.test-rhevm1 DEBUG] RhevM-2(27432):MainThread @virt.py:enqueue:351 - Report for config "test-rhevm1" gathered, putting to queue for sending
2016-03-16 02:21:08,453 [virtwho.main DEBUG] MainProcess(27421):MainThread @subscriptionmanager.py:_connect:121 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2016-03-16 02:21:08,642 [virtwho.main DEBUG] MainProcess(27421):MainThread @subscriptionmanager.py:hypervisorCheckIn:162 - Checking if server has capability 'hypervisor_async'
2016-03-16 02:21:08,785 [virtwho.main DEBUG] MainProcess(27421):MainThread @subscriptionmanager.py:hypervisorCheckIn:174 - Server does not have 'hypervisors_async' capability
2016-03-16 02:21:08,786 [virtwho.main INFO] MainProcess(27421):MainThread @subscriptionmanager.py:hypervisorCheckIn:185 - Sending update in hosts-to-guests mapping for config "test-rhevm1": 2 hypervisors and 1 guests found
2016-03-16 02:21:08,786 [virtwho.main DEBUG] MainProcess(27421):MainThread @subscriptionmanager.py:hypervisorCheckIn:186 - Host-to-guest mapping: {
    "fd26642f-62f5-4075-adce-3359e5a8bd10": [
        {
            "guestId": "a11f0b29-b430-4a53-978a-cae879ad1ef3", 
            "state": 1, 
            "attributes": {
                "active": 1, 
                "hypervisorVersion": "", 
                "virtWhoType": "rhevm", 
                "hypervisorType": "qemu"
            }
        }
    ], 
    "d64f48cc-5fda-45c3-a307-fa42ef1864c7": []
}
2016-03-16 02:21:09,057 [virtwho.main DEBUG] MainProcess(27421):MainThread @virtwho.py:send_report:161 - Report for config "test-rhevm1" sent
2016-03-16 02:21:11,114 [virtwho.test-esx1 DEBUG] Esx-1(27430):MainThread @virt.py:enqueue:351 - Report for config "test-esx1" gathered, putting to queue for sending
2016-03-16 02:21:11,118 [virtwho.main DEBUG] MainProcess(27421):MainThread @subscriptionmanager.py:_connect:121 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2016-03-16 02:21:11,286 [virtwho.main DEBUG] MainProcess(27421):MainThread @subscriptionmanager.py:hypervisorCheckIn:162 - Checking if server has capability 'hypervisor_async'
2016-03-16 02:21:11,616 [virtwho.main DEBUG] MainProcess(27421):MainThread @subscriptionmanager.py:hypervisorCheckIn:174 - Server does not have 'hypervisors_async' capability
2016-03-16 02:21:11,616 [virtwho.main INFO] MainProcess(27421):MainThread @subscriptionmanager.py:hypervisorCheckIn:185 - Sending update in hosts-to-guests mapping for config "test-esx1": 1 hypervisors and 1 guests found
2016-03-16 02:21:11,617 [virtwho.main DEBUG] MainProcess(27421):MainThread @subscriptionmanager.py:hypervisorCheckIn:186 - Host-to-guest mapping: {
    "86b2bd00-8bad-11e2-87f4-6c3be514699d": [
        {
            "guestId": "42060bb0-044b-dd8c-c111-361bb5b2f78c", 
            "state": 5, 
            "attributes": {
                "active": 0, 
                "hypervisorVersion": "6.0.0", 
                "virtWhoType": "esx", 
                "hypervisorType": "VMware ESXi"
            }
        }
    ]
}
2016-03-16 02:21:11,917 [virtwho.main DEBUG] MainProcess(27421):MainThread @virtwho.py:send_report:161 - Report for config "test-esx1" sent

Result:
Virt-who can send multi host/guest mapping info to server immediately.

Note:
Configure multi server/mode in /etc/sysconfig/virt-who and /etc/virt-who.d/, virt-who also can send mapping info simultaneously.
Comment 15 errata-xmlrpc 2016-05-10 19:57:12 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0859.html

Note You need to log in before you can comment on or make changes to this bug.