Bug 1271085 - virt-who shows "Communication with server failed" with RateLimitExceededException
virt-who shows "Communication with server failed" with RateLimitExceededExcep...
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: virt-who (Show other bugs)
x86_64 Linux
high Severity high
: rc
: ---
Assigned To: Radek Novacek
Jiri Herrmann
Depends On:
  Show dependency treegraph
Reported: 2015-10-13 03:13 EDT by Eko
Modified: 2016-11-30 19:36 EST (History)
8 users (show)

See Also:
Fixed In Version: virt-who-0.17-1.el7
Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-11-04 01:06:16 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2387 normal SHIPPED_LIVE virt-who bug fix and enhancement update 2016-11-03 09:53:39 EDT

  None (edit)
Description Eko 2015-10-13 03:13:51 EDT
Description of problem:
when migrated a guest to another host, virt-who will show RateLimitExceededException and can't send the mapping info to stage candlepin.

Version-Release number of selected component (if applicable):
 - python-rhsm-1.15.4-5.el7.x86_64
 - subscription-manager-1.15.9-14.el7.x86_64
 - virt-who-0.14-9.el7.noarch
 - RHEL7.2-20151008.0-Server-x86_64 + vdsm mode against Stage candlepin

How reproducible:

Steps to Reproduce:
1. prepare two hosts with vdsm mode
2. prepare a guest on host1 
2. register the two hosts and guest to stage candlepin
3. subscribe a unlimit pool for the two hosts
4. subscribe the bonus pool for the guest
5. migrate the guest to host2
6. check the rhsm.log on host2, following error message will be found:
2015-10-13 15:08:10,196 [INFO]  @subscriptionmanager.py:136 - Sending domain info: [
        "guestId": "0bff30b0-7c7d-4884-a8a2-6b3b43daa963", 
        "state": 5, 
        "attributes": {
            "active": 0, 
            "virtWhoType": "vdsm", 
            "hypervisorType": "qemu"
2015-10-13 15:08:13,497 [ERROR]  @virtwho.py:190 - Error in communication with subscription manager:
Traceback (most recent call last):
  File "/usr/share/virt-who/virtwho.py", line 171, in send
  File "/usr/share/virt-who/virtwho.py", line 196, in _sendGuestList
  File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 139, in sendVirtGuests
    self.connection.updateConsumer(self.uuid(), guest_uuids=serialized_guests)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 947, in updateConsumer
    ret = self.conn.request_put(method, params)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 653, in request_put
    return self._request("PUT", method, params)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 564, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 627, in validateResponse
    raise RateLimitExceededException(response['status'])

Actual results:
RateLimitExceededException will be found, and can't send the mapping info to stage candlepin

Expected results:
no error message found, the mapping info can be sent to stage candlepin normally.

Additional info:
Comment 2 Radek Novacek 2015-10-13 10:44:39 EDT
Chris, this will be fixed by your recent changes in the interval reporting, right?
Comment 3 Chris Snyder 2015-10-14 10:29:03 EDT
Yes, this will be fixed by my recent changes in the interval reporting.
Comment 4 Radek Novacek 2015-11-04 04:19:57 EST
I think that this error shouldn't appear when the reporting interval is set to some high value.

What VIRTWHO_INTERVAL did you used for testing? Can you retry it with some bigger values (60, 3600, etc.)?

Chris, do you know when is this RateLimitExceeded exception raised? What are the circumstances?
Comment 6 Eko 2015-11-05 04:06:32 EST
hi Radek,
with different interval time (3s, 5s, 15s, 30s, 60s), this exception still can be raised. 
As our meeting discussed, if we can't fixed this issue on rhel7.2, we should add it to the release note as a known issue.
Comment 9 Eko 2015-11-05 23:03:17 EST
Hi Chris & Radek,
I have changed the interval time to 600s, the exception is not found, but 10 minutes is very long for fetching the mapping info when adding or delete a guest. I still suggest to add this issue to release note, do you agree?
Comment 11 Radek Novacek 2015-11-10 01:49:19 EST
Chris, only thing we can do is to release a z-stream fix, but that would mean to backport the RateLimitExceeded exception handling to virt-who-0.14.
Comment 12 Radek Novacek 2016-05-17 08:58:30 EDT
Fixed in virt-who-0.17-1.el7.
Comment 14 Liushihui 2016-06-02 22:48:39 EDT
Verified it on virt-who-0.17-2.el7.noarch since virt-who's minimum refresh interval has updated to 60s, it hasn't shown RateLimitExceededException when add/delete/migrate vm or add/delete host on vcenter. Therefore, verify it.
Comment 16 errata-xmlrpc 2016-11-04 01:06:16 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.