Bug 1271085

Summary: virt-who shows "Communication with server failed" with RateLimitExceededException
Product: Red Hat Enterprise Linux 7 Reporter: Eko <hsun>
Component: virt-whoAssignee: Radek Novacek <rnovacek>
Status: CLOSED ERRATA QA Contact: Eko <hsun>
Severity: high Docs Contact: Jiri Herrmann <jherrman>
Priority: high    
Version: 7.2CC: csnyder, csulekha, hsun, jherrman, ldai, ovasik, sgao, shihliu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: virt-who-0.17-1.el7 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 05:06:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eko 2015-10-13 07:13:51 UTC
Description of problem:
when migrated a guest to another host, virt-who will show RateLimitExceededException and can't send the mapping info to stage candlepin.

Version-Release number of selected component (if applicable):
 - python-rhsm-1.15.4-5.el7.x86_64
 - subscription-manager-1.15.9-14.el7.x86_64
 - virt-who-0.14-9.el7.noarch
 - RHEL7.2-20151008.0-Server-x86_64 + vdsm mode against Stage candlepin

How reproducible:


Steps to Reproduce:
1. prepare two hosts with vdsm mode
2. prepare a guest on host1 
2. register the two hosts and guest to stage candlepin
3. subscribe a unlimit pool for the two hosts
4. subscribe the bonus pool for the guest
5. migrate the guest to host2
6. check the rhsm.log on host2, following error message will be found:
2015-10-13 15:08:10,196 [INFO]  @subscriptionmanager.py:136 - Sending domain info: [
    {
        "guestId": "0bff30b0-7c7d-4884-a8a2-6b3b43daa963", 
        "state": 5, 
        "attributes": {
            "active": 0, 
            "virtWhoType": "vdsm", 
            "hypervisorType": "qemu"
        }
    }
]
2015-10-13 15:08:13,497 [ERROR]  @virtwho.py:190 - Error in communication with subscription manager:
Traceback (most recent call last):
  File "/usr/share/virt-who/virtwho.py", line 171, in send
    self._sendGuestList(report)
  File "/usr/share/virt-who/virtwho.py", line 196, in _sendGuestList
    manager.sendVirtGuests(report.guests)
  File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 139, in sendVirtGuests
    self.connection.updateConsumer(self.uuid(), guest_uuids=serialized_guests)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 947, in updateConsumer
    ret = self.conn.request_put(method, params)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 653, in request_put
    return self._request("PUT", method, params)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 564, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 627, in validateResponse
    raise RateLimitExceededException(response['status'])
RateLimitExceededException


Actual results:
RateLimitExceededException will be found, and can't send the mapping info to stage candlepin

Expected results:
no error message found, the mapping info can be sent to stage candlepin normally.

Additional info:

Comment 2 Radek Novacek 2015-10-13 14:44:39 UTC
Chris, this will be fixed by your recent changes in the interval reporting, right?

Comment 3 Chris Snyder 2015-10-14 14:29:03 UTC
Yes, this will be fixed by my recent changes in the interval reporting.

Comment 4 Radek Novacek 2015-11-04 09:19:57 UTC
I think that this error shouldn't appear when the reporting interval is set to some high value.

What VIRTWHO_INTERVAL did you used for testing? Can you retry it with some bigger values (60, 3600, etc.)?

Chris, do you know when is this RateLimitExceeded exception raised? What are the circumstances?

Comment 6 Eko 2015-11-05 09:06:32 UTC
hi Radek,
with different interval time (3s, 5s, 15s, 30s, 60s), this exception still can be raised. 
As our meeting discussed, if we can't fixed this issue on rhel7.2, we should add it to the release note as a known issue.

Comment 9 Eko 2015-11-06 04:03:17 UTC
Hi Chris & Radek,
I have changed the interval time to 600s, the exception is not found, but 10 minutes is very long for fetching the mapping info when adding or delete a guest. I still suggest to add this issue to release note, do you agree?

Comment 11 Radek Novacek 2015-11-10 06:49:19 UTC
Chris, only thing we can do is to release a z-stream fix, but that would mean to backport the RateLimitExceeded exception handling to virt-who-0.14.

Comment 12 Radek Novacek 2016-05-17 12:58:30 UTC
Fixed in virt-who-0.17-1.el7.

Comment 14 Liushihui 2016-06-03 02:48:39 UTC
Verified it on virt-who-0.17-2.el7.noarch since virt-who's minimum refresh interval has updated to 60s, it hasn't shown RateLimitExceededException when add/delete/migrate vm or add/delete host on vcenter. Therefore, verify it.

Comment 16 errata-xmlrpc 2016-11-04 05:06:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2387.html