Bug 1284037

Summary: Traceback displayed on GoneException received from candlepin
Product: Red Hat Enterprise Linux 6 Reporter: Chris Snyder <csnyder>
Component: virt-whoAssignee: Radek Novacek <rnovacek>
Status: CLOSED ERRATA QA Contact: gaoshang <sgao>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.8CC: hsun, ovasik, rbalakri, shihliu
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: virt-who-0.16-1.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-10 23:56:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chris Snyder 2015-11-20 15:51:59 UTC
Description of problem:
As a part of Bug 1271158
a traceback is shown when virt-who references a consumer that has been deleted.

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Register a hypervisor to stage
2. Start virt-who
3. Delete the hypervisor using the stage candlepin site 
4. Look at rhsm.log 

Actual results:

2015-10-13 17:19:30,360 [ERROR]  @virtwho.py:190 - Error in communication with subscription manager:
Traceback (most recent call last):
  File "/usr/share/virt-who/virtwho.py", line 171, in send
    self._sendGuestList(report)
  File "/usr/share/virt-who/virtwho.py", line 196, in _sendGuestList
    manager.sendVirtGuests(report.guests)
  File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 139, in sendVirtGuests
    self.connection.updateConsumer(self.uuid(), guest_uuids=serialized_guests)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 947, in updateConsumer
    ret = self.conn.request_put(method, params)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 653, in request_put
    return self._request("PUT", method, params)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 564, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 597, in validateResponse
    parsed['displayMessage'], parsed['deletedId'])

Expected results:
No stack trace. There should also be a log message stating that communication with rhsm failed because the consumer no longer exists.

Comment 3 Radek Novacek 2015-11-24 14:26:39 UTC
The previous commit doesn't handle it very well, this one should be better:

https://github.com/virt-who/virt-who/commit/eb66da3d8f89307835dcb1cfbd5a715e4487d19e

Comment 4 Radek Novacek 2015-12-17 12:30:43 UTC
Fixed by rebase to virt-who-0.16-1.el6.

Comment 6 Liushihui 2016-02-26 02:55:12 UTC
Reopen it on virt-who-0.16-4.el6.noarch since virt-who cannot detect the hypervisor has been deleted also cannot send any error info in the log.

Rechecked version:
subscription-manager-1.16.8-2.el6.x86_64
python-rhsm-1.16.6-1.el6.x86_64
virt-who-0.16-3.el6.noarch

Rechecked process:
1 Register system to stage candlepin.
2 Configure virt-who monitor hyperv.
[root@sgi-xe500-01 run]# cat /etc/sysconfig/virt-who  | grep -v ^# | grep -v ^$
VIRTWHO_DEBUG=1
VIRTWHO_INTERVAL=2
VIRTWHO_HYPERV=1
VIRTWHO_HYPERV_OWNER=7715246
VIRTWHO_HYPERV_ENV=7715246
VIRTWHO_HYPERV_SERVER=10.73.5.212
VIRTWHO_HYPERV_USERNAME=administrator
VIRTWHO_HYPERV_PASSWORD=Welcome1
3 Restart virt-who 
[root@sgi-xe500-01 run]# service virt-who restart
4 Delete the system using the stage candlepin site
5 Check the virt-who log in rhsm.log
[root@sgi-xe500-01 run]# tail -f /var/log/rhsm/rhsm.log

Result:
virt-who cannot send log message stating that communication with rhsm failed because the consumer no longer exists. it repeat send normally info in the virt-who log as the following:
2016-02-25 20:56:25,935 [virtwho.env_cmdline DEBUG] HyperV-1(3085):MainThread @hyperv.py:retry_ntlm_negotitate:67 - Using NTLM authentication
2016-02-25 20:56:26,509 [virtwho.env_cmdline DEBUG] HyperV-1(3085):MainThread @hyperv.py:retry_ntlm_authenticate:80 - Sending NTLM authentication data
2016-02-25 20:56:26,810 [virtwho.env_cmdline DEBUG] HyperV-1(3085):MainThread @hyperv.py:retry_ntlm_authenticate:103 - NTLM authentication successful
2016-02-25 20:56:32,623 [virtwho.env_cmdline DEBUG] HyperV-1(3085):MainThread @virt.py:enqueue:351 - Report gathered, putting to queue for sending
2016-02-25 20:56:32,625 [virtwho.main INFO] MainProcess(3077):MainThread @virtwho.py:run:305 - Report for config "env/cmdline" haven't changed, not sending

Comment 7 Radek Novacek 2016-03-15 14:07:19 UTC
You have to do some change in host/guest association (start some VM), otherwise the report is not sent ("Report for config "env/cmdline" haven't changed, not sending") and this bug won't trigger.

Comment 8 Liushihui 2016-03-16 05:46:55 UTC
Checked it on virt-who-0.16-7.el6.noarch,when mapping info update, virt-who can show error info after delete system on satellite web UI, error info as the following:
2016-03-16 01:40:46,376 [virtwho.main ERROR] MainProcess(26864):MainThread @virtwho.py:send:203 - Unable to send data: Communication with subscription manager failed with code 500: undefined method `inject' for nil:NilClass

However, the error info is not clear, would you mind to stating that "communication with rhsm failed because the consumer no longer exists."

Comment 9 Radek Novacek 2016-03-17 07:40:22 UTC
This looks like you're facing some bug in candlepin. This error is returned by candlepin: "undefined method `inject' for nil:NilClass". What candlepin version do you use?

Comment 10 Liushihui 2016-03-30 06:34:23 UTC
Verified it on virt-who-0.16-7.el6.noarch against latest stage candlepin since it will state "Unable to send data: Communication with subscription manager failed: consumer no longer exists" if system has been delete from web UI when host/guest updating on hyperv/esx. Therefore, verify it.

Verified version:
virt-who-0.16-7.el6.noarch 
subscription-manager-1.16.8-7.el6.x86_64 
python-rhsm-1.16.6-1.el6.x86_64

Verified process:
1 Register system to stage candlepin.
2 Configure virt-who monitor hyperv.
[root@sgi-xe500-01 run]# cat /etc/sysconfig/virt-who  | grep -v ^# | grep -v ^$
VIRTWHO_DEBUG=1
VIRTWHO_INTERVAL=2
VIRTWHO_HYPERV=1
VIRTWHO_HYPERV_OWNER=7715246
VIRTWHO_HYPERV_ENV=7715246
VIRTWHO_HYPERV_SERVER=10.73.5.212
VIRTWHO_HYPERV_USERNAME=administrator
VIRTWHO_HYPERV_PASSWORD=Welcome1
3 Restart virt-who 
[root@sgi-xe500-01 run]# service virt-who restart
4 Delete the system using the stage candlepin site
5.In hyperv, start/stop/pause guests.
6 Check the virt-who log in rhsm.log
[root@sgi-xe500-01 run]# tail -f /var/log/rhsm/rhsm.log
2016-03-30 02:30:35,101 [virtwho.main INFO] MainProcess(12420):MainThread @subscriptionmanager.py:hypervisorCheckIn:185 - Sending update in hosts-to-guests mapping for config "env/cmdline": 1 hypervisors and 2 guests found
2016-03-30 02:30:35,101 [virtwho.main DEBUG] MainProcess(12420):MainThread @subscriptionmanager.py:hypervisorCheckIn:186 - Host-to-guest mapping: {
    "564D80F1-61F0-8510-87BF-57F9EC6122D7": [
        {
            "guestId": "C877953C-A19E-5149-B614-D97873C4CBD2", 
            "state": 1, 
            "attributes": {
                "active": 1, 
                "hypervisorVersion": "6.3.9600.16404", 
                "virtWhoType": "hyperv", 
                "hypervisorType": "hyperv"
            }
        }, 
        {
            "guestId": "8329390B-8359-F646-9176-27D738580E81", 
            "state": 5, 
            "attributes": {
                "active": 0, 
                "hypervisorVersion": "6.3.9600.16404", 
                "virtWhoType": "hyperv", 
                "hypervisorType": "hyperv"
            }
        }
    ]
}
2016-03-30 02:30:35,862 [virtwho.main ERROR] MainProcess(12420):MainThread @virtwho.py:send:203 - Unable to send data: Communication with subscription manager failed: consumer no longer exists
2016-03-30 02:30:35,863 [virtwho.main DEBUG] MainProcess(12420):MainThread @virtwho.py:send_report:168 - Report from "env/cmdline" failed to sent

Result:
No stack trace. There is a log message stating that "Communication with subscription manager failed: consumer no longer exists"

Comment 12 errata-xmlrpc 2016-05-10 23:56:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0859.html