Bug 1297623

Summary: "virt-who -o -d" can't exist immediately when failed to connect hypervisor
Product: Red Hat Enterprise Linux 6 Reporter: Liushihui <shihliu>
Component: virt-whoAssignee: Radek Novacek <rnovacek>
Status: CLOSED ERRATA QA Contact: Eko <hsun>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.8CC: gxing, hsun, ovasik, rbalakri, sgao, shihliu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: virt-who-0.16-2.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-10 23:57:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Liushihui 2016-01-12 03:29:06 UTC
Description of problem:
When virt-who run at one-shot time mode, if it failed to connect hypervisor, the virt-who process can't terminated until 60s

Version-Release number of selected component (if applicable):
virt-who-0.16-1.el6.noarch
subscription-manager-1.16.7-1.el6.x86_64
python-rhsm-1.16.5-1.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Register system to satellite
2. Make virt-who monitor remote libvirt which can't be connected, and run virt-who at one-shot time.
[root@lenovo-rd230-02 ~]# virt-who --libvirt --libvirt-owner=ACME_Corporation --libvirt-env=Library --libvirt-server=tyan-gt24-13.rhts.eng.bos.redhat.com --libvirt-username=root --libvirt-password= -o -d
2016-01-11 21:10:41,402 [virtwho.init INFO] MainProcess(25615):MainThread @virtwho.py:parseOptions:720 - Using reporter_id='lenovo-rd230-02.khw.lab.eng.bos.redhat.com'
2016-01-11 21:10:41,404 [virtwho.init INFO] MainProcess(25615):MainThread @virtwho.py:main:811 - Using configuration "env/cmdline" ("libvirt" mode)
2016-01-11 21:10:41,528 [virtwho.env_cmdline DEBUG] Libvirtd-1(25621):MainThread @virt.py:run:338 - Virt backend 'env/cmdline' started
2016-01-11 21:10:41,530 [virtwho.env_cmdline INFO] Libvirtd-1(25621):MainThread @libvirtd.py:_get_url:124 - Protocol is not specified in libvirt url, using qemu+ssh://
2016-01-11 21:10:41,531 [virtwho.env_cmdline INFO] Libvirtd-1(25621):MainThread @libvirtd.py:_get_url:135 - Libvirt path is not specified in the url, using /system
2016-01-11 21:10:41,531 [virtwho.env_cmdline INFO] Libvirtd-1(25621):MainThread @libvirtd.py:_connect:156 - Using libvirt url: qemu+ssh://root.eng.bos.redhat.com/system?no_tty=1
2016-01-11 21:10:41,591 [virtwho.env_cmdline ERROR] Libvirtd-1(25621):MainThread @libvirtd.py:_connect:164 - Error in libvirt backend
Traceback (most recent call last):
  File "/usr/share/virt-who/virt/libvirtd/libvirtd.py", line 162, in _connect
    v = libvirt.openReadOnly(url)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 247, in openReadOnly
    if ret is None:raise libvirtError('virConnectOpenReadOnly() failed')
libvirtError: Cannot recv data: Host key verification failed.: Connection reset by peer
2016-01-11 21:10:41,595 [virtwho.env_cmdline ERROR] Libvirtd-1(25621):MainThread @virt.py:run:348 - Virt backend 'env/cmdline' fails with error: Cannot recv data: Host key verification failed.: Connection reset by peer
2016-01-11 21:10:41,597 [virtwho.env_cmdline DEBUG] Libvirtd-1(25621):MainThread @virt.py:enqueue:331 - Report gathered, putting to queue for sending
2016-01-11 21:10:41,599 [virtwho.env_cmdline DEBUG] Libvirtd-1(25621):MainThread @virt.py:run:357 - Virt backend 'env/cmdline' stopped after sending one report
2016-01-11 21:10:41,601 [virtwho.main WARNING] MainProcess(25615):MainThread @virtwho.py:run:363 - Unable to collect report for config "env/cmdline"

==================After 60s, virt-who terminated=========================
2016-01-11 21:11:41,663 [virtwho.main DEBUG] MainProcess(25615):MainThread @virtwho.py:<module>:890 - virt-who terminated
2016-01-11 21:11:41,664 [virtwho.main DEBUG] MainProcess(25615):MainThread @virtwho.py:terminate:413 - virt-who is shutting down

Actual results:
Virt-who failed to connect with hypervisor, it can't stop virt-who process until 60s

Expected results:
virt-who process should terminate when run at one-shot time mode 

Additional info:
When virt-who connect remote libvirt successfully, it hasn't this problem.

Comment 1 Radek Novacek 2016-01-19 07:04:52 UTC
*** Bug 1298062 has been marked as a duplicate of this bug. ***

Comment 2 Radek Novacek 2016-01-28 13:04:01 UTC
Fixed in virt-who-0.16-2.el6.

Comment 4 Liushihui 2016-02-04 07:36:11 UTC
Verified it on virt-who-0.16-3.el6.noarch since virt-who can terminate when run at one-shot time mode. 
1. When virt-who failed to connect remote libvirt. it can terminate when run at one-shot time mode.
[root@intel-canoepass-10 ~]# virt-who --libvirt --libvirt-owner=ACME_Corporation --libvirt-env=Library --libvirt-server=ibm-hs22-04.lab.bos.redhat.com --libvirt-username=root --libvirt-password= -o -d
2016-02-04 02:27:20,560 [virtwho.init INFO] MainProcess(13625):MainThread @virtwho.py:parseOptions:630 - Using reporter_id='intel-canoepass-10.lab.bos.redhat.com'
2016-02-04 02:27:20,563 [virtwho.init INFO] MainProcess(13625):MainThread @virtwho.py:main:721 - Using configuration "env/cmdline" ("libvirt" mode)
2016-02-04 02:27:20,693 [virtwho.env_cmdline DEBUG] Libvirtd-1(13631):MainThread @virt.py:run:358 - Virt backend 'env/cmdline' started
2016-02-04 02:27:20,694 [virtwho.env_cmdline INFO] Libvirtd-1(13631):MainThread @libvirtd.py:_get_url:124 - Protocol is not specified in libvirt url, using qemu+ssh://
2016-02-04 02:27:20,695 [virtwho.env_cmdline INFO] Libvirtd-1(13631):MainThread @libvirtd.py:_get_url:135 - Libvirt path is not specified in the url, using /system
2016-02-04 02:27:20,695 [virtwho.env_cmdline INFO] Libvirtd-1(13631):MainThread @libvirtd.py:_connect:156 - Using libvirt url: qemu+ssh://root.bos.redhat.com/system?no_tty=1
2016-02-04 02:27:20,726 [virtwho.env_cmdline ERROR] Libvirtd-1(13631):MainThread @libvirtd.py:_connect:164 - Error in libvirt backend
Traceback (most recent call last):
  File "/usr/share/virt-who/virt/libvirtd/libvirtd.py", line 162, in _connect
    v = libvirt.openReadOnly(url)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 247, in openReadOnly
    if ret is None:raise libvirtError('virConnectOpenReadOnly() failed')
libvirtError: Cannot recv data: Host key verification failed.: Connection reset by peer
2016-02-04 02:27:20,730 [virtwho.env_cmdline ERROR] Libvirtd-1(13631):MainThread @virt.py:run:368 - Virt backend 'env/cmdline' fails with error: Cannot recv data: Host key verification failed.: Connection reset by peer
2016-02-04 02:27:20,730 [virtwho.env_cmdline DEBUG] Libvirtd-1(13631):MainThread @virt.py:enqueue:351 - Report gathered, putting to queue for sending
2016-02-04 02:27:20,732 [virtwho.env_cmdline DEBUG] Libvirtd-1(13631):MainThread @virt.py:run:377 - Virt backend 'env/cmdline' stopped after sending one report
2016-02-04 02:27:20,733 [virtwho.main WARNING] MainProcess(13625):MainThread @virtwho.py:run:302 - Unable to collect report for config "env/cmdline"
2016-02-04 02:27:20,739 [virtwho.main DEBUG] MainProcess(13625):MainThread @virtwho.py:<module>:805 - virt-who terminated
2016-02-04 02:27:20,739 [virtwho.main DEBUG] MainProcess(13625):MainThread @virtwho.py:terminate:352 - virt-who is shutting down
[root@intel-canoepass-10 ~]# service virt-who status
virt-who dead but subsys locked
[root@intel-canoepass-10 ~]# ps -ef|grep virt-who
root      13651  10536  0 02:27 pts/1    00:00:00 grep virt-who

2. When virt-who connect remote libvirt successfully. it also can terminate when run at one-shot time mode.
[root@intel-canoepass-11 ~]# virt-who --libvirt --libvirt-owner=ACME_Corporation --libvirt-env=Library --libvirt-server=intel-canoepass-10.lab.bos.redhat.com --libvirt-username=root --libvirt-password= -o -d
2016-02-04 02:34:45,931 [virtwho.init INFO] MainProcess(12140):MainThread @virtwho.py:parseOptions:630 - Using reporter_id='intel-canoepass-11.lab.bos.redhat.com'
2016-02-04 02:34:45,934 [virtwho.init INFO] MainProcess(12140):MainThread @virtwho.py:main:721 - Using configuration "env/cmdline" ("libvirt" mode)
2016-02-04 02:34:46,048 [virtwho.env_cmdline DEBUG] Libvirtd-1(12146):MainThread @virt.py:run:358 - Virt backend 'env/cmdline' started
2016-02-04 02:34:46,051 [virtwho.env_cmdline INFO] Libvirtd-1(12146):MainThread @libvirtd.py:_get_url:124 - Protocol is not specified in libvirt url, using qemu+ssh://
2016-02-04 02:34:46,051 [virtwho.env_cmdline INFO] Libvirtd-1(12146):MainThread @libvirtd.py:_get_url:135 - Libvirt path is not specified in the url, using /system
2016-02-04 02:34:46,052 [virtwho.env_cmdline INFO] Libvirtd-1(12146):MainThread @libvirtd.py:_connect:156 - Using libvirt url: qemu+ssh://root.bos.redhat.com/system?no_tty=1
2016-02-04 02:34:46,975 [virtwho.env_cmdline DEBUG] Libvirtd-1(12146):MainThread @libvirtd.py:_listDomains:236 - Libvirt domains found: 9ebbeb2c-e391-b586-1d29-5e579c190a31
2016-02-04 02:34:46,976 [virtwho.env_cmdline DEBUG] Libvirtd-1(12146):MainThread @virt.py:enqueue:351 - Report gathered, putting to queue for sending
2016-02-04 02:34:47,000 [virtwho.main DEBUG] MainProcess(12140):MainThread @subscriptionmanager.py:_connect:121 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2016-02-04 02:34:47,632 [virtwho.main DEBUG] MainProcess(12140):MainThread @subscriptionmanager.py:hypervisorCheckIn:162 - Checking if server has capability 'hypervisor_async'
2016-02-04 02:34:47,833 [virtwho.main DEBUG] MainProcess(12140):MainThread @subscriptionmanager.py:hypervisorCheckIn:174 - Server does not have 'hypervisors_async' capability
2016-02-04 02:34:47,833 [virtwho.main INFO] MainProcess(12140):MainThread @subscriptionmanager.py:hypervisorCheckIn:185 - Sending update in hosts-to-guests mapping for config "env/cmdline": 1 hypervisors and 1 guests found
2016-02-04 02:34:47,834 [virtwho.main DEBUG] MainProcess(12140):MainThread @subscriptionmanager.py:hypervisorCheckIn:186 - Host-to-guest mapping: {
    "f14a757e-c7ed-480e-9613-001e67659838": [
        {
            "guestId": "9ebbeb2c-e391-b586-1d29-5e579c190a31", 
            "state": 5, 
            "attributes": {
                "active": 0, 
                "hypervisorVersion": "0.12.1", 
                "virtWhoType": "libvirt", 
                "hypervisorType": "QEMU"
            }
        }
    ]
}
2016-02-04 02:34:47,980 [virtwho.env_cmdline DEBUG] Libvirtd-1(12146):MainThread @virt.py:run:377 - Virt backend 'env/cmdline' stopped after sending one report
2016-02-04 02:34:49,603 [virtwho.main DEBUG] MainProcess(12140):MainThread @virtwho.py:send_current_report:159 - Report for config "env/cmdline" sent
2016-02-04 02:34:49,606 [virtwho.main DEBUG] MainProcess(12140):MainThread @virtwho.py:<module>:805 - virt-who terminated
2016-02-04 02:34:49,606 [virtwho.main DEBUG] MainProcess(12140):MainThread @virtwho.py:terminate:352 - virt-who is shutting down

Comment 5 Liushihui 2016-02-15 06:55:32 UTC
It reproduced on virt-who-0.16-3.el6.noarch. Therefore, reopen it.

1. [root@ibm-x3630m4-01 ~]# virt-who --libvirt --libvirt-owner=ACME_Corporation --libvirt-env=Library --libvirt-server=sgi-xe500-01.rhts.eng.bos.redhat.com --libvirt-username=root --libvirt-password=-o -d
2016-02-15 01:50:09,564 [virtwho.main INFO] MainProcess(13730):MainThread @subscriptionmanager.py:hypervisorCheckIn:185 - Sending update in hosts-to-guests mapping for config "env/cmdline": 1 hypervisors and 1 guests found
2016-02-15 01:50:09,564 [virtwho.main DEBUG] MainProcess(13730):MainThread @subscriptionmanager.py:hypervisorCheckIn:186 - Host-to-guest mapping: {
    "00020003-0004-0005-0006-000700080009": [
        {
            "guestId": "bc0dca31-8eb2-714c-1493-651956ed3680", 
            "state": 5, 
            "attributes": {
                "active": 0, 
                "hypervisorVersion": "0.12.1", 
                "virtWhoType": "libvirt", 
                "hypervisorType": "QEMU"
            }
        }
    ]
}
2016-02-15 01:50:10,064 [virtwho.main DEBUG] MainProcess(13730):MainThread @virtwho.py:send_current_report:159 - Report for config "env/cmdline" sent
================================hang up =====================

2.Check virt-who's process
[root@ibm-x3630m4-01 ~]# ps -ef|grep virt-who
root     13730 13221  0 01:50 pts/2    00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py --libvirt --libvirt-owner=ACME_Corporation --libvirt-env=Library --libvirt-server=sgi-xe500-01.rhts.eng.bos.redhat.com --libvirt-username=root --libvirt-password=-o -d
root     13737 13730  0 01:50 pts/2    00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py --libvirt --libvirt-owner=ACME_Corporation --libvirt-env=Library --libvirt-server=sgi-xe500-01.rhts.eng.bos.redhat.com --libvirt-username=root --libvirt-password=-o -d
root     13745 13242  0 01:51 pts/3    00:00:00 grep virt-who

Comment 6 Radek Novacek 2016-02-18 11:55:25 UTC
Is "--libvirt-password=-o" this just a typo when reporting this bug? If not it is the cause of this problem. The "-o" is used as an password and not as a oneshot option.

Comment 7 Liushihui 2016-02-19 02:01:06 UTC
It definitely cause by the wrong command line, it work normally after run correct command line. Therefore, verify it.

Comment 9 errata-xmlrpc 2016-05-10 23:57:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0859.html