Bug 1293821

Summary: virt-who processes(PPID != 1) cannot be killed immediately after unregister system
Product: Red Hat Enterprise Linux 6 Reporter: Liushihui <shihliu>
Component: virt-whoAssignee: Radek Novacek <rnovacek>
Status: CLOSED ERRATA QA Contact: gaoshang <sgao>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.8CC: gxing, hsun, ovasik, rbalakri, sgao, shihliu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: virt-who-0.16-4.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-10 23:57:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Liushihui 2015-12-23 07:32:52 UTC
Description of problem:
After unregister system, virt-who processes which PPID is not 1 can not be killed until 30s. it also won't show system has been unregistered in the rhsm.log. 

Version-Release number of selected component (if applicable):
virt-who-0.16-1.el6.noarch
subscription-manager-1.16.6-1.el6.x86_64
python-rhsm-1.16.4-1.el6.x86_64

How reproducible:
80%

Steps to Reproduce:
1. Register system to Satellite6. make virt-who run at kvm mode, refresh interval is default.
[root@hp-z220-05 tmp]# subscription-manager register --username=admin --password=admin
Registering to: cisco-b200m1-04.rhts.eng.bos.redhat.com:443/rhsm
The system has been registered with ID: b9f6ca8a-3ae8-4b66-b293-847e39481ce0 
2. Start virt-who service 
[root@hp-z220-05 tmp]# service virt-who start
[root@hp-z220-05 tmp]# ps -ef|grep virt-who
root     10795     1  0 14:00 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root     11055 10795  0 14:05 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root     11060  9603  0 14:05 pts/0    00:00:00 grep virt-who
3. Unregister system. then check virt-who processes.
[root@hp-z220-05 tmp]# subscription-manager unregister
System has been unregistered.
[root@hp-z220-05 tmp]# ps -ef|grep virt-who
root     10795     1  0 14:00 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root     11055 10795  0 14:05 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root     11060  9603  0 14:05 pts/0    00:00:00 grep virt-who
4. Check virt-who's log in /var/log/rhsm/rhsm.log
# tail -f /var/log/rhsm/rhsm.log
5. After about 30s, check virt-who processes:
[root@hp-z220-05 tmp]# ps -ef|grep virt-who
root     10795     1  0 14:00 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root     11062  9603  0 14:06 pts/0    00:00:00 grep virt-who

Actual results:
It still show two virt-who processes after unregister system.
It will not show system is unregistered immediately.
2015-12-23 14:05:28,759 [virtwho.env_cmdline DEBUG] Libvirtd-6(11055):MainThread @libvirtd.py:_listDomains:238 - Libvirt domains found: 3c6e36b9-a88e-dacf-3d8d-782fcd8cbb39
2015-12-23 14:05:28,759 [virtwho.env_cmdline DEBUG] Libvirtd-6(11055):MainThread @virt.py:enqueue:331 - Report gathered, putting to queue for sending
2015-12-23 14:05:28,762 [virtwho.main DEBUG] MainProcess(10795):MainThread @virtwho.py:update_report_to_send:274 - Report for config "env/cmdline" updated
=============================================================================
==============After 30s, it will show "system is not registered "===========
=============================================================================                                                   
2015-12-23 14:05:59,110 [virtwho.main DEBUG] MainProcess(10795):MainThread @subscriptionmanager.py:_connect:112 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2015-12-23 14:05:59,111 [virtwho.main ERROR] MainProcess(10795):MainThread @virtwho.py:send:232 - Error in communication with subscription manager:
Traceback (most recent call last):
  File "/usr/share/virt-who/virtwho.py", line 222, in send
    self._sendGuestList(report)
  File "/usr/share/virt-who/virtwho.py", line 249, in _sendGuestList
    manager.sendVirtGuests(report.guests)
  File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 130, in sendVirtGuests
    self._connect()
  File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 115, in _connect
    "Unable to read certificate, system is not registered or you are not root")
SubscriptionManagerUnregisteredError: Unable to read certificate, system is not registered or you are not root

Expected results:
It should keep only one virt-who(PPID=1) process after unregister system. system has been unregistered should be shown on rhsm log immediately.

Additional info:

Comment 1 Radek Novacek 2016-01-28 13:04:23 UTC
Fixed in virt-who-0.16-2.el6.

Comment 3 Liushihui 2016-02-04 07:45:51 UTC
Reopen it on virt-who-0.16-3.el6.noarch since virt-who dead after unregister system. please see the detail as the following:

Checked version:
virt-who-0.16-3.el6.noarch
subscription-manager-1.16.8-1.el6.x86_64
python-rhsm-1.16.6-1.el6.x86_64

Checked processes:
1.Register system to satellite and run virt-who at libvirt mode.there are 2 virt-who threads.
[root@intel-canoepass-10 ~]# service virt-who status
virt-who (pid  13779) is running...
[root@intel-canoepass-10 ~]# ps -ef|grep virt-who 
root      13779      1  1 02:37 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root      13786  13779  0 02:37 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root      13808  10536  0 02:37 pts/1    00:00:00 grep virt-who
2. Unregister system,then check virt-who thread
[root@intel-canoepass-10 ~]# subscription-manager unregister
System has been unregistered.
[root@intel-canoepass-10 ~]# ps -ef|grep virt-who 
root      13818  10536  0 02:38 pts/1    00:00:00 grep virt-who
[root@intel-canoepass-10 ~]# service virt-who status
virt-who dead but subsys locked

Result:
virt-who dead after unregister system.

Expected result:
It should keep only one virt-who(PPID=1) process after unregister system.

Comment 4 Radek Novacek 2016-02-11 13:07:56 UTC
Fixed in virt-who-0.16-4.el6.

Comment 6 Liushihui 2016-02-25 07:47:14 UTC
Verified it on virt-who-0.16-4.el6.noarch since virt-who can work normally and system has been unregistered has shown on rhsm log immediately after unregister system. Therefore, verify it.

Verified version:
virt-who-0.16-4.el6.noarch
subscription-manager-1.16.8-2.el6.x86_64
python-rhsm-1.16.6-1.el6.x86_64

Verified process:
1.Register system to Satellite and run virt-who at libvirt mode.there are 2 virt-who threads.
[root@sgi-xe500-01 ~]# service virt-who restart
Stopping virt-who: [  OK  ]
Starting virt-who: [  OK  ]
[root@sgi-xe500-01 ~]# ps -ef|grep virt-who
root      1379     1  1 02:33 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root      1387  1379  0 02:33 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root      1392  8716  0 02:33 pts/0    00:00:00 grep virt-who

2. Unregister system,then check virt-who thread
[root@sgi-xe500-01 ~]# subscription-manager  unregister
System has been unregistered.
[root@sgi-xe500-01 ~]# ps -ef|grep virt-who
root      1379     1  0 02:33 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root      1397  1379  0 02:33 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root      1402  8716  0 02:34 pts/0    00:00:00 grep virt-who

3. Check virt-who's log after unregister system.
2016-02-25 02:35:43,162 [virtwho.env_cmdline DEBUG] HyperV-4(1494):MainThread @hyperv.py:getHostGuestMapping:521 - Unable to enumerate using root/virtualization namespace, trying root/virtualization/v2 namespace
2016-02-25 02:35:46,749 [virtwho.test-esx1 DEBUG] Esx-3(1492):MainThread @esx.py:_prepare:58 - Creating ESX event filter
2016-02-25 02:35:54,734 [virtwho.env_cmdline DEBUG] HyperV-4(1494):MainThread @virt.py:enqueue:351 - Report gathered, putting to queue for sending
2016-02-25 02:35:54,741 [virtwho.main DEBUG] MainProcess(1469):MainThread @subscriptionmanager.py:_connect:121 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2016-02-25 02:35:54,745 [virtwho.main ERROR] MainProcess(1469):MainThread @virtwho.py:_main:751 - Fatal error:
Traceback (most recent call last):
  File "/usr/share/virt-who/virtwho.py", line 748, in _main
    result = virtWho.run()
  File "/usr/share/virt-who/virtwho.py", line 333, in run
    self.send_current_report()
  File "/usr/share/virt-who/virtwho.py", line 153, in send_current_report
    if self.send(report):
  File "/usr/share/virt-who/virtwho.py", line 197, in send
    self._sendGuestAssociation(report)
  File "/usr/share/virt-who/virtwho.py", line 224, in _sendGuestAssociation
    manager.hypervisorCheckIn(report, self.options)
  File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 161, in hypervisorCheckIn
    self._connect(report.config)
  File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 124, in _connect
    "Unable to read certificate, system is not registered or you are not root")
SubscriptionManagerUnregisteredError: Unable to read certificate, system is not registered or you are not root
2016-02-25 02:35:54,747 [virtwho.main INFO] MainProcess(1469):MainThread @virtwho.py:_main:753 - Waiting for reload signal

4. Re-register system to satellite and check vir-who status and threads
[root@sgi-xe500-01 ~]# subscription-manager  register --username=admin --password=admin
Registering to: bigisis.rhts.eng.bos.redhat.com:443/rhsm
The system has been registered with ID: f901e50e-f52c-4595-a77c-3f364c2739a3 
[root@sgi-xe500-01 ~]# ps -ef|grep virt-who
root      1379     1  0 02:33 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root      1432  1379  0 02:34 ?        00:00:00 /usr/bin/python /usr/share/virt-who/virtwho.py
root      1442  8716  0 02:34 pts/0    00:00:00 grep virt-who
[root@sgi-xe500-01 ~]# service virt-who status
virt-who (pid  1469) is running...

Result:
After unregister system, virt-who run normally and "system has been unregistered" has shown on rhsm log immediately.Re-register system to server, virt-who still work normally.

Comment 8 errata-xmlrpc 2016-05-10 23:57:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0859.html