Bug 1336904

Summary: subscription-manager auto-heal prevents systems from healing due to a locked child process
Product: Red Hat Enterprise Linux 6 Reporter: Kathryn Dixon <kdixon>
Component: subscription-managerAssignee: Kevin Howell <khowell>
Status: CLOSED CURRENTRELEASE QA Contact: John Sefler <jsefler>
Severity: high Docs Contact:
Priority: high    
Version: 6.7CC: ajanorka, baitken, bcourt, brian.murrell, candlepin-bugs, cdonnell, cww, khowell, mike2win, redakkan, vrjain, xdmoon
Target Milestone: rcKeywords: Reopened, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: subscription-manager-1.17.14-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-07 15:14:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1306058, 1355878, 1374441, 1385841    
Attachments:
Description Flags
command hang attachement none

Description Kathryn Dixon 2016-05-17 18:35:37 UTC
Description of problem:

After subscriptions expire, and then new ones become available systems should heal themselves with a new subscription. In this case customer had around 10 systems that did not heal, and there was nothing in the logs that would tell us why. Almost all subscription-manager commands would work, however # subscription-manager refresh would hang. 

Version-Release number of selected component (if applicable):


How reproducible: 50%


Steps to Reproduce:
1. system is registered to portal or satellite
2. subscription becomes expired
3. rhsmcertd has a child process that seems to prevent us on making any changes to the system in the customer portal or satellite. You can add a subscription in the portal/webui but when you refresh it just hangs. Attaching straces

Actual results:

# subscription-manager refresh
Hangs and hangs until you control c out. No log helped us, until we did an strace. There should be something at the very least in the rhsm.log that has "system can't be refreshed do to another process" 


Expected results:

Auto-heal should heal the system, and if it can't the process should die out so you can manually add one.

Additional info:

Comment 4 Blair Aitken 2016-06-13 19:10:29 UTC
Hello,

Also noticed this behavior, possibly related to this bug:
my customer is experiencing this issue

* /etc/cron.daily/rhsmd hangs 
* this cron.daily to not complete, and will not run again until rhsmd pid is killed

strace of hang:

83700 03:48:52.150791 connect(4, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("10.7.50.197")}, 16) = 0 <0.000921>
83700 03:48:52.151821 sendto(4, "CONNECT subscription.rhn.redhat."..., 91, 0, NULL, 0) = 91 <0.000018>
83700 03:48:52.151954 recvfrom(4,

rhsmd does not timeout and remains hung until pid is killed.

Can anyone confirm this issue is associated with this bug, or does a new bug need to be opened?

Blair

Comment 5 Barnaby Court 2016-06-13 20:32:07 UTC
Blair,

That would be exactly the same connection hang mentioned in Comment 3. A new bug does not need to be opened.

Comment 9 Kevin Howell 2016-09-12 20:55:31 UTC
We've added a default timeout to the https connections in python-rhsm of 3 minutes. This addresses all of the known causes of rhsmcertd-worker hanging.

We've also addressed rhsmd itself not exiting properly (will attach PR).

In testing, I found that rhsmcertd-worker makes two consecutive requests; with default settings, it should take rhsmcertd-worker about 6 minutes total to timeout.

Comment 10 Kevin Howell 2016-09-13 14:18:02 UTC
*** Bug 1255812 has been marked as a duplicate of this bug. ***

Comment 15 ajanorka 2017-05-03 07:34:24 UTC
Hello,


customer has got the latest subscription manager package installed :-

# yum list subscr*
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Installed Packages
subscription-manager.x86_64                                                                      
1.18.10-1.el6                                                          
@rhel-6-server-rpms       

and still the command hangs :-

the refresh command never returns:

# time subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Current


real    0m0.733s
user    0m0.302s
sys     0m0.079s
# time subscription-manager refresh
^C
User interrupted process.


real    5m9.161s
user    0m0.003s
sys     0m0.004s
#

Customer has also shared the list of processes that are currently hung. 
Reopening the case. please let me know if you wuld like to see the list of hung processes.

Regards,
Anuja

Comment 16 Kevin Howell 2017-05-03 14:03:27 UTC
Anuja,

Please do share the list of hung processes. Also a fresh strace of the hung process might be helpful.

Thanks,
Kevin

Comment 18 ajanorka 2017-05-10 04:32:15 UTC
Created attachment 1277493 [details]
command hang attachement

Comment 22 Kevin Howell 2017-07-07 15:14:25 UTC
Closing this again, since there were fixes for root causes identified in this bug, and the all the identified related customer cases have been addressed (as far as I can tell). Please open a new bug if necessary for related issues.

Comment 23 Red Hat Bugzilla 2023-09-14 03:22:48 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days