RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1336904 - subscription-manager auto-heal prevents systems from healing due to a locked child process
Summary: subscription-manager auto-heal prevents systems from healing due to a locked ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: subscription-manager
Version: 6.7
Hardware: Unspecified
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Kevin Howell
QA Contact: John Sefler
URL:
Whiteboard:
: 1255812 (view as bug list)
Depends On:
Blocks: 1306058 1355878 1374441 CEE_Sat6_Top_BZs, GSS_Sat6_Top_Bugs
TreeView+ depends on / blocked
 
Reported: 2016-05-17 18:35 UTC by Kathryn Dixon
Modified: 2023-09-14 03:22 UTC (History)
12 users (show)

Fixed In Version: subscription-manager-1.17.14-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-07-07 15:14:25 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
command hang attachement (41.67 KB, text/plain)
2017-05-10 04:32 UTC, ajanorka
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github candlepin python-rhsm pull 170 0 'None' closed Awood/socket timeout 2021-01-07 10:58:48 UTC
Github candlepin subscription-manager pull 1456 0 'None' closed 1351370: Ensure rhsmd exits on exceptions 2021-01-07 10:58:52 UTC
Red Hat Knowledge Base (Solution) 1754373 0 None None None 2016-05-17 19:58:36 UTC

Description Kathryn Dixon 2016-05-17 18:35:37 UTC
Description of problem:

After subscriptions expire, and then new ones become available systems should heal themselves with a new subscription. In this case customer had around 10 systems that did not heal, and there was nothing in the logs that would tell us why. Almost all subscription-manager commands would work, however # subscription-manager refresh would hang. 

Version-Release number of selected component (if applicable):


How reproducible: 50%


Steps to Reproduce:
1. system is registered to portal or satellite
2. subscription becomes expired
3. rhsmcertd has a child process that seems to prevent us on making any changes to the system in the customer portal or satellite. You can add a subscription in the portal/webui but when you refresh it just hangs. Attaching straces

Actual results:

# subscription-manager refresh
Hangs and hangs until you control c out. No log helped us, until we did an strace. There should be something at the very least in the rhsm.log that has "system can't be refreshed do to another process" 


Expected results:

Auto-heal should heal the system, and if it can't the process should die out so you can manually add one.

Additional info:

Comment 4 Blair Aitken 2016-06-13 19:10:29 UTC
Hello,

Also noticed this behavior, possibly related to this bug:
my customer is experiencing this issue

* /etc/cron.daily/rhsmd hangs 
* this cron.daily to not complete, and will not run again until rhsmd pid is killed

strace of hang:

83700 03:48:52.150791 connect(4, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("10.7.50.197")}, 16) = 0 <0.000921>
83700 03:48:52.151821 sendto(4, "CONNECT subscription.rhn.redhat."..., 91, 0, NULL, 0) = 91 <0.000018>
83700 03:48:52.151954 recvfrom(4,

rhsmd does not timeout and remains hung until pid is killed.

Can anyone confirm this issue is associated with this bug, or does a new bug need to be opened?

Blair

Comment 5 Barnaby Court 2016-06-13 20:32:07 UTC
Blair,

That would be exactly the same connection hang mentioned in Comment 3. A new bug does not need to be opened.

Comment 9 Kevin Howell 2016-09-12 20:55:31 UTC
We've added a default timeout to the https connections in python-rhsm of 3 minutes. This addresses all of the known causes of rhsmcertd-worker hanging.

We've also addressed rhsmd itself not exiting properly (will attach PR).

In testing, I found that rhsmcertd-worker makes two consecutive requests; with default settings, it should take rhsmcertd-worker about 6 minutes total to timeout.

Comment 10 Kevin Howell 2016-09-13 14:18:02 UTC
*** Bug 1255812 has been marked as a duplicate of this bug. ***

Comment 15 ajanorka 2017-05-03 07:34:24 UTC
Hello,


customer has got the latest subscription manager package installed :-

# yum list subscr*
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Installed Packages
subscription-manager.x86_64                                                                      
1.18.10-1.el6                                                          
@rhel-6-server-rpms       

and still the command hangs :-

the refresh command never returns:

# time subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Current


real    0m0.733s
user    0m0.302s
sys     0m0.079s
# time subscription-manager refresh
^C
User interrupted process.


real    5m9.161s
user    0m0.003s
sys     0m0.004s
#

Customer has also shared the list of processes that are currently hung. 
Reopening the case. please let me know if you wuld like to see the list of hung processes.

Regards,
Anuja

Comment 16 Kevin Howell 2017-05-03 14:03:27 UTC
Anuja,

Please do share the list of hung processes. Also a fresh strace of the hung process might be helpful.

Thanks,
Kevin

Comment 18 ajanorka 2017-05-10 04:32:15 UTC
Created attachment 1277493 [details]
command hang attachement

Comment 22 Kevin Howell 2017-07-07 15:14:25 UTC
Closing this again, since there were fixes for root causes identified in this bug, and the all the identified related customer cases have been addressed (as far as I can tell). Please open a new bug if necessary for related issues.

Comment 23 Red Hat Bugzilla 2023-09-14 03:22:48 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.