Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1830084

Summary:	Network error, unable to connect to server. / ConnectionRefusedError: [Errno 111] Connection refused
Product:	Red Hat Enterprise Linux 8	Reporter:	John Sefler <jsefler>
Component:	subscription-manager	Assignee:	candlepin-bugs
Status:	CLOSED CURRENTRELEASE	QA Contact:	Red Hat subscription-manager QE Team <rhsm-qe>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	---	CC:	peasters, redakkan
Target Milestone:	rc	Flags:	pm-rhel: mirror+
Target Release:	8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-12-15 09:16:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description John Sefler 2020-04-30 19:43:36 UTC

Description of problem:

When attempting to use subscription-manager to register (and also subscribe) to the default entitlement server, hostname = [subscription.rhsm.redhat.com], registration sometimes fails with a "Network error, unable to connect to server. Please see /var/log/rhsm/rhsm.log for more information." and looking into the rhsm.log reveals a "ConnectionRefusedError: [Errno 111] Connection refused" regardless of my valid credentials.

This problem started occurring on April 17-ish, 2020 and has not gone away.

Version-Release number of selected component (if applicable):
  occurs on both RHEL7 and RHEL8
  subscription-manager: 1.26.16-1.el8 is what I am using on RHEL8


How reproducible:
  ***intermittent*** but frequent!

Steps to Reproduce:

# subscription-manager register --username=rhelentqe --password=REDACTED --org=12415135 --force
Unregistering from: subscription.rhsm.redhat.com:443/subscription
The system with UUID 1d3a05a4-3f27-42f5-b7e0-e8965c59b001 has been unregistered
All local data removed
Registering to: subscription.rhsm.redhat.com:443/subscription
Network error, unable to connect to server. Please see /var/log/rhsm/rhsm.log for more information.

# LINE_NUMBER=$(grep --line-number 'Making request:' /var/log/rhsm/rhsm.log | tail --lines=1 | cut --delimiter=':' --field=1); if [ -n "$LINE_NUMBER" ]; then tail -n +$LINE_NUMBER /var/log/rhsm/rhsm.log; fi;
2020-04-17 12:00:15,765 [DEBUG] subscription-manager:43929:MainThread @connection.py:571 - Making request: POST /subscription/consumers?owner=12415135
2020-04-17 12:00:15,851 [ERROR] subscription-manager:43929:MainThread @managercli.py:216 - Error during registration: [Errno 111] Connection refused
2020-04-17 12:00:15,851 [ERROR] subscription-manager:43929:MainThread @managercli.py:217 - [Errno 111] Connection refused
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 1370, in _do_command
type=self.options.consumertype
File "/usr/lib64/python3.6/site-packages/rhsmlib/services/register.py", line 91, in register
usage=usage
File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 1021, in registerConsumer
return self.conn.request_post(url, params)
File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 729, in request_post
return self._request("POST", method, params, headers=headers)
File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 752, in _request
info=info, headers=headers)
File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 583, in _request
conn.request(request_type, handler, body=body, headers=final_headers)
File "/usr/lib64/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib64/python3.6/http/client.py", line 974, in send
self.connect()
File "/usr/lib64/python3.6/http/client.py", line 1414, in connect
super().connect()
File "/usr/lib64/python3.6/http/client.py", line 946, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
raise err
File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused



Additional info:
I have also caught same error while attempting to auto subscribe as shown here...

# subscription-manager subscribe --auto
Network error, unable to connect to server. Please see /var/log/rhsm/rhsm.log for more information.

# LINE_NUMBER=$(grep --line-number 'Making request:' /var/log/rhsm/rhsm.log | tail --lines=1 | cut --delimiter=':' --field=1); if [ -n "$LINE_NUMBER" ]; then tail -n +$LINE_NUMBER /var/log/rhsm/rhsm.log; fi;
2020-04-30 15:02:02,824 [DEBUG] subscription-manager:156849:MainThread @connection.py:571 - Making request: GET /subscription/consumers/e1b93bb4-f0e3-4f88-8dc9-e364f09567fd/certificates/serials
2020-04-30 15:02:02,909 [ERROR] subscription-manager:156849:MainThread @entcertlib.py:121 - [Errno 111] Connection refused
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 119, in perform
    expected = self._get_expected_serials()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 292, in _get_expected_serials
    exp = self.get_certificate_serials_list()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 272, in get_certificate_serials_list
    reply = self.uep.getCertificateSerials(identity.uuid)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 1301, in getCertificateSerials
    return self.conn.request_get(method)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 726, in request_get
    return self._request("GET", method, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 752, in _request
    info=info, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 583, in _request
    conn.request(request_type, handler, body=body, headers=final_headers)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/lib64/python3.6/http/client.py", line 1414, in connect
    super().connect()
  File "/usr/lib64/python3.6/http/client.py", line 946, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
2020-04-30 15:02:02,911 [ERROR] subscription-manager:156849:MainThread @entcertlib.py:122 - Cannot modify subscriptions while disconnected
2020-04-30 15:02:02,911 [ERROR] subscription-manager:156849:MainThread @managercli.py:216 - Unable to attach: 
2020-04-30 15:02:02,911 [ERROR] subscription-manager:156849:MainThread @managercli.py:217 - 
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 119, in perform
    expected = self._get_expected_serials()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 292, in _get_expected_serials
    exp = self.get_certificate_serials_list()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 272, in get_certificate_serials_list
    reply = self.uep.getCertificateSerials(identity.uuid)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 1301, in getCertificateSerials
    return self.conn.request_get(method)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 726, in request_get
    return self._request("GET", method, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 752, in _request
    info=info, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 583, in _request
    conn.request(request_type, handler, body=body, headers=final_headers)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/lib64/python3.6/http/client.py", line 1414, in connect
    super().connect()
  File "/usr/lib64/python3.6/http/client.py", line 946, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 1871, in _do_command
    report = self.entcertlib.update()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/certlib.py", line 32, in update
    self.report = self.locker.run(self._do_update)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/certlib.py", line 18, in run
    return action()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 42, in _do_update
    return action.perform()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 123, in perform
    raise Disconnected()
subscription_manager.entcertlib.Disconnected

Comment 1 John Sefler 2020-05-08 13:46:40 UTC

And here is another example that just occurred while I was removing repo overrides...

# subscription-manager repo-override --remove-all
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 119, in perform
    expected = self._get_expected_serials()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 292, in _get_expected_serials
    exp = self.get_certificate_serials_list()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 272, in get_certificate_serials_list
    reply = self.uep.getCertificateSerials(identity.uuid)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 1301, in getCertificateSerials
    return self.conn.request_get(method)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 726, in request_get
    return self._request("GET", method, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 752, in _request
    info=info, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 583, in _request
    conn.request(request_type, handler, body=body, headers=final_headers)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/lib64/python3.6/http/client.py", line 1414, in connect
    super().connect()
  File "/usr/lib64/python3.6/http/client.py", line 946, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/sbin/subscription-manager", line 11, in <module>
    load_entry_point('subscription-manager==1.26.16', 'console_scripts', 'subscription-manager')()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/subscription_manager.py", line 87, in main
    return managercli.ManagerCLI().main()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 2996, in main
    ret = CLI.main(self)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/cli.py", line 183, in main
    return cmd.main()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 536, in main
    return_code = self._do_command()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 2812, in _do_command
    self.entcertlib.update()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/certlib.py", line 32, in update
    self.report = self.locker.run(self._do_update)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/certlib.py", line 18, in run
    return action()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 42, in _do_update
    return action.perform()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 123, in perform
    raise Disconnected()
subscription_manager.entcertlib.Disconnected

Comment 2 John Sefler 2020-05-08 13:53:03 UTC

The ConnectionRefusedError can happen at anytime regardless of my individual system's load on the Red Hat entitlement servers.  My understanding is that the issue is caused by a recently applied GLOBAL connection load limit applied at the load balancer layer.  During bursts of high incoming connection requests, the load balancer appears to refuse connections making subscription-manager fail.  The only choice I have as a subscription-manager user is to... try again.

Comment 3 Patrick Easters 2020-05-11 15:51:52 UTC

The global rate limits causing the connection refused errors has been removed. Can you confirm if this is still happening?

Comment 4 John Sefler 2020-06-15 13:13:32 UTC

Moving this to VERIFIED.  Automated tests are no longer encountering the ConnectionRefusedError.

Comment 5 John Sefler 2020-06-15 13:19:24 UTC

Final comment... There was no code change to component subscription-manager.  This issue was introduced by IT infrastructure and addressed by IT infrastructure.