Bug 1830084

Summary:	Network error, unable to connect to server. / ConnectionRefusedError: [Errno 111] Connection refused
Product:	Red Hat Enterprise Linux 8	Reporter:	John Sefler <jsefler>
Component:	subscription-manager	Assignee:	candlepin-bugs
Status:	CLOSED CURRENTRELEASE	QA Contact:	Red Hat subscription-manager QE Team <rhsm-qe>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	---	CC:	peasters, redakkan
Target Milestone:	rc
Target Release:	8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-12-15 09:16:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description John Sefler 2020-04-30 19:43:36 UTC

Description of problem:

When attempting to use subscription-manager to register (and also subscribe) to the default entitlement server, hostname = [subscription.rhsm.redhat.com], registration sometimes fails with a "Network error, unable to connect to server. Please see /var/log/rhsm/rhsm.log for more information." and looking into the rhsm.log reveals a "ConnectionRefusedError: [Errno 111] Connection refused" regardless of my valid credentials.

This problem started occurring on April 17-ish, 2020 and has not gone away.

Version-Release number of selected component (if applicable):
  occurs on both RHEL7 and RHEL8
  subscription-manager: 1.26.16-1.el8 is what I am using on RHEL8


How reproducible:
  ***intermittent*** but frequent!

Steps to Reproduce:

# subscription-manager register --username=rhelentqe --password=REDACTED --org=12415135 --force
Unregistering from: subscription.rhsm.redhat.com:443/subscription
The system with UUID 1d3a05a4-3f27-42f5-b7e0-e8965c59b001 has been unregistered
All local data removed
Registering to: subscription.rhsm.redhat.com:443/subscription
Network error, unable to connect to server. Please see /var/log/rhsm/rhsm.log for more information.

# LINE_NUMBER=$(grep --line-number 'Making request:' /var/log/rhsm/rhsm.log | tail --lines=1 | cut --delimiter=':' --field=1); if [ -n "$LINE_NUMBER" ]; then tail -n +$LINE_NUMBER /var/log/rhsm/rhsm.log; fi;
2020-04-17 12:00:15,765 [DEBUG] subscription-manager:43929:MainThread @connection.py:571 - Making request: POST /subscription/consumers?owner=12415135
2020-04-17 12:00:15,851 [ERROR] subscription-manager:43929:MainThread @managercli.py:216 - Error during registration: [Errno 111] Connection refused
2020-04-17 12:00:15,851 [ERROR] subscription-manager:43929:MainThread @managercli.py:217 - [Errno 111] Connection refused
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 1370, in _do_command
type=self.options.consumertype
File "/usr/lib64/python3.6/site-packages/rhsmlib/services/register.py", line 91, in register
usage=usage
File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 1021, in registerConsumer
return self.conn.request_post(url, params)
File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 729, in request_post
return self._request("POST", method, params, headers=headers)
File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 752, in _request
info=info, headers=headers)
File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 583, in _request
conn.request(request_type, handler, body=body, headers=final_headers)
File "/usr/lib64/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib64/python3.6/http/client.py", line 974, in send
self.connect()
File "/usr/lib64/python3.6/http/client.py", line 1414, in connect
super().connect()
File "/usr/lib64/python3.6/http/client.py", line 946, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
raise err
File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused



Additional info:
I have also caught same error while attempting to auto subscribe as shown here...

# subscription-manager subscribe --auto
Network error, unable to connect to server. Please see /var/log/rhsm/rhsm.log for more information.

# LINE_NUMBER=$(grep --line-number 'Making request:' /var/log/rhsm/rhsm.log | tail --lines=1 | cut --delimiter=':' --field=1); if [ -n "$LINE_NUMBER" ]; then tail -n +$LINE_NUMBER /var/log/rhsm/rhsm.log; fi;
2020-04-30 15:02:02,824 [DEBUG] subscription-manager:156849:MainThread @connection.py:571 - Making request: GET /subscription/consumers/e1b93bb4-f0e3-4f88-8dc9-e364f09567fd/certificates/serials
2020-04-30 15:02:02,909 [ERROR] subscription-manager:156849:MainThread @entcertlib.py:121 - [Errno 111] Connection refused
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 119, in perform
    expected = self._get_expected_serials()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 292, in _get_expected_serials
    exp = self.get_certificate_serials_list()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 272, in get_certificate_serials_list
    reply = self.uep.getCertificateSerials(identity.uuid)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 1301, in getCertificateSerials
    return self.conn.request_get(method)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 726, in request_get
    return self._request("GET", method, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 752, in _request
    info=info, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 583, in _request
    conn.request(request_type, handler, body=body, headers=final_headers)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/lib64/python3.6/http/client.py", line 1414, in connect
    super().connect()
  File "/usr/lib64/python3.6/http/client.py", line 946, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
2020-04-30 15:02:02,911 [ERROR] subscription-manager:156849:MainThread @entcertlib.py:122 - Cannot modify subscriptions while disconnected
2020-04-30 15:02:02,911 [ERROR] subscription-manager:156849:MainThread @managercli.py:216 - Unable to attach: 
2020-04-30 15:02:02,911 [ERROR] subscription-manager:156849:MainThread @managercli.py:217 - 
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 119, in perform
    expected = self._get_expected_serials()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 292, in _get_expected_serials
    exp = self.get_certificate_serials_list()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 272, in get_certificate_serials_list
    reply = self.uep.getCertificateSerials(identity.uuid)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 1301, in getCertificateSerials
    return self.conn.request_get(method)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 726, in request_get
    return self._request("GET", method, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 752, in _request
    info=info, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 583, in _request
    conn.request(request_type, handler, body=body, headers=final_headers)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/lib64/python3.6/http/client.py", line 1414, in connect
    super().connect()
  File "/usr/lib64/python3.6/http/client.py", line 946, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 1871, in _do_command
    report = self.entcertlib.update()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/certlib.py", line 32, in update
    self.report = self.locker.run(self._do_update)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/certlib.py", line 18, in run
    return action()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 42, in _do_update
    return action.perform()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 123, in perform
    raise Disconnected()
subscription_manager.entcertlib.Disconnected

Comment 1 John Sefler 2020-05-08 13:46:40 UTC

And here is another example that just occurred while I was removing repo overrides...

# subscription-manager repo-override --remove-all
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 119, in perform
    expected = self._get_expected_serials()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 292, in _get_expected_serials
    exp = self.get_certificate_serials_list()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 272, in get_certificate_serials_list
    reply = self.uep.getCertificateSerials(identity.uuid)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 1301, in getCertificateSerials
    return self.conn.request_get(method)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 726, in request_get
    return self._request("GET", method, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 752, in _request
    info=info, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 583, in _request
    conn.request(request_type, handler, body=body, headers=final_headers)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/lib64/python3.6/http/client.py", line 1414, in connect
    super().connect()
  File "/usr/lib64/python3.6/http/client.py", line 946, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/sbin/subscription-manager", line 11, in <module>
    load_entry_point('subscription-manager==1.26.16', 'console_scripts', 'subscription-manager')()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/subscription_manager.py", line 87, in main
    return managercli.ManagerCLI().main()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 2996, in main
    ret = CLI.main(self)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/cli.py", line 183, in main
    return cmd.main()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 536, in main
    return_code = self._do_command()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 2812, in _do_command
    self.entcertlib.update()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/certlib.py", line 32, in update
    self.report = self.locker.run(self._do_update)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/certlib.py", line 18, in run
    return action()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 42, in _do_update
    return action.perform()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/entcertlib.py", line 123, in perform
    raise Disconnected()
subscription_manager.entcertlib.Disconnected

Comment 2 John Sefler 2020-05-08 13:53:03 UTC

The ConnectionRefusedError can happen at anytime regardless of my individual system's load on the Red Hat entitlement servers.  My understanding is that the issue is caused by a recently applied GLOBAL connection load limit applied at the load balancer layer.  During bursts of high incoming connection requests, the load balancer appears to refuse connections making subscription-manager fail.  The only choice I have as a subscription-manager user is to... try again.

Comment 3 Patrick Easters 2020-05-11 15:51:52 UTC

The global rate limits causing the connection refused errors has been removed. Can you confirm if this is still happening?

Comment 4 John Sefler 2020-06-15 13:13:32 UTC

Moving this to VERIFIED.  Automated tests are no longer encountering the ConnectionRefusedError.

Comment 5 John Sefler 2020-06-15 13:19:24 UTC

Final comment... There was no code change to component subscription-manager.  This issue was introduced by IT infrastructure and addressed by IT infrastructure.