2183533 – rhcd disconnects from console dot. Satellite source displays status of Unavailable.

Bug 2183533 - rhcd disconnects from console dot. Satellite source displays status of Unavailable. [NEEDINFO]

Summary: rhcd disconnects from console dot. Satellite source displays status of Unavai...

Keywords:
Status:	NEW
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	RH Cloud - Cloud Connector
Sub Component:
Version:	Unspecified
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	medium
Target Milestone:	Unspecified
Assignee:	satellite6-bugs
QA Contact:	Satellite QE Team
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2023-03-31 14:22 UTC by Matthew Yee
Modified:	2023-07-11 00:22 UTC (History)
CC List:	19 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:
Flags:	dajohnso: needinfo? (dhorton)

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	ESSNTL-4456	0	None	None	None	2023-05-17 15:52:15 UTC
Red Hat Issue Tracker	SAT-18147	0	None	None	None	2023-06-06 16:17:26 UTC

Description Matthew Yee 2023-03-31 14:22:26 UTC

Description of problem:
This is an intermittent problem where rhcd on the Satelltie server will disconnect. "systemctl status rhcd.service" shows the service is running but there is an additional message that describes the daemon as disconnected. In console dot, the Satellite server will display a status in "Sources" as unavailable. The fix is relatively easy, ranging from re-running cloud connector setup to restarting rhcd. 

Additionally, it is impossible to apply remediation playbooks from console dot. 

Version-Release number of selected component (if applicable):


How reproducible:
Intermittent.

Steps to Reproduce:
1. Check the status of the rhcd service. If it is not running or it is running and showing a message that it is disconnected, restart the service. 
2.
3.

Actual results:


Expected results:
rhcd should not disconnect.

Additional info:

Comment 1 Brad Buckingham 2023-04-03 13:05:19 UTC

Hi Shim,

Would this be an issue within Satellite, Insights or other?

Thanks!

Comment 3 dhorton 2023-04-10 14:22:51 UTC

Please gather the rhc logs and the rhc client-id when this happens next time.

The rhc client-id comes from the cert:
sudo openssl x509 -in /etc/pki/consumer/cert.pem -noout -text | grep Subject

You should be able to get the rhc logs with this command:
journalctl -u rhcd

Comment 7 Link Dupont 2023-04-11 15:05:19 UTC

What is the output of `rpm -q rhc` on the host this is occurring?

Comment 23 Mark Milano 2023-06-08 13:12:57 UTC

(In reply to dhorton from comment #3)
> Please gather the rhc logs and the rhc client-id when this happens next time.
> 
> The rhc client-id comes from the cert:
> sudo openssl x509 -in /etc/pki/consumer/cert.pem -noout -text | grep Subject
> 
> You should be able to get the rhc logs with this command:
> journalctl -u rhcd

# rpm -q rhc
rhc-0.2.2-1.el8.x86_64

# openssl x509 -in /etc/pki/consumer/cert.pem -noout -text | grep Subject
        Subject: O = 4425915, CN = 1e6b516d-70ff-40c1-a97a-134c119f3ca6
        Subject Public Key Info:
            X509v3 Subject Key Identifier:
            X509v3 Subject Alternative Name:


-- Logs begin at Fri 2023-05-12 02:45:01 CDT, end at Thu 2023-06-08 08:08:12 CDT. --
[rhcd] 2023/05/14 00:42:42 connection lost unexpectedly: pingresp not received, disconnecting
[rhcd] 2023/05/19 19:51:25 connection lost unexpectedly: read tcp 10.22.152.68:38526->104.117.244.34:443: read: connection reset by peer
[rhcd] 2023/05/20 09:52:36 connection lost unexpectedly: read tcp 10.22.152.68:43842->104.117.244.24:443: read: connection reset by peer
[rhcd] 2023/05/22 00:39:52 connection lost unexpectedly: pingresp not received, disconnecting
[rhcd] 2023/05/29 17:09:43 connection lost unexpectedly: pingresp not received, disconnecting
[rhcd] 2023/05/29 22:27:04 connection lost unexpectedly: read tcp 10.22.152.68:54490->23.222.28.17:443: read: connection reset by peer
[rhcd] 2023/05/30 09:01:56 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:01:56 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:01:56 handler registration failed: <nil>
[rhcd] 2023/05/30 09:02:01 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:02:01 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:01 handler registration failed: <nil>
[rhcd] 2023/05/30 09:02:11 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:02:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:11 handler registration failed: <nil>
[rhcd] 2023/05/30 09:02:26 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:02:26 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:26 handler registration failed: <nil>
[rhcd] 2023/05/30 09:02:46 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:02:46 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:46 handler registration failed: <nil>
[rhcd] 2023/05/30 09:03:11 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:03:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:03:11 handler registration failed: <nil>
[rhcd] 2023/05/30 09:03:11 failed to start worker '/usr/libexec/rhc/rhc-package-manager-worker' too many times
Stopping Remote Host Configuration daemon...
[rhcd] 2023/05/30 09:20:13 cannot kill process: os: process already finished
rhcd.service: Succeeded.
Stopped Remote Host Configuration daemon.
Started Remote Host Configuration daemon.
[rhcd] 2023/06/01 06:10:23 connection lost unexpectedly: read tcp 10.22.152.68:36548->23.47.52.17:443: read: connection reset by peer
Stopping Remote Host Configuration daemon...
rhcd.service: Succeeded.
Stopped Remote Host Configuration daemon.
Started Remote Host Configuration daemon.
[rhcd] 2023/06/06 00:15:17 connection lost unexpectedly: read tcp 10.22.152.68:34254->23.47.52.17:443: read: connection reset by peer
[rhcd] 2023/06/06 18:49:57 connection lost unexpectedly: read tcp 10.22.152.68:39208->23.47.52.19:443: read: connection reset by peer
[rhcd] 2023/06/07 00:10:33 connection lost unexpectedly: read tcp 10.22.152.68:44070->23.64.115.140:443: read: connection reset by peer

Comment 27 Mark Milano 2023-06-13 13:13:29 UTC

Latest logs

-- Logs begin at Tue 2023-05-16 16:40:11 CDT, end at Tue 2023-06-13 08:08:15 CDT. --
May 19 19:51:25 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/19 19:51:25 connection lost unexpectedly: read tcp 10.22.152.68:38526->104.117.244.34:443: read: connection reset by peer
May 20 09:52:36 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/20 09:52:36 connection lost unexpectedly: read tcp 10.22.152.68:43842->104.117.244.24:443: read: connection reset by peer
May 22 00:39:52 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/22 00:39:52 connection lost unexpectedly: pingresp not received, disconnecting
May 29 17:09:43 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/29 17:09:43 connection lost unexpectedly: pingresp not received, disconnecting
May 29 22:27:04 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/29 22:27:04 connection lost unexpectedly: read tcp 10.22.152.68:54490->23.222.28.17:443: read: connection reset by peer
May 30 09:01:56 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:01:56 worker failed to register for handler package-manager
May 30 09:01:56 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:01:56 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:01:56 handler registration failed: <nil>
May 30 09:02:01 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:01 worker failed to register for handler package-manager
May 30 09:02:01 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:01 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:01 handler registration failed: <nil>
May 30 09:02:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:11 worker failed to register for handler package-manager
May 30 09:02:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:11 handler registration failed: <nil>
May 30 09:02:26 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:26 worker failed to register for handler package-manager
May 30 09:02:26 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:26 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:26 handler registration failed: <nil>
May 30 09:02:46 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:46 worker failed to register for handler package-manager
May 30 09:02:46 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:46 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:46 handler registration failed: <nil>
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 worker failed to register for handler package-manager
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:03:11 handler registration failed: <nil>
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 failed to start worker '/usr/libexec/rhc/rhc-package-manager-worker' too many times
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Stopping Remote Host Configuration daemon...
May 30 09:20:13 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:20:13 cannot kill process: os: process already finished
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: rhcd.service: Succeeded.
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Stopped Remote Host Configuration daemon.
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Started Remote Host Configuration daemon.
Jun 01 06:10:23 rceeitaprhs001.realpage.com rhcd[3898246]: [rhcd] 2023/06/01 06:10:23 connection lost unexpectedly: read tcp 10.22.152.68:36548->23.47.52.17:443: read: connection reset by peer
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Stopping Remote Host Configuration daemon...
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: rhcd.service: Succeeded.
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Stopped Remote Host Configuration daemon.
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Started Remote Host Configuration daemon.
Jun 06 00:15:17 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/06 00:15:17 connection lost unexpectedly: read tcp 10.22.152.68:34254->23.47.52.17:443: read: connection reset by peer
Jun 06 18:49:57 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/06 18:49:57 connection lost unexpectedly: read tcp 10.22.152.68:39208->23.47.52.19:443: read: connection reset by peer
Jun 07 00:10:33 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/07 00:10:33 connection lost unexpectedly: read tcp 10.22.152.68:44070->23.64.115.140:443: read: connection reset by peer
Jun 13 07:34:42 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/13 07:34:42 connection lost unexpectedly: read tcp 10.22.152.68:33616->23.64.115.139:443: read: connection reset by peer

Comment 29 Mark Milano 2023-06-20 19:12:34 UTC

-- Logs begin at Wed 2023-05-24 07:17:11 CDT, end at Tue 2023-06-20 14:12:11 CDT. --
May 29 17:09:43 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/29 17:09:43 connection lost unexpectedly: pingresp not received, disconnecting
May 29 22:27:04 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/29 22:27:04 connection lost unexpectedly: read tcp 10.22.152.68:54490->23.222.28.17:443: read: connection rese>
May 30 09:01:56 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:01:56 worker failed to register for handler package-manager
May 30 09:01:56 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:01:56 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:01:56 handler registration failed: <ni>
May 30 09:02:01 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:01 worker failed to register for handler package-manager
May 30 09:02:01 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:01 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:01 handler registration failed: <ni>
May 30 09:02:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:11 worker failed to register for handler package-manager
May 30 09:02:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:11 handler registration failed: <ni>
May 30 09:02:26 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:26 worker failed to register for handler package-manager
May 30 09:02:26 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:26 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:26 handler registration failed: <ni>
May 30 09:02:46 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:46 worker failed to register for handler package-manager
May 30 09:02:46 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:46 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:46 handler registration failed: <ni>
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 worker failed to register for handler package-manager
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:03:11 handler registration failed: <ni>
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 failed to start worker '/usr/libexec/rhc/rhc-package-manager-worker' too many times
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Stopping Remote Host Configuration daemon...
May 30 09:20:13 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:20:13 cannot kill process: os: process already finished
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: rhcd.service: Succeeded.
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Stopped Remote Host Configuration daemon.
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Started Remote Host Configuration daemon.
Jun 01 06:10:23 rceeitaprhs001.realpage.com rhcd[3898246]: [rhcd] 2023/06/01 06:10:23 connection lost unexpectedly: read tcp 10.22.152.68:36548->23.47.52.17:443: read: connection reset>
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Stopping Remote Host Configuration daemon...
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: rhcd.service: Succeeded.
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Stopped Remote Host Configuration daemon.
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Started Remote Host Configuration daemon.
Jun 06 00:15:17 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/06 00:15:17 connection lost unexpectedly: read tcp 10.22.152.68:34254->23.47.52.17:443: read: connection reset>
Jun 06 18:49:57 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/06 18:49:57 connection lost unexpectedly: read tcp 10.22.152.68:39208->23.47.52.19:443: read: connection reset>
Jun 07 00:10:33 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/07 00:10:33 connection lost unexpectedly: read tcp 10.22.152.68:44070->23.64.115.140:443: read: connection res>
Jun 13 07:34:42 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/13 07:34:42 connection lost unexpectedly: read tcp 10.22.152.68:33616->23.64.115.139:443: read: connection res>
Jun 13 11:37:19 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/13 11:37:19 connection lost unexpectedly: read tcp 10.22.152.68:41612->23.64.115.140:443: read: connection res>
Jun 13 18:13:29 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/13 18:13:29 connection lost unexpectedly: read tcp 10.22.152.68:46586->23.64.115.154:443: read: connection res>
Jun 16 07:13:25 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/16 07:13:25 connection lost unexpectedly: read tcp 10.22.152.68:33952->23.47.52.17:443: read: connection reset>
Jun 16 11:24:33 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/16 11:24:33 connection lost unexpectedly: read tcp 10.22.152.68:50014->23.47.52.26:443: read: connection reset>

Note You need to log in before you can comment on or make changes to this bug.