Bug 2183533

Summary: rhcd disconnects from console dot. Satellite source displays status of Unavailable.
Product: Red Hat Satellite Reporter: Matthew Yee <myee>
Component: RH Cloud - Cloud ConnectorAssignee: satellite6-bugs <satellite6-bugs>
Status: NEW --- QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: medium Docs Contact:
Priority: high    
Version: UnspecifiedCC: achadha, ahumbe, aruzicka, cmarinea, dajohnso, dhorton, ehelms, gmccullo, jaudet, jlindgre, jpathan, ktordeur, link, mark.milano, rexwhite, rlavi, saydas, shivagup, sshtein
Target Milestone: UnspecifiedKeywords: PrioBumpGSS, Triaged
Target Release: UnusedFlags: dajohnso: needinfo? (dhorton)
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Matthew Yee 2023-03-31 14:22:26 UTC 
Description of problem:
This is an intermittent problem where rhcd on the Satelltie server will disconnect. "systemctl status rhcd.service" shows the service is running but there is an additional message that describes the daemon as disconnected. In console dot, the Satellite server will display a status in "Sources" as unavailable. The fix is relatively easy, ranging from re-running cloud connector setup to restarting rhcd. 

Additionally, it is impossible to apply remediation playbooks from console dot. 

Version-Release number of selected component (if applicable):


How reproducible:
Intermittent.

Steps to Reproduce:
1. Check the status of the rhcd service. If it is not running or it is running and showing a message that it is disconnected, restart the service. 
2.
3.

Actual results:


Expected results:
rhcd should not disconnect.

Additional info:
Comment 1 Brad Buckingham 2023-04-03 13:05:19 UTC 
Hi Shim,

Would this be an issue within Satellite, Insights or other?

Thanks!
Comment 3 dhorton 2023-04-10 14:22:51 UTC 
Please gather the rhc logs and the rhc client-id when this happens next time.

The rhc client-id comes from the cert:
sudo openssl x509 -in /etc/pki/consumer/cert.pem -noout -text | grep Subject

You should be able to get the rhc logs with this command:
journalctl -u rhcd
Comment 7 Link Dupont 2023-04-11 15:05:19 UTC 
What is the output of `rpm -q rhc` on the host this is occurring?
Comment 23 Mark Milano 2023-06-08 13:12:57 UTC 
(In reply to dhorton from comment #3)
> Please gather the rhc logs and the rhc client-id when this happens next time.
> 
> The rhc client-id comes from the cert:
> sudo openssl x509 -in /etc/pki/consumer/cert.pem -noout -text | grep Subject
> 
> You should be able to get the rhc logs with this command:
> journalctl -u rhcd

# rpm -q rhc
rhc-0.2.2-1.el8.x86_64

# openssl x509 -in /etc/pki/consumer/cert.pem -noout -text | grep Subject
        Subject: O = 4425915, CN = 1e6b516d-70ff-40c1-a97a-134c119f3ca6
        Subject Public Key Info:
            X509v3 Subject Key Identifier:
            X509v3 Subject Alternative Name:


-- Logs begin at Fri 2023-05-12 02:45:01 CDT, end at Thu 2023-06-08 08:08:12 CDT. --
[rhcd] 2023/05/14 00:42:42 connection lost unexpectedly: pingresp not received, disconnecting
[rhcd] 2023/05/19 19:51:25 connection lost unexpectedly: read tcp 10.22.152.68:38526->104.117.244.34:443: read: connection reset by peer
[rhcd] 2023/05/20 09:52:36 connection lost unexpectedly: read tcp 10.22.152.68:43842->104.117.244.24:443: read: connection reset by peer
[rhcd] 2023/05/22 00:39:52 connection lost unexpectedly: pingresp not received, disconnecting
[rhcd] 2023/05/29 17:09:43 connection lost unexpectedly: pingresp not received, disconnecting
[rhcd] 2023/05/29 22:27:04 connection lost unexpectedly: read tcp 10.22.152.68:54490->23.222.28.17:443: read: connection reset by peer
[rhcd] 2023/05/30 09:01:56 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:01:56 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:01:56 handler registration failed: <nil>
[rhcd] 2023/05/30 09:02:01 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:02:01 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:01 handler registration failed: <nil>
[rhcd] 2023/05/30 09:02:11 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:02:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:11 handler registration failed: <nil>
[rhcd] 2023/05/30 09:02:26 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:02:26 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:26 handler registration failed: <nil>
[rhcd] 2023/05/30 09:02:46 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:02:46 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:46 handler registration failed: <nil>
[rhcd] 2023/05/30 09:03:11 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:03:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:03:11 handler registration failed: <nil>
[rhcd] 2023/05/30 09:03:11 failed to start worker '/usr/libexec/rhc/rhc-package-manager-worker' too many times
Stopping Remote Host Configuration daemon...
[rhcd] 2023/05/30 09:20:13 cannot kill process: os: process already finished
rhcd.service: Succeeded.
Stopped Remote Host Configuration daemon.
Started Remote Host Configuration daemon.
[rhcd] 2023/06/01 06:10:23 connection lost unexpectedly: read tcp 10.22.152.68:36548->23.47.52.17:443: read: connection reset by peer
Stopping Remote Host Configuration daemon...
rhcd.service: Succeeded.
Stopped Remote Host Configuration daemon.
Started Remote Host Configuration daemon.
[rhcd] 2023/06/06 00:15:17 connection lost unexpectedly: read tcp 10.22.152.68:34254->23.47.52.17:443: read: connection reset by peer
[rhcd] 2023/06/06 18:49:57 connection lost unexpectedly: read tcp 10.22.152.68:39208->23.47.52.19:443: read: connection reset by peer
[rhcd] 2023/06/07 00:10:33 connection lost unexpectedly: read tcp 10.22.152.68:44070->23.64.115.140:443: read: connection reset by peer
Comment 27 Mark Milano 2023-06-13 13:13:29 UTC 
Latest logs

-- Logs begin at Tue 2023-05-16 16:40:11 CDT, end at Tue 2023-06-13 08:08:15 CDT. --
May 19 19:51:25 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/19 19:51:25 connection lost unexpectedly: read tcp 10.22.152.68:38526->104.117.244.34:443: read: connection reset by peer
May 20 09:52:36 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/20 09:52:36 connection lost unexpectedly: read tcp 10.22.152.68:43842->104.117.244.24:443: read: connection reset by peer
May 22 00:39:52 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/22 00:39:52 connection lost unexpectedly: pingresp not received, disconnecting
May 29 17:09:43 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/29 17:09:43 connection lost unexpectedly: pingresp not received, disconnecting
May 29 22:27:04 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/29 22:27:04 connection lost unexpectedly: read tcp 10.22.152.68:54490->23.222.28.17:443: read: connection reset by peer
May 30 09:01:56 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:01:56 worker failed to register for handler package-manager
May 30 09:01:56 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:01:56 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:01:56 handler registration failed: <nil>
May 30 09:02:01 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:01 worker failed to register for handler package-manager
May 30 09:02:01 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:01 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:01 handler registration failed: <nil>
May 30 09:02:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:11 worker failed to register for handler package-manager
May 30 09:02:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:11 handler registration failed: <nil>
May 30 09:02:26 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:26 worker failed to register for handler package-manager
May 30 09:02:26 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:26 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:26 handler registration failed: <nil>
May 30 09:02:46 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:46 worker failed to register for handler package-manager
May 30 09:02:46 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:46 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:46 handler registration failed: <nil>
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 worker failed to register for handler package-manager
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:03:11 handler registration failed: <nil>
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 failed to start worker '/usr/libexec/rhc/rhc-package-manager-worker' too many times
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Stopping Remote Host Configuration daemon...
May 30 09:20:13 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:20:13 cannot kill process: os: process already finished
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: rhcd.service: Succeeded.
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Stopped Remote Host Configuration daemon.
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Started Remote Host Configuration daemon.
Jun 01 06:10:23 rceeitaprhs001.realpage.com rhcd[3898246]: [rhcd] 2023/06/01 06:10:23 connection lost unexpectedly: read tcp 10.22.152.68:36548->23.47.52.17:443: read: connection reset by peer
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Stopping Remote Host Configuration daemon...
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: rhcd.service: Succeeded.
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Stopped Remote Host Configuration daemon.
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Started Remote Host Configuration daemon.
Jun 06 00:15:17 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/06 00:15:17 connection lost unexpectedly: read tcp 10.22.152.68:34254->23.47.52.17:443: read: connection reset by peer
Jun 06 18:49:57 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/06 18:49:57 connection lost unexpectedly: read tcp 10.22.152.68:39208->23.47.52.19:443: read: connection reset by peer
Jun 07 00:10:33 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/07 00:10:33 connection lost unexpectedly: read tcp 10.22.152.68:44070->23.64.115.140:443: read: connection reset by peer
Jun 13 07:34:42 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/13 07:34:42 connection lost unexpectedly: read tcp 10.22.152.68:33616->23.64.115.139:443: read: connection reset by peer
Comment 29 Mark Milano 2023-06-20 19:12:34 UTC 
-- Logs begin at Wed 2023-05-24 07:17:11 CDT, end at Tue 2023-06-20 14:12:11 CDT. --
May 29 17:09:43 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/29 17:09:43 connection lost unexpectedly: pingresp not received, disconnecting
May 29 22:27:04 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/29 22:27:04 connection lost unexpectedly: read tcp 10.22.152.68:54490->23.222.28.17:443: read: connection rese>
May 30 09:01:56 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:01:56 worker failed to register for handler package-manager
May 30 09:01:56 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:01:56 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:01:56 handler registration failed: <ni>
May 30 09:02:01 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:01 worker failed to register for handler package-manager
May 30 09:02:01 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:01 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:01 handler registration failed: <ni>
May 30 09:02:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:11 worker failed to register for handler package-manager
May 30 09:02:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:11 handler registration failed: <ni>
May 30 09:02:26 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:26 worker failed to register for handler package-manager
May 30 09:02:26 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:26 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:26 handler registration failed: <ni>
May 30 09:02:46 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:46 worker failed to register for handler package-manager
May 30 09:02:46 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:46 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:46 handler registration failed: <ni>
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 worker failed to register for handler package-manager
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:03:11 handler registration failed: <ni>
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 failed to start worker '/usr/libexec/rhc/rhc-package-manager-worker' too many times
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Stopping Remote Host Configuration daemon...
May 30 09:20:13 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:20:13 cannot kill process: os: process already finished
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: rhcd.service: Succeeded.
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Stopped Remote Host Configuration daemon.
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Started Remote Host Configuration daemon.
Jun 01 06:10:23 rceeitaprhs001.realpage.com rhcd[3898246]: [rhcd] 2023/06/01 06:10:23 connection lost unexpectedly: read tcp 10.22.152.68:36548->23.47.52.17:443: read: connection reset>
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Stopping Remote Host Configuration daemon...
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: rhcd.service: Succeeded.
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Stopped Remote Host Configuration daemon.
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Started Remote Host Configuration daemon.
Jun 06 00:15:17 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/06 00:15:17 connection lost unexpectedly: read tcp 10.22.152.68:34254->23.47.52.17:443: read: connection reset>
Jun 06 18:49:57 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/06 18:49:57 connection lost unexpectedly: read tcp 10.22.152.68:39208->23.47.52.19:443: read: connection reset>
Jun 07 00:10:33 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/07 00:10:33 connection lost unexpectedly: read tcp 10.22.152.68:44070->23.64.115.140:443: read: connection res>
Jun 13 07:34:42 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/13 07:34:42 connection lost unexpectedly: read tcp 10.22.152.68:33616->23.64.115.139:443: read: connection res>
Jun 13 11:37:19 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/13 11:37:19 connection lost unexpectedly: read tcp 10.22.152.68:41612->23.64.115.140:443: read: connection res>
Jun 13 18:13:29 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/13 18:13:29 connection lost unexpectedly: read tcp 10.22.152.68:46586->23.64.115.154:443: read: connection res>
Jun 16 07:13:25 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/16 07:13:25 connection lost unexpectedly: read tcp 10.22.152.68:33952->23.47.52.17:443: read: connection reset>
Jun 16 11:24:33 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/16 11:24:33 connection lost unexpectedly: read tcp 10.22.152.68:50014->23.47.52.26:443: read: connection reset>