Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2183533

Summary: rhcd disconnects from console dot. Satellite source displays status of Unavailable.
Product: Red Hat Satellite Reporter: Matthew Yee <myee>
Component: RH CloudAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED MIGRATED QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: medium Docs Contact:
Priority: high    
Version: UnspecifiedCC: achadha, ahumbe, amiagarw, aruzicka, benoit.martin-poitras, cmarinea, dajohnso, dhorton, ehelms, gmccullo, jaudet, jlindgre, jpathan, ktordeur, link, mark.milano, rexwhite, rlavi, saydas, shivagup, sshtein
Target Milestone: UnspecifiedKeywords: MigratedToJIRA, PrioBumpGSS, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-06-06 16:12:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Matthew Yee 2023-03-31 14:22:26 UTC 
Description of problem:
This is an intermittent problem where rhcd on the Satelltie server will disconnect. "systemctl status rhcd.service" shows the service is running but there is an additional message that describes the daemon as disconnected. In console dot, the Satellite server will display a status in "Sources" as unavailable. The fix is relatively easy, ranging from re-running cloud connector setup to restarting rhcd. 

Additionally, it is impossible to apply remediation playbooks from console dot. 

Version-Release number of selected component (if applicable):


How reproducible:
Intermittent.

Steps to Reproduce:
1. Check the status of the rhcd service. If it is not running or it is running and showing a message that it is disconnected, restart the service. 
2.
3.

Actual results:


Expected results:
rhcd should not disconnect.

Additional info:
Comment 1 Brad Buckingham 2023-04-03 13:05:19 UTC 
Hi Shim,

Would this be an issue within Satellite, Insights or other?

Thanks!
Comment 3 dhorton 2023-04-10 14:22:51 UTC 
Please gather the rhc logs and the rhc client-id when this happens next time.

The rhc client-id comes from the cert:
sudo openssl x509 -in /etc/pki/consumer/cert.pem -noout -text | grep Subject

You should be able to get the rhc logs with this command:
journalctl -u rhcd
Comment 7 Link Dupont 2023-04-11 15:05:19 UTC 
What is the output of `rpm -q rhc` on the host this is occurring?
Comment 23 Mark Milano 2023-06-08 13:12:57 UTC 
(In reply to dhorton from comment #3)
> Please gather the rhc logs and the rhc client-id when this happens next time.
> 
> The rhc client-id comes from the cert:
> sudo openssl x509 -in /etc/pki/consumer/cert.pem -noout -text | grep Subject
> 
> You should be able to get the rhc logs with this command:
> journalctl -u rhcd

# rpm -q rhc
rhc-0.2.2-1.el8.x86_64

# openssl x509 -in /etc/pki/consumer/cert.pem -noout -text | grep Subject
        Subject: O = 4425915, CN = 1e6b516d-70ff-40c1-a97a-134c119f3ca6
        Subject Public Key Info:
            X509v3 Subject Key Identifier:
            X509v3 Subject Alternative Name:


-- Logs begin at Fri 2023-05-12 02:45:01 CDT, end at Thu 2023-06-08 08:08:12 CDT. --
[rhcd] 2023/05/14 00:42:42 connection lost unexpectedly: pingresp not received, disconnecting
[rhcd] 2023/05/19 19:51:25 connection lost unexpectedly: read tcp 10.22.152.68:38526->104.117.244.34:443: read: connection reset by peer
[rhcd] 2023/05/20 09:52:36 connection lost unexpectedly: read tcp 10.22.152.68:43842->104.117.244.24:443: read: connection reset by peer
[rhcd] 2023/05/22 00:39:52 connection lost unexpectedly: pingresp not received, disconnecting
[rhcd] 2023/05/29 17:09:43 connection lost unexpectedly: pingresp not received, disconnecting
[rhcd] 2023/05/29 22:27:04 connection lost unexpectedly: read tcp 10.22.152.68:54490->23.222.28.17:443: read: connection reset by peer
[rhcd] 2023/05/30 09:01:56 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:01:56 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:01:56 handler registration failed: <nil>
[rhcd] 2023/05/30 09:02:01 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:02:01 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:01 handler registration failed: <nil>
[rhcd] 2023/05/30 09:02:11 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:02:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:11 handler registration failed: <nil>
[rhcd] 2023/05/30 09:02:26 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:02:26 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:26 handler registration failed: <nil>
[rhcd] 2023/05/30 09:02:46 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:02:46 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:46 handler registration failed: <nil>
[rhcd] 2023/05/30 09:03:11 worker failed to register for handler package-manager
[rhcd] 2023/05/30 09:03:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:03:11 handler registration failed: <nil>
[rhcd] 2023/05/30 09:03:11 failed to start worker '/usr/libexec/rhc/rhc-package-manager-worker' too many times
Stopping Remote Host Configuration daemon...
[rhcd] 2023/05/30 09:20:13 cannot kill process: os: process already finished
rhcd.service: Succeeded.
Stopped Remote Host Configuration daemon.
Started Remote Host Configuration daemon.
[rhcd] 2023/06/01 06:10:23 connection lost unexpectedly: read tcp 10.22.152.68:36548->23.47.52.17:443: read: connection reset by peer
Stopping Remote Host Configuration daemon...
rhcd.service: Succeeded.
Stopped Remote Host Configuration daemon.
Started Remote Host Configuration daemon.
[rhcd] 2023/06/06 00:15:17 connection lost unexpectedly: read tcp 10.22.152.68:34254->23.47.52.17:443: read: connection reset by peer
[rhcd] 2023/06/06 18:49:57 connection lost unexpectedly: read tcp 10.22.152.68:39208->23.47.52.19:443: read: connection reset by peer
[rhcd] 2023/06/07 00:10:33 connection lost unexpectedly: read tcp 10.22.152.68:44070->23.64.115.140:443: read: connection reset by peer
Comment 27 Mark Milano 2023-06-13 13:13:29 UTC 
Latest logs

-- Logs begin at Tue 2023-05-16 16:40:11 CDT, end at Tue 2023-06-13 08:08:15 CDT. --
May 19 19:51:25 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/19 19:51:25 connection lost unexpectedly: read tcp 10.22.152.68:38526->104.117.244.34:443: read: connection reset by peer
May 20 09:52:36 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/20 09:52:36 connection lost unexpectedly: read tcp 10.22.152.68:43842->104.117.244.24:443: read: connection reset by peer
May 22 00:39:52 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/22 00:39:52 connection lost unexpectedly: pingresp not received, disconnecting
May 29 17:09:43 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/29 17:09:43 connection lost unexpectedly: pingresp not received, disconnecting
May 29 22:27:04 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/29 22:27:04 connection lost unexpectedly: read tcp 10.22.152.68:54490->23.222.28.17:443: read: connection reset by peer
May 30 09:01:56 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:01:56 worker failed to register for handler package-manager
May 30 09:01:56 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:01:56 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:01:56 handler registration failed: <nil>
May 30 09:02:01 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:01 worker failed to register for handler package-manager
May 30 09:02:01 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:01 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:01 handler registration failed: <nil>
May 30 09:02:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:11 worker failed to register for handler package-manager
May 30 09:02:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:11 handler registration failed: <nil>
May 30 09:02:26 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:26 worker failed to register for handler package-manager
May 30 09:02:26 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:26 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:26 handler registration failed: <nil>
May 30 09:02:46 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:46 worker failed to register for handler package-manager
May 30 09:02:46 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:46 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:46 handler registration failed: <nil>
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 worker failed to register for handler package-manager
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:03:11 handler registration failed: <nil>
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 failed to start worker '/usr/libexec/rhc/rhc-package-manager-worker' too many times
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Stopping Remote Host Configuration daemon...
May 30 09:20:13 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:20:13 cannot kill process: os: process already finished
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: rhcd.service: Succeeded.
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Stopped Remote Host Configuration daemon.
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Started Remote Host Configuration daemon.
Jun 01 06:10:23 rceeitaprhs001.realpage.com rhcd[3898246]: [rhcd] 2023/06/01 06:10:23 connection lost unexpectedly: read tcp 10.22.152.68:36548->23.47.52.17:443: read: connection reset by peer
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Stopping Remote Host Configuration daemon...
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: rhcd.service: Succeeded.
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Stopped Remote Host Configuration daemon.
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Started Remote Host Configuration daemon.
Jun 06 00:15:17 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/06 00:15:17 connection lost unexpectedly: read tcp 10.22.152.68:34254->23.47.52.17:443: read: connection reset by peer
Jun 06 18:49:57 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/06 18:49:57 connection lost unexpectedly: read tcp 10.22.152.68:39208->23.47.52.19:443: read: connection reset by peer
Jun 07 00:10:33 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/07 00:10:33 connection lost unexpectedly: read tcp 10.22.152.68:44070->23.64.115.140:443: read: connection reset by peer
Jun 13 07:34:42 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/13 07:34:42 connection lost unexpectedly: read tcp 10.22.152.68:33616->23.64.115.139:443: read: connection reset by peer
Comment 29 Mark Milano 2023-06-20 19:12:34 UTC 
-- Logs begin at Wed 2023-05-24 07:17:11 CDT, end at Tue 2023-06-20 14:12:11 CDT. --
May 29 17:09:43 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/29 17:09:43 connection lost unexpectedly: pingresp not received, disconnecting
May 29 22:27:04 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/29 22:27:04 connection lost unexpectedly: read tcp 10.22.152.68:54490->23.222.28.17:443: read: connection rese>
May 30 09:01:56 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:01:56 worker failed to register for handler package-manager
May 30 09:01:56 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:01:56 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:01:56 handler registration failed: <ni>
May 30 09:02:01 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:01 worker failed to register for handler package-manager
May 30 09:02:01 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:01 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:01 handler registration failed: <ni>
May 30 09:02:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:11 worker failed to register for handler package-manager
May 30 09:02:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:11 handler registration failed: <ni>
May 30 09:02:26 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:26 worker failed to register for handler package-manager
May 30 09:02:26 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:26 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:26 handler registration failed: <ni>
May 30 09:02:46 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:46 worker failed to register for handler package-manager
May 30 09:02:46 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:02:46 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:02:46 handler registration failed: <ni>
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 worker failed to register for handler package-manager
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 [/usr/libexec/rhc/rhc-package-manager-worker] 2023/05/30 09:03:11 handler registration failed: <ni>
May 30 09:03:11 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:03:11 failed to start worker '/usr/libexec/rhc/rhc-package-manager-worker' too many times
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Stopping Remote Host Configuration daemon...
May 30 09:20:13 rceeitaprhs001.realpage.com rhcd[1568616]: [rhcd] 2023/05/30 09:20:13 cannot kill process: os: process already finished
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: rhcd.service: Succeeded.
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Stopped Remote Host Configuration daemon.
May 30 09:20:13 rceeitaprhs001.realpage.com systemd[1]: Started Remote Host Configuration daemon.
Jun 01 06:10:23 rceeitaprhs001.realpage.com rhcd[3898246]: [rhcd] 2023/06/01 06:10:23 connection lost unexpectedly: read tcp 10.22.152.68:36548->23.47.52.17:443: read: connection reset>
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Stopping Remote Host Configuration daemon...
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: rhcd.service: Succeeded.
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Stopped Remote Host Configuration daemon.
Jun 05 08:51:16 rceeitaprhs001.realpage.com systemd[1]: Started Remote Host Configuration daemon.
Jun 06 00:15:17 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/06 00:15:17 connection lost unexpectedly: read tcp 10.22.152.68:34254->23.47.52.17:443: read: connection reset>
Jun 06 18:49:57 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/06 18:49:57 connection lost unexpectedly: read tcp 10.22.152.68:39208->23.47.52.19:443: read: connection reset>
Jun 07 00:10:33 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/07 00:10:33 connection lost unexpectedly: read tcp 10.22.152.68:44070->23.64.115.140:443: read: connection res>
Jun 13 07:34:42 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/13 07:34:42 connection lost unexpectedly: read tcp 10.22.152.68:33616->23.64.115.139:443: read: connection res>
Jun 13 11:37:19 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/13 11:37:19 connection lost unexpectedly: read tcp 10.22.152.68:41612->23.64.115.140:443: read: connection res>
Jun 13 18:13:29 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/13 18:13:29 connection lost unexpectedly: read tcp 10.22.152.68:46586->23.64.115.154:443: read: connection res>
Jun 16 07:13:25 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/16 07:13:25 connection lost unexpectedly: read tcp 10.22.152.68:33952->23.47.52.17:443: read: connection reset>
Jun 16 11:24:33 rceeitaprhs001.realpage.com rhcd[1825159]: [rhcd] 2023/06/16 11:24:33 connection lost unexpectedly: read tcp 10.22.152.68:50014->23.47.52.26:443: read: connection reset>
Comment 33 bmartinp@gmail.com 2023-10-12 13:34:45 UTC 
Do you think that if in my company we have a firewall that analyzes SSL connections and this firewall closes dormant connections after 1 hour, changing a kernel parameter to ensure that connections keep alive more quickly could solve the problem? I have the same issue and applied this small patch, and I'll check if I still have disconnection problems. I'd still like to get your opinion on the subject. 

net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_time = 600




Thanks!
Comment 36 Amit Agarwal 2024-02-13 14:18:31 UTC 
Customer is looking for an update, Can we have a update on this for a permanent fix as this issue is pending since long.
Comment 38 Eric Helms 2024-06-06 16:12:01 UTC 
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "SAT-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.
Comment 39 Red Hat Bugzilla 2024-10-05 04:25:57 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days