Bug 1485297

Summary: REX creates zombie ssh processes when ProxyCommand is used for a tunnel (Satellite 6.2)
Product: Red Hat Satellite Reporter: Lukas Zapletal <lzap>
Component: Remote ExecutionAssignee: Adam Ruzicka <aruzicka>
Status: CLOSED CURRENTRELEASE QA Contact: Lukas Pramuk <lpramuk>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.2.10CC: aruzicka, bbuckingham, bkearney, carlospec, cduryee, cwelton, ehelms, inecas, ktordeur, lpramuk, lzap, mmccune, oshtaier, pmoravec, sthirugn
Target Milestone: UnspecifiedKeywords: FieldEngineering, PrioBumpField, Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard: hotfix, hotfix_delivered
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1461088 Environment:
Last Closed: 2018-03-08 22:19:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1461088    
Bug Blocks: 1465787    

Comment 2 Satellite Program 2017-08-25 12:03:17 UTC
Upstream bug assigned to aruzicka

Comment 3 Satellite Program 2017-08-25 12:03:21 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/20018 has been resolved.

Comment 7 Lukas Zapletal 2017-08-28 12:25:28 UTC
UPDATED HOTFIX INSTRUCTIONS FOR SATELLITE 6.2 AND CAPSULE:

On Satellite Server:

1) Upgrade package net-ssh:

yum upgrade http://people.redhat.com/~lzapleta/hotfix/ssh-zombie-1485297/sat-6.2/tfm-rubygem-net-ssh-4.0.1-2.el7sat.noarch.rpm http://people.redhat.com/~lzapleta/hotfix/ssh-zombie-1485297/sat-6.2/tfm-rubygem-net-ssh-doc-4.0.1-2.el7sat.noarch.rpm

2) Check for all running tasks via Monitor - Tasks, there should be only two running tasks left. Wait if there is long queue (e.g. regular syncs or promotions).

3) Restart foreman-task and foreman-proxy service

systemctl restart foreman-tasks foreman-proxy

4) Check tasks in Monitor - Tasks. Kill zombie processes if necessary

On each Satellite Capsule with ReX capability enabled:

Perform steps (1), (2)

3) Restart proxy:

systemctl restart foreman-proxy

AFTER UPGRADE

Perform Remote Execution harmless test on multiple hosts.

ROLLBACK INSTRUCTIONS

Downgrade tfm-rubygem-net-ssh and tfm-rubygem-net-ssh-doc to previous version and restart services.

NOTES

There is also a hotfix for non-scl package which I believe is not used, but publishing the build as well just for sure:

http://people.redhat.com/~lzapleta/hotfix/ssh-zombie-1485297/sat-6.2/non-scl/

Comment 8 Lukas Zapletal 2017-08-30 11:19:48 UTC
UPDATE:

Only upgrade tfm-rubygem-net-ssh package the tfm-rubygem-net-ssh-doc subpackage is not needed when not installed. If you attempt to install the -doc subpackage be prepared to pull some extra dependencies from rhel-server-rhscl-7-rpms repo.

Comment 20 Mike McCune 2018-03-08 22:19:51 UTC
This is resolved in Satellite 6.3. If you have upgraded and are still
seeing this issue, please feel free to re-open and provide additional
information about the issue.