Bug 429329

Summary: IPMI hardware steals ports; bind to these ports so that they are not used by accident
Product: Red Hat Enterprise Linux 5 Reporter: Jeff Moyer <jmoyer>
Component: OpenIPMIAssignee: Jan Safranek <jsafrane>
Status: CLOSED ERRATA QA Contact:
Severity: low Docs Contact:
Priority: low    
Version: 5.1CC: agospoda, dkovalsk, ebrown, jlayton, jmoyer, kvolny, rlerch, rvokal
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
some IPMI-enabled hardware makes use of UDP ports 623 (ASF Remote Management and Control Protocol) and 664 (ASF Secure Remote Management and Control Protocol), which corrupts other traffic on these ports, causing symptoms such as autofs mounts hanging. The OpenIPMI package provides a configuration file for xinetd that prevents other services from using these ports, so that they do not interfere with IPMI. On affected systems, the fix has to be enabled manually by setting "disabled = no" for the appropriate port(s) in /etc/xinetd.d/rmcp and (re)starting the xinetd service. (BZ#429329)
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-02 09:59:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 513501    

Description Jeff Moyer 2008-01-18 18:26:12 UTC
+++ This bug was initially created as a clone of Bug #429327 +++

Description of problem:

Check out this posting on lkml: 
  http://lkml.org/lkml/2006/10/10/239

I ran into problems with my autofs test due to the fact that we set
the min resv port down to 500 in order to get 500 mounts in rapid
succession.  On systems with IPMI enabled, we end up with a hung mount
process.  It sends SYN packets to the server, who then responds with
ACK, but the client never sees the ACK (I guess IPMI eats that).

After some discussion with Jeff Layton and Andy Gospodarek, we concluded that
the best way forward would be to add support to the userspace IPMI daemon to
bind to these ports when it knows they are in use.  To date, we have seen ports
663 and 664 used by the IPMI hardware/firmware.

Version-Release number of selected component (if applicable):
I have seen this on the released packages for RHEL 4 Update 5 and RHEL 4 Update 6.

How reproducible:
100% if you have IPMI-enabled hardware.

Steps to Reproduce:
Run the /CoreOS/autofs/bugzillas test suite, subtest 128966 on hardware with
IPMI enabled.
  
Actual results:
One mount process will be hung indefinitely in the D state.

Expected results:
The test should finish mounting and unmount 800 nfs file systems.

Additional info:
Andy got a program from intel that does the bind.  I'll attach it here for
reference.

-- Additional comment from jmoyer on 2008-01-18 13:25 EST --
Created an attachment (id=292182)
code to bind to an ipmi port

Comment 1 Jan Safranek 2009-03-05 09:45:59 UTC
wouldn't it be better to configure e.g. xinetd to "listen" on the port? I found it stupid to create & maintain new daemon, even if it is such small thing.

Comment 2 Andy Gospodarek 2009-03-05 14:09:09 UTC
Can xinetd reserve ports without having a real binary to run and handle requests on that port?

Comment 4 RHEL Program Management 2009-03-16 09:45:07 UTC
Development Management has reviewed and declined this request.  You may appeal
this decision by reopening this request.

Comment 6 Jan Safranek 2009-03-20 13:20:51 UTC
OpenIPMI now ships xinetd service, which, when enabled, binds to UDP port 623 and does not allow portmap or any other service to bind to the port. The service does not do anything useful, it just silently consumes all incoming UDP packets.

As consequence, OpenIPMI.rpm now depends on xinetd.

I thought about shipping the service file as /usr/share/doc/OpenIPMI/rmcp.xinetd, which would require user to manually install xinetd and copy the service configuration to /etc/xinetd/ when he/she is interested in the service. This solution would relax the dependency on xinetd, on the other hand I consider the manual copying less user friendly and it makes future update of the config file difficult.

Comment 9 Jeff Layton 2009-03-20 13:48:04 UTC
What about ports 663 and 664? It seems like those were the ports that I've usually seen hardware silently steal. Should they be added to this config too?

Comment 11 Jeff Moyer 2009-03-20 13:50:48 UTC
(In reply to comment #9)
> What about ports 663 and 664? It seems like those were the ports that I've
> usually seen hardware silently steal. Should they be added to this config too?  

Given that those are the two ports that caused this bug to be filed, I would hope so!

Comment 12 Jan Safranek 2009-03-20 13:56:56 UTC
(In reply to comment #9)
> What about ports 663 and 664? 

664 is ASF Secure Remote Management and Control Protocol, I agree it should be blocked and I'll add it to the service.

But what is port 663? IPMI specification does not reference this port and IANA says "PureNoise"...

Comment 17 Karel Volný 2009-07-27 15:04:39 UTC
Could we add a release note stating that this "fix" has to be enabled by hand, i.e. by changing the "disable = yes" line within the /etc/xinet.d/rmcp file?

Comment 19 Karel Volný 2009-07-27 15:30:25 UTC
Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

New Contents:
Some IPMI-enabled hardware makes use of UDP ports 623 (ASF Remote Management and Control Protocol) and 664 (ASF Secure Remote Management and Control Protocol) which corrupts other traffic on these ports, causing symptomes like autofs mounts hanging. The OpenIPMI package provides a configuration file for xinetd that prevents other services from using these ports, so that they do not interfere with IPMI. On affected systems, the fix has to be enabled manually by setting "disabled = no" for the appropriate port(s) in /etc/xinetd.d/rmcp and (re)starting xinetd service.

Comment 22 Ryan Lerch 2009-08-18 21:36:50 UTC
Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1 +1,9 @@
-Some IPMI-enabled hardware makes use of UDP ports 623 (ASF Remote Management and Control Protocol) and 664 (ASF Secure Remote Management and Control Protocol) which corrupts other traffic on these ports, causing symptomes like autofs mounts hanging. The OpenIPMI package provides a configuration file for xinetd that prevents other services from using these ports, so that they do not interfere with IPMI. On affected systems, the fix has to be enabled manually by setting "disabled = no" for the appropriate port(s) in /etc/xinetd.d/rmcp and (re)starting xinetd service.+some IPMI-enabled hardware makes use of UDP ports 623 (ASF Remote
+Management and Control Protocol) and 664 (ASF Secure Remote Management and
+Control Protocol), which corrupts other traffic on these ports, causing
+symptoms such as autofs mounts hanging. The OpenIPMI package provides a
+configuration file for xinetd that prevents other services from using these
+ports, so that they do not interfere with IPMI. On affected systems, the
+fix has to be enabled manually by setting "disabled = no" for the
+appropriate port(s) in /etc/xinetd.d/rmcp and (re)starting the xinetd
+service. (BZ#429329)

Comment 23 errata-xmlrpc 2009-09-02 09:59:40 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-1312.html

Comment 24 Ed Brown 2009-09-09 22:10:29 UTC
It was a heavy-handed fix to require xinetd to be installed.  Of course when xinetd installs, it defaults to running.  Basic configuration guidelines 101: turn off and uninstall daemons you don't need, but now that would break this pseudo-dependency.  And it IS a false dependency: OpenIPMI does NOT require xinetd.  But not installing xinetd is no longer an option, and unfortunately it's not even so simple as to disable xinetd globally, because we do use it some places.  So this will now require completely new workarounds in our configuration management settings to differentiate and handle the different cases, with the risk of mistakes and disabling xinetd where it is needed.

Manual intervention is already required to enable this fix, why would it have been too much to both edit a file AND copy a file?  Couldn't it have been accomplished with rpm triggers?  And when will RedHat GET that such a fix as this is security significant, unacceptable to your government customers and probably most corporate customers with a clue?  (Sorry, but my aggravation is high because this isn't the first time that updates have resulted in new and unrelated daemons installed and running.)

Comment 25 Ed Brown 2009-09-09 22:34:20 UTC
Actually, it was just simply a mistake to do this.  Other packages put files into /etc/xinetd.d/ without requiring xinetd itself.  /etc/xinetd.d/ is owned by the filesystem package.