Bug 429327

Summary: IPMI hardware steals ports; bind to these ports so that they are not used by accident
Product: Red Hat Enterprise Linux 4 Reporter: Jeff Moyer <jmoyer>
Component: OpenIPMIAssignee: Jan Safranek <jsafrane>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: low Docs Contact:
Priority: low    
Version: 4.6CC: agospoda, jlayton, pknirsch
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-02-11 14:10:51 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
code to bind to an ipmi port none

Description Jeff Moyer 2008-01-18 18:25:10 UTC
Description of problem:

Check out this posting on lkml: 
  http://lkml.org/lkml/2006/10/10/239

I ran into problems with my autofs test due to the fact that we set
the min resv port down to 500 in order to get 500 mounts in rapid
succession.  On systems with IPMI enabled, we end up with a hung mount
process.  It sends SYN packets to the server, who then responds with
ACK, but the client never sees the ACK (I guess IPMI eats that).

After some discussion with Jeff Layton and Andy Gospodarek, we concluded that
the best way forward would be to add support to the userspace IPMI daemon to
bind to these ports when it knows they are in use.  To date, we have seen ports
663 and 664 used by the IPMI hardware/firmware.

Version-Release number of selected component (if applicable):
I have seen this on the released packages for RHEL 4 Update 5 and RHEL 4 Update 6.

How reproducible:
100% if you have IPMI-enabled hardware.

Steps to Reproduce:
Run the /CoreOS/autofs/bugzillas test suite, subtest 128966 on hardware with
IPMI enabled.
  
Actual results:
One mount process will be hung indefinitely in the D state.

Expected results:
The test should finish mounting and unmount 800 nfs file systems.

Additional info:
Andy got a program from intel that does the bind.  I'll attach it here for
reference.

Comment 1 Jeff Moyer 2008-01-18 18:25:11 UTC
Created attachment 292182 [details]
code to bind to an ipmi port

Comment 2 Jeff Moyer 2008-02-05 19:23:58 UTC
Phil,

As mentioned, this problem affects our internal test environment.  It would be
of great help to us if this problem was addressed.  It's been suggested to me
that the IPMI hardware should not be configured to use the same IP address as
the host.  That makes a lot of sense to me, but it appears that there are at
least some installations that do not heed that advice.  Can you comment on what
you feel is an appropriate fix for this problem?

Thanks.

Comment 3 Phil Knirsch 2008-04-18 13:58:30 UTC
Thats a tricky problem.

Of course the right choice is to have different IP addresses, but as you
mentioned in the real world this sometimes doesn't happen.

A possible solution that we're looking into for Fedora 10 at the moment is a
daemon called portreserve. With this services can at bootup time reserve
specific ports that won't be randomly allocated by other apps or services and
can actively be "claimed" later when the system is running and the specific
service needs the port to operate.

This could be used to basically reserve those 2 ports automatically during bootup.

The chances though that this change will make it into RHEL-4 is highly unlikely
as it quite drastically changes bootup behavior. Possibly for RHEL-5 but the
main aim is to have it for RHEL-6.

Hope this clears it up a bit.

Read ya, Phil

Comment 4 Andy Gospodarek 2008-04-18 14:28:39 UTC
I think portreserve is a decent solution (as long as someone takes the 5 minutes
to extend it to support UDP as well since IPMI uses UDP).

I don't think this would be a terrible idea in RHEL either.  Once in rawhide
maybe EPEL would be a good place to try it out?

Comment 5 Phil Knirsch 2008-04-18 15:03:02 UTC
That sounds definitely reasonable. As soon as we have something ready i'll
update the bugzilla here.

Read ya, Phil

Comment 6 Phil Knirsch 2008-08-18 16:53:55 UTC
We're now shipping portreserve in Fedora 10, so please take a look at that and report if that would fix your problems.

Thanks,

Read ya, Phil

Comment 7 Andy Gospodarek 2008-08-18 17:15:21 UTC
Does portreserve now contain support for udp ports as well?  If it's still tcp only that isn't helpful for this problem.

Comment 8 Phil Knirsch 2008-08-18 17:18:40 UTC
I'll ping the engineer that is taking care of it.

Read ya, Phil

Comment 9 Phil Knirsch 2008-08-25 15:46:23 UTC
Ok, portreserve in the current form in Fedora supports TCP and UDP ports, so you should all be set.

If you have any more detailed questions about portreserve please send an email to Tim Waugh <twaugh>, the maintainer of the package.

Thanks,

Read ya, Phil

Comment 10 Jeff Moyer 2009-02-10 16:33:22 UTC
I think this has been sufficiently addressed.  I have a program that does this reservation for my test infrastructure.  I think you can close this as nextrelease or whatever seems appropriate.

Thanks!

Comment 11 Phil Knirsch 2009-02-11 14:10:51 UTC
Alright, closing this as nextrelease then.

Thanks Jeffrey.