Red Hat Bugzilla – Bug 429327
IPMI hardware steals ports; bind to these ports so that they are not used by accident
Last modified: 2009-02-11 09:10:51 EST
Description of problem: Check out this posting on lkml: http://lkml.org/lkml/2006/10/10/239 I ran into problems with my autofs test due to the fact that we set the min resv port down to 500 in order to get 500 mounts in rapid succession. On systems with IPMI enabled, we end up with a hung mount process. It sends SYN packets to the server, who then responds with ACK, but the client never sees the ACK (I guess IPMI eats that). After some discussion with Jeff Layton and Andy Gospodarek, we concluded that the best way forward would be to add support to the userspace IPMI daemon to bind to these ports when it knows they are in use. To date, we have seen ports 663 and 664 used by the IPMI hardware/firmware. Version-Release number of selected component (if applicable): I have seen this on the released packages for RHEL 4 Update 5 and RHEL 4 Update 6. How reproducible: 100% if you have IPMI-enabled hardware. Steps to Reproduce: Run the /CoreOS/autofs/bugzillas test suite, subtest 128966 on hardware with IPMI enabled. Actual results: One mount process will be hung indefinitely in the D state. Expected results: The test should finish mounting and unmount 800 nfs file systems. Additional info: Andy got a program from intel that does the bind. I'll attach it here for reference.
Created attachment 292182 [details] code to bind to an ipmi port
Phil, As mentioned, this problem affects our internal test environment. It would be of great help to us if this problem was addressed. It's been suggested to me that the IPMI hardware should not be configured to use the same IP address as the host. That makes a lot of sense to me, but it appears that there are at least some installations that do not heed that advice. Can you comment on what you feel is an appropriate fix for this problem? Thanks.
Thats a tricky problem. Of course the right choice is to have different IP addresses, but as you mentioned in the real world this sometimes doesn't happen. A possible solution that we're looking into for Fedora 10 at the moment is a daemon called portreserve. With this services can at bootup time reserve specific ports that won't be randomly allocated by other apps or services and can actively be "claimed" later when the system is running and the specific service needs the port to operate. This could be used to basically reserve those 2 ports automatically during bootup. The chances though that this change will make it into RHEL-4 is highly unlikely as it quite drastically changes bootup behavior. Possibly for RHEL-5 but the main aim is to have it for RHEL-6. Hope this clears it up a bit. Read ya, Phil
I think portreserve is a decent solution (as long as someone takes the 5 minutes to extend it to support UDP as well since IPMI uses UDP). I don't think this would be a terrible idea in RHEL either. Once in rawhide maybe EPEL would be a good place to try it out?
That sounds definitely reasonable. As soon as we have something ready i'll update the bugzilla here. Read ya, Phil
We're now shipping portreserve in Fedora 10, so please take a look at that and report if that would fix your problems. Thanks, Read ya, Phil
Does portreserve now contain support for udp ports as well? If it's still tcp only that isn't helpful for this problem.
I'll ping the engineer that is taking care of it. Read ya, Phil
Ok, portreserve in the current form in Fedora supports TCP and UDP ports, so you should all be set. If you have any more detailed questions about portreserve please send an email to Tim Waugh <twaugh@redhat.com>, the maintainer of the package. Thanks, Read ya, Phil
I think this has been sufficiently addressed. I have a program that does this reservation for my test infrastructure. I think you can close this as nextrelease or whatever seems appropriate. Thanks!
Alright, closing this as nextrelease then. Thanks Jeffrey.