Bug 460410 - xm create fails to add vif PHYSDEV match rules for a domU with multiple network interfaces
xm create fails to add vif PHYSDEV match rules for a domU with multiple netw...
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen (Show other bugs)
All Linux
medium Severity medium
: rc
: ---
Assigned To: Jiri Denemark
Virtualization Bugs
Depends On:
Blocks: 477162
  Show dependency treegraph
Reported: 2008-08-28 01:38 EDT by Michael Kearey
Modified: 2010-10-23 00:07 EDT (History)
6 users (show)

See Also:
Fixed In Version: xen-3.0.3-87.el5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 503043 (view as bug list)
Last Closed: 2009-09-02 06:07:28 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
patch to resolve iptables contention in xen network scripts (3.06 KB, patch)
2008-08-28 01:40 EDT, Michael Kearey
no flags Details | Diff
Serialize iptables calls in hotplug scripts (818 bytes, patch)
2009-05-27 12:42 EDT, Jiri Denemark
no flags Details | Diff

  None (edit)
Description Michael Kearey 2008-08-28 01:38:05 EDT
Description of problem:
When a domU is configured to have multiple network interfaces, it may fail to create the iptables rules to allow each of the domU's interfaces to bridge the dom0's firewall.

It seems to be a resources issue with iptables and the Xen scripts for networking - multiple network interfaces means that the iptables command is run in a very quick sequence or even concurrently to add vif PHYSDEV match rules resulting in resource contention.

Version-Release number of selected component (if applicable):

How reproducible:
A domU with 2 network interfaces I observed the problem 50% of the time.
A domU with 3 network interfaces I observed the problem 100% of the time 

Steps to Reproduce:
1. Set up a domU guest with more than one network interface - for example edit the /etc/xen/domuGuest  file and add line :
vif = [ '', '', '' ]

2. Start the domU with xm create 

Actual results:

After starting the domU we observe the iptables firewall rules :

ACCEPT     all  --  anywhere             anywhere            PHYSDEV match --physdev-in vif1.0
ACCEPT     all  --  anywhere             anywhere            PHYSDEV match --physdev-in vif1.1

And in the /var/log/messages we see:

iptables -A FORWARD -m physdev --physdev-in BLAH BLAH -j ACCEPT failed.
If you are using iptables, this may affect networking for guest domains.

If we strace the iptables command itself we see:

5087  setsockopt(3, SOL_IP, 0x40 /* IP_??? */, "filter.., 3840) = -1 EAGAIN (Resource temporarily unavailable)
5087  write(2, "iptables: Unknown error 18446744073709551615n", 45) = 45
5087  exit_group(1)                     = ?

Expected results:
We should see the xm create command for the domU run completely, create the domain and the PHYSDEV match rules for each vif should be present. Also no errors in /var/log/messages  

Additional info:

A patch to resolve the problem is attached. I adds a simple locking mechanism around the iptables command as it is run by the xen network scripts. It ensures that the iptables commands are  run one at a time.
Comment 1 Michael Kearey 2008-08-28 01:40:32 EDT
Created attachment 315173 [details]
patch to resolve iptables contention in xen network scripts
Comment 5 Jiri Denemark 2009-05-27 12:34:33 EDT
It's actually a bug in iptables, IMHO. If you run something like

for ((i=0;i<100;i++)); do iptables -A FORWARD -m physdev --physdev-in vif7.$i -j ACCEPT & done

quite a few of the calls will fail. Should we clone this bug for iptables?

Anyway we can easily workaround it by serializing iptables calls...
Comment 6 Jiri Denemark 2009-05-27 12:42:11 EDT
Created attachment 345650 [details]
Serialize iptables calls in hotplug scripts
Comment 7 Michal Novotny 2009-06-08 03:52:42 EDT
Fix built into xen-3.0.3-87.el5
Comment 9 Chris Ward 2009-07-03 14:07:20 EDT
~~ Attention - RHEL 5.4 Beta Released! ~~

RHEL 5.4 Beta has been released! There should be a fix present in the Beta release that addresses this particular request. Please test and report back results here, at your earliest convenience. RHEL 5.4 General Availability release is just around the corner!

If you encounter any issues while testing Beta, please describe the issues you have encountered and set the bug into NEED_INFO. If you encounter new issues, please clone this bug to open a new issue and request it be reviewed for inclusion in RHEL 5.4 or a later update, if it is not of urgent severity.

Please do not flip the bug status to VERIFIED. Only post your verification results, and if available, update Verified field with the appropriate value.

Questions can be posted to this bug or your customer or partner representative.
Comment 10 zhanghaiyan 2009-07-30 03:00:56 EDT
Verified on xen- PASS
Comment 12 errata-xmlrpc 2009-09-02 06:07:28 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.