Bug 1270117

Summary: Pass -w flag to iptables to make it wait for xtables lock
Product: Red Hat OpenStack Reporter: Lucy Bopf <lbopf>
Component: openstack-ironic-inspectorAssignee: Dmitry Tantsur <dtantsur>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 7.0 (Kilo)CC: apevec, dtantsur, hbrock, lhh, mburns, morazi, rhel-osp-director-maint, slinaber
Target Milestone: gaKeywords: Triaged
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-ironic-inspector-2.2.2-2.el7ost Doc Type: Bug Fix
Doc Text:
Previously, periodic iptables calls made by Ironic Inspector did not contain the -w option, which instructs iptables to wait for the xtables lock. As a consequence, periodic iptables updates occasionally failed. This update adds the -w option to the iptables calls, which prevents the periodic iptables updates from failing.
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-07 21:41:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lucy Bopf 2015-10-09 02:17:05 UTC
Description of problem:
During hardware introspection, iptables commands are failing because something else is holding the xtables lock. The output of 'journalctl -l -u openstack-ironic-discoverd -f' politely suggests using the -w option:

Oct 09 12:06:48 localhost.localdomain ironic-discoverd[3483]: DEBUG:ironic_discoverd.firewall:Running iptables ('-D', 'INPUT', '-i', 'br-ens4', '-p', 'udp', '--dport', '67', '-j', 'discovery_temp')
Oct 09 12:06:48 localhost.localdomain ironic-discoverd[3483]: DEBUG:ironic_discoverd.firewall:Running iptables ('-A', 'discovery_temp', '-m', 'mac', '--mac-source', u'52:54:00:f5:c6:44', '-j', 'DROP')
Oct 09 12:06:48 localhost.localdomain ironic-discoverd[3483]: DEBUG:ironic_discoverd.firewall:ignoring failed iptables ('-D', 'INPUT', '-i', 'br-ens4', '-p', 'udp', '--dport', '67', '-j', 'discovery_temp'):
Oct 09 12:06:48 localhost.localdomain ironic-discoverd[3483]: iptables: No chain/target/match by that name.
Oct 09 12:06:48 localhost.localdomain ironic-discoverd[3483]: ERROR:ironic_discoverd.firewall:iptables ('-A', 'discovery_temp', '-m', 'mac', '--mac-source', u'52:54:00:f5:c6:44', '-j', 'DROP') failed:
Oct 09 12:06:48 localhost.localdomain ironic-discoverd[3483]: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?


The problem is described upstream here: https://bugs.launchpad.net/ironic-inspector/+bug/1484110

The solution in that bug was to add a patch that passes the -w flag to iptables. I believe the same solution applies here.


Version-Release number of selected component (if applicable):
openstack-ironic-discoverd-1.1.0-5.el7ost.noarch
openstack-ironic-api-2015.1.0-9.el7ost.noarch
openstack-ironic-conductor-2015.1.0-9.el7ost.noarch

How reproducible:
For me, every time I've run hardware introspection.

Steps to Reproduce:
1. Run hardware introspection on a bare metal node.
2. Check the output of 'journalctl -l -u openstack-ironic-discoverd -f'
3.

Actual results:
iptables commands fail.

Expected results:
iptables commands wait for the lock to be released.

Additional info:

Comment 3 Dmitry Tantsur 2015-10-12 15:43:01 UTC
I'll provide a simplified backport of the upstream commit https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=3f7054ed4de0da80320c55ec42b1464d88bceae8

Comment 6 Dmitry Tantsur 2016-01-11 12:14:34 UTC
was fixed in liberty final

Comment 9 Alexander Chuzhoy 2016-02-24 15:04:06 UTC
Verified:

Environment:
openstack-ironic-inspector-2.2.4-1.el7ost.noarch

Running "journalctl -l -u openstack-ironic-inspector", see lines like:
Feb 24 10:00:23 instack.localdomain sudo[4401]: ironic-inspector : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/ironic-inspector-rootwrap /etc/ironic-inspector/rootwrap.conf iptables -w -E ironic-inspector_temp ironic-inspector


Where "-w" presents,

Comment 11 errata-xmlrpc 2016-04-07 21:41:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0604.html