Bug 1125075 - Failure in iptables-save: Table does not exist
Summary: Failure in iptables-save: Table does not exist
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhel-osp-installer
Version: Foreman (RHEL 6)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ga
: Installer
Assignee: John Eckersberg
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-31 03:09 UTC by John Eckersberg
Modified: 2014-08-21 18:07 UTC (History)
8 users (show)

Fixed In Version: rhel-osp-installer-0.1.6-6.el6ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-08-21 18:07:12 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1090 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform Enhancement Advisory 2014-08-22 15:28:08 UTC
Red Hat Bugzilla 1125136 None None None Never

Internal Links: 1125136

Description John Eckersberg 2014-07-31 03:09:02 UTC
Description of problem:
Puppet failed with a neutron/gre non-HA install with the following error:

Execution of '/usr/sbin/iptables-save' returned 1: iptables-save v1.4.21: Cannot initialize: Table does not exist (do you need to insmod?)
Jul 31 01:17:40 mac5254008803ab.example.org puppet-agent[3861]: (/Stage[main]/Quickstack::Neutron::Firewall::Gre/Firewall[002 gre]/ensure) change from absent to present failed: Execution of '/usr/sbin/iptables-save' returned 1: iptables-save v1.4.21: Cannot initialize: Table does not exist (do you need to insmod?)

This caused the deployment to pause.

Running puppet a second time manually caused the resource to execute correctly, and allowed the deployment to continue after manually resuming in the Foreman UI.


Version-Release number of selected component (if applicable):
iptables-1.4.21-13.el7.x86_64 (on networker)
rhel-osp-installer-0.1.6-4.el6ost.noarch
openstack-puppet-modules-2014.1-19.9.el6ost.noarch

Comment 3 Ivan Chavero 2014-07-31 18:19:23 UTC
in my tests with packstack i haven't found this problem, could you be more specific in how to recreate this problem. checking 1125136 it seems like it might be a staypuft problem.

Comment 4 John Eckersberg 2014-07-31 21:58:49 UTC
I need to run, so I don't have time for a lengthy explanation, but:

17:55:59         eck | i am 99% positive the iptables service is buggy
17:56:13         eck | it is a oneshot service under systemd
17:56:36         eck | so i suspect it forks off to do its start up and then systemd reports it to be successfully started
17:56:53         eck | then puppet runs iptables-save before it's really initialized all the way
17:56:54         eck | and that fails

Comment 5 Ivan Chavero 2014-08-01 02:10:00 UTC
This appears to be an iptables problem. This would be better fixed in the staypuft puppet manifest instead of the puppet module.

Comment 6 John Eckersberg 2014-08-01 21:16:48 UTC
I've figured out what is happening here.  I think this is really a systemd bug, (although there are any number of ways we can work around it).  Here's the scenario:

The host has firewalld installed and running.

The service unit contains: Conflicts=iptables.service ip6tables.service ebtables.service

However, the ordering of the conflict operation seems to be wrong.  Here's the journal while executing 'systemctl start iptables.service':

Aug 01 15:05:03 mac5254008803ab.example.org systemd[1]: Stopping firewalld - dynamic firewall daemon...
Aug 01 15:05:03 mac5254008803ab.example.org systemd[1]: Starting IPv4 firewall with iptables...
Aug 01 15:05:03 mac5254008803ab.example.org systemd[1]: Started IPv4 firewall with iptables.
Aug 01 15:05:04 mac5254008803ab.example.org kernel: Ebtables v2.0 unregistered
Aug 01 15:05:04 mac5254008803ab.example.org systemd[1]: Stopped firewalld - dynamic firewall daemon.

(Note that although systemd initiates the process of stopping firewalld, it does not wait for it to complete before it then starts iptables.)

So what happens with puppet involved is:

1. Puppet runs 'systemctl start iptables.service'
2. Systemd begins stopping firewalld
3. Systemd begins starting iptables
4. Systemd determines iptables is started
5. Puppet returns from (1)
6. Puppet continues the catalog and begins inserting iptables rules
7. The puppet firewall provider runs iptables-save to persist new rules
8. iptables-save is in the process of executing, and has handles into the kernel netfilter bits
9. Firewalld is still stopping.  At this point it's unregistering its netfilter hooks in the kernel.
10. Some resource that iptables-save has open (in my testing it was always the 'nat' table) now becomes invalid.
11. iptables-save aborts
12. Puppet fails

Comment 7 John Eckersberg 2014-08-01 21:27:58 UTC
The easiest way to make this stop being a problem in rhel-osp is probably to remove the firewalld package in the kickstart template.  As such, I'm going to reassign this over to rhel-osp-installer for an expedited fix.

Comment 13 Omri Hochman 2014-08-07 11:34:49 UTC
Verified with:

ruby193-rubygem-staypuft-0.1.22-1.el6ost.noarch
openstack-puppet-modules-2014.1-19.9.el6ost.noarch
rhel-osp-installer-0.1.6-5.el6ost.noarch

Unable to reproduce foreman puddle 2014-07-31.1.

Comment 14 errata-xmlrpc 2014-08-21 18:07:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1090.html


Note You need to log in before you can comment on or make changes to this bug.