Bug 1027122 - Running lokkit tools on node will block openshift-iptables-port-proxy service starting.
Running lokkit tools on node will block openshift-iptables-port-proxy service...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Pod (Show other bugs)
2.0.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Brenton Leanhardt
libra bugs
:
Depends On: 1032798
Blocks: 1038831
  Show dependency treegraph
 
Reported: 2013-11-06 03:30 EST by Johnny Liu
Modified: 2017-03-08 12 EST (History)
8 users (show)

See Also:
Fixed In Version: rubygem-openshift-origin-node-1.17.5.8-1 rubygem-openshift-origin-common-1.17.2.7-1
Doc Type: Bug Fix
Doc Text:
The lokkit tool removed critical iptables rules required by OpenShift Enterprise, resulting in node outages. This bug fix updates the oo-diagnostics tool to detect if the lokkit or system-config-firewall tools are used, and advises the user not to use lokkit. If lokkit is required by the user, oo-diagnostics also provides a snippet that users can add to their lokkit configuration to allow interoperation with OpenShift Enterprise. Outages due to conflicting iptables configurations are now reduced.
Story Points: ---
Clone Of:
: 1038831 (view as bug list)
Environment:
Last Closed: 2014-02-25 10:41:14 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Johnny Liu 2013-11-06 03:30:07 EST
Description of problem:
Running lokkit tools on node will flush the existing iptable rules, and save it to /etc/sysconfig/iptables, then try to start openshift-iptables-port-proxy service to restore openshift iptable rules, it failed, due to "rhc-app-comm" chain is flushed by lokkit. That means even user reboot node, the iptables rules for openshift still can not be restored. Suggest the action about adding "rhc-app-comm" chain should be completed in openshift-iptables-port-proxy service.

Version-Release number of selected component (if applicable):
2.0/2013-11-05.1

How reproducible:
Always

Steps to Reproduce:
1.Setup env using openshift.sh installation script
2.Create a scalable applications
3.Show current iptable rules:
# iptables-save
# Generated by iptables-save v1.4.7 on Wed Nov  6 01:13:42 2013
*nat
:PREROUTING ACCEPT [12:456]
:POSTROUTING ACCEPT [1441:94556]
:OUTPUT ACCEPT [1441:94556]
-A PREROUTING -d 192.168.59.195/32 -p tcp -m tcp --dport 38046 -j DNAT --to-destination 127.1.245.129:8080 
-A PREROUTING -d 192.168.59.195/32 -p tcp -m tcp --dport 38047 -j DNAT --to-destination 127.1.245.130:8080 
-A PREROUTING -d 192.168.59.195/32 -p tcp -m tcp --dport 38051 -j DNAT --to-destination 127.1.246.1:8080 
-A OUTPUT -d 192.168.59.195/32 -p tcp -m tcp --dport 38046 -j DNAT --to-destination 127.1.245.129:8080 
-A OUTPUT -d 192.168.59.195/32 -p tcp -m tcp --dport 38047 -j DNAT --to-destination 127.1.245.130:8080 
-A OUTPUT -d 192.168.59.195/32 -p tcp -m tcp --dport 38051 -j DNAT --to-destination 127.1.246.1:8080 
COMMIT
# Completed on Wed Nov  6 01:13:42 2013
# Generated by iptables-save v1.4.7 on Wed Nov  6 01:13:42 2013
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [17178:8463798]
:rhc-app-comm - [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p icmp -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -j rhc-app-comm 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8000 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8443 -j ACCEPT 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited 
-A rhc-app-comm -d 127.1.246.1/32 -p tcp -m conntrack --ctstate NEW -m tcp --dport 8080 -m comment --comment "38051" -j ACCEPT 
-A rhc-app-comm -d 127.1.246.1/32 -p tcp -m tcp --dport 8080 -m comment --comment "38051" -j ACCEPT 
-A rhc-app-comm -d 127.1.245.130/32 -p tcp -m conntrack --ctstate NEW -m tcp --dport 8080 -m comment --comment "38047" -j ACCEPT 
-A rhc-app-comm -d 127.1.245.130/32 -p tcp -m tcp --dport 8080 -m comment --comment "38047" -j ACCEPT 
-A rhc-app-comm -d 127.1.245.129/32 -p tcp -m conntrack --ctstate NEW -m tcp --dport 8080 -m comment --comment "38046" -j ACCEPT 
-A rhc-app-comm -d 127.1.245.129/32 -p tcp -m tcp --dport 8080 -m comment --comment "38046" -j ACCEPT 
COMMIT
# Completed on Wed Nov  6 01:13:42 2013
4.Run lokkit tools to enable port.
# lokkit --port=11111:tcp
5.Show iptable rules again, all iptable rules about opoenshift are flushed.
# iptables-save
# Generated by iptables-save v1.4.7 on Wed Nov  6 01:14:58 2013
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [160:79708]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p icmp -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8000 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8443 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 11111 -j ACCEPT 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited 
COMMIT
# Completed on Wed Nov  6 01:14:58 2013
6. After step 4, /etc/sysconfig/iptables is rewrote, and old one is backed up.
# diff /etc/sysconfig/iptables /etc/sysconfig/iptables.old 
6a7
> :rhc-app-comm - [0:0]
9a11
> -A INPUT -j rhc-app-comm
15d16
< -A INPUT -m state --state NEW -m tcp -p tcp --dport 11111 -j ACCEPT
7. Try to start openshift-iptables-port-proxy service, failed.
# /etc/init.d/openshift-iptables-port-proxy start
iptables-restore: line 2 failed

Actual results:
Fail to start openshift-iptables-port-proxy service, iptable rules about openshift can not be restored.

Expected results:
Add "rhc-app-comm" chain action should be splited into openshift-iptables-port-proxy.

Additional info:
Comment 2 Brenton Leanhardt 2013-11-22 12:18:27 EST
I talked with the system-config-firewall maintainer about this a bit and here's a possible solution:

* OSE install could modify /etc/sysconfig/system-config-firewall and add: 

--custom-rules=ipv4:filter:/etc/openshift/iptables.filter.rules
--custom-rules=ipv4:nat:/etc/openshift/iptables.nat.rules

This would cause the rules to be added just before the right before the reject rules at the end.  This is slightly different than today where we inject '-A INPUT -j rhc-app-comm' before the location where admins would normally make their customizations.  This _shouldn't_ be a problem because OpenShift gear ports shouldn't conflict with common system services (if they do there will be other problems)

* We still have a problem with making sure lokkit doesn't nuke the chain.  I think we would add a new custom-rule file that would handle the chain creation.

* There is testing needed with 'service iptables save'.  If an admin were to run that then the chain creation would be written to /etc/sysconfig/iptables.  When lokkit tries to readd the chain it will fail.
Comment 3 Brenton Leanhardt 2013-11-22 13:48:37 EST
I added the following to /etc/sysconfig/system-config-firewall:

--custom-rules=ipv4:filter:/etc/openshift/system-config-firewall
--custom-rules=ipv4:filter:/etc/openshift/iptables.filter.rules
--custom-rules=ipv4:nat:/etc/openshift/iptables.nat.rules

The content of /etc/openshift/system-config-firewall is:

-N rhc-app-comm
-A INPUT -j rhc-app-comm

With this in place the following works as expected:

# manage /etc/sysconfig/iptables with lokkit and overwrite/reload that file
$ lokkit --service=ssh

# see if our chain still exists
$ service openshift-iptables-port-proxy status
The OpenShift iptables port proxy is enabled.

# stop the chain
$ service openshift-iptables-port-proxy stop
$ service openshift-iptables-port-proxy status
ERROR: A difference has been detected between state of /etc/openshift/iptables.filter.rules and the rhc-app-comm iptables chain.

# try again and verify it loads the chain back
$ lokkit --service=ssh
$ service openshift-iptables-port-proxy status
The OpenShift iptables port proxy is enabled.

# verify iptables-save works as expected
$ service iptables save
$ lokkit --service=ssh
$service openshift-iptables-port-proxy status
The OpenShift iptables port proxy is enabled.

I think we can consider putting the needed logic in openshift.sh.  Alternatively we sould add it to the rubygem-openshift-origin-node package.  I'd prefer to do the latter upstream and test it first.
Comment 5 Brenton Leanhardt 2013-12-05 10:18:52 EST
We need to verify that oo-diagnostics catches the missing iptables chain.
Comment 7 Brenton Leanhardt 2014-01-29 12:09:32 EST
For clarification, the proposed fix for this is to advise admins not to use lokkit.  We will detect that in oo-diagnostics.  However, since it ships with RHEL we will do our best to maintain compatibility.  To that in oo-diagnostics will also provide advice on how to configure OSE to work with lokkit (as best it can).
Comment 10 Peter Ruan 2014-01-31 02:56:20 EST
verified with puddle-2-0-3-2014-01-30

[root@broker sysconfig]# oo-diagnostics 
WARN: test_node_profiles_districts_from_broker
        No districts are defined. Districts should be used in any production installation.
        Please consult the Administration Guide.

FAIL: test_services_enabled
      The following service(s) are not currently started:
        openshift-iptables-port-proxy
      These services are required for OpenShift functionality.

WARN: test_system_config_firewall
         Using system-config-firewall and lokkit with OpenShift is not recommended.
         To continue using lokkit please ensure the following custom rules are 
         installed in /etc/sysconfig/system-config-firewall:

         --custom-rules=ipv4:filter:/etc/openshift/system-config-firewall-compat
         --custom-rules=ipv4:filter:/etc/openshift/iptables.filter.rules
         --custom-rules=ipv4:nat:/etc/openshift/iptables.nat.rules

WARN: test_altered_package_owned_configs
          The mlocate package is not installed. mlocate is not a required runtime package; however,
          you may install mlocate to enable further diagnostics checking.

WARN: test_broker_certificate
Using a self-signed certificate for the broker
WARN: test_yum_configuration
        oo-admin-yum-validator reported some possible problems
        with your package source configuration:
--------------------------------------------------------------
      No roles have been specified. Attempting to guess the roles for this system...
If the roles listed below are incorrect or incomplete, please re-run this script with the appropriate --role arguments
    node
    broker
    client
If this system will be providing the JBossEAP cartridge, re-run this command with the --role=node-eap argument
Detected OpenShift Enterprise repository subscription managed by Red Hat Subscription Manager.
Detected installed OpenShift Enterprise version 2.0

Checking if yum-plugin-priorities is installed
Checking channel/repository priorities
Resolving repository/channel/subscription priority conflicts
To resolve conflicting repositories, update repo priority by running:
# yum-config-manager --setopt=openshift_client_tools_extra.priority=40 openshift_client_tools_extra --save
# yum-config-manager --setopt=openshift_node_extra.priority=40 openshift_node_extra --save
# yum-config-manager --setopt=openshift_infra_extra.priority=40 openshift_infra_extra --save
Please re-run this tool after making any recommended repairs to this system

--------------------------------------------------------------
        Incorrect package source configuration could lead to
        failure to install the correct RPMs.

5 WARNINGS
1 ERRORS
Comment 12 errata-xmlrpc 2014-02-25 10:41:14 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0209.html

Note You need to log in before you can comment on or make changes to this bug.