Hide Forgot
The lokkit tool removed critical iptables rules required by OpenShift Enterprise, resulting in node outages. This bug fix updates the oo-diagnostics tool to detect if the lokkit or system-config-firewall tools are used, and advises the user not to use lokkit. If lokkit is required by the user, oo-diagnostics also provides a snippet that users can add to their lokkit configuration to allow interoperation with OpenShift Enterprise. Outages due to conflicting iptables configurations are now reduced.
Description of problem: Running lokkit tools on node will flush the existing iptable rules, and save it to /etc/sysconfig/iptables, then try to start openshift-iptables-port-proxy service to restore openshift iptable rules, it failed, due to "rhc-app-comm" chain is flushed by lokkit. That means even user reboot node, the iptables rules for openshift still can not be restored. Suggest the action about adding "rhc-app-comm" chain should be completed in openshift-iptables-port-proxy service. Version-Release number of selected component (if applicable): 2.0/2013-11-05.1 How reproducible: Always Steps to Reproduce: 1.Setup env using openshift.sh installation script 2.Create a scalable applications 3.Show current iptable rules: # iptables-save # Generated by iptables-save v1.4.7 on Wed Nov 6 01:13:42 2013 *nat :PREROUTING ACCEPT [12:456] :POSTROUTING ACCEPT [1441:94556] :OUTPUT ACCEPT [1441:94556] -A PREROUTING -d 192.168.59.195/32 -p tcp -m tcp --dport 38046 -j DNAT --to-destination 127.1.245.129:8080 -A PREROUTING -d 192.168.59.195/32 -p tcp -m tcp --dport 38047 -j DNAT --to-destination 127.1.245.130:8080 -A PREROUTING -d 192.168.59.195/32 -p tcp -m tcp --dport 38051 -j DNAT --to-destination 127.1.246.1:8080 -A OUTPUT -d 192.168.59.195/32 -p tcp -m tcp --dport 38046 -j DNAT --to-destination 127.1.245.129:8080 -A OUTPUT -d 192.168.59.195/32 -p tcp -m tcp --dport 38047 -j DNAT --to-destination 127.1.245.130:8080 -A OUTPUT -d 192.168.59.195/32 -p tcp -m tcp --dport 38051 -j DNAT --to-destination 127.1.246.1:8080 COMMIT # Completed on Wed Nov 6 01:13:42 2013 # Generated by iptables-save v1.4.7 on Wed Nov 6 01:13:42 2013 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [17178:8463798] :rhc-app-comm - [0:0] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -j rhc-app-comm -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 8000 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 8443 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited -A rhc-app-comm -d 127.1.246.1/32 -p tcp -m conntrack --ctstate NEW -m tcp --dport 8080 -m comment --comment "38051" -j ACCEPT -A rhc-app-comm -d 127.1.246.1/32 -p tcp -m tcp --dport 8080 -m comment --comment "38051" -j ACCEPT -A rhc-app-comm -d 127.1.245.130/32 -p tcp -m conntrack --ctstate NEW -m tcp --dport 8080 -m comment --comment "38047" -j ACCEPT -A rhc-app-comm -d 127.1.245.130/32 -p tcp -m tcp --dport 8080 -m comment --comment "38047" -j ACCEPT -A rhc-app-comm -d 127.1.245.129/32 -p tcp -m conntrack --ctstate NEW -m tcp --dport 8080 -m comment --comment "38046" -j ACCEPT -A rhc-app-comm -d 127.1.245.129/32 -p tcp -m tcp --dport 8080 -m comment --comment "38046" -j ACCEPT COMMIT # Completed on Wed Nov 6 01:13:42 2013 4.Run lokkit tools to enable port. # lokkit --port=11111:tcp 5.Show iptable rules again, all iptable rules about opoenshift are flushed. # iptables-save # Generated by iptables-save v1.4.7 on Wed Nov 6 01:14:58 2013 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [160:79708] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 8000 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 8443 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 11111 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT # Completed on Wed Nov 6 01:14:58 2013 6. After step 4, /etc/sysconfig/iptables is rewrote, and old one is backed up. # diff /etc/sysconfig/iptables /etc/sysconfig/iptables.old 6a7 > :rhc-app-comm - [0:0] 9a11 > -A INPUT -j rhc-app-comm 15d16 < -A INPUT -m state --state NEW -m tcp -p tcp --dport 11111 -j ACCEPT 7. Try to start openshift-iptables-port-proxy service, failed. # /etc/init.d/openshift-iptables-port-proxy start iptables-restore: line 2 failed Actual results: Fail to start openshift-iptables-port-proxy service, iptable rules about openshift can not be restored. Expected results: Add "rhc-app-comm" chain action should be splited into openshift-iptables-port-proxy. Additional info:
I talked with the system-config-firewall maintainer about this a bit and here's a possible solution: * OSE install could modify /etc/sysconfig/system-config-firewall and add: --custom-rules=ipv4:filter:/etc/openshift/iptables.filter.rules --custom-rules=ipv4:nat:/etc/openshift/iptables.nat.rules This would cause the rules to be added just before the right before the reject rules at the end. This is slightly different than today where we inject '-A INPUT -j rhc-app-comm' before the location where admins would normally make their customizations. This _shouldn't_ be a problem because OpenShift gear ports shouldn't conflict with common system services (if they do there will be other problems) * We still have a problem with making sure lokkit doesn't nuke the chain. I think we would add a new custom-rule file that would handle the chain creation. * There is testing needed with 'service iptables save'. If an admin were to run that then the chain creation would be written to /etc/sysconfig/iptables. When lokkit tries to readd the chain it will fail.
I added the following to /etc/sysconfig/system-config-firewall: --custom-rules=ipv4:filter:/etc/openshift/system-config-firewall --custom-rules=ipv4:filter:/etc/openshift/iptables.filter.rules --custom-rules=ipv4:nat:/etc/openshift/iptables.nat.rules The content of /etc/openshift/system-config-firewall is: -N rhc-app-comm -A INPUT -j rhc-app-comm With this in place the following works as expected: # manage /etc/sysconfig/iptables with lokkit and overwrite/reload that file $ lokkit --service=ssh # see if our chain still exists $ service openshift-iptables-port-proxy status The OpenShift iptables port proxy is enabled. # stop the chain $ service openshift-iptables-port-proxy stop $ service openshift-iptables-port-proxy status ERROR: A difference has been detected between state of /etc/openshift/iptables.filter.rules and the rhc-app-comm iptables chain. # try again and verify it loads the chain back $ lokkit --service=ssh $ service openshift-iptables-port-proxy status The OpenShift iptables port proxy is enabled. # verify iptables-save works as expected $ service iptables save $ lokkit --service=ssh $service openshift-iptables-port-proxy status The OpenShift iptables port proxy is enabled. I think we can consider putting the needed logic in openshift.sh. Alternatively we sould add it to the rubygem-openshift-origin-node package. I'd prefer to do the latter upstream and test it first.
We need to verify that oo-diagnostics catches the missing iptables chain.
For clarification, the proposed fix for this is to advise admins not to use lokkit. We will detect that in oo-diagnostics. However, since it ships with RHEL we will do our best to maintain compatibility. To that in oo-diagnostics will also provide advice on how to configure OSE to work with lokkit (as best it can).
verified with puddle-2-0-3-2014-01-30 [root@broker sysconfig]# oo-diagnostics WARN: test_node_profiles_districts_from_broker No districts are defined. Districts should be used in any production installation. Please consult the Administration Guide. FAIL: test_services_enabled The following service(s) are not currently started: openshift-iptables-port-proxy These services are required for OpenShift functionality. WARN: test_system_config_firewall Using system-config-firewall and lokkit with OpenShift is not recommended. To continue using lokkit please ensure the following custom rules are installed in /etc/sysconfig/system-config-firewall: --custom-rules=ipv4:filter:/etc/openshift/system-config-firewall-compat --custom-rules=ipv4:filter:/etc/openshift/iptables.filter.rules --custom-rules=ipv4:nat:/etc/openshift/iptables.nat.rules WARN: test_altered_package_owned_configs The mlocate package is not installed. mlocate is not a required runtime package; however, you may install mlocate to enable further diagnostics checking. WARN: test_broker_certificate Using a self-signed certificate for the broker WARN: test_yum_configuration oo-admin-yum-validator reported some possible problems with your package source configuration: -------------------------------------------------------------- No roles have been specified. Attempting to guess the roles for this system... If the roles listed below are incorrect or incomplete, please re-run this script with the appropriate --role arguments node broker client If this system will be providing the JBossEAP cartridge, re-run this command with the --role=node-eap argument Detected OpenShift Enterprise repository subscription managed by Red Hat Subscription Manager. Detected installed OpenShift Enterprise version 2.0 Checking if yum-plugin-priorities is installed Checking channel/repository priorities Resolving repository/channel/subscription priority conflicts To resolve conflicting repositories, update repo priority by running: # yum-config-manager --setopt=openshift_client_tools_extra.priority=40 openshift_client_tools_extra --save # yum-config-manager --setopt=openshift_node_extra.priority=40 openshift_node_extra --save # yum-config-manager --setopt=openshift_infra_extra.priority=40 openshift_infra_extra --save Please re-run this tool after making any recommended repairs to this system -------------------------------------------------------------- Incorrect package source configuration could lead to failure to install the correct RPMs. 5 WARNINGS 1 ERRORS
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0209.html