Description of problem: I can create unscaled applications. While creating scaled it fails with the error: "Unable to complete the requested operation due to: Failed to correctly execute all parallel operations." The node side error states: "openshift.rb:310:in `rescue in with_container_from_args' System proxy set for 38051=>127.1.246.1:8080 failed(1): stdout: -I rhc-app-comm 1 -d 127.1.246.1 -p tcp --dport 8080 -j ACCEPT -m comment --comment 38051 stderr: iptables: No chain/target/match by that name." This is due to /etc/sysconfig/iptables file is missing. The openshift.sh script tries to input "rhc-app-comm - [0:0]" into the file and fails and never complains. So openshift-iptables-port-proxy cant start. Manually editing the file, saving and starting openshift-iptables-port-proxy resolves the problem. Version-Release number of selected component (if applicable): redhat 6.5 beta OSE 2.2 beta drop2 rubygem-openshift-origin-node-1.17.0-1.git.112.d026e45.el6op.noarch How reproducible: Have installed more than 5 instances of 3 host OSE 2.0 beta 2 drop 2. All of then have the same issue. Steps to Reproduce: 1.use openshift.sh for OSE 2.0 beta from github tio install OSE 2. Do not manually start iptables services 3. The script then cant find /etc/sysconfig/iptables and fails to edit the file with required "rhc-app-comm - [0:0]" and never complains. 4. create and app: pass 5. create a scaled app: "fail" Actual results: Chain rhc-app-comm does not exist. Cant create iptable rules for gears. Expected results: Chain rhc-app-comm should exist. Iptables rules should be created for scaling applications.
I realize nearly two months have passed but by any chance do you remember how this system was configured? openshift.sh is mentioned in the description but I wanted to double check you didn't mean openshift.ks. The reason I ask is that it's not apparent to me how /etc/sysconfig/iptables could be missing. See: https://github.com/brenton/openshift-extras/blob/master/enterprise/install-scripts/generic/openshift.sh#L1302 That's where the sed command is run. At that point /etc/sysconfig/iptables needs to exist. This part is triggered from https://github.com/brenton/openshift-extras/blob/master/enterprise/install-scripts/generic/openshift.sh#L2565. You can see a few lines above that enable_services_on_node is called: https://github.com/brenton/openshift-extras/blob/master/enterprise/install-scripts/generic/openshift.sh#L2549 That method uses lokkit to open up the required ports on a node (if lokkit isn't installed the installation will abort). If openshift.sh is being run in bash then $lokkit will be set to "lokkit". If it's running in kickstart mode then it runs with "lokkit --nostart" which I have found does not actually create /etc/sysconfig/iptables. However that doesn't appear to be your issue. Ultimately, what I'm trying to say is that steps 1 and 2 in the bug description aren't yet making sense to me since running lokkit will both create /etc/sysconfig/iptables and immediately start the firewall.
I have used openshift.sh from github to install on many systems and manually needed to write to the iptables file.
(In reply to Brenton Leanhardt from comment #2) It was just brought to my attention that I accidentally linked to my personal repo in Comment #2. Below are the links I intended: > https://github.com/brenton/openshift-extras/blob/master/enterprise/install- > scripts/generic/openshift.sh#L1302 https://github.com/openshift/openshift-extras/blob/enterprise-2.0/enterprise/install-scripts/generic/openshift.sh#L1362 > > You can see a few lines above that enable_services_on_node is called: > https://github.com/brenton/openshift-extras/blob/master/enterprise/install- > scripts/generic/openshift.sh#L2549 > https://github.com/openshift/openshift-extras/blob/enterprise-2.0/enterprise/install-scripts/generic/openshift.sh#L2812
We've had reports of another customer who has hit this problem. If anyone experiencing the problem could attach the output of running openshift.sh with the -x flag it would be extremely helpful in letting us figure out where the problem exists.
It works, when create a stock /etc/sysconfig/iptables before running `openshift.sh` ======== cat <<EOF > /etc/sysconfig/iptables *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT EOF ========
Did you happen to capture a the output of 'openshift.sh -x' when in the case of it not working?
Also, could you provide the contents of /etc/sysconfig/system-config-firewall at the moment you run into this problem?
Brenton, would it be reasonable to drop configure_port_proxy from openshift.sh entirely and instead add ":rhc-app-comm - [0:0]" to the head of /etc/openshift/iptables.filter.rules and /etc/openshift/iptables.nat.rules, or better, add it to the iptables-restore commands in the openshift-iptables-port-proxy initscript? That will create the chain if it does not already exist, and it will not cause problems if the chain does already exist. I can put together a PR and test it out. Of course, even if we sort out the port-proxy issue, I'm still concerned that the other configuration rules that openshift.sh might not be taking effect if /etc/sysconfig/iptables is empty.
I'm pretty sure the script will fail in some way if the chain already exists. I feel like I tried adding the chain to our .rules files when working on Bug #1027122. That's not to say we shouldn't improve how openshift-iptables-port-proxy loads the rules. I like the idea of handling it in /usr/sbin/oo-admin-ctl-iptables-port-proxy just before the iptables-restore. Let's see what we can find from the affected users to see if /etc/sysconfig/system-config-firewall indeed has the firewall disabled. We'll probably want to check a few things like that and report useful error messages in oo-admin-ctl-iptables-port-proxy.
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/e25d78ffeb7d937f008bbd6fc326cfce7619a9d6 Various iptables integration fixes * oo-diagnostics will warn if /etc/sysconfig/iptables is missing * oo-diagnostics will advise admins not to use lokkit (and offer helpful advice if they must) * oo-admin-ctl-uptables-port-proxy will ensure the rhc-app-comm chain exist * oo-admin-ctl-uptables-port-proxy's status method was improved to better handle custom NAT rules This is to address: Bug 1032798 - unable to create scaled app because of misconfigured iptables method Bug 1027122 - Running lokkit tools on node will block openshift-iptables-port-proxy service starting. Bug 1053639 - Following deployment instructions for OSE2.0 node seems to cause iptables issues with openshift-iptables-port-proxy service start The latter has been very problematic because manual installs would tell users to run a sed command. If commands are copied wrong or out of order it can lead to a broken setup. Lokkit commands are much easier to copy/paste from a wiki.
verified with puddle 2014-01-30 by following these steps # Launch another devenv that can reach the one where you are testing. The reason it has to be in the AWS is because of our how default security group is setup. To reach these high numbered ports the call must be made from inside the EC2. #You should be able to reach the web app from this new machine: curl -k -H "Host: mynodejsapp1-demo.dev.rhcloud.com" http://$AWS_INTERNAL_DNS:35532 # Delete the rhc-app-comm INPUT rule (on the first devenv) iptables -D INPUT -j rhc-app-comm # This should no longer work from the second machine: curl -k -H "Host: mynodejsapp1-demo.dev.rhcloud.com" http://$AWS_INTERNAL_DNS:35532 # This should report the missing INPUT rule service openshift-iptables-port-proxy status # This should fix it and the curl command should now work service openshift-iptables-port-proxy start (it will say the chain already exist. This is OK) curl -k -H "Host: mynodejsapp1-demo.dev.rhcloud.com" http://$AWS_INTERNAL_DNS:35532
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0209.html