Hide Forgot
Description of problem: Multiple firewalld error messages in OpenShift master node /var/log/messages Version-Release number of selected component (if applicable): QCI-1.0-RHEL-7-20160815.t.0 How reproducible: 100% Steps to Reproduce: 1. Install/configure QCI 2. Deploy OpenShift on RHV 3. ssh ocp_master_node 4. Examine /var/log/messages Actual results: date time host docker-current: time="2016-08-17T14:07:32.716683867Z" level=info msg="Firewalld running: true" date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory#012#012Try `iptables -h' or 'iptables --help' for more information. date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory#012#012Try `iptables -h' or 'iptables --help' for more information. date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory#012#012Try `iptables -h' or 'iptables --help' for more information. date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?). date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?). date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -F DOCKER' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -X DOCKER' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -F DOCKER' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -n -L DOCKER' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -n -L DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?). date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?). date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?). date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name. date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -j DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. Expected results: No errors, firewall properly configured Additional info:
Similar messages on worker node
Dylan, Please look at this and determine if it is an issue we need to fix for GA.
It appears this is an issue with iptables rules not being present when docker-storage-setup runs. On a successful deployment, the DOCKER and DOCKER-ISOLATION chain exists, and I believe are instantiated once we start docker after docker-storage-setup runs. This appears to be a common issue encountered with Docker (https://github.com/docker/docker/issues/1871). Firewalld is disabled during OSE installation but enabled at the start. I think we can eliminate these messages if we stop firewalld before running docker-storage-setup / install docker. This appears to be a log cleanup issue, shouldn't impact deployment. Will see if this is an easy change.
I could not reproduce this as of 8/31. We made some changes to the post-install process, it is possible something changed where this is not showing up in the logs anymore. Moving to post ga.
I was able to reproduce this with ISO QCI-1.1-RHEL-7-20161215.t.0. Now it seems the errors are classified as warnings, and the text of the error message is suppressed. All of the 'COMMAND_FAILED' messages are still in the log. This is how it looks in /var/log/messages: " Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -F DOCKER' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -X DOCKER' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -F DOCKER' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -F DOCKER-ISOLATION' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -n -L DOCKER' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER-ISOLATION' failed: Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed: Dec 16 16:16:25 rhvocp-ose-master1 docker-current: time="2016-12-16T16:16:25.308310531Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" Dec 16 16:16:26 rhvocp-ose-master1 NetworkManager[693]: <info> [1481904986.8594] manager: (docker0): new Bridge device (/org/freedesktop/NetworkManager/Devices/2) Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed: Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed: Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -j DOCKER-ISOLATION' failed: " I'm setting the status back to NEW, so we can decide on the appropriate action.
Was able to replicate the warning seen in comment #6 using QCI-1.1-RHEL-7-20170106.t.0-QCI-x86_64-dvd1.iso
After speaking openshift devs, we are informed that the firewalld warnings from /var/log/messages have no adverse effects on the deployment. This is just a byproduct of docker starting prior to the installer configuring the environment to use iptables. Once iptables are configured, the installer would restart the appropriate services as needed. Marking this to ON_QA to verify that the warnings have no adverse to the deployment.
Since these warnings are deemed harmless we are closing this bug as verified. Compose: QCI-1.1-RHEL-7-20170111.t.8
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:0335