Bug 1367832

Summary: [OCP] Master and worker nodes: Multiple firewalld error messages
Product: Red Hat Quickstart Cloud Installer Reporter: Thom Carlin <tcarlin>
Component: Installation - OpenShiftAssignee: dgao
Status: CLOSED ERRATA QA Contact: James Olin Oden <joden>
Severity: medium Docs Contact: Derek <dcadzow>
Priority: unspecified    
Version: 1.0CC: apagac, arubin, bthurber, dgao, jesusr, jmontleo, joden, kdube
Target Milestone: gaKeywords: Triaged
Target Release: 1.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-28 01:38:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Thom Carlin 2016-08-17 15:13:57 UTC
Description of problem:

Multiple firewalld error messages in OpenShift master node /var/log/messages

Version-Release number of selected component (if applicable):

QCI-1.0-RHEL-7-20160815.t.0

How reproducible:

100%

Steps to Reproduce:
1. Install/configure QCI
2. Deploy OpenShift on RHV
3. ssh ocp_master_node
4. Examine /var/log/messages

Actual results:

date time host docker-current: time="2016-08-17T14:07:32.716683867Z" level=info msg="Firewalld running: true"
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory#012#012Try `iptables -h' or 'iptables --help' for more information.
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory#012#012Try `iptables -h' or 'iptables --help' for more information.
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory#012#012Try `iptables -h' or 'iptables --help' for more information.
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -F DOCKER' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -X DOCKER' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -F DOCKER' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -n -L DOCKER' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -n -L DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:32 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.
date time host firewalld: 2016-08-17 14:07:34 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -j DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.

Expected results:

No errors, firewall properly configured

Additional info:

Comment 1 Thom Carlin 2016-08-17 16:24:25 UTC
Similar messages on worker node

Comment 2 John Matthews 2016-08-17 18:41:13 UTC
Dylan,

Please look at this and determine if it is an issue we need to fix for GA.

Comment 4 Dylan Murray 2016-08-17 19:07:47 UTC
It appears this is an issue with iptables rules not being present when docker-storage-setup runs. On a successful deployment, the DOCKER and DOCKER-ISOLATION chain exists, and I believe are instantiated once we start docker after docker-storage-setup runs. This appears to be a common issue encountered with Docker (https://github.com/docker/docker/issues/1871). Firewalld is disabled during OSE installation but enabled at the start. I think we can eliminate these messages if we stop firewalld before running docker-storage-setup / install docker. 

This appears to be a log cleanup issue, shouldn't impact deployment. Will see if this is an easy change.

Comment 5 Dylan Murray 2016-08-31 19:27:43 UTC
I could not reproduce this as of 8/31. We made some changes to the post-install process, it is possible something changed where this is not showing up in the logs anymore. Moving to post ga.

Comment 6 Antonin Pagac 2016-12-19 10:13:09 UTC
I was able to reproduce this with ISO QCI-1.1-RHEL-7-20161215.t.0.

Now it seems the errors are classified as warnings, and the text of the error message is suppressed. All of the 'COMMAND_FAILED' messages are still in the log. This is how it looks in /var/log/messages:

"
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -F DOCKER' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -X DOCKER' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -F DOCKER' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -F DOCKER-ISOLATION' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -n -L DOCKER' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER-ISOLATION' failed:
Dec 16 16:16:24 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed:
Dec 16 16:16:25 rhvocp-ose-master1 docker-current: time="2016-12-16T16:16:25.308310531Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Dec 16 16:16:26 rhvocp-ose-master1 NetworkManager[693]: <info>  [1481904986.8594] manager: (docker0): new Bridge device (/org/freedesktop/NetworkManager/Devices/2)
Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed:
Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed:
Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed:
Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed:
Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed:
Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed:
Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed:
Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed:
Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed:
Dec 16 16:16:26 rhvocp-ose-master1 firewalld: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -j DOCKER-ISOLATION' failed:
"

I'm setting the status back to NEW, so we can decide on the appropriate action.

Comment 7 dgao 2017-01-10 00:34:36 UTC
Was able to replicate the warning seen in comment #6 using 

QCI-1.1-RHEL-7-20170106.t.0-QCI-x86_64-dvd1.iso

Comment 8 dgao 2017-01-11 15:30:06 UTC
After speaking openshift devs, we are informed that the firewalld warnings from /var/log/messages have no adverse effects on the deployment. This is just a byproduct of docker starting prior to the installer configuring the environment to use iptables. Once iptables are configured, the installer would restart the appropriate services as needed.

Marking this to ON_QA to verify that the warnings have no adverse to the deployment.

Comment 9 James Olin Oden 2017-01-12 21:56:43 UTC
Since these warnings are deemed harmless we are closing this bug as verified.

Compose:  QCI-1.1-RHEL-7-20170111.t.8

Comment 11 errata-xmlrpc 2017-02-28 01:38:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:0335