Bug 1304004 - 1.3.2: Installation: After blacklisting nf-conntrack modules, cannot start firewalld
1.3.2: Installation: After blacklisting nf-conntrack modules, cannot start fi...
Status: CLOSED DUPLICATE of bug 1290923
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: Documentation (Show other bugs)
1.3.2
Unspecified Unspecified
unspecified Severity high
: rc
: 1.3.2
Assigned To: ceph-docs@redhat.com
ceph-qe-bugs
:
Depends On:
Blocks: 1290923
  Show dependency treegraph
 
Reported: 2016-02-02 11:34 EST by Harish NV Rao
Modified: 2016-02-08 16:20 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-08 16:20:37 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Harish NV Rao 2016-02-02 11:34:55 EST
Description of problem:

  After blacklisting nf-conntrack modules, cannot start firewalld

Version-Release number of selected component (if applicable):1.3.2


How reproducible: always


Steps to Reproduce:
1. Follow the instructions to install calamari from doc: http://10.34.3.139:8080/view/Ceph/job/doc-Red_Hat_Ceph_Storage-Installation_Guide_RHEL%20%28html-single%29/lastSuccessfulBuild/artifact/index.html.
2. As part of instructions, setup firewall on cluster nodes
3. Follow the step to blacklist nf-conntrack modules
4. Reboot the node as per instructions in the doc
5. After node comes up, run any firewalld command on it and it fails as firewalld is not running. 
6. Try starting the firewalld. It fails too.

Actual results:

Expected results:

Additional info:
Log:
---
[cephuser@magna092 ~]$ sudo systemctl start firewalld
[cephuser@magna092 ~]$ sudo systemctl enable firewalld
[cephuser@magna092 ~]$ sudo firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent
success
[cephuser@magna092 ~]$ sudo firewall-cmd --reload
success
[cephuser@magna092 ~]$ firewall-cmd --zone=public --list-ports
6800-7300/tcp
[cephuser@magna092 ~]$ sudo vi /etc/modprobe.d/conntrack.conf
[cephuser@magna092 ~]$ cat /etc/modprobe.d/conntrack.conf
blacklist nf_conntrack
blacklist nf_conntrack_ipv6
blacklist xt_conntrack
blacklist nf_conntrack_ftp
blacklist xt_state
blacklist iptable_nat
blacklist ipt_REDIRECT
blacklist nf_nat
blacklist nf_conntrack_ipv4
[cephuser@magna092 ~]$ sudo reboot
:
:
Warning: Permanently added 'magna092,10.8.128.92' (ECDSA) to the list of known hosts.
cephuser@magna092's password: 
Last login: Tue Feb  2 15:15:51 2016 from 10.8.128.2
[cephuser@magna092 ~]$ firewall-cmd --zone=public --list-ports
FirewallD is not running
[cephuser@magna092 ~]$ sudo systemctl start firewalld
^C (this command did not complete. Had to issue ctrl-c)

[cephuser@magna092 ~]$ sudo lsmod | grep conntrack
nf_conntrack          105737  2 nf_nat,nf_nat_ipv4
Comment 2 Ken Dreyer (Red Hat) 2016-02-03 18:36:37 EST
Kyle, in bz 1290923 you recommended blacklisting nf_conntrack, but I'm wondering if that is too blunt a hammer. It seems that this breaks firewalld.

http://ceph-users.ceph.narkive.com/koTRsuPb/nf-conntrack-overflow-crashes-osds

...this mailing list post indicates that setting the following sysctl values should do the trick:

net.netfilter.nf_conntrack_max = 1024000
net.nf_conntrack_max = 1024000

What do you think?
Comment 3 Ken Dreyer (Red Hat) 2016-02-04 16:26:12 EST
Kyle confirmed that we should pursue the sysctl option.

Harish can you please un-do the work to blacklist nf-conntrack, and then set 

net.netfilter.nf_conntrack_max = 1024000
net.nf_conntrack_max = 1024000

in /etc/sysctl.conf? To make the setting effective you can reboot (or run `sudo sysctl -p`, although you'd probably want to reboot anyway to un-blacklist the nf_conntrack modules).

to verify the changes, execute

sudo sysctl -a | grep nf_conntrack
Comment 4 Harish NV Rao 2016-02-05 04:51:16 EST
Hi Ken and Kyle,

Thanks for the clarification. Firewalld runs after setting the values mentioned in comment 3.

I am marking this bz as doc bz. 

The documentation needs to be changed in the "Blacklist the nf-conntrack modules" of http://10.34.3.139:8080/view/Ceph/job/doc-Red_Hat_Ceph_Storage-Installation_Guide_RHEL%20%28html-single%29/lastSuccessfulBuild/artifact/index.html#install-selinux.

Can you please check the existing documentation above and provide the right doc text to doc team based on new change?

Regards,
Harish

LOG:
----
[cephuser@magna023 ~]$ cat /etc/sysctl.conf 
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

net.netfilter.nf_conntrack_max = 1024000
net.nf_conntrack_max = 1024000

[cephuser@magna023 ~]$  sudo sysctl -a | grep conntrack_max
net.netfilter.nf_conntrack_max = 1024000
net.nf_conntrack_max = 1024000

[cephuser@magna023 ~]$ sudo service firewalld status
Redirecting to /bin/systemctl status  firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2016-02-05 09:39:43 UTC; 4min 4s ago
 Main PID: 1553 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─1553 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Feb 05 09:39:40 magna023 systemd[1]: Starting firewalld - dynamic firewall daemon...
Feb 05 09:39:43 magna023 systemd[1]: Started firewalld - dynamic firewall daemon.

[cephuser@magna023 ~]$ sudo firewall-cmd --zone=public --list-ports
6800-7300/tcp
Comment 8 Harish NV Rao 2016-02-07 03:42:42 EST
Hi Ken,

I followed the steps you provided in the gitlab on one of my cluster nodes. Firewalld comes up fine after the reboot and firewall settings are retained.

One correction needed to the steps is that all systemctl cmds should be with sudo. Please change the doc accordingly.

Regards,
Harish
Comment 9 Ken Dreyer (Red Hat) 2016-02-08 16:20:37 EST
Thanks Harish! I've added sudo to the systemctl commands in my latest version.

For simplicity, I'm going to go ahead and close this as a dup of Kyle's original docs bug, 1290923, and we can track progress there.

*** This bug has been marked as a duplicate of bug 1290923 ***

Note You need to log in before you can comment on or make changes to this bug.