Bug 1304004

Summary: 1.3.2: Installation: After blacklisting nf-conntrack modules, cannot start firewalld
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Harish NV Rao <hnallurv>
Component: DocumentationAssignee: ceph-docs <ceph-docs>
Status: CLOSED DUPLICATE QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 1.3.2CC: flucifre, kdreyer, ksquizza, ngoswami
Target Milestone: rc   
Target Release: 1.3.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-08 21:20:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1290923    

Description Harish NV Rao 2016-02-02 16:34:55 UTC
Description of problem:

  After blacklisting nf-conntrack modules, cannot start firewalld

Version-Release number of selected component (if applicable):1.3.2


How reproducible: always


Steps to Reproduce:
1. Follow the instructions to install calamari from doc: http://10.34.3.139:8080/view/Ceph/job/doc-Red_Hat_Ceph_Storage-Installation_Guide_RHEL%20%28html-single%29/lastSuccessfulBuild/artifact/index.html.
2. As part of instructions, setup firewall on cluster nodes
3. Follow the step to blacklist nf-conntrack modules
4. Reboot the node as per instructions in the doc
5. After node comes up, run any firewalld command on it and it fails as firewalld is not running. 
6. Try starting the firewalld. It fails too.

Actual results:

Expected results:

Additional info:
Log:
---
[cephuser@magna092 ~]$ sudo systemctl start firewalld
[cephuser@magna092 ~]$ sudo systemctl enable firewalld
[cephuser@magna092 ~]$ sudo firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent
success
[cephuser@magna092 ~]$ sudo firewall-cmd --reload
success
[cephuser@magna092 ~]$ firewall-cmd --zone=public --list-ports
6800-7300/tcp
[cephuser@magna092 ~]$ sudo vi /etc/modprobe.d/conntrack.conf
[cephuser@magna092 ~]$ cat /etc/modprobe.d/conntrack.conf
blacklist nf_conntrack
blacklist nf_conntrack_ipv6
blacklist xt_conntrack
blacklist nf_conntrack_ftp
blacklist xt_state
blacklist iptable_nat
blacklist ipt_REDIRECT
blacklist nf_nat
blacklist nf_conntrack_ipv4
[cephuser@magna092 ~]$ sudo reboot
:
:
Warning: Permanently added 'magna092,10.8.128.92' (ECDSA) to the list of known hosts.
cephuser@magna092's password: 
Last login: Tue Feb  2 15:15:51 2016 from 10.8.128.2
[cephuser@magna092 ~]$ firewall-cmd --zone=public --list-ports
FirewallD is not running
[cephuser@magna092 ~]$ sudo systemctl start firewalld
^C (this command did not complete. Had to issue ctrl-c)

[cephuser@magna092 ~]$ sudo lsmod | grep conntrack
nf_conntrack          105737  2 nf_nat,nf_nat_ipv4

Comment 2 Ken Dreyer (Red Hat) 2016-02-03 23:36:37 UTC
Kyle, in bz 1290923 you recommended blacklisting nf_conntrack, but I'm wondering if that is too blunt a hammer. It seems that this breaks firewalld.

http://ceph-users.ceph.narkive.com/koTRsuPb/nf-conntrack-overflow-crashes-osds

...this mailing list post indicates that setting the following sysctl values should do the trick:

net.netfilter.nf_conntrack_max = 1024000
net.nf_conntrack_max = 1024000

What do you think?

Comment 3 Ken Dreyer (Red Hat) 2016-02-04 21:26:12 UTC
Kyle confirmed that we should pursue the sysctl option.

Harish can you please un-do the work to blacklist nf-conntrack, and then set 

net.netfilter.nf_conntrack_max = 1024000
net.nf_conntrack_max = 1024000

in /etc/sysctl.conf? To make the setting effective you can reboot (or run `sudo sysctl -p`, although you'd probably want to reboot anyway to un-blacklist the nf_conntrack modules).

to verify the changes, execute

sudo sysctl -a | grep nf_conntrack

Comment 4 Harish NV Rao 2016-02-05 09:51:16 UTC
Hi Ken and Kyle,

Thanks for the clarification. Firewalld runs after setting the values mentioned in comment 3.

I am marking this bz as doc bz. 

The documentation needs to be changed in the "Blacklist the nf-conntrack modules" of http://10.34.3.139:8080/view/Ceph/job/doc-Red_Hat_Ceph_Storage-Installation_Guide_RHEL%20%28html-single%29/lastSuccessfulBuild/artifact/index.html#install-selinux.

Can you please check the existing documentation above and provide the right doc text to doc team based on new change?

Regards,
Harish

LOG:
----
[cephuser@magna023 ~]$ cat /etc/sysctl.conf 
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

net.netfilter.nf_conntrack_max = 1024000
net.nf_conntrack_max = 1024000

[cephuser@magna023 ~]$  sudo sysctl -a | grep conntrack_max
net.netfilter.nf_conntrack_max = 1024000
net.nf_conntrack_max = 1024000

[cephuser@magna023 ~]$ sudo service firewalld status
Redirecting to /bin/systemctl status  firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2016-02-05 09:39:43 UTC; 4min 4s ago
 Main PID: 1553 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─1553 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Feb 05 09:39:40 magna023 systemd[1]: Starting firewalld - dynamic firewall daemon...
Feb 05 09:39:43 magna023 systemd[1]: Started firewalld - dynamic firewall daemon.

[cephuser@magna023 ~]$ sudo firewall-cmd --zone=public --list-ports
6800-7300/tcp

Comment 8 Harish NV Rao 2016-02-07 08:42:42 UTC
Hi Ken,

I followed the steps you provided in the gitlab on one of my cluster nodes. Firewalld comes up fine after the reboot and firewall settings are retained.

One correction needed to the steps is that all systemctl cmds should be with sudo. Please change the doc accordingly.

Regards,
Harish

Comment 9 Ken Dreyer (Red Hat) 2016-02-08 21:20:37 UTC
Thanks Harish! I've added sudo to the systemctl commands in my latest version.

For simplicity, I'm going to go ahead and close this as a dup of Kyle's original docs bug, 1290923, and we can track progress there.

*** This bug has been marked as a duplicate of bug 1290923 ***