Description of problem: ssh into a server behind NAT (in my case it's the undercloud VM inside Red Hat Openstack 13) - will be blocked by firewalld of the RHEL host, even when configuring port forwarding. That's the objective: ssh from laptop --> Host with port forwarding in firewall --> Get directly into guest (172.16.0.2, behind host NAT). Version-Release number of selected component (if applicable): RedHatEnterpriseServer 7.5 3.10.0-862.3.2.el7.x86_64 How reproducible: Always. Steps to Reproduce: Configure firewalld on Host, to have ssh tunnel into guest (undercloud 172.16.0.2): # firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=172.16.0.2' # firewall-cmd --permanent --zone=public --add-masquerade # firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -d 0.0.0.0/0 -j ACCEPT # firewall-cmd --permanent --direct --add-rule ipv4 filter OUTPUT 0 -d 0.0.0.0/0 -j ACCEPT # firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -d 0.0.0.0/0 -j ACCEPT # firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 0 -d 0.0.0.0/0 -p tcp --dport 222 -j DNAT --to-destination 172.16.0.2:22 # firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="0.0.0.0/0" forward-port port="222" protocol="tcp" to-port="22" to-addr='"172.16.0.2" # firewall-cmd --reload # firewall-cmd --list-all public (active) target: default icmp-block-inversion: no interfaces: enp4s0f0 sources: services: ssh dhcpv6-client ports: 8139/tcp protocols: masquerade: yes forward-ports: port=222:proto=tcp:toport=22:toaddr=172.16.0.2 source-ports: icmp-blocks: rich rules: rule family="ipv4" destination address="0.0.0.0/0" forward-port port="222" protocol="tcp" to-port="22" to-addr="172.16.0.2" # firewall-cmd --direct --get-all-rules ipv4 filter INPUT 0 -d 0.0.0.0/0 -j ACCEPT ipv4 filter OUTPUT 0 -d 0.0.0.0/0 -j ACCEPT ipv4 filter FORWARD 0 -d 0.0.0.0/0 -j ACCEPT ipv4 nat PREROUTING 0 -d 0.0.0.0/0 -p tcp --dport 222 -j DNAT --to-destination 172.16.0.2:22 Actual results: Now, when trying to connect to the guest - from my laptop, via host port 222 - the ssh connection is refused: ssh -l stack my-host -p 222 ssh: connect to host my-host port 222: Connection refused Expected results: SSH should work, and log into the guest (172.16.0.2). Workaround - Use iptables instead of firewald: # iptables -I OUTPUT -d 0.0.0.0/0 -j ACCEPT # iptables -I FORWARD -d 0.0.0.0/0 -j ACCEPT # iptables -I INPUT -d 0.0.0.0/0 -j ACCEPT # iptables -t nat -I PREROUTING -d 0.0.0.0/0 -p tcp --dport 222 -j DNAT --to-destination 172.16.0.2:22 However, it will not be saved on host reboot. Additional info: - Opened issue in https://serverfault.com/questions/915257/ssh-port-forwarding-with-firewall-cmd - This bug might be related to #904098.
Why are you adding the --direct rules? Forward ports should be enough to do what you want. You also should not need the rich rule.
(In reply to Eric Garver from comment #2) > Why are you adding the --direct rules? Forward ports should be enough to do > what you want. You also should not need the rich rule. I did try any of the options above [Forward ports | Direct rule | Rich rule], separately, and in combination - but none resolved it.
Can you show the output of the following? - iptables -t filter -L - iptables -t nat -L firewalld's forward-ports feature uses packet marks. It's possible OpenStack is overwriting them.
Please also show - iptables -t mangle -L
(In reply to Eric Garver from comment #6) > Please also show > > - iptables -t mangle -L [root@panther24 ~]# iptables -t filter -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- 10.0.0.0/24 anywhere udp dpt:6234 /* Infrared: vbmc ports from overcloud */ ACCEPT udp -- 10.0.0.0/24 anywhere udp dpt:6233 /* Infrared: vbmc ports from overcloud */ ACCEPT udp -- 10.0.0.0/24 anywhere udp dpt:6232 /* Infrared: vbmc ports from overcloud */ ACCEPT udp -- 10.0.0.0/24 anywhere udp dpt:6231 /* Infrared: vbmc ports from overcloud */ ACCEPT udp -- 10.0.0.0/24 anywhere udp dpt:6230 /* Infrared: vbmc ports from overcloud */ ACCEPT udp -- panther24.lab.eng.tlv2.redhat.com anywhere udp dpt:6234 /* Infrared: vbmc ports */ ACCEPT udp -- panther24.lab.eng.tlv2.redhat.com anywhere udp dpt:6233 /* Infrared: vbmc ports */ ACCEPT udp -- panther24.lab.eng.tlv2.redhat.com anywhere udp dpt:6232 /* Infrared: vbmc ports */ ACCEPT udp -- panther24.lab.eng.tlv2.redhat.com anywhere udp dpt:6231 /* Infrared: vbmc ports */ ACCEPT udp -- panther24.lab.eng.tlv2.redhat.com anywhere udp dpt:6230 /* Infrared: vbmc ports */ ACCEPT udp -- 172.16.0.80 anywhere udp dpt:6234 /* Infrared: vbmc ports */ ACCEPT udp -- 172.16.0.80 anywhere udp dpt:6233 /* Infrared: vbmc ports */ ACCEPT udp -- 172.16.0.80 anywhere udp dpt:6232 /* Infrared: vbmc ports */ ACCEPT udp -- 172.16.0.80 anywhere udp dpt:6231 /* Infrared: vbmc ports */ ACCEPT udp -- 172.16.0.80 anywhere udp dpt:6230 /* Infrared: vbmc ports */ ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT tcp -- anywhere 10.0.0.107 /* Infrared: forward overcloud dashboard */ ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable ACCEPT all -- anywhere 172.16.0.0/24 ctstate RELATED,ESTABLISHED ACCEPT all -- 172.16.0.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable ACCEPT all -- anywhere 10.0.0.0/24 ctstate RELATED,ESTABLISHED ACCEPT all -- 10.0.0.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:bootpc ACCEPT udp -- anywhere anywhere udp dpt:bootpc ACCEPT udp -- anywhere anywhere udp dpt:bootpc ACCEPT udp -- anywhere anywhere udp dpt:bootpc [root@panther24 ~]# [root@panther24 ~]# iptables -t nat -L Chain PREROUTING (policy ACCEPT) target prot opt source destination DNAT tcp -- anywhere panther24.lab.eng.tlv2.redhat.com tcp dpt:webcache /* Infrared: forward overcloud dashboard */ to:10.0.0.107:80 Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination RETURN all -- 172.16.0.0/24 224.0.0.0/24 RETURN all -- 172.16.0.0/24 255.255.255.255 MASQUERADE tcp -- 172.16.0.0/24 !172.16.0.0/24 masq ports: 1024-65535 MASQUERADE udp -- 172.16.0.0/24 !172.16.0.0/24 masq ports: 1024-65535 MASQUERADE all -- 172.16.0.0/24 !172.16.0.0/24 RETURN all -- 10.0.0.0/24 224.0.0.0/24 RETURN all -- 10.0.0.0/24 255.255.255.255 MASQUERADE tcp -- 10.0.0.0/24 !10.0.0.0/24 masq ports: 1024-65535 MASQUERADE udp -- 10.0.0.0/24 !10.0.0.0/24 masq ports: 1024-65535 MASQUERADE all -- 10.0.0.0/24 !10.0.0.0/24 RETURN all -- 192.168.122.0/24 224.0.0.0/24 RETURN all -- 192.168.122.0/24 255.255.255.255 MASQUERADE tcp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535 MASQUERADE udp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535 MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 [root@panther24 ~]# [root@panther24 ~]# iptables -t mangle -L Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination CHECKSUM udp -- anywhere anywhere udp dpt:bootpc CHECKSUM fill CHECKSUM udp -- anywhere anywhere udp dpt:bootpc CHECKSUM fill CHECKSUM udp -- anywhere anywhere udp dpt:bootpc CHECKSUM fill [root@panther24 ~]#
The firewalld rules are not present. This means either a) firewalld is not running or b) OpenStack flushed all the rules after firewalld added them. Which explains what you're seeing. Do you have anything in /var/log/firewalld? It might give us clues.
Created attachment 1498872 [details] Problem reproduced. Including /var/log/firewalld and iptables trace
[nmanos@nmanos ~]$ ssh root.eng.tlv2.redhat.com Last login: Mon Oct 29 10:07:52 2018 from dhcp-4-77.tlv.redhat.com ** ** ** NOTE: This server is under puppet control! Local changes may be lost. ** ** ** ** You are on titan08.lab.eng.tlv2.redhat.com This host belongs to the Default hostgroup owned by Noam Manos (nmanos) CPU: 2 Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz RAM: 125.63 GiB OS: RedHat 7.5 ** ** ** NOTE: This server is under Puppet control! Local changes may be lost. ** ** ** ** [root@titan08 ~]# ll -a /var/log/firewalld -rw-r--r--. 1 root root 0 Aug 30 11:31 /var/log/firewalld [root@titan08 ~]# cat /var/log/firewalld [root@titan08 ~]# [root@titan08 ~]# [root@titan08 ~]# systemctl status firewalld ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2018-10-03 15:21:57 IDT; 3 weeks 5 days ago Docs: man:firewalld(1) Main PID: 1090 (firewalld) Tasks: 2 CGroup: /system.slice/firewalld.service └─1090 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid Oct 03 15:21:57 titan08.lab.eng.tlv2.redhat.com systemd[1]: Starting firewalld - dynamic firewall daemon... Oct 03 15:21:57 titan08.lab.eng.tlv2.redhat.com systemd[1]: Started firewalld - dynamic firewall daemon. [root@titan08 ~]# # Login to guest undercloud (ssh to port 22): [root@titan08 ~]# ssh stack@undercloud-0 Last login: Mon Oct 29 04:10:46 2018 from 172.16.0.1 [stack@undercloud-0 ~]$ [stack@undercloud-0 ~]$ [stack@undercloud-0 ~]$ pwd /home/stack [stack@undercloud-0 ~]$ exit # Get undercloud ip and login to undercloud ip (ssh to port 22): [root@titan08 ~]# ipv4_undercloud=`virsh net-dhcp-leases management| grep undercloud| awk '{print $5}'| grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" ` [root@titan08 ~]# echo $ipv4_undercloud 172.16.0.83 [root@titan08 ~]# ssh stack.0.83 Last login: Tue Oct 30 05:20:26 2018 from 172.16.0.1 [stack@undercloud-0 ~]$ [stack@undercloud-0 ~]$ pwd /home/stack [stack@undercloud-0 ~]$ exit # Configure firewalld on Host, to have ssh tunnel into guest (ipv4_undercloud = 172.16.0.83): [root@titan08 ~]# firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=$ipv4_undercloud success [root@titan08 ~]# firewall-cmd --permanent --zone=public --add-masquerade success [root@titan08 ~]# firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -d 0.0.0.0/0 -j ACCEPT success [root@titan08 ~]# firewall-cmd --permanent --direct --add-rule ipv4 filter OUTPUT 0 -d 0.0.0.0/0 -j ACCEPT success [root@titan08 ~]# firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -d 0.0.0.0/0 -j ACCEPT success [root@titan08 ~]# firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 0 -d 0.0.0.0/0 -p tcp --dport 222 -j DNAT --to-destination $ipv4_undercloud:22 success [root@titan08 ~]# firewall-cmd --permanent --add-rich-rule="rule family=ipv4 source address=0.0.0.0/0 forward-port port=222 protocol=tcp to-port=22 to-addr=$ipv4_undercloud" success [root@titan08 ~]# firewall-cmd --reload success [root@titan08 ~]# firewall-cmd --list-all public (active) target: default icmp-block-inversion: no interfaces: ens1f0 sources: services: ssh dhcpv6-client ports: protocols: masquerade: yes forward-ports: port=222:proto=tcp:toport=22:toaddr=172.16.0.83 source-ports: icmp-blocks: rich rules: rule family="ipv4" source address="0.0.0.0/0" forward-port port="222" protocol="tcp" to-port="22" to-addr="172.16.0.83" [root@titan08 ~]# firewall-cmd --direct --get-all-rules ipv4 filter OUTPUT 0 -d 0.0.0.0/0 -j ACCEPT ipv4 filter FORWARD 0 -d 0.0.0.0/0 -j ACCEPT ipv4 nat PREROUTING 0 -d 0.0.0.0/0 -p tcp --dport 222 -j DNAT --to-destination 172.16.0.83:22 ipv4 filter INPUT 0 -d 0.0.0.0/0 -j ACCEPT [root@titan08 ~]# exit # Try to connect to guest undercloud with ssh tunnel on port 222 - connection refused: [nmanos@nmanos ~]$ ssh stack.eng.tlv2.redhat.com -p 222 ssh: connect to host titan08.lab.eng.tlv2.redhat.com port 222: Connection refused [nmanos@nmanos ~]$ ssh -l stack titan08.lab.eng.tlv2.redhat.com -p 222 ssh: connect to host titan08.lab.eng.tlv2.redhat.com port 222: Connection refused # --------------------------- [root@titan08 ~]# [root@titan08 ~]# systemctl status firewalld -l ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2018-10-03 15:21:57 IDT; 3 weeks 5 days ago Docs: man:firewalld(1) Main PID: 1090 (firewalld) Tasks: 2 CGroup: /system.slice/firewalld.service └─1090 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid Oct 30 11:18:19 titan08.lab.eng.tlv2.redhat.com firewalld[1090]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --destination 192.168.122.0/24 --out-interface virbr0 --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Oct 30 11:18:19 titan08.lab.eng.tlv2.redhat.com firewalld[1090]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --source 192.168.122.0/24 --in-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Oct 30 11:18:19 titan08.lab.eng.tlv2.redhat.com firewalld[1090]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --in-interface virbr0 --out-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Oct 30 11:18:19 titan08.lab.eng.tlv2.redhat.com firewalld[1090]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --out-interface virbr0 --jump REJECT' failed: iptables: No chain/target/match by that name. Oct 30 11:18:19 titan08.lab.eng.tlv2.redhat.com firewalld[1090]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --in-interface virbr0 --jump REJECT' failed: iptables: No chain/target/match by that name. Oct 30 11:18:19 titan08.lab.eng.tlv2.redhat.com firewalld[1090]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Oct 30 11:18:19 titan08.lab.eng.tlv2.redhat.com firewalld[1090]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol tcp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Oct 30 11:18:19 titan08.lab.eng.tlv2.redhat.com firewalld[1090]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete OUTPUT --out-interface virbr0 --protocol udp --destination-port 68 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Oct 30 11:18:19 titan08.lab.eng.tlv2.redhat.com firewalld[1090]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Oct 30 11:18:19 titan08.lab.eng.tlv2.redhat.com firewalld[1090]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). [root@titan08 ~]#
As I mentioned in comment 8, none of the firewalld rules are present. Something must have removed them. I would guess it occurred implicitly by something using iptables-restore without the "-n" option. AFAIK, OpenStack deployment guides suggest disabling firewalld. I guess they don't play nice. https://docs.openstack.org/project-deploy-guide/openstack-ansible/draft/deploymenthost.html#configure-the-operating-system-centos I don't think there is anything we can fix on the firewalld side. Maybe you should reassign this to OpenStack.
As suggested by Eric, this issue is related to OSP deployment, thus changing bug owners to TripleO. Note that it is known to effect OSP 14 and bellow.
(In reply to Noam Manos from comment #12) > As suggested by Eric, this issue is related to OSP deployment, thus changing > bug owners to TripleO. > > Note that it is known to effect OSP 14 and bellow. No, this is not correct based on how I interpret the bug. You're trying to add firewall rules on the host (titan08) that is hosting a vm called undercloud-0. undercloud-0 is where OpenStack is installed. But there are no OpenStack rpm's or configuration done on the physical host itself (titan08). In that case, it seems that this is entirely about the configuration of the firewall of the physical host and not the OpenStack vm. As Eric pointed out, the rules you are trying to add with firewalld are not getting saved on the physical host (titan08). Is my understanding correct? When you install OpenStack on a VM, it makes no changes to the physical host itself.