Bug 1079088 - RFE: network: add explicit XML support for virtual network port forwarding
Summary: RFE: network: add explicit XML support for virtual network port forwarding
Status: NEW
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt (Show other bugs)
(Show other bugs)
Version: unspecified
Hardware: All Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
: 995480 1031813 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-03-21 00:27 UTC by prismalytics
Modified: 2018-07-18 14:25 UTC (History)
24 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

Description prismalytics 2014-03-21 00:27:27 UTC
Hello:

I performed the following, which doesn't work, on a kvmHost to enable incoming TCP requests destined for kvmHost:8888 to be forwarded to, say, kvmGuest01:8888? (i.e. to one of it's KVM guests). 

=========================================
Noting that:
=========================================
    - the active zone is the public zone
    - the host network is 192.168.0.0/24
    - the kvmHost IP address is 192.168.0.15
    - the guest network is 192.168.122.0/24
    - the the kvmGuest01 IP address is 192.168.122.180
    - and SELinux is disabled.
=========================================

I tried the following commands (on kvmHost), which does not work:

  root@kvmHost# firewall-cmd --zone=public --add-port=8888/tcp
  root@kvmHost# firewall-cmd --zone=public --add-masquerade
  root@kvmHost# firewall-cmd --zone=public --add-forward-port=port=8888:proto=tcp:toport=8888:toaddr=192.168.122.180

# I also tried using different host/quest ports, just in case:
  root@kvmHost# firewall-cmd --zone=public --add-forward-port=port=18888:proto=tcp:toport=8888:toaddr=192.168.122.180

Perhaps the above commands are necessary, but apparently they are not sufficient. For example, when I do this:

user@kvmHost$ telnet kvmGuest01 8888
Trying 192.168.122.180...
Connected to kvmGuest01.
Escape character is '^]'.
^]<cr>
user@kvmHost$

The above works as expected because we are specifying kvmGuest01 directly (not forwarding). But the following does not work, and is what I need to work (i.e. to forward from kvmHost to kvmGuest01):

user@kvmHost$ telnet kvmHost 8888 -or- telnet kvmHost 18888
Trying 192.168.0.15...
telnet: connect to address 192.168.0.15: Connection refused

Note that the above firewall commands were applied ony to Fedora kvmHost.

I don't think this works in a KVM/QEMU setting. I tried everything I know and have read about. Thanks in advance.

NMV

Comment 1 Jiri Popelka 2014-03-21 15:41:28 UTC
Thank you for the description Noel,

yes, I can reproduce it. 
I'm not sure now what causes it, but I guess it's some unexpected combination of firewalld's and libvirtd's rules.
I played recently (bug #1076857) with port-forwarding in firewalld and I believe that itself works ok.

I'm away next week, but Thomas will hopefully take a look at this.

Comment 2 prismalytics 2014-03-21 18:47:35 UTC
Hi Jiri (and friends):

Thank you for looking into it. I hope you enjoy your time away next week. :)

I think you're right about some interaction between firewalld and libvirtd
casing the issue.

I'll add one or two side notes here that I didn't mention in my opening
Comment, because I didn't want to distract from it.

And they are that I also tried various lower-level alternatives to using firewall-cmd, including:

    - The KVM/QEMU method described here: http://bit.ly/1kQ6OtO

      That doesn't work because, oddly, when I edit the XML definition file
      for the VM (i.e. kvmGuest01.xml) as prescribed in that article
      (via 'root@kvmHost# virsh edit kvmGuest01'), when I exit out of that
      vi(1)-like session, libvirtd undoes all of the changes I made. The VM
      is shutdown when I do this, of course. So next I tried editing the file
      manually with vi(1) by shutting down the libvirtd service first, then
      making the edits, and then starting up the libvirtd service again. Sadly,
      as soon as I start libvirtd, it too undoes the changes I made. And note 
      that I verified the syntactic correctness of my XML edits using
      xmllint(1), so the roolbacks weren't due to syntax errors. Libvirtd just
      wants to be too smart I guess. :). Either way, this isn't an elegant
      way of doing this these days (several years ago maybe).


    - Next, I tried old-school manual iptables(8) entry additions on 'kvmHost'.
      In this case, no matter what combinations I tried I couldn't get that
      to work either.

It's possible that I didn't get the iptables(8) entries quite correct and/or
it's possible that I missed a step while editing the 'kvmGuest01.xml' file
and that is why libvirtd keeps reverting my changes.

Which leads me to the reason I decided to file this bug: While those
additional under-the-hood methods may work, employing them kind of gets away
from -- in my estimation -- one of the goals of firewalld/firewall-cmd, which
is to streamline and simplify the administration of firewall behaviors. :).

So I said maybe the gals/guys at Fedora/Redhat can help fix this more elegantly
with firewalld.

I personally make heavy use of Fedora VMs (both KVM & LXC) for big-data
environment simulations (Storm, Hadoop, etc.), and so I desperately need this port-forwarding between kvmHost and kvmGuests to work.

I hope that additional context helps. =:)

Thank you!
Noel

Comment 3 prismalytics 2014-04-07 20:12:04 UTC
Hello:

Will there be any progress on this bug? Appreciated.

Thank you,
Noel

Comment 4 Jiri Popelka 2014-04-10 14:02:57 UTC
*** Bug 995480 has been marked as a duplicate of this bug. ***

Comment 5 Jiri Popelka 2014-04-10 14:03:15 UTC
*** Bug 1031813 has been marked as a duplicate of this bug. ***

Comment 6 Jiri Popelka 2014-04-10 14:04:29 UTC
Actually we've already had 2 such reports, bug #1031813 and especially bug #995480, which reveals the culprit of the problem and possible "work-around".

I'll summarize it here. In the next example my kvmHost has em1 iface and kvmGuest has virbr0 ('NAT to em1' network, 192.168.0.0/24),
both ifaces being in default - public zone.

# iptables-save -t filter

-A FORWARD -d 192.168.0.0/24 -i em1 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
 /\ 
 |   added by libvirtd  
-----------------------------------
 |   added by firewalld 
 \/
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD_IN_ZONES -i em1 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FWDI_public -j FWDI_public_allow
-A FWDI_public_allow -m conntrack --ctstate NEW -m mark --mark 0x64 -j ACCEPT

Libvirtd inserts all its rules on top of the FORWARD chain and the '-o virbr0 -j REJECT' prevents firewalld's port-forwarding rule ('--mark 0x64 -j ACCEPT') to be applied.

As bug #995480, comment #1 suggests the work-around could be in deleting the libvirtd's "-o virbr0 -j REJECT" rule.
One might probably put
iptables -D FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
into /etc/libvirt/hooks/qemu as suggested by
http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections

But I'm not sure what/where would be the correct solution, it'd probably need bigger redesign.
Libvirtd might probably avoid using the '-o virbr0 -j REJECT' rule if it uses firewalld because firewalld already has
'-j REJECT --reject-with icmp-host-prohibited' at the bottom of FORWARD chain. Any ideas, Thomas ?

Comment 7 Thomas Woerner 2014-06-03 13:47:08 UTC
Is this REJECT rule for -A FORWARD -o virbr0 really needed? firewalld is rejecting everything that is not covered by another rule in the end.

Assigning to libvirt for verification.

Comment 8 Laine Stump 2014-06-03 15:23:53 UTC
First, some background. Here is a description of libvirt's use of iptables:

  http://libvirt.org/firewall.html

In particular, the interesting part is under the heading "The virtual network driver".

As you mention above, I've had success with the rules added by the script at
http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections

Of course this works because in addition to the DNAT rule in the nat table, an ACCEPT rule for the particular port and IP address is at the top of the chain.

As for the two REJECT rules that libvirt adds, they are added to ensure that there is no inbound traffic to the guests on a nat network (other than from the host itself, or from other guests on the same network). Because libvirt has no information about any network security beyond that which it adds itself, it needs to add all the rules necessary to ensure guests on the network are allowed access only as defined in the libvirt documentation. In particular, guests on a libvirt network with <forward mode='nat'/> should not get any incoming connections from the outside.

You may at first think that simply by virtue of being in a private network space, the guest wouldn't be reachable. However, a simple experiment shows that an attacker on the same subnet as the host can simply add a route with the private network as destination and the virtualization host's IP as gateway, and if the rule in question is missing, will be able to (e.g.) ssh into a guest on the private network.

(Note that with my Fedora 20 machine this didn't work until I also removed another rule apparently added by firewalld:

  -A FORWARD -j REJECT --reject-with icmp-host-prohibited

this was the very last of the FORWARD rules. Still, libvirt historically has not been able to rely on any other entity taking care of any aspect of network security for them, so we can't just assume that rule is in place.)

Aside from that, another thing this rule is *intended* to protect against (but fails in some cases) is attempts to reach one private nat-mode network from another. In practice, the ordering of the rules ends up allow the connection in one direction but not the other, but fortunately that hasn't seemed to bother anybody yet, so it has never been a high priority to fix (if you have ideas, we'd love to hear them!).

So, I think that yes the REJECT rule is necessary, but we are open to alternatives that give the same effect. In particular, we would be *very* happy to see a solution that also solved the problem mentioned in the previous paragraph.

Comment 9 Laine Stump 2014-06-03 15:31:28 UTC
(In reply to nmvega from comment #2)
> And they are that I also tried various lower-level alternatives to using
> firewall-cmd, including:
> 
>     - The KVM/QEMU method described here: http://bit.ly/1kQ6OtO
> 
>       That doesn't work because, oddly, when I edit the XML definition file
>       for the VM (i.e. kvmGuest01.xml) as prescribed in that article
>       (via 'root@kvmHost# virsh edit kvmGuest01'), when I exit out of that
>

I haven't taken the time to see exactly what went wrong that caused your changes to be removed (libvirt does silently discard any elements that it doesn't recognize, so possibly the elements were added at the wrong place), but I did want to point out that the method of forwarding ports described in that article will only work for user mode networking, not for network connections that are made using tap devices and a Linux host bridge (as is the case with libvirt's virtual networks).

BTW, in the end I think what libvirt would most like to have would be a new element for <interface> that would allow specifying port forwarding directly in the guest config (similar to the way you can add packet filter rules with a <filterref>). There was a patch to do that only for usermode networking a year or so ago, but it languished because nobody would commit to the syntax without confirming that the same syntax would also work for port forwarding in the case of tap-based network devices.

Comment 10 Daniel Berrange 2014-06-03 15:46:14 UTC
> BTW, in the end I think what libvirt would most like to have would be a new
> element for <interface> that would allow specifying port forwarding directly
> in the guest config (similar to the way you can add packet filter rules with
> a <filterref>). There was a patch to do that only for usermode networking a
> year or so ago, but it languished because nobody would commit to the syntax
> without confirming that the same syntax would also work for port forwarding
> in the case of tap-based network devices.

Yes, people have been asking for port forwarding for our virtual networks for sooooooooo many years, it is well overdue for us to actually give them what they want, rather than force them to try to set it up behind libvirt's back and hit problems there.

Comment 11 prismalytics 2014-08-22 20:20:58 UTC
Hello again...

Seeing that this is taking a while (and with no immediate resolution in site), I have a question, because **for several years now** I keep slamming into this *show-stopping* issue whenever I need to simulate some BIG-DATA cluster completely inside a KVM network, and need to interact with it from an outside that network.

My latest issie/problem,
my question,
My hope for a work-around offed by my friends here:

My KVM machines are used to simulate a HADOOP/YARN compute & storage cluster. YARN-aware application drivers/clients sitting on the KVM Host (not Guest), issue requests to the YARN ResourceManager (RM) -- which runs inside the KVM network -- for cluster resources on which to run their application. The YARN RM then allocates cluster resousces, launches the application *somewhere* in the cluster, and finally supplies the extarnal application driver/client with a KVM-internal IP and *arbitrary* PORT that it should connect to, to interact with the now running distributed application... say, to receive standard output.

Since it is not possible to know which port will be returned by YARN, what can I do to allow *ALL* ports above port 1024? Is there a simple firewall-cmd syntax to do this? Maybe I missed it somewhere.

P.S. Port Fwarding still doesn't work, but that's a separate issue (the one I filed this bug about). But my question here, is a different one.

Thank you,
NMV

Comment 12 Aurelien Bompard 2015-01-17 11:55:01 UTC
This bug is still valid on F21, it took me a few hours to understand where that was coming from.

Comment 13 Cole Robinson 2015-04-28 14:44:24 UTC
Setting to NEW, since ASSIGNED state was just inherited from the firewalld componet

Comment 14 Aurelien Bompard 2015-09-05 08:14:23 UTC
This bug is still valid on F22.

Comment 15 wrob0123 2015-12-11 00:09:06 UTC
Sounds like the libvirt hooks (https://libvirt.org/hooks.html) are not supported in Fedora or RHEL. I would like to have some sort of that capability. 

Specifically I am interested in changing a macvtap device to have ALLMULTI turned on after libvirt creates it.

Comment 16 Cole Robinson 2016-03-17 01:02:51 UTC
Repurposing to track port forwarding for virtual networks, like dan suggests in comment #10

Comment 17 Gerardo Rosales 2017-11-19 12:32:39 UTC
Hi guys,

I'm getting the same bug with a centos 7 lab machine trying to do the same as the first comment.

Is there any update regarding it? Maybe a way to work around it?

Thanks in advanced.

Comment 18 Gerardo Rosales 2017-11-19 13:16:22 UTC
I found a way to workaround the issue using socat

'socat TCP-LISTEN:<LOCAL PORT>,fork TCP:<VM IP>:<VM PORT>'

The server is using the script hack in here:

http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections

and also I've allowed the same host port on Firewalld.

With all that in place I'm able to RDP the windows VM successfully outside the network.

I'm not sure if all above is needed. I'll test taking out things from the current setup to be sure.


Note You need to log in before you can comment on or make changes to this bug.