Thank you for the description Noel,

yes, I can reproduce it. 
I'm not sure now what causes it, but I guess it's some unexpected combination of firewalld's and libvirtd's rules.
I played recently (bug #1076857) with port-forwarding in firewalld and I believe that itself works ok.

I'm away next week, but Thomas will hopefully take a look at this.

Hi Jiri (and friends):

Thank you for looking into it. I hope you enjoy your time away next week. :)

I think you're right about some interaction between firewalld and libvirtd
casing the issue.

I'll add one or two side notes here that I didn't mention in my opening
Comment, because I didn't want to distract from it.

And they are that I also tried various lower-level alternatives to using firewall-cmd, including:

    - The KVM/QEMU method described here: http://bit.ly/1kQ6OtO

      That doesn't work because, oddly, when I edit the XML definition file
      for the VM (i.e. kvmGuest01.xml) as prescribed in that article
      (via 'root@kvmHost# virsh edit kvmGuest01'), when I exit out of that
      vi(1)-like session, libvirtd undoes all of the changes I made. The VM
      is shutdown when I do this, of course. So next I tried editing the file
      manually with vi(1) by shutting down the libvirtd service first, then
      making the edits, and then starting up the libvirtd service again. Sadly,
      as soon as I start libvirtd, it too undoes the changes I made. And note 
      that I verified the syntactic correctness of my XML edits using
      xmllint(1), so the roolbacks weren't due to syntax errors. Libvirtd just
      wants to be too smart I guess. :). Either way, this isn't an elegant
      way of doing this these days (several years ago maybe).


    - Next, I tried old-school manual iptables(8) entry additions on 'kvmHost'.
      In this case, no matter what combinations I tried I couldn't get that
      to work either.

It's possible that I didn't get the iptables(8) entries quite correct and/or
it's possible that I missed a step while editing the 'kvmGuest01.xml' file
and that is why libvirtd keeps reverting my changes.

Which leads me to the reason I decided to file this bug: While those
additional under-the-hood methods may work, employing them kind of gets away
from -- in my estimation -- one of the goals of firewalld/firewall-cmd, which
is to streamline and simplify the administration of firewall behaviors. :).

So I said maybe the gals/guys at Fedora/Redhat can help fix this more elegantly
with firewalld.

I personally make heavy use of Fedora VMs (both KVM & LXC) for big-data
environment simulations (Storm, Hadoop, etc.), and so I desperately need this port-forwarding between kvmHost and kvmGuests to work.

I hope that additional context helps. =:)

Thank you!
Noel

Hello:

Will there be any progress on this bug? Appreciated.

Thank you,
Noel

*** Bug 995480 has been marked as a duplicate of this bug. ***

*** Bug 1031813 has been marked as a duplicate of this bug. ***

Actually we've already had 2 such reports, bug #1031813 and especially bug #995480, which reveals the culprit of the problem and possible "work-around".

I'll summarize it here. In the next example my kvmHost has em1 iface and kvmGuest has virbr0 ('NAT to em1' network, 192.168.0.0/24),
both ifaces being in default - public zone.

# iptables-save -t filter

-A FORWARD -d 192.168.0.0/24 -i em1 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
 /\ 
 |   added by libvirtd  
-----------------------------------
 |   added by firewalld 
 \/
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD_IN_ZONES -i em1 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FWDI_public -j FWDI_public_allow
-A FWDI_public_allow -m conntrack --ctstate NEW -m mark --mark 0x64 -j ACCEPT

Libvirtd inserts all its rules on top of the FORWARD chain and the '-o virbr0 -j REJECT' prevents firewalld's port-forwarding rule ('--mark 0x64 -j ACCEPT') to be applied.

As bug #995480, comment #1 suggests the work-around could be in deleting the libvirtd's "-o virbr0 -j REJECT" rule.
One might probably put
iptables -D FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
into /etc/libvirt/hooks/qemu as suggested by
http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections

But I'm not sure what/where would be the correct solution, it'd probably need bigger redesign.
Libvirtd might probably avoid using the '-o virbr0 -j REJECT' rule if it uses firewalld because firewalld already has
'-j REJECT --reject-with icmp-host-prohibited' at the bottom of FORWARD chain. Any ideas, Thomas ?

Is this REJECT rule for -A FORWARD -o virbr0 really needed? firewalld is rejecting everything that is not covered by another rule in the end.

Assigning to libvirt for verification.

First, some background. Here is a description of libvirt's use of iptables:

  http://libvirt.org/firewall.html

In particular, the interesting part is under the heading "The virtual network driver".

As you mention above, I've had success with the rules added by the script at
http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections

Of course this works because in addition to the DNAT rule in the nat table, an ACCEPT rule for the particular port and IP address is at the top of the chain.

As for the two REJECT rules that libvirt adds, they are added to ensure that there is no inbound traffic to the guests on a nat network (other than from the host itself, or from other guests on the same network). Because libvirt has no information about any network security beyond that which it adds itself, it needs to add all the rules necessary to ensure guests on the network are allowed access only as defined in the libvirt documentation. In particular, guests on a libvirt network with <forward mode='nat'/> should not get any incoming connections from the outside.

You may at first think that simply by virtue of being in a private network space, the guest wouldn't be reachable. However, a simple experiment shows that an attacker on the same subnet as the host can simply add a route with the private network as destination and the virtualization host's IP as gateway, and if the rule in question is missing, will be able to (e.g.) ssh into a guest on the private network.

(Note that with my Fedora 20 machine this didn't work until I also removed another rule apparently added by firewalld:

  -A FORWARD -j REJECT --reject-with icmp-host-prohibited

this was the very last of the FORWARD rules. Still, libvirt historically has not been able to rely on any other entity taking care of any aspect of network security for them, so we can't just assume that rule is in place.)

Aside from that, another thing this rule is *intended* to protect against (but fails in some cases) is attempts to reach one private nat-mode network from another. In practice, the ordering of the rules ends up allow the connection in one direction but not the other, but fortunately that hasn't seemed to bother anybody yet, so it has never been a high priority to fix (if you have ideas, we'd love to hear them!).

So, I think that yes the REJECT rule is necessary, but we are open to alternatives that give the same effect. In particular, we would be *very* happy to see a solution that also solved the problem mentioned in the previous paragraph.

(In reply to nmvega from comment #2)
> And they are that I also tried various lower-level alternatives to using
> firewall-cmd, including:
> 
>     - The KVM/QEMU method described here: http://bit.ly/1kQ6OtO
> 
>       That doesn't work because, oddly, when I edit the XML definition file
>       for the VM (i.e. kvmGuest01.xml) as prescribed in that article
>       (via 'root@kvmHost# virsh edit kvmGuest01'), when I exit out of that
>

I haven't taken the time to see exactly what went wrong that caused your changes to be removed (libvirt does silently discard any elements that it doesn't recognize, so possibly the elements were added at the wrong place), but I did want to point out that the method of forwarding ports described in that article will only work for user mode networking, not for network connections that are made using tap devices and a Linux host bridge (as is the case with libvirt's virtual networks).

BTW, in the end I think what libvirt would most like to have would be a new element for <interface> that would allow specifying port forwarding directly in the guest config (similar to the way you can add packet filter rules with a <filterref>). There was a patch to do that only for usermode networking a year or so ago, but it languished because nobody would commit to the syntax without confirming that the same syntax would also work for port forwarding in the case of tap-based network devices.

> BTW, in the end I think what libvirt would most like to have would be a new
> element for <interface> that would allow specifying port forwarding directly
> in the guest config (similar to the way you can add packet filter rules with
> a <filterref>). There was a patch to do that only for usermode networking a
> year or so ago, but it languished because nobody would commit to the syntax
> without confirming that the same syntax would also work for port forwarding
> in the case of tap-based network devices.

Yes, people have been asking for port forwarding for our virtual networks for sooooooooo many years, it is well overdue for us to actually give them what they want, rather than force them to try to set it up behind libvirt's back and hit problems there.

Hello again...

Seeing that this is taking a while (and with no immediate resolution in site), I have a question, because **for several years now** I keep slamming into this *show-stopping* issue whenever I need to simulate some BIG-DATA cluster completely inside a KVM network, and need to interact with it from an outside that network.

My latest issie/problem,
my question,
My hope for a work-around offed by my friends here:

My KVM machines are used to simulate a HADOOP/YARN compute & storage cluster. YARN-aware application drivers/clients sitting on the KVM Host (not Guest), issue requests to the YARN ResourceManager (RM) -- which runs inside the KVM network -- for cluster resources on which to run their application. The YARN RM then allocates cluster resousces, launches the application *somewhere* in the cluster, and finally supplies the extarnal application driver/client with a KVM-internal IP and *arbitrary* PORT that it should connect to, to interact with the now running distributed application... say, to receive standard output.

Since it is not possible to know which port will be returned by YARN, what can I do to allow *ALL* ports above port 1024? Is there a simple firewall-cmd syntax to do this? Maybe I missed it somewhere.

P.S. Port Fwarding still doesn't work, but that's a separate issue (the one I filed this bug about). But my question here, is a different one.

Thank you,
NMV

This bug is still valid on F21, it took me a few hours to understand where that was coming from.

Setting to NEW, since ASSIGNED state was just inherited from the firewalld componet

This bug is still valid on F22.

Sounds like the libvirt hooks (https://libvirt.org/hooks.html) are not supported in Fedora or RHEL. I would like to have some sort of that capability. 

Specifically I am interested in changing a macvtap device to have ALLMULTI turned on after libvirt creates it.

Repurposing to track port forwarding for virtual networks, like dan suggests in comment #10

Hi guys,

I'm getting the same bug with a centos 7 lab machine trying to do the same as the first comment.

Is there any update regarding it? Maybe a way to work around it?

Thanks in advanced.

I found a way to workaround the issue using socat

'socat TCP-LISTEN:<LOCAL PORT>,fork TCP:<VM IP>:<VM PORT>'

The server is using the script hack in here:

http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections

and also I've allowed the same host port on Firewalld.

With all that in place I'm able to RDP the windows VM successfully outside the network.

I'm not sure if all above is needed. I'll test taking out things from the current setup to be sure.

Thank you for reporting this issue to the libvirt project. Unfortunately we have been unable to resolve this issue due to insufficient maintainer capacity and it will now be closed. This is not a reflection on the possible validity of the issue, merely the lack of resources to investigate and address it, for which we apologise. If you none the less feel the issue is still important, you may choose to report it again at the new project issue tracker https://gitlab.com/libvirt/libvirt/-/issues The project also welcomes contribution from anyone who believes they can provide a solution.