Bug 1817022

Summary: docker0 interface has to be in trusted zone
Product: [Fedora] Fedora Reporter: sedrubal <fedora>
Component: moby-engineAssignee: Olivier Lemasle <o.lemasle>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 32CC: berend.de.schouwer, bugzilla, egarver, ja, jmontleo, lee.jnk, o.lemasle, philbates35, psutter, rabin, robinlee.sysu, tn, yajo.sk8
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: moby-engine-19.03.8-2.ce.gitafacb8b.fc32 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-23 02:43:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description sedrubal 2020-03-25 12:16:10 UTC
Description of problem:
Since the upgrade to fedora 32 beta, all my docker containers don't have access to the internet (except ICMP / ping). The problem is, that the docker0 interface was in 'public' zone but has to be in 'trusted' zone.

Version-Release number of selected component (if applicable):
0.8.1

How reproducible:


Steps to Reproduce:
1. Install Fedora 32
2. Install moby-engine
3. docker run --rm -it fedora bash
4. try to access the internet (dnf upgrade)
5. run: firewall-cmd --permanent --zone=trusted --add-interface=docker0
6. do it again

Actual results:
The container does not have internet.

Expected results:
Docker should work out of the box.

Additional info:
https://success.docker.com/article/firewalld-problems-with-container-to-container-network-communications

Comment 1 Eric Garver 2020-03-25 12:42:03 UTC
This is known. See this Change:
   https://fedoraproject.org/wiki/Changes/firewalld_default_to_nftables

It must be fixed on the docker/moby side. What needs to be done is outlined in the Change.
See this fesco ticket or the decision:
   https://pagure.io/fesco/issue/2231#comment-601798

Alternatively, move over to using podman as it does the right thing.

I'm reassigning this to moby-engine.

Comment 2 Eric Garver 2020-03-25 12:50:21 UTC
Alternatively, users may revert to using the iptables backend. However, the firewalld maintainers do not recommend this as a long term solution.

In /etc/firewalld/firewalld.conf:

  # FirewallBackend
  # Selects the firewall backend implementation.
  # Choices are:
  #       - nftables (default)
  #       - iptables (iptables, ip6tables, ebtables and ipset)
  FirewallBackend=iptables

Then restart firewalld:

  # systemctl restart firewalld

Comment 3 Yajo 2020-05-04 10:40:20 UTC
At this point, podman isn't able to replace docker completely. And Fedora Workstation is supposed to be a... workstation. Spending a whole work day fighting it isn't fun. 😕

There's no docs on this also. Users should be clearly warned that docker will stop working after the upgrade IMHO.

Upstream issues:

- https://github.com/moby/moby/issues/26824
- https://github.com/docker/for-linux/issues/955

To make it work immediately, apply workaround from first comment, but also do the fiewall-cmd command without --permanent to be applied immediately. Basically:

firewall-cmd --zone=trusted --add-interface=docker0
firewall-cmd --zone=trusted --add-interface=docker0 --permanent

Comment 4 Yajo 2020-05-04 11:02:23 UTC
Oops sorry that doesn't work.

The workaround to be used is the one explained in comment 2.

Comment 5 Eric Garver 2020-05-04 12:27:50 UTC
(In reply to Yajo from comment #3)
> At this point, podman isn't able to replace docker completely.

Can you clarify here? I was under the impression that podman is fully capable of replacing docker.

Comment 6 Yajo 2020-05-04 22:05:36 UTC
It can replace tools tht used docker CLI, but not those that used docker API. One of those, docker compose, is essential in my company's devel workflow.
I hope with podman 2 I'm really able to dump Docker.

Comment 7 Olivier Lemasle 2020-05-06 10:28:18 UTC
*** Bug 1830188 has been marked as a duplicate of this bug. ***

Comment 8 Olivier Lemasle 2020-05-06 11:11:55 UTC
- It is indeed possible to add docker0 to the trusted zone (firewall-cmd --zone=trusted --add-interface=docker0) and it fixes the issue IF the docker bridge name is docker0.

@Yajo, why do you say "Oops sorry that doesn't work"?

- However, I also tried to use iptables-nft, but I'm a little bit lost... iptables-nft should work as a CLI-compatible iptables and communicates with nftables (https://www.redhat.com/en/blog/using-iptables-nft-hybrid-linux-firewall).

1) On a fresh installation where resolution fails:

  $ sudo docker run --rm fedora curl http://google.com
  curl: (6) Could not resolve host: google.com

2) I select iptables-nft as the iptables alternative:

  $ iptables -V
  iptables v1.8.4 (legacy)

  $ sudo alternatives --config iptables

  There are 2 programs which provide 'iptables'.

    Selection    Command
  -----------------------------------------------
  *+ 1           /usr/sbin/iptables-legacy
     2           /usr/sbin/iptables-nft

  Enter to keep the current selection[+], or type selection number: 2

  $ iptables -V
  iptables v1.8.4 (nf_tables)

3) Then I flush nft. This action is performed by "systemctl restart nftables" but I don't really understand why I have to do it here... as I'm unsure why service "nftables" is disabled.

  sudo nft flush ruleset

4) Then, after restarting Docker engine:

  sudo systemctl restart docker

5) Containers have network access:

  $ sudo docker run --rm fedora curl -s http://google.com
  <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
  <TITLE>301 Moved</TITLE></HEAD><BODY>
  <H1>301 Moved</H1>
  The document has moved
  <A HREF="http://www.google.com/">here</A>.
  </BODY></HTML>

@Eric Garver: could it be a solution?

Comment 9 Eric Garver 2020-05-06 13:38:37 UTC
(In reply to Olivier Lemasle from comment #8)
> @Eric Garver: could it be a solution?

No. "nft flush ruleset" above flushed all the firewalld rules. That's why it works. :)

Comment 10 Fedora Update System 2020-05-07 19:17:10 UTC
FEDORA-2020-a658d9abde has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-a658d9abde

Comment 11 Fedora Update System 2020-05-08 04:10:10 UTC
FEDORA-2020-a658d9abde has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-a658d9abde`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-a658d9abde

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Yajo 2020-05-08 07:00:44 UTC
(In reply to Olivier Lemasle from comment #8)
> @Yajo, why do you say "Oops sorry that doesn't work"?

According to my tests, a simple container command like that above works, but Docker was still unable to create networks and use them properly.

Comment 13 Fedora Update System 2020-05-23 02:43:50 UTC
FEDORA-2020-a658d9abde has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 14 Jason Montleon 2020-05-26 16:49:57 UTC
The fix seems to break libvirt and cause interfaces to stop being added to any zones if you implemented the workaround so it's probably a good idea to remove the workaround once you update to the new package.

https://bugzilla.redhat.com/show_bug.cgi?id=1829090#c3

Comment 15 philbates35 2020-05-27 11:31:06 UTC
I can see that the latest version of the moby-engine package pushed to Fedora 32 repo (19.03.8-2.ce.gitafacb8b.fc32) now provides its own docker zone out of the box that docker0 is assigned to:

$ firewall-cmd --get-active-zones
FedoraWorkstation
  interfaces: wlp2s0
docker
  interfaces: docker0
libvirt
  interfaces: virbr0

However this still wasn't enough to allow didn't get my web container to communicate with my db container specified in my docker-compose. I still had to additionally run:

$ firewall-cmd --zone=FedoraWorkstation --add-masquerade

Found a reference to this workaround here: https://github.com/docker/for-linux/issues/955#issuecomment-628407032. Is there anything the moby-engine package could do differently so that containers defined in a docker-compose can communicate with each other out of the box?