1557048 – [docker] cannot create networks simultaneously

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1557048 - [docker] cannot create networks simultaneously

Summary: [docker] cannot create networks simultaneously

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	docker
Sub Component:
Version:	7.4
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Daniel Walsh
QA Contact:	atomic-bugs@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-03-15 21:17 UTC by Carl George
Modified:	2021-02-15 07:37 UTC (History)
CC List:	3 users (show)
Fixed In Version:	docker-1.13.1-58.git87f2fab.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-02-15 07:37:58 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	projectatomic docker pull 304	0	None	closed	BACKPORT: Serialize non-atomic jump rule programming in bridge	2020-07-27 20:33:18 UTC
Github	projectatomic docker pull 305	0	None	closed	BACKPORT: Serialize non-atomic jump rule programming in bridge	2020-07-27 20:33:18 UTC

Description Carl George 2018-03-15 21:17:04 UTC

Description of problem:
In both RHEL and Fedora, the docker daemon is not able to accept simultaneous requests to create docker networks. I first observed this behavior in the drone CI software [1], which uses separate docker networks for each task in a matrix build. One of the matrix tasks will succeed, but the others will fail with this error:

Error response from daemon: unable to remove jump to DOCKER-ISOLATION rule in FORWARD chain: (COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -j DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. )

I was also able to reproduce this error just by simultaneously running multiple `docker network create ...` commands via tmux synchronized panes (see Steps to Reproduce).

Version-Release number of selected component (if applicable):
docker-1.13.1-53.git774336d.el7
docker-1.13.1-44.git584d391.fc27

How reproducible:
Always when multiple commands are run simultaneously. Not reproducible when only one command is run.

Steps to reproduce:
1. start tmux session
2. tmux command: split-window
3. in each pane, type `docker network create ...` with a unique network name
4. tmux command: set-window-option synchronize-panes
5. hit enter

Actual results:
One network is created, the other fails.

Expected results:
Both networks be created.

How to fix:
I was able to track this down to an upstream issue [2], which has already been fixed in docker/libnetwork and bundled in with docker 17.04. It boils down to this commit [3] from docker/libnetwork#1658 [4]. I created a patch [5] based on that commit, and it applies cleanly to the RHEL SRPM and produces RPMS [6] that do not suffer from this bug. I'm happy to send this patch as a pull request directly into projectatomic/docker if so desired.

Additional info:
[1]: https://discourse.drone.io/t/matrix-builds-network-concurrency-issue/1936
[2]: https://github.com/moby/moby/issues/25393
[3]: https://github.com/docker/libnetwork/commit/fe741120dbacc1c720a80812de1d65811695f5d6
[4]: https://github.com/docker/libnetwork/pull/1658
[5]: https://github.com/carlwgeorge/docker/commit/660a257754ab0e43e1e778baf8a625f1542fd334
[6]: https://copr.fedorainfracloud.org/coprs/carlwgeorge/docker/

Comment 2 Daniel Walsh 2018-03-16 11:26:00 UTC

Can you open a pr against projectatomic/docker to fix this issue.

Comment 3 Carl George 2018-03-16 12:29:48 UTC

Absolutely.  I noticed the Fedora package builds the docker-1.13.1 branch, and RHEL builds the docker-1.13.1-rhel branch.  Should I send separate pull requests for each?  Or just one pull request and let yall cherry-pick to the other later?

Comment 4 Daniel Walsh 2018-03-16 13:35:02 UTC

Both, thanks.

Comment 5 Carl George 2018-04-11 18:36:51 UTC

https://git.centos.org/commitdiff/rpms!docker.git/92d3dcc2011f7a3ae01f4fcd7e15cdfcccc60f88

Comment 6 Carl George 2018-04-11 21:01:40 UTC

This was released in RHEL 7.5, but I don't see an option to close the bug.

Comment 9 RHEL Program Management 2021-02-15 07:37:58 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Note You need to log in before you can comment on or make changes to this bug.