Bug 1094823 - Nova Network fails to associate floating ip because ebtables race issue
Summary: Nova Network fails to associate floating ip because ebtables race issue
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 4.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 5.0 (RHEL 7)
Assignee: Brent Eagles
QA Contact: yeylon@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-05-06 14:39 UTC by Pavel Sedlák
Modified: 2019-09-09 13:22 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-12-15 15:25:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1316621 0 None None None Never
OpenStack gerrit 136217 0 None None None Never

Description Pavel Sedlák 2014-05-06 14:39:53 UTC
When trying to associate floating ip, when using nova network, it may fail with 

> BadRequest: Bad request
> Details: {'message': 'Error. Unable to associate floating ip', 'code': '400'}

Reason is that rootwrap call to ebtables failed.

> Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ebtables -t nat -I PREROUTING --logical-in br100 -p ipv4 --ip-src 192.168.32.2 ! --ip-dst 192.168.32.0/22 -j redirect --redirect-target ACCEPT
> Exit code: 255
> Stdout: ''
> Stderr: "The kernel doesn't support a certain ebtables extension, consider recompiling your kernel or insmod the extension.\n" to caller

Same issue may actually happen with Icehouse on RHEL7:

> Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ebtables -t nat -I PREROUTING --logical-in br100 -p ipv4 --ip-src 192.168.32.10 ! --ip-dst 192.168.32.0/22 -j redirect --redirect-target ACCEPT
> Exit code: 255
> Stdout: ''
> Stderr: "Unable to update the kernel. Two possible causes:\n1. Multiple ebtables programs were executing simultaneously.
>  The ebtables\n userspace tool doesn't by default support multiple ebtables programs running\n concurrently.
>  The ebtables option --concurrent or a tool like flock can be\n used to support concurrent scripts that update
>  the ebtables kernel tables.\n2. The kernel doesn't support a certain ebtables extension, consider\n
>  recompiling your kernel or insmod the extension.\n.\n"

So it's not related only to RHOS4/RHEL 6.5.

This happens randomly, and only like once per whole run of tempest (lot of servers created, floating ip's associated etc, so no issue with kernel support etc).

As the rootwrap calls seems to be synchronized with locks, there has to be something else running ebtables at the same time.
Possible offender could be libvirt/nwfilter, or similar out-of-openstack part.


Version-Release number of selected component (if applicable):
> ebtables.x86_64                    2.0.9-6.el6
> openstack-nova-network.noarch      2013.2.3-7.el6ost

Comment 5 Jaroslav Henner 2014-09-16 18:49:47 UTC
This is causing failures in random tests in our gate, so it is impossible to create any skip pattern, I am considering calling this an urgent severity issue.

Comment 6 Brent Eagles 2014-11-24 17:12:29 UTC
Upstream patch is up for review.


Note You need to log in before you can comment on or make changes to this bug.