Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1094823

Summary: Nova Network fails to associate floating ip because ebtables race issue
Product: Red Hat OpenStack Reporter: Pavel Sedlák <psedlak>
Component: openstack-novaAssignee: Brent Eagles <beagles>
Status: CLOSED WONTFIX QA Contact: yeylon <yeylon>
Severity: high Docs Contact:
Priority: high    
Version: 4.0CC: eglynn, jhenner, ndipanov, sgordon, srevivo, yeylon
Target Milestone: ---Keywords: Automation, ZStream
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-15 15:25:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Sedlák 2014-05-06 14:39:53 UTC
When trying to associate floating ip, when using nova network, it may fail with 

> BadRequest: Bad request
> Details: {'message': 'Error. Unable to associate floating ip', 'code': '400'}

Reason is that rootwrap call to ebtables failed.

> Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ebtables -t nat -I PREROUTING --logical-in br100 -p ipv4 --ip-src 192.168.32.2 ! --ip-dst 192.168.32.0/22 -j redirect --redirect-target ACCEPT
> Exit code: 255
> Stdout: ''
> Stderr: "The kernel doesn't support a certain ebtables extension, consider recompiling your kernel or insmod the extension.\n" to caller

Same issue may actually happen with Icehouse on RHEL7:

> Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ebtables -t nat -I PREROUTING --logical-in br100 -p ipv4 --ip-src 192.168.32.10 ! --ip-dst 192.168.32.0/22 -j redirect --redirect-target ACCEPT
> Exit code: 255
> Stdout: ''
> Stderr: "Unable to update the kernel. Two possible causes:\n1. Multiple ebtables programs were executing simultaneously.
>  The ebtables\n userspace tool doesn't by default support multiple ebtables programs running\n concurrently.
>  The ebtables option --concurrent or a tool like flock can be\n used to support concurrent scripts that update
>  the ebtables kernel tables.\n2. The kernel doesn't support a certain ebtables extension, consider\n
>  recompiling your kernel or insmod the extension.\n.\n"

So it's not related only to RHOS4/RHEL 6.5.

This happens randomly, and only like once per whole run of tempest (lot of servers created, floating ip's associated etc, so no issue with kernel support etc).

As the rootwrap calls seems to be synchronized with locks, there has to be something else running ebtables at the same time.
Possible offender could be libvirt/nwfilter, or similar out-of-openstack part.


Version-Release number of selected component (if applicable):
> ebtables.x86_64                    2.0.9-6.el6
> openstack-nova-network.noarch      2013.2.3-7.el6ost

Comment 5 Jaroslav Henner 2014-09-16 18:49:47 UTC
This is causing failures in random tests in our gate, so it is impossible to create any skip pattern, I am considering calling this an urgent severity issue.

Comment 6 Brent Eagles 2014-11-24 17:12:29 UTC
Upstream patch is up for review.