Bug 985954

I also enconter this bug with the following scnarios:

1. When associating a floating ip to an instance, it takes long time (maybe ~20 seconds) to the newly-associated to appear in the instance's IP Address view.

2. When dissociating a floating ip from and instance, the floating ip remains in the instance's IP Address view although operation succeeded and neither floating is not pingable anymore nor appears in the cli's "nova list" command. Note that moving to another vertical tab and getting back to the "Manage Compute"->"Instances" tab doesn't help, eventually the floating ip usually disappears (it's not consistent), I've tried few times and I got into situation that the floating IP remained for minutes, even when logging out and logging in to Horizon again.

(In reply to Rami Vaknin from comment #2)
> I also enconter this bug with the following scnarios:
> 
> 1. When associating a floating ip to an instance, it takes long time (maybe
> ~20 seconds) to the newly-associated to appear in the instance's IP Address
> view.
> 
> 2. When dissociating a floating ip from and instance, the floating ip
> remains in the instance's IP Address view although operation succeeded and
> neither floating is not pingable anymore nor appears in the cli's "nova
> list" command. Note that moving to another vertical tab and getting back to
> the "Manage Compute"->"Instances" tab doesn't help, eventually the floating
> ip usually disappears (it's not consistent), I've tried few times and I got
> into situation that the floating IP remained for minutes, even when logging
> out and logging in to Horizon again.

2) Is probably caused by nova caching. compare the output of 
nova floating-ip-list
quantum floating-ip-list

*** Bug 1028898 has been marked as a duplicate of this bug. ***

Failed.

Tested on rhos 4.0, puddle 2013-11-15.1

1. I've created several instances and floating IPs in cli
2. I logged in into horizon and started to assign floating IPs to instances at time 00:00
3. I finished to assign 8 floating IPs at time 1:40 (1 minute and 20 seconds)
4. I saw all 8 floating IPs associated with instance in horizon at time 7:04 which means that the issue is deeper now.

Forgot to add that I did refresh the horizon every few seconds.

I am interested to understand if this is a Neutron issue or an Horizon one.
Please reproduce it using the Neutron CLI and update if you can reproduce the issue.

I reproduced it using the cli, it takes a looooot of time to the floating ip to appear in the "nova list" output.
Tested on 4.0/2013-11-15.1.

This seems to be at the core of an upstream test failure for basic network functionality. I've submitted a patch upstream that amends the basic network scenario to "work around" slow assignment, which isn't awesome from the point of view of this bug. However it *does* highlight just how bad it is at the moment.

This is, in a sense, an end-to-end/round-trip performance issue. While it might be mitigated by some small self-contained changes, a rudimentary analysis indicates that it is caused by design and implementation choices. It is not likely something that can be declared *fixed* in an async fix or patch release.

What I would like to do is to push this to 5.0, but in the meantime analyze the timing of each stage of the interactions in operations like this in environments and scenarios where the behavior is *terrible*. If we can isolate the segments that are the worst offenders we can a.) create tests to gauge improvements and avoid performance critical regressions in the future, and b.) identify contributing issues, create bugs, fix them etc. and finally c.) repeat (a.).

There is another upstream fix I'm adding a link to that promises to further speed up floating ip allocation.  I haven't tested it though.

On my all-in-one devstack setup:
When assigning a floating IP, I was able to ping it through the DNAT in under a second, however it took roughly 23 seconds for the association to show up in nova list and in Horizon.

This is a very uninformed guess - But this hints to an issue somewhere in Nova, or between Nova and Neutron, and not an issue with Neutron itself.

I'll continue to investigate.

This bug shouldn't happen using Nova Networking (Unverified, but with a very high probability). With Neutron it does.

So what's happening is this:
Horizon uses Neutron directly to associate a floating IP. Neutron doesn't notify Nova that an instance's networking information has changed, and Horizon doesn't update Nova either. However, Horizon takes the instances networking information from Nova.

Nova uses a strange mechanism to update its instances networking information. It is detailed in _heal_instance_info_cache at nova/compute/manager.py:4388. A periodic task (60 seconds by default) is ran, and updates *one* instance's networking information at a time (By making API calls to Neutron). This means that when associating three floating IPs, it can take up to 3 minutes for the last floating IP to be properly displayed in Horizon, and this should scale linearly.

Solutions:
1) Nova's 'heal cache mechanism' could update networking information for *all* instances and not only one. Any instance networking changes will still be reflected in Horizon from 1 to 60 seconds after the operation. I don't think that this is acceptable user experience.
2) Change Horizon so that it takes networking information directly from Neutron. Taking this information from Nova should be avoided in other places as well. Horizon could accomplish this using the same API calls that Nova makes to Neutron in nova/network/neutronv2/api.py:992:_build_network_info_model. Alternatively, we could add a Neutron verb that accepts a port ID and makes the appropriate queries to retrieve all networking information for that port. Horizon could then use that verb on every port on every instance (It would first obtain instance -> ports mappings from Nova). This is simpler and more efficient than how it's currently implemented in Nova.
3) Perhaps implement both solutions so everything else that relies on Nova's cached instance's networking information will benefit.

Filed an upstream bug on Horizon.

Pushed a patch to Horizon to fix the bug.

"nova list" still takes a lot of time to update from Neutron, but that's a separate issue.

After a discussion with Ofer and Assaf we agreed that this fix does not need to be backported to Havana.
Horizon does not reflect the state in Neutron correctly and the information on the floating IP can be pulled from Neutron directly.

Upstream patch merged to master, will be available in I3.

Tested on IceHouse 


python-neutronclient-2.3.4-1.el7ost.noarch
openstack-neutron-2014.1-20.el7ost.noarch
openstack-neutron-openvswitch-2014.1-20.el7ost.noarch
python-neutron-2014.1-20.el7ost.noarch
openstack-neutron-ml2-2014.1-20.el7ost.noarch

python-django-horizon-2014.1-6.el7ost.noarch

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0848.html