Created attachment 1115847 [details]
ip_addr_flush_reproducer.sh

Hi Phil,

I can confirm it. Tested on all rhel-6 architectures and x86_64 rhel-7. There is a huge performance regression between rhel-6 and rhel-7.

I ran a few more tests:

RHEL7, upstream kernel, RHEL iproute: 0m53.384s
RHEL7, upstream kernel, upstream iproute: 1m2.577s
RHEL6, RHEL6 kernel, RHEL iproute: 0m8.199s
RHEL6, RHEL6 kernel, upstream iproute: 0m7.594s

So this very much looks like a kernel issue, at least iproute version seems unrelated.

As the output in comment 4 shows, iproute flushes many more addresses on RHEL7 than on RHEL6. This reminded me of the 'promote_secondaries' sysctl setting, which is indeed disabled in RHEL6 and enabled in RHEL7. Running the test again in RHEL7 with promote_secondaries disabled helps in run time, but shows a new error:

flushing all ip addresses
Failed to send flush request: No buffer space available
Flush terminated

real	0m31.644s
user	0m0.000s
sys	0m31.305s

This needs further investigation, as well as a possible way to disable promote_secondaries temporarily while flushing the interface as it hinders operation.

Regarding the error message printed with promote_secondaries=0:

recv() in rtnl_send_check() sets errno to ENOBUFS. This call is just an early check for errors, so it comes from the kernel. Another prove to this is that with upstream kernel the error message does not show.

Despite what one might think, the flush completes in both cases, so this is again rather a cosmetic issue.

As for the performance issue, I have created perf records for promote_secondaries on/off on RHEL kernel, preliminary analysis did not yield a result yet, though. Address deletion is obviously quite complicated due to the necessary management of routing table adjustments.

Created attachment 1149429 [details]
perf record for promote_secondaries=1

Created attachment 1149430 [details]
perf record for promote_secondaries=0

Removing devel_ack+ since it is still unclear where the performance regression comes from and whether it can be fixed or simply has to be accepted as a side effect of increased complexity in RHEL7 kernel over RHEL6.

Thanks to Phil for providing a reproducer.

Gave it a run on RHEL7 and FC27 VMs. With 40k addresses the flush took:

* 1m2.566s on RHEL7, 3.10.0-861.el7, iproute-4.11.0-14.el7
* 0m15.713s on FC27, 4.15.7-300.fc27, iproute-4.15.0-1.fc27

So latest(ish) stable kernel is not as fast as RHEL6 (basing on Phil's report here, I haven't tested it yet), but much better than RHEL7.

Also, on Fedora the flush happens in just 2 rounds, just as on RHEL6, and not all the addresses are accounted for.

Will look what has changed upstream that could have brought back the performance.

(In reply to Jakub Sitnicki from comment #17)
> Thanks to Phil for providing a reproducer.
> 
> Gave it a run on RHEL7 and FC27 VMs. With 40k addresses the flush took:
> 
> * 1m2.566s on RHEL7, 3.10.0-861.el7, iproute-4.11.0-14.el7
> * 0m15.713s on FC27, 4.15.7-300.fc27, iproute-4.15.0-1.fc27
> 

The above measurement on FC27 was wrong. While I was running the FC27 kernel and FC27 user-space, systemd was not running so net.ipv4.conf.default.promote_secondaries was not enabled. This was the reason of seen speed-up. Once I've enabled promoting secondaries, the flush took as long as on RHEL7.

NB: I did not get the "Failed to send flush request: No buffer space available" error message brought up by Phil on either RHEL7 or FC27.

In the end is this a performance regression at all?

Comparing flushing with promoting secondaries disabled (the default on RHEL6) with flushing with promoting secondaries enabled (the default on RHEL7) does not look to me like comparing apples to apples.

In the kernel we can't remove all addresses in one run when we have to do promotions. Essentially we are testing and comparing two different operations. 

Please note that the default does not come from changes in the kernel, it was introduced in systemd 216 as stated in NEWS file:

CHANGES WITH 216:
[...]
        * The default sysctl.d/ snippets will now set

                net.ipv4.conf.default.promote_secondaries=1

          This has the benefit of no flushing secondary IP addresses
          when primary addresses are removed.

And this is indeed what we have in RHEL7 (and FC27):

[root@localhost ~]# rpm -q systemd
systemd-219-57.el7.x86_64
[root@localhost ~]# grep promote_secondaries /usr/lib/sysctl.d/50-default.conf
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1

@Jaroslav Aster, what is your take on this? Since the default sysctl has changed between RHEL6 and RHEL7, I'm inclined to close it as NOTABUG. At least from kernel/iproute point of view.

As per earlier Phil's suggestion. I think we can turn this into an RFE and attempt to convince upstream to make the flush with promote_secondaries on faster. I've prepared a PoC that introduces a new flag to be used with RTM_DELADDR requests to let the kernel know that we're going for a flush and it seems to work.

It will make it easier to motivate the change if there is a use-case (among Layered Products perhaps?) for it. Now, I'm not sure if this BZ is a result of QE investigation or feedback from a customer. Can anyone shed some light on the background of this BZ?

Proposed a new flag for RTM_DELADDR requests that would speed-up the flush upstream:

https://marc.info/?l=linux-netdev&m=152836638902888

Hi Jakub,

I agree with you, it does not seem like a regression. My suggestion is close this bug as NOTABUG and create a new one for RFE.

I think, there is no customer behind it. It is a result of qe work.

(In reply to Jakub Sitnicki from comment #21)
> Proposed a new flag for RTM_DELADDR requests that would speed-up the flush
> upstream:
> 
> https://marc.info/?l=linux-netdev&m=152836638902888

It has been suggested by Michal Kubecek that the new 'FLUSH' flag handling can be improved so that all interface addresses are flushed in one go in response to a single request from userspace:

https://marc.info/?l=linux-netdev&m=152836923403698&w=2

To differentiate a flush request from an address removal request, the former would carry no addresses as netlink attributes and would have the flush flag set.

It seems worth pursing. The expected benefits are:
- increased performance of the flush operation when secondary address promotion is enabled (as demonstrated in proposed patch),
- easier to use API for userspace to remove all addresses from an interface; at the moment multiple requests have to be issued to flush all addresses.

Moving the BZ back to NEW as I will be no longer working on it.

Interestingly, current upstream kernel already supports RTM_DELADDR request without any address specified (at least in IPv4). The effect is that kernel's inet_rtm_deladdr() removes all addresses. Sadly, this doesn't increase performance over the current method.

Assuming that address promotion in kernel is the actual bottleneck, this is what has to be improved (or eliminated) while flushing (obviously).

OTOH, if the above works since reasonably old kernels, a fallback from passing the new FLUSH flag and omitting addresses from the request might not be required.

Moving it to RHEL-7.8.

This will be fixed for rhel8 (issue cloned here: https://bugzilla.redhat.com/show_bug.cgi?id=1641094).
Feel free to reopen if needed.