Bug 1086512

Summary: "ip link show" commands display Message Truncated
Product: Red Hat Enterprise Linux 6 Reporter: Michael Samuel <mik>
Component: iprouteAssignee: Phil Sutter <psutter>
Status: CLOSED ERRATA QA Contact: Jaroslav Aster <jaster>
Severity: low Docs Contact:
Priority: medium    
Version: 6.5CC: ebenes, gprocunier, gwatson, kzhang, laine, mik, mkalinin, rkhan, ssekidde
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: iproute-2.6.32-54.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-11 00:18:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
iproute-2.6.32-33.notrunc.el6_6.x86_64 none

Description Michael Samuel 2014-04-11 03:51:52 UTC
On a system with a large number of network interfaces (82 physical + 1 bridge + 1 vlan) "ip link show" commands (including "ip link show dev") show "Message truncated" 5 times.

Reducing the number of interfaces made this problem go away.

The problem appears to stem from lib/netlink.c one of the iov bufs being too small.

Comment 3 Greg Procunier 2015-03-31 02:21:36 UTC
We also experience this issue on our Cisco UCS servers.

We associated a 80 member dynamic connection pool on our VIC-1240/1280 cards to the 3 static nics and now iproute spams this 'Message truncated' error.

Comment 4 Pavel Šimerda (pavlix) 2015-04-07 20:55:36 UTC
(In reply to Michael Samuel from comment #0)
> The problem appears to stem from lib/netlink.c one of the iov bufs being too
> small.

I've seen the netlink buffer issue with other network configuration tools as well.

Comment 5 Pavel Šimerda (pavlix) 2015-04-16 23:16:25 UTC
I expected that thousands of dummy interfaces would be enough for reproducing this but apparently not. Therefore while it's easy to change buffer sizes, it's hard to know which ones are actually insufficient if we can't test the change.

What I can say is that there are two 16k buffers and two 8k buffers in the rhel6 and rhel7 libnetlink.c source file as found using `grep 'buf\['` but in upstream the latter two are also 16k so there apparently is a difference from upstream.

Comment 8 Phil Sutter 2015-10-15 15:34:08 UTC
I can not reproduce the issue locally. Could anyone perhaps get me access to a test system exposing the problem?

Comment 9 Rashid Khan 2015-10-23 22:37:29 UTC
Hi Michael, 
Can you please send a reliable reproducer for this. 
If we do not hear from you soon, then we might not be able to include it in our next RHEL6.x release. 
Thanks

Comment 10 Michael Samuel 2015-10-26 03:58:35 UTC
I no longer have access to the affected system, as I've changed jobs.

It should be reproducible by just creating a system (maybe a VM) with a lot of network interfaces.

Comment 11 Phil Sutter 2015-10-26 11:59:20 UTC
Hi Michael,

(In reply to Michael Samuel from comment #10)
> I no longer have access to the affected system, as I've changed jobs.
> 
> It should be reproducible by just creating a system (maybe a VM) with a lot
> of network interfaces.

OK. As stated, we can't reproduce this so I'm closing it for now. Feel free to reopen in case you run into this problem again, of course.

Comment 12 Laine Stump 2016-01-13 16:00:28 UTC
This was seen on a system running RHEL6.7 with 80 VFs, a couple of PFs, and 15 macvtap interfaces (created by libvirt and attached to 15 of the VFs), so I'm reopening.

Comment 14 Laine Stump 2016-01-13 19:20:37 UTC
As I suspected, this seems to be specific to certain types of interfaces. I did a test where I created 150 bridges attached to 150 vlan interfaces, and "ip -d link show" continued to work properly.

Most likely it is related to the number of SRIOV virtual functions (which would affect the number of elements in the IFLA_VFINFO_LIST, which is all put into a single netlink response message when an IFLA_EXT_MASK with RTEXT_FILTER_VF is put in the RTM_GETLINK request that precedes it). I only have an 82576 card available though, so the maximum VFs I can get on my system is 14, which is much too low to see the problem.

Comment 18 Phil Sutter 2016-01-14 18:22:43 UTC
Created attachment 1114921 [details]
iproute-2.6.32-33.notrunc.el6_6.x86_64

Comment 36 errata-xmlrpc 2016-05-11 00:18:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0895.html