1086512 – "ip link show" commands display Message Truncated

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1086512 - "ip link show" commands display Message Truncated

Summary: "ip link show" commands display Message Truncated

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	iproute
Sub Component:
Version:	6.5
Hardware:	Unspecified
OS:	Linux
Priority:	medium
Severity:	low
Target Milestone:	rc
Target Release:	---
Assignee:	Phil Sutter
QA Contact:	Jaroslav Aster
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2014-04-11 03:51 UTC by Michael Samuel
Modified:	2016-05-11 00:18 UTC (History)
CC List:	9 users (show)
Fixed In Version:	iproute-2.6.32-54.el6
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-05-11 00:18:49 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
iproute-2.6.32-33.notrunc.el6_6.x86_64 (363.78 KB, application/x-rpm) 2016-01-14 18:22 UTC, Phil Sutter	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	1304840	1	None	None	None	2021-01-20 06:05:38 UTC
Red Hat Product Errata	RHBA-2016:0895	0	normal	SHIPPED_LIVE	iproute bug fix and enhancement update	2016-05-10 22:50:11 UTC

Internal Links: 1304840

Description Michael Samuel 2014-04-11 03:51:52 UTC

On a system with a large number of network interfaces (82 physical + 1 bridge + 1 vlan) "ip link show" commands (including "ip link show dev") show "Message truncated" 5 times.

Reducing the number of interfaces made this problem go away.

The problem appears to stem from lib/netlink.c one of the iov bufs being too small.

Comment 3 Greg Procunier 2015-03-31 02:21:36 UTC

We also experience this issue on our Cisco UCS servers.

We associated a 80 member dynamic connection pool on our VIC-1240/1280 cards to the 3 static nics and now iproute spams this 'Message truncated' error.

Comment 4 Pavel Šimerda (pavlix) 2015-04-07 20:55:36 UTC

(In reply to Michael Samuel from comment #0)
> The problem appears to stem from lib/netlink.c one of the iov bufs being too
> small.

I've seen the netlink buffer issue with other network configuration tools as well.

Comment 5 Pavel Šimerda (pavlix) 2015-04-16 23:16:25 UTC

I expected that thousands of dummy interfaces would be enough for reproducing this but apparently not. Therefore while it's easy to change buffer sizes, it's hard to know which ones are actually insufficient if we can't test the change.

What I can say is that there are two 16k buffers and two 8k buffers in the rhel6 and rhel7 libnetlink.c source file as found using `grep 'buf\['` but in upstream the latter two are also 16k so there apparently is a difference from upstream.

Comment 8 Phil Sutter 2015-10-15 15:34:08 UTC

I can not reproduce the issue locally. Could anyone perhaps get me access to a test system exposing the problem?

Comment 9 Rashid Khan 2015-10-23 22:37:29 UTC

Hi Michael, 
Can you please send a reliable reproducer for this. 
If we do not hear from you soon, then we might not be able to include it in our next RHEL6.x release. 
Thanks

Comment 10 Michael Samuel 2015-10-26 03:58:35 UTC

I no longer have access to the affected system, as I've changed jobs.

It should be reproducible by just creating a system (maybe a VM) with a lot of network interfaces.

Comment 11 Phil Sutter 2015-10-26 11:59:20 UTC

Hi Michael,

(In reply to Michael Samuel from comment #10)
> I no longer have access to the affected system, as I've changed jobs.
> 
> It should be reproducible by just creating a system (maybe a VM) with a lot
> of network interfaces.

OK. As stated, we can't reproduce this so I'm closing it for now. Feel free to reopen in case you run into this problem again, of course.

Comment 12 Laine Stump 2016-01-13 16:00:28 UTC

This was seen on a system running RHEL6.7 with 80 VFs, a couple of PFs, and 15 macvtap interfaces (created by libvirt and attached to 15 of the VFs), so I'm reopening.

Comment 14 Laine Stump 2016-01-13 19:20:37 UTC

As I suspected, this seems to be specific to certain types of interfaces. I did a test where I created 150 bridges attached to 150 vlan interfaces, and "ip -d link show" continued to work properly.

Most likely it is related to the number of SRIOV virtual functions (which would affect the number of elements in the IFLA_VFINFO_LIST, which is all put into a single netlink response message when an IFLA_EXT_MASK with RTEXT_FILTER_VF is put in the RTM_GETLINK request that precedes it). I only have an 82576 card available though, so the maximum VFs I can get on my system is 14, which is much too low to see the problem.

Comment 18 Phil Sutter 2016-01-14 18:22:43 UTC

Created attachment 1114921 [details]
iproute-2.6.32-33.notrunc.el6_6.x86_64

Comment 36 errata-xmlrpc 2016-05-11 00:18:49 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0895.html

Note You need to log in before you can comment on or make changes to this bug.