RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1275542 - It reports "RTNETLINK answers: Invalid argument" internal error sometimes when restore guests after upgrading the host OS from RHEL7.1 to RHEL7.2.
Summary: It reports "RTNETLINK answers: Invalid argument" internal error sometimes whe...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: iproute
Version: 7.2
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Phil Sutter
QA Contact: BaseOS QE Security Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-10-27 08:23 UTC by Fangge Jin
Modified: 2017-08-17 17:16 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-17 17:16:34 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
libvirtd log (1.16 MB, application/x-gzip)
2015-10-27 08:26 UTC, Fangge Jin 		no flags Details
guest xml (3.06 KB, application/x-gzip)
2015-10-27 08:27 UTC, Fangge Jin 		no flags Details
View All

Description Fangge Jin 2015-10-27 08:23:50 UTC 
Description of problem:
It reports "RTNETLINK answers: Invalid argument" internal error when restore guests after upgrading the host OS from RHEL7.1 to RHEL7.2.

Version-Release number of selected component (if applicable):
RHEL7.1 host:
kernel-3.10.0-229.el7.x86_64
libvirt-1.2.8-16.el7.x86_64
qemu-kvm-rhev-2.1.2-23.el7.x86_64
iproute-3.10.0-21.el7.x86_64

Upgrade to RHEL7.2:
kernel-3.10.0-324.el7.x86_64
libvirt-1.2.17-13.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7.x86_64
iproute-3.10.0-54.el7.x86_64

How reproducible:
I'm not sure, I tried for several times, sometimes it can be reproduced, while sometimes it can't, maybe 50%.

Steps to Reproduce:
On RHEL7.1 host:
1.Edit the default network to include the QoS elements:
# virsh net-dumpxml default
<network>
  <name>default</name>
...
  <bandwidth>
    <inbound average='1000' peak='5000' burst='5120'/>
    <outbound average='128' peak='256' burst='256'/>
  </bandwidth>

2.Prepare several running guests which include QoS elements for network interface:
    <interface type='network'>
      <source network='default'/>
...
      <bandwidth>
        <inbound average='1000' peak='5000' floor='200' burst='1024'/>
        <outbound average='128' peak='256' burst='256'/>
      </bandwidth>
...

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     rhel6.7-1-managedsave          running
 5     rhel6.7-1-save                 running
 8     rhel7.2-managedsave            running
 9     rhel7.2-save                   running
 -     rhel6.7-managedsave            shut off
 -     rhel6.7-save                   shut off
 -     rhel7.2-1-managedsave          shut off
 -     rhel7.2-1-save                 shut off

3.Do managedsave and save:
# virsh managedsave rhel6.7-1-managedsave
# virsh save rhel6.7-1-save rhel6.7-1.save
# virsh managedsave rhel7.2-managedsave
# virsh save rhel7.2-save rhel7.2.save

4.Upgrade the host to RHEL7.2:
# yum update -y

5.After upgrading completed, try to restore the guests:
# virsh start rhel6.7-1-managedsave
error: Failed to start domain rhel6.7-1-managedsave
error: internal error: Child process (/sbin/tc filter add dev virbr0 protocol ip prio 2 handle 800::5 u32 match u16 0x0800 0xffff at -2 match u32 0x00c63b95 0xffffffff at -12 match u16 0x5228 0xffff at -14 flowid 1:5) unexpected exit status 2: 2015-10-26 10:38:25.500+0000: 32536: debug : virFileClose:102 : Closed fd 26
2015-10-26 10:38:25.500+0000: 32536: debug : virFileClose:102 : Closed fd 28
2015-10-26 10:38:25.500+0000: 32536: debug : virFileClose:102 : Closed fd 24
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

# virsh restore rhel6.7-1.save
error: Failed to restore domain from rhel6.7-1.save
error: internal error: Child process (/sbin/tc filter add dev virbr0 protocol ip prio 2 handle 800::6 u32 match u16 0x0800 0xffff at -2 match u32 0x00c74b95 0xffffffff at -12 match u16 0x5254 0xffff at -14 flowid 1:6) unexpected exit status 2: 2015-10-26 10:38:54.284+0000: 32551: debug : virFileClose:102 : Closed fd 26
2015-10-26 10:38:54.284+0000: 32551: debug : virFileClose:102 : Closed fd 28
2015-10-26 10:38:54.284+0000: 32551: debug : virFileClose:102 : Closed fd 24
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

# virsh start rhel7.2-managedsave
error: Failed to start domain rhel7.2-managedsave
error: internal error: Child process (/sbin/tc filter add dev virbr0 protocol ip prio 2 handle 800::7 u32 match u16 0x0800 0xffff at -2 match u32 0x00d33b95 0xffffffff at -12 match u16 0x5254 0xffff at -14 flowid 1:7) unexpected exit status 2: 2015-10-26 10:39:04.681+0000: 32565: debug : virFileClose:102 : Closed fd 26
2015-10-26 10:39:04.681+0000: 32565: debug : virFileClose:102 : Closed fd 28
2015-10-26 10:39:04.681+0000: 32565: debug : virFileClose:102 : Closed fd 24
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

# virsh restore rhel7.2.save
error: Failed to restore domain from rhel7.2.save
error: internal error: Child process (/sbin/tc filter add dev virbr0 protocol ip prio 2 handle 800::8 u32 match u16 0x0800 0xffff at -2 match u32 0x00d73b95 0xffffffff at -12 match u16 0x5254 0xffff at -14 flowid 1:8) unexpected exit status 2: 2015-10-26 10:39:16.008+0000: 32580: debug : virFileClose:102 : Closed fd 26
2015-10-26 10:39:16.008+0000: 32580: debug : virFileClose:102 : Closed fd 28
2015-10-26 10:39:16.008+0000: 32580: debug : virFileClose:102 : Closed fd 24
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

6.Try to start new guest which includes interface Qos elements, also failed:
# virsh start rhel7.2-1-managedsave
error: Failed to start domain rhel7.2-1-managedsave
error: internal error: Child process (/sbin/tc filter add dev virbr0 protocol ip prio 2 handle 800::9 u32 match u16 0x0800 0xffff at -2 match u32 0x12c63b95 0xffffffff at -12 match u16 0x5254 0xffff at -14 flowid 1:9) unexpected exit status 2: 2015-10-26 10:40:38.041+0000: 32751: debug : virFileClose:102 : Closed fd 25
2015-10-26 10:40:38.041+0000: 32751: debug : virFileClose:102 : Closed fd 27
2015-10-26 10:40:38.041+0000: 32751: debug : virFileClose:102 : Closed fd 23
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

7.Remove the interface Qos elements from the guest xml, restore guest successfully:
# virsh sace-image-edit rhel7.2.save
State file rhel7.2.save edited.

#virsh restore rhel7.2.save
Domain restored from rhel7.2.save

Or reboot the host immediately after upgrading, then restore guest, also succeed:
# virsh restore rhel6.7-1.save
Domain restored from rhel6.7-1.save

# virsh start rhel7.2-managedsave
Domain rhel7.2-managedsave started


Actual results:
As steps

Expected results:
Guests restore successfully without any error after upgrading host os.

Additional info:
Comment 2 Fangge Jin 2015-10-27 08:26:17 UTC 
Created attachment 1086751 [details]
libvirtd log
Comment 3 Fangge Jin 2015-10-27 08:27:21 UTC 
Created attachment 1086752 [details]
guest xml
Comment 4 Michal Privoznik 2016-06-10 09:34:39 UTC 
This looks like a kernel/iproute problem to me. Moreover, if libvirt is spawning the same command over and over again and sometimes it succeeds and sometimes fails. Switching over to iproute.
Comment 5 Phil Sutter 2016-06-10 12:07:53 UTC 
Hi Michal,

(In reply to Michal Privoznik from comment #4)
> This looks like a kernel/iproute problem to me. Moreover, if libvirt is
> spawning the same command over and over again and sometimes it succeeds and
> sometimes fails. Switching over to iproute.

Can you please list the exact command which is spawned by libvirt and sometimes fails?

Thanks, Phil
Comment 6 Phil Sutter 2016-07-14 13:11:18 UTC 
(In reply to Phil Sutter from comment #5)
> Hi Michal,
> 
> (In reply to Michal Privoznik from comment #4)
> > This looks like a kernel/iproute problem to me. Moreover, if libvirt is
> > spawning the same command over and over again and sometimes it succeeds and
> > sometimes fails. Switching over to iproute.
> 
> Can you please list the exact command which is spawned by libvirt and
> sometimes fails?

Ah, reading helps - it's all stated above, sorry for the noise.


(In reply to JinFangge from comment #0)
> 5.After upgrading completed, try to restore the guests:
> # virsh start rhel6.7-1-managedsave
> error: Failed to start domain rhel6.7-1-managedsave
> error: internal error: Child process (/sbin/tc filter add dev virbr0
> protocol ip prio 2 handle 800::5 u32 match u16 0x0800 0xffff at -2 match u32
> 0x00c63b95 0xffffffff at -12 match u16 0x5228 0xffff at -14 flowid 1:5)
> unexpected exit status 2: 2015-10-26 10:38:25.500+0000: 32536: debug :
> virFileClose:102 : Closed fd 26
> 2015-10-26 10:38:25.500+0000: 32536: debug : virFileClose:102 : Closed fd 28
> 2015-10-26 10:38:25.500+0000: 32536: debug : virFileClose:102 : Closed fd 24
> RTNETLINK answers: Invalid argument
> We have an error talking to the kernel

It looks like the qdisc/class setup of virbr0 is incorrect in that situation
so that the quoted tc filter command does not apply. Sadly there is no way to
get more error details from tc, so we have to do some black box analysis:

Can you please run the following commands before and after the failure:

# ip -d addr show dev virbr0
# tc qd show dev virbr0
# tc class show dev virbr0
# tc filter show dev virbr0

Thanks, Phil
Comment 7 Fangge Jin 2016-07-15 15:10:22 UTC 
I will have a try, but it's hard to reproduce the bug.
Note You need to log in before you can comment on or make changes to this bug.