Bug 1275542 - It reports "RTNETLINK answers: Invalid argument" internal error sometimes when restore guests after upgrading the host OS from RHEL7.1 to RHEL7.2.
It reports "RTNETLINK answers: Invalid argument" internal error sometimes whe...
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: iproute (Show other bugs)
7.2
x86_64 Linux
medium Severity medium
: rc
: ---
Assigned To: Phil Sutter
BaseOS QE Security Team
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-27 04:23 EDT by Fangge Jin
Modified: 2017-08-17 13:16 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-17 13:16:34 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
libvirtd log (1.16 MB, application/x-gzip)
2015-10-27 04:26 EDT, Fangge Jin
no flags Details
guest xml (3.06 KB, application/x-gzip)
2015-10-27 04:27 EDT, Fangge Jin
no flags Details

  None (edit)
Description Fangge Jin 2015-10-27 04:23:50 EDT
Description of problem:
It reports "RTNETLINK answers: Invalid argument" internal error when restore guests after upgrading the host OS from RHEL7.1 to RHEL7.2.

Version-Release number of selected component (if applicable):
RHEL7.1 host:
kernel-3.10.0-229.el7.x86_64
libvirt-1.2.8-16.el7.x86_64
qemu-kvm-rhev-2.1.2-23.el7.x86_64
iproute-3.10.0-21.el7.x86_64

Upgrade to RHEL7.2:
kernel-3.10.0-324.el7.x86_64
libvirt-1.2.17-13.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7.x86_64
iproute-3.10.0-54.el7.x86_64

How reproducible:
I'm not sure, I tried for several times, sometimes it can be reproduced, while sometimes it can't, maybe 50%.

Steps to Reproduce:
On RHEL7.1 host:
1.Edit the default network to include the QoS elements:
# virsh net-dumpxml default
<network>
  <name>default</name>
...
  <bandwidth>
    <inbound average='1000' peak='5000' burst='5120'/>
    <outbound average='128' peak='256' burst='256'/>
  </bandwidth>

2.Prepare several running guests which include QoS elements for network interface:
    <interface type='network'>
      <source network='default'/>
...
      <bandwidth>
        <inbound average='1000' peak='5000' floor='200' burst='1024'/>
        <outbound average='128' peak='256' burst='256'/>
      </bandwidth>
...

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     rhel6.7-1-managedsave          running
 5     rhel6.7-1-save                 running
 8     rhel7.2-managedsave            running
 9     rhel7.2-save                   running
 -     rhel6.7-managedsave            shut off
 -     rhel6.7-save                   shut off
 -     rhel7.2-1-managedsave          shut off
 -     rhel7.2-1-save                 shut off

3.Do managedsave and save:
# virsh managedsave rhel6.7-1-managedsave
# virsh save rhel6.7-1-save rhel6.7-1.save
# virsh managedsave rhel7.2-managedsave
# virsh save rhel7.2-save rhel7.2.save

4.Upgrade the host to RHEL7.2:
# yum update -y

5.After upgrading completed, try to restore the guests:
# virsh start rhel6.7-1-managedsave
error: Failed to start domain rhel6.7-1-managedsave
error: internal error: Child process (/sbin/tc filter add dev virbr0 protocol ip prio 2 handle 800::5 u32 match u16 0x0800 0xffff at -2 match u32 0x00c63b95 0xffffffff at -12 match u16 0x5228 0xffff at -14 flowid 1:5) unexpected exit status 2: 2015-10-26 10:38:25.500+0000: 32536: debug : virFileClose:102 : Closed fd 26
2015-10-26 10:38:25.500+0000: 32536: debug : virFileClose:102 : Closed fd 28
2015-10-26 10:38:25.500+0000: 32536: debug : virFileClose:102 : Closed fd 24
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

# virsh restore rhel6.7-1.save
error: Failed to restore domain from rhel6.7-1.save
error: internal error: Child process (/sbin/tc filter add dev virbr0 protocol ip prio 2 handle 800::6 u32 match u16 0x0800 0xffff at -2 match u32 0x00c74b95 0xffffffff at -12 match u16 0x5254 0xffff at -14 flowid 1:6) unexpected exit status 2: 2015-10-26 10:38:54.284+0000: 32551: debug : virFileClose:102 : Closed fd 26
2015-10-26 10:38:54.284+0000: 32551: debug : virFileClose:102 : Closed fd 28
2015-10-26 10:38:54.284+0000: 32551: debug : virFileClose:102 : Closed fd 24
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

# virsh start rhel7.2-managedsave
error: Failed to start domain rhel7.2-managedsave
error: internal error: Child process (/sbin/tc filter add dev virbr0 protocol ip prio 2 handle 800::7 u32 match u16 0x0800 0xffff at -2 match u32 0x00d33b95 0xffffffff at -12 match u16 0x5254 0xffff at -14 flowid 1:7) unexpected exit status 2: 2015-10-26 10:39:04.681+0000: 32565: debug : virFileClose:102 : Closed fd 26
2015-10-26 10:39:04.681+0000: 32565: debug : virFileClose:102 : Closed fd 28
2015-10-26 10:39:04.681+0000: 32565: debug : virFileClose:102 : Closed fd 24
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

# virsh restore rhel7.2.save
error: Failed to restore domain from rhel7.2.save
error: internal error: Child process (/sbin/tc filter add dev virbr0 protocol ip prio 2 handle 800::8 u32 match u16 0x0800 0xffff at -2 match u32 0x00d73b95 0xffffffff at -12 match u16 0x5254 0xffff at -14 flowid 1:8) unexpected exit status 2: 2015-10-26 10:39:16.008+0000: 32580: debug : virFileClose:102 : Closed fd 26
2015-10-26 10:39:16.008+0000: 32580: debug : virFileClose:102 : Closed fd 28
2015-10-26 10:39:16.008+0000: 32580: debug : virFileClose:102 : Closed fd 24
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

6.Try to start new guest which includes interface Qos elements, also failed:
# virsh start rhel7.2-1-managedsave
error: Failed to start domain rhel7.2-1-managedsave
error: internal error: Child process (/sbin/tc filter add dev virbr0 protocol ip prio 2 handle 800::9 u32 match u16 0x0800 0xffff at -2 match u32 0x12c63b95 0xffffffff at -12 match u16 0x5254 0xffff at -14 flowid 1:9) unexpected exit status 2: 2015-10-26 10:40:38.041+0000: 32751: debug : virFileClose:102 : Closed fd 25
2015-10-26 10:40:38.041+0000: 32751: debug : virFileClose:102 : Closed fd 27
2015-10-26 10:40:38.041+0000: 32751: debug : virFileClose:102 : Closed fd 23
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

7.Remove the interface Qos elements from the guest xml, restore guest successfully:
# virsh sace-image-edit rhel7.2.save
State file rhel7.2.save edited.

#virsh restore rhel7.2.save
Domain restored from rhel7.2.save

Or reboot the host immediately after upgrading, then restore guest, also succeed:
# virsh restore rhel6.7-1.save
Domain restored from rhel6.7-1.save

# virsh start rhel7.2-managedsave
Domain rhel7.2-managedsave started


Actual results:
As steps

Expected results:
Guests restore successfully without any error after upgrading host os.

Additional info:
Comment 2 Fangge Jin 2015-10-27 04:26 EDT
Created attachment 1086751 [details]
libvirtd log
Comment 3 Fangge Jin 2015-10-27 04:27 EDT
Created attachment 1086752 [details]
guest xml
Comment 4 Michal Privoznik 2016-06-10 05:34:39 EDT
This looks like a kernel/iproute problem to me. Moreover, if libvirt is spawning the same command over and over again and sometimes it succeeds and sometimes fails. Switching over to iproute.
Comment 5 Phil Sutter 2016-06-10 08:07:53 EDT
Hi Michal,

(In reply to Michal Privoznik from comment #4)
> This looks like a kernel/iproute problem to me. Moreover, if libvirt is
> spawning the same command over and over again and sometimes it succeeds and
> sometimes fails. Switching over to iproute.

Can you please list the exact command which is spawned by libvirt and sometimes fails?

Thanks, Phil
Comment 6 Phil Sutter 2016-07-14 09:11:18 EDT
(In reply to Phil Sutter from comment #5)
> Hi Michal,
> 
> (In reply to Michal Privoznik from comment #4)
> > This looks like a kernel/iproute problem to me. Moreover, if libvirt is
> > spawning the same command over and over again and sometimes it succeeds and
> > sometimes fails. Switching over to iproute.
> 
> Can you please list the exact command which is spawned by libvirt and
> sometimes fails?

Ah, reading helps - it's all stated above, sorry for the noise.


(In reply to JinFangge from comment #0)
> 5.After upgrading completed, try to restore the guests:
> # virsh start rhel6.7-1-managedsave
> error: Failed to start domain rhel6.7-1-managedsave
> error: internal error: Child process (/sbin/tc filter add dev virbr0
> protocol ip prio 2 handle 800::5 u32 match u16 0x0800 0xffff at -2 match u32
> 0x00c63b95 0xffffffff at -12 match u16 0x5228 0xffff at -14 flowid 1:5)
> unexpected exit status 2: 2015-10-26 10:38:25.500+0000: 32536: debug :
> virFileClose:102 : Closed fd 26
> 2015-10-26 10:38:25.500+0000: 32536: debug : virFileClose:102 : Closed fd 28
> 2015-10-26 10:38:25.500+0000: 32536: debug : virFileClose:102 : Closed fd 24
> RTNETLINK answers: Invalid argument
> We have an error talking to the kernel

It looks like the qdisc/class setup of virbr0 is incorrect in that situation
so that the quoted tc filter command does not apply. Sadly there is no way to
get more error details from tc, so we have to do some black box analysis:

Can you please run the following commands before and after the failure:

# ip -d addr show dev virbr0
# tc qd show dev virbr0
# tc class show dev virbr0
# tc filter show dev virbr0

Thanks, Phil
Comment 7 Fangge Jin 2016-07-15 11:10:22 EDT
I will have a try, but it's hard to reproduce the bug.

Note You need to log in before you can comment on or make changes to this bug.