Bug 844287

Summary: Cryptic error message when 'virsh detach-device' of a network device fails due to missing mac address
Product: Red Hat Enterprise Linux 6 Reporter: Alex Jia <ajia>
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED WONTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: high    
Version: 6.4CC: acathrow, dallan, dyuan, jdenemar, mzhan
Target Milestone: rcKeywords: Upstream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-04-04 21:00:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex Jia 2012-07-30 08:29:16 UTC
Description of problem:
Libvirt allows to hot-plug a 'hostdev' type network interface into the guest via 'virsh attach-device', but don't allows to hot-unplug 'hostdev' type network interface from the guest via 'virsh detach-device'.

Version-Release number of selected component (if applicable):
# rpm -q libvirt qemu-kvm-rhev
libvirt-0.9.13-3.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.297.el6_3.x86_64

How reproducible:
always.

Steps to Reproduce:

# lspci|grep 82576
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)


# modprobe -r kvm_intel kvm
# modprobe kvm allow_unsafe_assigned_interrupts=1
# modprobe kvm_intel

# virsh start myVF
Domain myVF started

# readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
/sys/bus/pci/drivers/igbvf

# cat hostdev.xml 
    <interface type='hostdev' managed='yes'>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
      </source>
    </interface>

# virsh attach-device myVF hostdev.xml 
Device attached successfully

Notes, can hot-plug a 'hostdev' type network interface into guest.

# virsh dumpxml myVF
<domain type='kvm' id='9'>
  <name>myVF</name>
  ......
  <devices>
  ......
    <interface type='network'>
      <mac address='52:54:00:81:94:f7'/>
      <source network='default'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:ec:52:44'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
  ......
  </devices>
  ......
</domain>

# readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
/sys/bus/pci/drivers/pci-stub

# virsh detach-device myVF hostdev.xml 
error: Failed to detach device from hostdev.xml
error: operation failed: network device 52:54:00:a4:8b:17 not found

Notes, there are 2 questions in here, the one is libvirt allows to use 'virsh attach-device' to attach a interface XML to guest, but don't allow detach it from guest via 'virsh detach-device'. the other is the error information also is weird, I haven't specified any MAC address in hostdev.xml.


  
Actual results:
# virsh detach-device myVF hostdev.xml 
error: Failed to detach device from hostdev.xml
error: operation failed: network device 52:54:00:a4:8b:17 not found

Expected results:
Allow to hot-unplug 'hostdev' type network interface from the guest via 'virsh detach-device'

Additional info:

Comment 2 Laine Stump 2012-07-30 15:47:46 UTC
> 
> # cat hostdev.xml 
>     <interface type='hostdev' managed='yes'>
>       <source>
>         <address type='pci' domain='0x0000' bus='0x03' slot='0x10'
> function='0x1'/>
>       </source>
>     </interface>

Network devices are uniquely identified by their MAC address. You didn't specify a MAC address in this xml, so a random mac was automatically generated (this is exactly what would happen for any of the other types of network devices). Since that mac doesn't match the mac that you assigned to the device when you attached it, the detach failed to find the device.

 
> # virsh dumpxml myVF
> <domain type='kvm' id='9'>
>   <name>myVF</name>
>   ......
>     <interface type='hostdev' managed='yes'>
>       <mac address='52:54:00:ec:52:44'/>

You need to put this mac address in the xml you send to detach-device.

> Notes, there are 2 questions in here, the one is libvirt allows to use
> 'virsh attach-device' to attach a interface XML to guest, but don't allow
> detach it from guest via 'virsh detach-device'. the other is the error
> information also is weird, I haven't specified any MAC address in
> hostdev.xml.

And that is exactly the problem. I do think that the error message in this case could be less mysterious though, so I'm leaving the bug open and changing the summary accordingly

Note that you could also detach this interface with:

   virsh detach-interface myVF hostdev 52:54:00:ec:52:44

Comment 3 Alex Jia 2012-07-31 02:49:41 UTC
(In reply to comment #2)
> > 
> > # cat hostdev.xml 
> >     <interface type='hostdev' managed='yes'>
> >       <source>
> >         <address type='pci' domain='0x0000' bus='0x03' slot='0x10'
> > function='0x1'/>
> >       </source>
> >     </interface>
> 
> Network devices are uniquely identified by their MAC address. You didn't
> specify a MAC address in this xml, so a random mac was automatically
> generated (this is exactly what would happen for any of the other types of
> network devices). Since that mac doesn't match the mac that you assigned to
> the device when you attached it, the detach failed to find the device.
> 
>  
> > # virsh dumpxml myVF
> > <domain type='kvm' id='9'>
> >   <name>myVF</name>
> >   ......
> >     <interface type='hostdev' managed='yes'>
> >       <mac address='52:54:00:ec:52:44'/>
> 
> You need to put this mac address in the xml you send to detach-device.
> 
> > Notes, there are 2 questions in here, the one is libvirt allows to use
> > 'virsh attach-device' to attach a interface XML to guest, but don't allow
> > detach it from guest via 'virsh detach-device'. the other is the error
> > information also is weird, I haven't specified any MAC address in
> > hostdev.xml.

It's different from detaching a host network device from the guest via  <hostdev>...</hostdev> XML configuration, we don't need to specify a MAC in <hostdev>...</hostdev> section, so maybe, we should document it.   


> 
> And that is exactly the problem. I do think that the error message in this
> case could be less mysterious though, so I'm leaving the bug open and
> changing the summary accordingly
> 
> Note that you could also detach this interface with:
> 
>    virsh detach-interface myVF hostdev 52:54:00:ec:52:44

Yeah, I see, but firstly, {attach, detach}-device is a pair of operation, and {attach, detach}-interface is also. In addition, it exists a bug for the above operation(see bug 843016), although the device can be detached from the guest.

Thanks for your comments.

Comment 4 Laine Stump 2012-07-31 05:32:11 UTC
(In reply to comment #3)

> It's different from detaching a host network device from the guest via 
> <hostdev>...</hostdev> XML configuration, we don't need to specify a MAC in
> <hostdev>...</hostdev> section, so maybe, we should document it.   

Yes, because it isn't <hostdev>, it is an <interface>, and <interface> devices always require a MAC address in the detach XML - that is what is matched when searching for which device to detach. (<hostdev> devices don't even *have* a mac address; they search for matching source PCI address). Similarly, when you detach a disk, you need to specify the <target dev='xxx'/>). Of course, if you just include the entire element as it exists at the time in the domain's xml, then you will always have all the information necessary to find the device.

You are correct though that if the search index for each type of detachable device isn't documented somewhere, it should be.

> 
> 
> > 
> > And that is exactly the problem. I do think that the error message in this
> > case could be less mysterious though, so I'm leaving the bug open and
> > changing the summary accordingly
> > 
> > Note that you could also detach this interface with:
> > 
> >    virsh detach-interface myVF hostdev 52:54:00:ec:52:44
> 
> Yeah, I see, but firstly, {attach, detach}-device is a pair of operation,
> and {attach, detach}-interface is also.

You imply that if attach-device is used to attach an <interface> device, that detach-device must be used to detach it. That is not true. detach-device and detach-interface are both just frontends for the libvirt function virDomainDetachDevice(), just as attach-device and attach-interface are both frontends for virDomainAttachDevice(). It is perfectly acceptable to use attach-device to attach an <interface>, but then use detach-interface to detach it (or the other way around).

At any rate, I wasn't suggesting that command as some sort of solution to a problem, just pointing out its existence.


By the way, here's another interesting point (which kind of contradicts what I said before about this not being a <hostdev>) - although the device you've attached is an <interface>, it's also internally kept on the list of <hostdev> devices for the domain, so it can also be detached with this xml:

  <hostdev type='pci' managed='yes'>
    <source>
      <address domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
    </source>
  </hostdev>

The device detach code for <hostdev> is intelligent enough to notice that the PCI device that you're describing was actually attached as <interface type='hostdev'>, and calls out to the <interface> detach code instead. (Of course you can't do this with attach-device, as it wouldn't get a MAC address set before assignment to the guest, and the whole point of <interface type='hostdev'> is to provide a place to specify the MAC address (and set it))

Comment 6 Alex Jia 2012-08-01 05:22:24 UTC
BTW, I added MAC address into hostdev.xml then use 'virsh detach-device' to hot-unplug the PCI device from the guest, although the result is successful, but the device also hasn't been returned to host:

# cat hostdev.xml
    <interface type='hostdev' managed='yes'>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
      </source>
    </interface>

Notes, no MAC address.

# virsh attach-device myVF hostdev.xml 
Device attached successfully

# readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
/sys/bus/pci/drivers/pci-stub

# cat hostdev.xml
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:82:2c:f4'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
      </source>
    </interface>

Notes, add MAC address line.

# virsh detach-device myVF hostdev.xml 
Device detached successfully

# readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
/sys/devices/pci0000:00/0000:00:01.0/0000:03:10.1/driver

Notes, it's known bug 843016.

Comment 7 Laine Stump 2012-08-01 07:41:13 UTC
I will just repeat here that the only bug I've seen described (other thanThe other problem already being tracked in Bug 843016) is that the error message is cryptic when no MAC address is specified in a detach-device command for an <interface> device. Other than that, from everything I see here, it's all operating as expected.

Comment 8 Alex Jia 2012-08-01 08:15:27 UTC
(In reply to comment #7)
> I will just repeat here that the only bug I've seen described (other thanThe
> other problem already being tracked in Bug 843016) is that the error message
> is cryptic when no MAC address is specified in a detach-device command for
> an <interface> device. Other than that, from everything I see here, it's all
> operating as expected.

Yeah, absolutely agree.

Comment 10 Laine Stump 2013-05-23 05:17:26 UTC
This is a bit involved to fix. There are two possible ways to fix it:

1) modify virsh to parse the <interface> xml prior to sending it to libvirt, notice that the <mac address> element is missing, then search the existing interface list for the domain and see if there is one-and-only-one interface that matches the rest of the provided <interface> xml; if so, it would add an  <mac address='xx:xx:xx:xx:xx:xx'/> element to the xml and send it to. virDomainDetachDeviceFlags()

2) make the changes on the libvirt side:

A) modify virDomainNetDefParseXML() to recognize a new flag VIR_DOMAIN_INTERNAL_NO_DEFAULTS_PARSE - if that flag is passed in, the bit of code that auto-generates a mac address when one isn't specified would be skipped.

B) add a "mac_present" bool to virNetDef (since mac isn't a pointer, and it's possible (although highly unlikely) that someone may actually use 00:00:00:00:00:00 as a mac address

C) modify virDomainNetFindIdx() to not require a MAC address - if none is given, match on guest-side PCI address (if neither is given, return an error).

Comment 11 Laine Stump 2013-07-02 07:06:37 UTC
Upstream libvirt does a much better job of this due to the following patch:


commit def31e4c5812aff869ebe00be6c4896920223012
Author: Laine Stump <laine>
Date:   Thu Oct 25 16:03:35 2012 -0400

    qemu: fix attach/detach of netdevs with matching mac addrs
    
    This resolves:
    
       https://bugzilla.redhat.com/show_bug.cgi?id=862515
    
    which describes inconsistencies in dealing with duplicate mac
    addresses on network devices in a domain.

But still always generates a new random MAC address for any <interface> xml definition that doesn't have one. Fixing this would require making the call to virDomainNetGenerateMAC() optional in the domain interface parser

At any rate, the backport effort of existing patches, plus the new work is too much for the available capacity for RHEL 6.5, so I'm moving it to 6.6.

Comment 14 RHEL Program Management 2014-04-04 21:00:44 UTC
Development Management has reviewed and declined this request.
You may appeal this decision by reopening this request.