Bug 880483

Two potential issues here:

1) people who've been using networks that are marked as inactive will get a surprise when they upgrade.

2) A restart of libvirtd switches active unmanaged networks to inactive (unless they have autostart set), since there is no readily available "clue" to determine if a network is active or not. Apparently more information needs to be stored in the network state xml.

Moving to 6.6 due to lack of capacity.

Hi,

If forward mode is any of bridge/hostdev/passthrough/private, will reproduce that issue. But I can't reproduce it on nat(default virtual network) and route forward mode. 

XML config as below:

1. route mode
<network>
     <name>route-net</name>
     <bridge name="virbr2" />
     <forward mode="route" dev="eth0"/>
     <ip address="192.168.125.1" netmask="255.255.255.0">
       <dhcp>
          <range start="192.168.125.2" end="192.168.125.254" />
       </dhcp>
     </ip>
</network>

2. bridge mode
<network>
  <name>bridge-net</name>
  <forward dev='eth0' mode='bridge'>
    <interface dev='eth0'/>
  </forward>
</network>

3.hostdev mode
<network>
   <name>hostdev-net</name>
   <forward mode='hostdev' managed='yes'>
     <pf dev='eth1'/>                           
   </forward>
</network>

4.passthrough mode
<network>
   <name>passthrough-net</name>
   <forward mode='passthrough'>
   <interface dev='eth10'/>
   <interface dev='eth11'/>                       
   </forward>
</network>

5.private mode
<network>
   <name>private-net</name>
   <forward mode='private'>
   <interface dev='eth10'/>
   <interface dev='eth11'/>                       
   </forward>
</network>

Steps:
1. Define above virtual networks;(Do not start them.)
2. Add them to domain;
3. Start the domain.

Expected result:
For nat and route, like below:
[root@sriov2 ~]# virsh start r6_1
error: Failed to start domain r6_1
error: internal error Network 'default' is not active.

For the rest of modes, domain can start normally, and the inactive virtual network works well in domain.

*** Bug 1002427 has been marked as a duplicate of this bug. ***

This bug was not selected to be addressed in Red Hat Enterprise Linux 6. We will look at it again within the Red Hat Enterprise Linux 7 product.

*** Bug 1002438 has been marked as a duplicate of this bug. ***

A series of patches has been proposed upstream:

https://www.redhat.com/archives/libvir-list/2014-April/msg00454.html

The following patches, which fix this and a few other problems with networks (and which must all be applied together) have been pushed to upstream git:

commit 440beeb7acfe45014be753760ffcc12606462134
Author: Laine Stump <laine>
Date:   Tue Apr 22 16:48:54 2014 +0300

    network: fix virNetworkObjAssignDef and persistence

commit 8aaa5b68efd58117951d76a67f20798f959a17fa
Author: Laine Stump <laine>
Date:   Fri Apr 4 14:21:13 2014 +0300

    network: create statedir during driver initialization

commit b9e95491d120d05737b77e2c4dc14ba5dfac5fb0
Author: Laine Stump <laine>
Date:   Fri Apr 4 16:48:54 2014 +0300

    network: change location of network state xml files

commit 411c5486685b81d0c04ad7c83ac3552ec011968c
Author: Laine Stump <laine>
Date:   Wed Apr 9 17:16:45 2014 +0300

    network: set macvtap/hostdev networks active if their state file exists

commit 34cc3b2f106e296df5e64309620c79d16fd76c85
Author: Laine Stump <laine>
Date:   Thu Apr 10 14:44:07 2014 +0300

    network: centralize check for active network during interface attach

I found libvirt can not save network status files to the new location, and remove them from the old location.

[root@localhost libvirt_wget]# rpm -q libvirt
libvirt-1.1.1-29.el7.x86_64
[root@localhost libvirt_wget]#
[root@localhost libvirt_wget]# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 bridge-net           active     no            yes
 default              active     yes           yes
 route-net            active     no            yes

[root@localhost libvirt_wget]# ll /var/lib/libvirt/network/
total 12
-rw-------. 1 root root 501 Nov 21 16:56 bridge-net.xml
-rw-------. 1 root root 760 Nov 21 16:56 default.xml
-rw-------. 1 root root 738 Nov 21 16:56 route-net.xml
[root@localhost libvirt_wget]# ll /var/run/libvirt/network/
total 8
-rw-r--r--. 1 root root 6 Nov 21 16:56 default.pid
-rw-r--r--. 1 root root 0 Nov 21 16:56 nwfilter.leases
-rw-r--r--. 1 root root 6 Nov 21 16:56 route-net.pid

[root@localhost libvirt-1.2.8]# rpm -q libvirt
libvirt-1.2.8-7.el7.x86_64

[root@localhost libvirt-1.2.8-7.el7]# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service
[root@localhost libvirt-1.2.8-7.el7]# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 bridge-net           inactive   no            yes
 default              inactive   yes           yes
 route-net            inactive   no            yes

[root@localhost libvirt-1.2.8-7.el7]# ll /var/lib/libvirt/network/
total 12
-rw-------. 1 root root 501 Nov 21 17:12 bridge-net.xml
-rw-------. 1 root root 760 Nov 21 17:11 default.xml
-rw-------. 1 root root 738 Nov 21 17:11 route-net.xml
[root@localhost libvirt-1.2.8-7.el7]# ll /var/run/libvirt/network/
total 8
-rw-r--r--. 1 root root 6 Nov 21 17:11 default.pid
-rw-r--r--. 1 root root 0 Nov 21 18:14 nwfilter.leases
-rw-r--r--. 1 root root 6 Nov 21 17:11 route-net.pid


My debugging info:
[root@localhost libvirt-1.2.8]# rpm -q libvirt
libvirt-1.2.8-7.el7.x86_64

Breakpoint 2, networkStateInitialize (privileged=true, callback=0x55555556ade0 <daemonInhibitCallback>, opaque=0x5555557e6f70) at network/bridge_driver.c:559
510	    while ((direrr = virDirRead(dir, &entry, oldStateDir)) > 0) {
(gdb) 
512	        if (entry->d_type != DT_REG ||
(gdb) p *entry
$28 = {
  d_ino = 135884899, 
  d_off = 25, 
  d_reclen = 32, 
  d_type = 0 '\000', 
  d_name = "default.xml\000\000dp\031\b\000\000\000\000#\000\000\000\000\000\000\000(\000\000route-net.xml\000^^^^^^\000hp\031\b\000\000\000\000\000\002\000\000\000\000\000\000(\000\000bridge-net.xml\000^^^^^\000", '^' <repeats 163 times>
}
(gdb)

I used a small c program to check the d_type of file in network folder.
[root@localhost libvirt-1.2.8]# ./123 /var/lib/libvirt/network/
--------------- nread=160 ---------------
i-node#  file type  d_reclen  d_off   d_name
135884903  directory    24          4  .
68219432  directory    24         18  ..
135884899  ???          32         25  default.xml
135884900  ???          40         35  route-net.xml
135884904  ???          40        512  bridge-net.xml


[root@localhost libvirt-1.2.8]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Tue Nov 18 09:45:04 2014
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/rhel-root   /                       xfs     defaults        0 0
UUID=40589125-bead-4e52-8ccf-c52991546bed /boot                   xfs     defaults        0 0
/dev/mapper/rhel-home   /home                   xfs     defaults        0 0
/dev/mapper/rhel-swap   swap                    swap    defaults        0 0

Is it a potential bug? Or should I change my file system to ext2/ext3/ext4?

+++++++++++++++++++++++
       The d_type field is implemented since Linux 2.6.4.  It occupies a
       space that was previously a zero-filled padding byte in the
       linux_dirent structure.  Thus, on kernels before 2.6.3, attempting to
       access this field always provides the value 0 (DT_UNKNOWN).

       Currently, only some filesystems (among them: Btrfs, ext2, ext3, and
       ext4) have full support for returning the file type in d_type.  All
       applications must properly handle a return of DT_UNKNOWN.
from http://man7.org/linux/man-pages/man2/getdents.2.html
+++++++++++++++++++++++

No, you shouldn't need to change your FS type, I just only had ext filesystems when I was testing so didn't see this. Please file a bug about this and assign it to me.

Nice catch, by the way!

Thanks, filed Bug 1167145 - networkMigrateStateFiles function does not work on xfs file system due to using unsupported t_type field

I'll check the basic functions for the bug later, but upgrade/downgrade scenarios will be followed up in bug 1167145.

I can not start a guest with a inactive virtual network.

[root@ibm-x3850x5-06 880583]# rpm -q libvirt
libvirt-1.2.8-7.el7.x86_64

<1> Define virtual network according to comment 3, and make sure them are inactive.
[root@ibm-x3850x5-06 880583]# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 bridge-net           inactive   no            yes
 default              inactive   yes           yes
 hostdev-net          inactive   no            yes
 passthrough-net      inactive   no            yes
 private-net          inactive   no            yes
 route-net            inactive   no            yes

<2> Start guest using those defined virtual networks
For nat mode:
[root@ibm-x3850x5-06 880583]# virsh start r7
error: Failed to start domain r7
error: Requested operation is not valid: network 'default' is not active

For route mode:
[root@ibm-x3850x5-06 880583]# virsh dumpxml r7| grep /interface -B5
    <interface type='network'>
      <mac address='02:54:00:36:c6:d0'/>
      <source network='route-net'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
[root@ibm-x3850x5-06 880583]# virsh start r7
error: Failed to start domain r7
error: Requested operation is not valid: network 'route-net' is not active

For bridge mode:
[root@ibm-x3850x5-06 880583]# virsh dumpxml r7| grep /interface -B5
    <interface type='network'>
      <mac address='02:54:00:36:c6:d0'/>
      <source network='bridge-net'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
[root@ibm-x3850x5-06 880583]# virsh start r7
error: Failed to start domain r7
error: Requested operation is not valid: network 'bridge-net' is not active

For hostdev mode:
[root@ibm-x3850x5-06 880583]# virsh dumpxml r7| grep /interface -B5
    <interface type='network'>
      <mac address='02:54:00:36:c6:d0'/>
      <source network='hostdev-net'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
[root@ibm-x3850x5-06 880583]# virsh start r7
error: Failed to start domain r7
error: Requested operation is not valid: network 'hostdev-net' is not active

For passthrough mode:
[root@ibm-x3850x5-06 880583]# virsh dumpxml r7| grep /interface -B5
    <interface type='network'>
      <mac address='02:54:00:36:c6:d0'/>
      <source network='passthrough-net'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
[root@ibm-x3850x5-06 880583]# virsh start r7
error: Failed to start domain r7
error: Requested operation is not valid: network 'passthrough-net' is not active

For private mode:
[root@ibm-x3850x5-06 880583]# virsh dumpxml r7| grep /interface -B5
    <interface type='network'>
      <mac address='02:54:00:36:c6:d0'/>
      <source network='private-net'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
[root@ibm-x3850x5-06 880583]# virsh start r7
error: Failed to start domain r7
error: Requested operation is not valid: network 'private-net' is not active

Testing the bug on hotplug and lxc scenarios, work well.

[root@ibm-x3850x5-06 880583]# rpm -q libvirt
libvirt-1.2.8-7.el7.x86_64

<1> Hotplug a device using a inactive virtual network(attach-device/attach-interface).

[root@ibm-x3850x5-06 880583]# cat bridge.xml
    <interface type='network'>
      <source network='bridge-net'/>
      <model type='virtio'/>
    </interface>
[root@ibm-x3850x5-06 880583]# virsh attach-device r7 bridge.xml 
error: Failed to attach device from bridge.xml
error: Requested operation is not valid: network 'bridge-net' is not active

[root@ibm-x3850x5-06 880583]# cat passthrough.xml
    <interface type='network'>
      <source network='passthrough-net'/>
      <model type='virtio'/>
    </interface>
[root@ibm-x3850x5-06 880583]# virsh attach-device r7 passthrough.xml 
error: Failed to attach device from passthrough.xml
error: Requested operation is not valid: network 'passthrough-net' is not active

[root@ibm-x3850x5-06 880583]# virsh attach-interface  r7 network route-net
error: Failed to attach interface
error: Requested operation is not valid: network 'route-net' is not active

[root@ibm-x3850x5-06 880583]# virsh attach-interface  r7 network hostdev-net
error: Failed to attach interface
error: Requested operation is not valid: network 'hostdev-net' is not active

Note: nat/route/bridge/hostdev/private/passthrough have same testing results.

For a lxc guest
[root@ibm-x3850x5-06 880583]# virsh -c lxc:/// dumpxml lxc  | grep /interface -B3
    <interface type='network'>
      <mac address='52:54:00:f2:2c:ac'/>
      <source network='bridge-net'/>
    </interface>
[root@ibm-x3850x5-06 880583]# virsh -c lxc:/// start lxc
error: Failed to start domain lxc
error: Requested operation is not valid: network 'bridge-net' is not active

[root@ibm-x3850x5-06 880583]# virsh -c lxc:/// edit lxc
Domain lxc XML configuration edited.

[root@ibm-x3850x5-06 880583]# virsh -c lxc:/// dumpxml lxc  | grep /interface -B3
    <interface type='network'>
      <mac address='52:54:00:f2:2c:ac'/>
      <source network='private-net'/>
    </interface>
[root@ibm-x3850x5-06 880583]# virsh -c lxc:/// start lxc
error: Failed to start domain lxc
error: Requested operation is not valid: network 'private-net' is not active

According to comment 16 and 17, move the bug to Verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html