Created attachment 712390 [details]
tcpdump on VF interface p4p1_58

Please reassign/dispatch as appropriate.

Some explanation.

1)  You should not be using passthru mode macvtap/macvlan on a device that
you wish to use on the host as well.  Passthru is a special mode that clones
devices real mac address and give that to the guest to use.  So if you try
to use both the real device and the passthru device that is on top of it, it
will not work as 2 devices share the mac address.

2) Virtual Functions work differently, and depending on the driver, there are
conditions when Physical Functions will not honour the creation of a macvlan.
I don't see driver info here, but when I tried to reproduce with ixgbevf driver,
the PF would not allow me to create macvlan if the mac of the VF was set using
 ip link set eth0 vf 2 mac xx:xx:xx:xx:xx:xx

3) When the VF mac was not set, the device associated with the VF had a random address assigned to it.  In this state, according to driver guys, the device is
not fully configured.  You have to change it's mac address first.  Once that was
done, everything worked.

In the end, I think this is an invalid configuration where you are trying to use
VF in the host as well as trying a passthru macvlan.

Can you doublecheck your configuration.  It would help to know which physical device/driver you are using.

Thanks
-vlad

The system I used to open this bug is not available for me now. I'll update with my result once I gain this system again.

(In reply to Vlad Yasevich from comment #4)
> Some explanation.
> 
> 1)  You should not be using passthru mode macvtap/macvlan on a device that
> you wish to use on the host as well.  Passthru is a special mode that clones
> devices real mac address and give that to the guest to use.  So if you try
> to use both the real device and the passthru device that is on top of it, it
> will not work as 2 devices share the mac address.
> 
> 2) Virtual Functions work differently, and depending on the driver, there are
> conditions when Physical Functions will not honour the creation of a macvlan.
> I don't see driver info here, but when I tried to reproduce with ixgbevf
> driver,
> the PF would not allow me to create macvlan if the mac of the VF was set
> using
>  ip link set eth0 vf 2 mac xx:xx:xx:xx:xx:xx
> 
> 3) When the VF mac was not set, the device associated with the VF had a
> random address assigned to it.  In this state, according to driver guys, the
> device is
> not fully configured.  You have to change it's mac address first.  Once that
> was
> done, everything worked.
> 
> In the end, I think this is an invalid configuration where you are trying to
> use
> VF in the host as well as trying a passthru macvlan.
> 
> Can you doublecheck your configuration.  It would help to know which
> physical device/driver you are using.
> 
I used ixgbe to test this.
 
# lspci | grep 82599
05:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

In my test:
echo 2 to one of PF to bring up VFs:

# ls /sys/bus/pci/devices/0000\:05\:00.0/net
p1p1

# echo 2 > /sys/bus/pci/devices/0000\:05\:00.0/sriov_numvfs

# ls -l /sys/bus/pci/devices/0000\:05\:00.0/virtfn{0,1}
lrwxrwxrwx. 1 root root 0 Oct 28 03:15 /sys/bus/pci/devices/0000:05:00.0/virtfn0 -> ../0000:05:10.0
lrwxrwxrwx. 1 root root 0 Oct 28 03:15 /sys/bus/pci/devices/0000:05:00.0/virtfn1 -> ../0000:05:10.2

# ip link show p1p1
7: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 00:1b:21:c3:d0:3c brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on
    vf 1 MAC 00:00:00:00:00:00, spoof checking on

# ip link set p1p1 vf 0 mac 64:1b:21:c3:d0:21

# ip link show p1p1
7: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 00:1b:21:c3:d0:3c brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 64:1b:21:c3:d0:21, spoof checking on
    vf 1 MAC 00:00:00:00:00:00, spoof checking on

# ip addr show p1p1_0
14: p1p1_0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether a2:70:73:41:33:17 brd ff:ff:ff:ff:ff:ff

# ip link add link p1p1_0 dev macvtap0 type macvtap mode passthru

# ip link set macvtap0 address 26:a5:6b:87:90:12 up

# ip -d link show macvtap0
16: macvtap0@p1p1_0: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc pfifo_fast state LOWERLAYERDOWN mode DEFAULT qlen 500
    link/ether 26:a5:6b:87:90:12 brd ff:ff:ff:ff:ff:ff promiscuity 0 
    macvtap  mode passthru 


Launch a qemu-kvm instance:
# /usr/libexec/qemu-kvm -M pc-i440fx-rhel7.0.0 -cpu Nehalem -enable-kvm -m 4096 -smp 4 -nodefaults -rtc base=utc -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/rhel6.5.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk1,id=virtio-disk1,bootindex=1 -netdev tap,id=hostnet0,vhost=on,fd=16 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=26:a5:6b:87:90:12,bus=pci.0 16<>/dev/tap16 -device usb-tablet,id=input0 -spice port=5900,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -monitor stdio -spice port=5900,disable-ticketing


As you could see above, I could setup macvtap on a VF and the underlying VF doesn't own an IP. But in guest, it still failed to get an IP. Correct me if I misunderstand what you meant. Thanks.


> Thanks
> -vlad

(In reply to Chao Yang from comment #6)
> I used ixgbe to test this.
>  
> # lspci | grep 82599
> 05:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+
> Network Connection (rev 01)
> 05:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+
> Network Connection (rev 01)
> 05:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller
> Virtual Function (rev 01)
> 05:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller
> Virtual Function (rev 01)
> 
> In my test:
> echo 2 to one of PF to bring up VFs:
> 
> # ls /sys/bus/pci/devices/0000\:05\:00.0/net
> p1p1
> 
> # echo 2 > /sys/bus/pci/devices/0000\:05\:00.0/sriov_numvfs
> 
> # ls -l /sys/bus/pci/devices/0000\:05\:00.0/virtfn{0,1}
> lrwxrwxrwx. 1 root root 0 Oct 28 03:15
> /sys/bus/pci/devices/0000:05:00.0/virtfn0 -> ../0000:05:10.0
> lrwxrwxrwx. 1 root root 0 Oct 28 03:15
> /sys/bus/pci/devices/0000:05:00.0/virtfn1 -> ../0000:05:10.2
> 
> # ip link show p1p1
> 7: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode
> DEFAULT qlen 1000
>     link/ether 00:1b:21:c3:d0:3c brd ff:ff:ff:ff:ff:ff
>     vf 0 MAC 00:00:00:00:00:00, spoof checking on
>     vf 1 MAC 00:00:00:00:00:00, spoof checking on
> 
> # ip link set p1p1 vf 0 mac 64:1b:21:c3:d0:21

This will set a flag in a VF stating that it is statically configured and
this will forbid the creation of macvlans on it.
Code:
static int ixgbe_set_vf_macvlan_msg(struct ixgbe_adapter *adapter,
                                    u32 *msgbuf, u32 vf)
{
        u8 *new_mac = ((u8 *)(&msgbuf[1]));
        int index = (msgbuf[0] & IXGBE_VT_MSGINFO_MASK) >>
                    IXGBE_VT_MSGINFO_SHIFT;
        int err;

        if (adapter->vfinfo[vf].pf_set_mac && index > 0) {
                e_warn(drv,
                       "VF %d requested MACVLAN filter but is administratively denied\n",
                       vf);
                return -1;
        }
       ....
}

> 
> # ip link show p1p1
> 7: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode
> DEFAULT qlen 1000
>     link/ether 00:1b:21:c3:d0:3c brd ff:ff:ff:ff:ff:ff
>     vf 0 MAC 64:1b:21:c3:d0:21, spoof checking on
>     vf 1 MAC 00:00:00:00:00:00, spoof checking on
> 
> # ip addr show p1p1_0
> 14: p1p1_0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
>     link/ether a2:70:73:41:33:17 brd ff:ff:ff:ff:ff:ff
> 
> # ip link add link p1p1_0 dev macvtap0 type macvtap mode passthru
> 
> # ip link set macvtap0 address 26:a5:6b:87:90:12 up

This command should generate an error message above.  This command attempts
to program the new address into the PF mac filter table and that is
forbidden since VF was statically configured.

> 
> # ip -d link show macvtap0
> 16: macvtap0@p1p1_0: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500
> qdisc pfifo_fast state LOWERLAYERDOWN mode DEFAULT qlen 500
>     link/ether 26:a5:6b:87:90:12 brd ff:ff:ff:ff:ff:ff promiscuity 0 
>     macvtap  mode passthru 
> 
> 
> Launch a qemu-kvm instance:
> # /usr/libexec/qemu-kvm -M pc-i440fx-rhel7.0.0 -cpu Nehalem -enable-kvm -m
> 4096 -smp 4 -nodefaults -rtc base=utc -device
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=/home/rhel6.5.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,
> cache=none,werror=stop,rerror=stop,aio=native -device
> virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk1,id=virtio-disk1,
> bootindex=1 -netdev tap,id=hostnet0,vhost=on,fd=16 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=26:a5:6b:87:90:12,bus=pci.0
> 16<>/dev/tap16 -device usb-tablet,id=input0 -spice
> port=5900,disable-ticketing,seamless-migration=on -vga qxl -global
> qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -monitor stdio
> -spice port=5900,disable-ticketing
> 
> 
> As you could see above, I could setup macvtap on a VF and the underlying VF
> doesn't own an IP. But in guest, it still failed to get an IP. Correct me if
> I misunderstand what you meant. Thanks.
> 

I hope above description made this clear.

Now, if you do NOT run the command:
> # ip link set p1p1 vf 0 mac 64:1b:21:c3:d0:21

Then everything should work fine.  However, if you decide to NOT change the
mac address on the macvtap device, then you first have to set the mac
address on the VF device:
 ip link set p1p1_0 addr <your addr>

Then you should be able to configure the macvtap device.

To make sure that things are programmed correctly, you can double check with
the command:
 /sbin/bridge fdb show dev p1p1_0

You can also omit the 'dev' parameter to get fdb entries for all devices.
This command will show you which mac addresses the PFs and VFs know about
and will forward to.  If you do not see you mac addresses in the list, then
there may be something wrong in the driver.

-vlad

Closing this as Not A bug.  Hope the above explanation made it clear.

To summarise, a passthru mode macvlan/macvtap device can not be used on the
interface that is expected to be used by the host as well.