Bug 688848

Are you sure dhcp request were sent by the guest? Is there a chance the guest is not configured in /etc/sysconfig/networking correctly and it uses a static IP?

What happens when you run dhclient ethX in the guest?

(1) Please provide lspci (-vvv) of bcm device in:
    (a) host, before assigned to guest
    (b) guest, after assigned to guest

(2) please provide /etc/sysconfig/network-scripts/ifcfg* files from host & guest

(3) please provide xml used to hot-add bcm to guest;

(4) is the machine accessible that I can login & do some poking, testing with existing rpm's mentioned in this bz?

(5) dmesg of host would be nice as well (to see config message related to bcm device); booting with "debug' on command line would generate best output.

(6) qemu log of guest (after started, after hot-plug add of bcm).

Note: the lspci provided in Description shows the driver to be pci-stub.

That means the device was manual unbound from the host driver & bound to pci-stub.
*Please*, use virsh-attach & proper (managed pci) xml spec to de-assign from host and re-assign to guest.

Also, please provide a *simple* lspci (no switches) of the host & guest; host pre-device-assignment and post-device-assignment.

Q: is this the only network card on the host?

(In reply to comment #2)
> Are you sure dhcp request were sent by the guest? Is there a chance the guest
> is not configured in /etc/sysconfig/networking correctly and it uses a static
> IP?
> 
> What happens when you run dhclient ethX in the guest?

Dor, 
 I did nothing in /etc/sysconfig/networking, host lost its IP(82576 holds it) once I de-assign BCM5709 then re-assign it to guest. Since I didn't do anything in guest, I think guest gets its IP via dhcp. Once I finish what Don tells me, I will update is ASAP.

Thanks

Dor, Don
 On this intel host, there are two nic cards, BCM5709 and 82576, it is weird that eth0 has the same IP addr with switch(switch interface is eth5). I guess its the problem cause guest hijacks switch's IP. Will attach network configure files.

# ifconfig switch
switch    Link encap:Ethernet  HWaddr 00:1B:21:66:2A:ED  
          inet addr:10.66.83.210  Bcast:10.66.83.255  Mask:255.255.254.0
          inet6 addr: fe80::21b:21ff:fe66:2aed/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9026 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4252 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:12263846 (11.6 MiB)  TX bytes:630117 (615.3 KiB)
# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 84:2B:2B:6C:8E:65  
          inet addr:10.66.83.210  Bcast:10.66.83.255  Mask:255.255.254.0
          inet6 addr: fe80::862b:2bff:fe6c:8e65/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:601 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:139290 (136.0 KiB)  TX bytes:1036 (1.0 KiB)
          Interrupt:34 Memory:f2000000-f2012800 
 
# brctl show
bridge name	bridge id		STP enabled	interfaces
switch		8000.001b21662aed	no		eth5
virbr0		8000.525400baa2d9	yes		virbr0-nic

lspci|grep Eth
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
03:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
04:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

 # ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 84:2b:2b:6c:8e:65 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 84:2b:2b:6c:8e:67 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 84:2b:2b:6c:8e:69 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 84:2b:2b:6c:8e:6b brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1b:21:66:2a:ec brd ff:ff:ff:ff:ff:ff
7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1b:21:66:2a:ed brd ff:ff:ff:ff:ff:ff
8: switch: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:1b:21:66:2a:ed brd ff:ff:ff:ff:ff:ff
9: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 52:54:00:ba:a2:d9 brd ff:ff:ff:ff:ff:ff
10: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
    link/ether 52:54:00:ba:a2:d9 brd ff:ff:ff:ff:ff:ff

Created attachment 487258 [details]
host network configure files

Created attachment 487259 [details]
host device message

Did you post the correct host dmesg log?

It does _not_ show the intel_iommu=on flag/switch set on the kernel cmdline,
which is necessary for device assignment.  Additionally, I asked for it with 'debug' added to the cmdline.
I'm also looking for dmesg of guest, as well as output of "lspci" from guest.

So, if you didn't boot the kernel with intel_iommu=on, device assignment won't work (and explains why the device still looks like it has host IP address -- it's still tied to the host!).   Also, if you do add that line, make sure VTd is turned on in the BIOS (or else the command line switch won't do anything either).

btw: I logged onto 10.66.83.210/redhat;  I could not find the guest image on that system, and it already changed from qemu-kvm<*>-150 to -151.

Where is the guest image so I can duplicate your test on that system????

(In reply to comment #11)
> Did you post the correct host dmesg log?
> 
> It does _not_ show the intel_iommu=on flag/switch set on the kernel cmdline,
> which is necessary for device assignment.  Additionally, I asked for it with
> 'debug' added to the cmdline.
> I'm also looking for dmesg of guest, as well as output of "lspci" from guest.
> 
> So, if you didn't boot the kernel with intel_iommu=on, device assignment won't
> work (and explains why the device still looks like it has host IP address --
> it's still tied to the host!).   Also, if you do add that line, make sure VTd
> is turned on in the BIOS (or else the command line switch won't do anything
> either).
> 
> btw: I logged onto 10.66.83.210/redhat;  I could not find the guest image on
> that system, and it already changed from qemu-kvm<*>-150 to -151.
> 
> Where is the guest image so I can duplicate your test on that system????

Don, when I log into that machine, found bridge was using eth0 which is to be de-assigned, so I launched a new bridge(bridge then set up on eth5), when restart network, bridge has same IP with eth0, so I didn't continue to de-assign the eth0. 
Many QE want to use this machine to do test work, so the environment is changing and will not available until next week. I will let you know once this machine is ready, sorry for my careless.

Please provide the xml file used to hot-plug the bcm5709 from the host to the guest.

If the device assignment is using 02:00.1 or 03:00.1, this code in the bcm5709 will cause the guest to use fcn-0 params (like fcn-0's mac-addr!):
        if ((reg & BNX2_SHM_HDR_SIGNATURE_SIG_MASK) ==
            BNX2_SHM_HDR_SIGNATURE_SIG) {
                u32 off = PCI_FUNC(pdev->devfn) << 2;

                bp->shmem_base = bnx2_reg_rd_ind(bp, BNX2_SHM_HDR_ADDR_0 + off);
        } else
                bp->shmem_base = HOST_VIEW_SHMEM_BASE;

Note: that if fcn==1 is assigned to guest, it will be mapped to something
       like 00:05.0 in the guest.  
      In that case, the guest will setup the improper shmem_base value
      and all cause the guest-based, shmem_base value (for 0[2,3]:00.1)
      to be the host-based shmem_base value (for 0[2,3]:00.0).

If the test was using fcn=1, please retry with .1 as the host network i/f and fcn=0 as the assigned-dev to the guest.

If that works, pls re-assign this bug to kernel-maint.

(In reply to comment #14)
> Please provide the xml file used to hot-plug the bcm5709 from the host to the
> guest.
> 
> If the device assignment is using 02:00.1 or 03:00.1, this code in the bcm5709
> will cause the guest to use fcn-0 params (like fcn-0's mac-addr!):
>         if ((reg & BNX2_SHM_HDR_SIGNATURE_SIG_MASK) ==
>             BNX2_SHM_HDR_SIGNATURE_SIG) {
>                 u32 off = PCI_FUNC(pdev->devfn) << 2;
> 
>                 bp->shmem_base = bnx2_reg_rd_ind(bp, BNX2_SHM_HDR_ADDR_0 +
> off);
>         } else
>                 bp->shmem_base = HOST_VIEW_SHMEM_BASE;
> 
> Note: that if fcn==1 is assigned to guest, it will be mapped to something
>        like 00:05.0 in the guest.  
>       In that case, the guest will setup the improper shmem_base value
>       and all cause the guest-based, shmem_base value (for 0[2,3]:00.1)
>       to be the host-based shmem_base value (for 0[2,3]:00.0).
> 
> If the test was using fcn=1, please retry with .1 as the host network i/f and
> fcn=0 as the assigned-dev to the guest.
> 
> If that works, pls re-assign this bug to kernel-maint.

Don,
 I de-assign BCM5709 manually, not use virsh. But will update the result.

Reproduced again on kernel 2.6.32-341.el6.x86_64. Booting a rhel5.9 i686 guest with Broadcom Corporation NetXtreme II BCM5709 assigned, after failed to acquire an IP from dhcp server, host lost network. 
I have bridge setup on em1, "kernel: bnx2 0000:01:00.0: em1: NIC Copper Link is Down" was observed in host dmesg while guest trying to acquire IP. From guest dmesg, I noticed the nic I assigned to guest owned the same mac address as em1 did. Both host and guest dmesg will be attached.

# ethtool -i em1
driver: bnx2
version: 2.2.3
firmware-version: 5.2.7 bc 5.2.2 NCSI 2.0.8
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

# ifconfig em1
em1       Link encap:Ethernet  HWaddr 00:25:64:FD:22:61  
          inet6 addr: fe80::225:64ff:fefd:2261/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11181 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23819 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:927601 (905.8 KiB)  TX bytes:7886961 (7.5 MiB)

qemu-kvm CLI:
-device pci-assign,host=01:00.1,id=hostnet1 -device pci-assign,host=22:00.0,id=hostnet2 -device pci-assign,host=23:00.0,id=hostnet3 -device pci-assign,host=23:00.1,id=hostnet4

# lspci | grep Eth
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
22:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
23:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
23:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

Created attachment 644658 [details]
dmesg from host and guest

(In reply to comment #21)
> Reproduced again on kernel 2.6.32-341.el6.x86_64. Booting a rhel5.9 i686
> guest with Broadcom Corporation NetXtreme II BCM5709 assigned, after failed
> to acquire an IP from dhcp server, host lost network. 
> I have bridge setup on em1, "kernel: bnx2 0000:01:00.0: em1: NIC Copper Link
> is Down" was observed in host dmesg while guest trying to acquire IP. From
> guest dmesg, I noticed the nic I assigned to guest owned the same mac
> address as em1 did. Both host and guest dmesg will be attached.
> 
> # ethtool -i em1
> driver: bnx2
> version: 2.2.3
> firmware-version: 5.2.7 bc 5.2.2 NCSI 2.0.8
> bus-info: 0000:01:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
> 
> # ifconfig em1
> em1       Link encap:Ethernet  HWaddr 00:25:64:FD:22:61  
>           inet6 addr: fe80::225:64ff:fefd:2261/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:11181 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:23819 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000 
>           RX bytes:927601 (905.8 KiB)  TX bytes:7886961 (7.5 MiB)
> 
> qemu-kvm CLI:
> -device pci-assign,host=01:00.1,id=hostnet1 -device
> pci-assign,host=22:00.0,id=hostnet2 -device
> pci-assign,host=23:00.0,id=hostnet3 -device
> pci-assign,host=23:00.1,id=hostnet4
> 
> # lspci | grep Eth
> 01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709
> Gigabit Ethernet (rev 20)
> 01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709
> Gigabit Ethernet (rev 20)
> 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709
> Gigabit Ethernet (rev 20)
> 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709
> Gigabit Ethernet (rev 20)
> 22:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet
> Controller (Copper) (rev 06)
> 23:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network
> Connection (rev 01)
> 23:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network
> Connection (rev 01)

In this case, the ***rhel5.9*** bcm5709 driver must be updated to the same level as the rhel6 version. for the assignment to work, b/c the RHEL5 & earlier RHEL6 drivers were not multi-function agnostic in their register accesses.
This *WONT WORK* with rhel5 guest driver.