Hide Forgot
Description of problem: Installed the latest ipxe package(ipxe-roms-qemu-20160127-1.git6366fa7a.el7) to host. Boot vm and use ipxe to get IP address. But switch can not get TFTP package in 240 seconds. If use ipxe-roms-qemu-20130517-6.gitc4bce43.el7 package to install host then switch will get tftp package after 4 seconds. Notes: I found ipxe-20130517-7.gitc4bce43.el7 version to fix bug 1196352 from changelog. so maybe bug 1196352 cause this regression problem. Version-Release number of selected component (if applicable): ipxe-roms-qemu-20160127-1.git6366fa7a.el7 qemu-kvm-1.5.3-109.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.install ipxe ipxe-roms-qemu-20160127-1.git6366fa7a.el7 2. Boot vm with cli /usr/libexec/qemu-kvm \ -name 'virt-tests-vm1' \ -sandbox off \ -machine pc \ -nodefaults \ -vga qxl \ -device intel-hda,bus=pci.0,addr=03 \ -device hda-duplex \ -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20160318-205235-KyO2KCHF,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/tmp/monitor-catch_monitor-20160318-205235-KyO2KCHF,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idrBoJ4T \ -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20160318-205235-KyO2KCHF,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -device virtio-serial-pci,id=virtio_serial_pci0,bus=pci.0,addr=04 \ -chardev socket,id=devvs,path=/tmp/virtio_port-vs-20160318-205235-KyO2KCHF,server,nowait \ -device virtserialport,chardev=devvs,name=vs,id=vs,bus=virtio_serial_pci0.0 \ -chardev socket,id=seabioslog_id_20160318-205235-KyO2KCHF,path=/tmp/seabios-20160318-205235-KyO2KCHF,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20160318-205235-KyO2KCHF,iobase=0x402 \ -device nec-usb-xhci,id=usb1,bus=pci.0,addr=05 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=06 \ -drive id=drive_pxe,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=/home/autotest/client/tests/virt/shared/data/images/pxe-test.qcow2 \ -device scsi-hd,id=pxe,drive=drive_pxe \ -device virtio-net-pci,mac=9a:07:08:09:0a:0b,id=id0H41P6,vectors=4,netdev=idoUBpgw,bus=pci.0,addr=07 \ -netdev tap,id=idoUBpgw,vhost=on \ -m 8192 \ -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \ -cpu 'SandyBridge',+kvm_pv_unhalt \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -spice port=3000,password=123456,addr=0,tls-port=3200,x509-dir=/tmp/spice_x509d,tls-channel=main,tls-channel=inputs,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4 \ -rtc base=utc,clock=host,driftfix=slew \ -boot order=cdn,once=n,menu=off,strict=off \ -enable-kvm \ -vnc :1 3.time tcpdump -i switch|grep tftp Actual results: can not get tftp package in 4 minutes. Expected results: should get tftp package in 10 seconds #time tcpdump -i switch|grep tftp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on switch, link-type EN10MB (Ethernet), capture size 65535 bytes 17:08:50.128195 IP dhcp-66-145-201.nay.redhat.com.1024 > lab-02.rhts.eng.nay.redhat.com.tftp: 40 RRQ "pxelinux.0" octet blksize 1432 tsize 0 17:08:50.174571 IP dhcp-66-145-201.nay.redhat.com.49152 > lab-02.rhts.eng.nay.redhat.com.tftp: 79 RRQ "pxelinux.cfg/00000000-0000-0000-0000-000000000000" octet tsize 0 blksize 1408 17:08:50.176103 IP dhcp-66-145-201.nay.redhat.com.49153 > lab-02.rhts.eng.nay.redhat.com.tftp: 63 RRQ "pxelinux.cfg/01-9a-07-08-09-0a-0b" octet tsize 0 blksize 1408 17:08:50.177623 IP dhcp-66-145-201.nay.redhat.com.49154 > lab-02.rhts.eng.nay.redhat.com.tftp: 51 RRQ "pxelinux.cfg/0A4291C9" octet tsize 0 blksize 1408 ............... Additional info:
As ipxe-roms-qemu-20130517-6.gitc4bce43.el7 works well. it can get tftp packages in 4 seconds. ipxe-roms-qemu-20130517-7.gitc4bce43.el7~ipxe-roms-qemu-20160127-1.git6366fa7a.el7encountered this bug. so set regression to keywords.
Created attachment 1138966 [details] vm ipxe
The main reason is that virtio-net can not get dhcp packages for long time. It is easy to get tftp package once guest get ip address. so changed bug summary tftp/dhcp/s.
Created attachment 1148155 [details] virtio-net rom with DHCP debugging enabled
Hi Fu Xiang Chun, ipxe-roms-qemu-20160127-1.git6366fa7a.el7 uses higher DHCP timeouts but it should still be able to finish all DHCP steps within 4 minutes as long as the server is responsive and the link solid. On my network, the time to finish DHCP went up from 4 seconds to 12 seconds, in line with the PXE spec and the goal of bug 1196352. Could you please re-run your scenario with the attached .rom? It is the current upstream iPXE with DHCP debugging enabled. It should help us figure out what's going on, and whether this issue hasn't been fixed already. Thanks! Ladi
(In reply to Ladi Prosek from comment #7) > Hi Fu Xiang Chun, > > ipxe-roms-qemu-20160127-1.git6366fa7a.el7 uses higher DHCP timeouts but it > should still be able to finish all DHCP steps within 4 minutes as long as > the server is responsive and the link solid. > > On my network, the time to finish DHCP went up from 4 seconds to 12 seconds, > in line with the PXE spec and the goal of bug 1196352. > > Could you please re-run your scenario with the attached .rom? It is the > current upstream iPXE with DHCP debugging enabled. It should help us figure > out what's going on, and whether this issue hasn't been fixed already. > > Thanks! > Ladi Ladi, As only one special host can reproduce this bug. I need to reserve it before testing. So late reply to your needinfo. result: still can reproduce this bug with your 1af41000.rom file(need about 260 seconds). I added 2 screenshots to attachment. Additional: QE found one host can reproduce this bug. host info: 1.processor : 7 vendor_id : GenuineIntel cpu family : 6 model : 58 model name : Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz stepping : 9 microcode : 0x1b cpu MHz : 2466.195 cache size : 8192 KB physical id : 0 siblings : 8 core id : 3 cpu cores : 4 apicid : 7 initial apicid : 7 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt bogomips : 6784.52 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: 2.# dmidecode |grep BIOS SMBIOS 2.7 present. BIOS Information BIOS is upgradeable BIOS shadowing is allowed BIOS boot specification is supported BIOS Revision: 1.2 BIOS Language Information If you need more information. please let me know.
Created attachment 1148887 [details] sreenshot 1
Created attachment 1148888 [details] guest screenshot 2
Thanks Fu Xiang Chun for the additional information. So here's what's happening. This machine is receiving STP packets like this one: IEEE 802.3 Ethernet Destination: Spanning-tree-(for-bridges)_00 (01:80:c2:00:00:00) Address: Spanning-tree-(for-bridges)_00 (01:80:c2:00:00:00) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast) Source: JuniperN_f6:5c:a3 (3c:61:04:f6:5c:a3) Address: JuniperN_f6:5c:a3 (3c:61:04:f6:5c:a3) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Length: 39 Padding: 00000000000000 Logical-Link Control DSAP: Spanning Tree BPDU (0x42) IG Bit: Individual SSAP: Spanning Tree BPDU (0x42) CR Bit: Command Control field: U, func=UI (0x03) 000. 00.. = Command: Unnumbered Information (0x00) .... ..11 = Frame type: Unnumbered frame (0x03) Spanning Tree Protocol Protocol Identifier: Spanning Tree Protocol (0x0000) Protocol Version Identifier: Rapid Spanning Tree (2) BPDU Type: Rapid/Multiple Spanning Tree (0x02) BPDU flags: 0x0e (Port Role: Designated, Proposal) 0... .... = Topology Change Acknowledgment: No .0.. .... = Agreement: No ..0. .... = Forwarding: No ...0 .... = Learning: No .... 11.. = Port Role: Designated (3) .... ..1. = Proposal: Yes .... ...0 = Topology Change: No Root Identifier: 32768 / 160 / 3c:61:04:f6:5c:81 Root Bridge Priority: 32768 Root Bridge System ID Extension: 160 Root Bridge System ID: JuniperN_f6:5c:81 (3c:61:04:f6:5c:81) Root Path Cost: 0 Bridge Identifier: 32768 / 160 / 3c:61:04:f6:5c:81 Bridge Priority: 32768 Bridge System ID Extension: 160 Bridge System ID: JuniperN_f6:5c:81 (3c:61:04:f6:5c:81) Port identifier: 0x8221 Message Age: 0 Max Age: 20 Hello Time: 2 Forward Delay: 15 Version 1 Length: 0 Because the Forwarding bit is not set, iPXE treats the link as blocked and waits for it to unblock. This functionality was added upstream quite recently: https://git.ipxe.org/ipxe.git/commitdiff/d73982f098db9fdedb28a3826eb97a6832eac1e4 which explains why this looks like a regression. It has nothing to do with bug 1196352 / longer timeouts. Maybe the Juniper device where the packets originate (3c:61:04:f6:5c:a3) is misconfigured. Maybe only that one port this machine is connected to is broken. Either way, not an iPXE bug.
Can you please follow up with IT / netops?
(In reply to Ladi Prosek from comment #14) > Can you please follow up with IT / netops? Thanks for Ladi's detailed explanation. In order to confirm this problem. I have sent a ticket to IT. and will update reason to bz once IT reply me. https://engineering.redhat.com/rt/Ticket/Display.html?id=399935 Another. I still have 2 small problems. P1. why ipxe-roms-qemu-20130517-6.gitc4bce43.el7 works? (Ａccording to my understanding. This version shouldn't work) p2. Can upstream's patch fix this problem? If it can. then It should be a valid bug before patch is backported.
(In reply to FuXiangChun from comment #15) > (In reply to Ladi Prosek from comment #14) > > Can you please follow up with IT / netops? > > Thanks for Ladi's detailed explanation. In order to confirm this problem. I > have sent a ticket to IT. and will update reason to bz once IT reply me. > > https://engineering.redhat.com/rt/Ticket/Display.html?id=399935 Thank you! > Another. I still have 2 small problems. > > P1. why ipxe-roms-qemu-20130517-6.gitc4bce43.el7 works? (Ａccording to my > understanding. This version shouldn't work) This version of the package doesn't contain the code that defers DHCP if the link is blocked as reported by the STP protocol, it was added in 2015 (upstream). In other words what you hit is a feature that's missing in the 2013 package. > p2. Can upstream's patch fix this problem? If it can. then It should be a > valid bug before patch is backported. Reverting the defer-DHCP-on-STP-blocked-link feature would be extremely unlikely to succeed. And again, this is not a bug, iPXE works as intended here.