Bug 607294 - libvirt network <tftp> element does not enable tftp port
Summary: libvirt network <tftp> element does not enable tftp port
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.0
Hardware: All
OS: Linux
Target Milestone: rc
: 6.0
Assignee: Eric Blake
QA Contact: Virtualization Bugs
Depends On: 580215
TreeView+ depends on / blocked
Reported: 2010-06-23 17:56 UTC by Eric Blake
Modified: 2010-11-10 21:05 UTC (History)
17 users (show)

Clone Of: 580215
Last Closed: 2010-11-10 21:05:30 UTC

Attachments (Terms of Use)
pxe-tftp boot failed (212.91 KB, image/png)
2010-07-30 11:55 UTC, weizhang
no flags Details

Description Eric Blake 2010-06-23 17:56:09 UTC
+++ This bug was initially created as a clone of Bug #580215 +++

Cloning from F13 over to RHEL6; problem is also present in libvirt-0.8.1-9.el6.x86_64

Description of problem:
When creating a virtual network, libvirt starts an instance of dnsmasq. The description for defining a network via xml says that including a tftp tag within the dhcp tag will let the user point that instance of dnsmasq to a different directory for a tftp root. However, when I try to do this the instance of dnsmasq does not allow pxe booting of the ISO image I've contained in the supplied directory.

Version-Release number of selected component (if applicable):

(mcpierce@mcpierce-desktop:Podcast)$ rpm -q libvirt

How reproducible:

Steps to Reproduce:
1. Use livecd-iso-to-pxeboot to create a tftp root directory for a LiveCD image.
2. Create a virtual interface and tell it to use the created directory as a tftp root.
3. Start a virtual machine and have it boot from the network.
Actual results:

The VM never PXE boots since dnsmasq never offers the LiveCD image.

Expected results:

The VM should PXE book the LiveCD image.

Additional info:

The virtual network is defined with the following XML:

<forward mode='nat' />
<bridge name='testbr5310' stp='on' forwardDelay='0' />
<ip address='' netmask=''>
<tftp root='/tmp/tmp.IpID1ZkCQ2' />

--- Additional comment from apevec@redhat.com on 2010-06-17 06:21:08 EDT ---

This seems to work with libvirt-0.7.7-4.fc13

nobody   11062     1  0 12:13 ?        S      0:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file=  --listen-address --except-interface lo --dhcp-range, --dhcp-lease-max=253 --enable-tftp --tftp-root /var/lib/dnsmasq/tftp --dhcp-boot pxelinux.0

# virsh net-dumpxml default
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0' />
  <ip address='' netmask=''>
    <tftp root='/var/lib/dnsmasq/tftp' />
      <range start='' end='' />
      <bootp file='pxelinux.0' />

But tftp port 69 is not allowed in the iptables, like ports for dns 53 and bootp 67.

Workaround is to insert it manually, then VM PXE boot worked:
# iptables -I INPUT -i virbr0 -p udp -m udp --dport 69 -j ACCEPT

--- Additional comment from apevec@redhat.com on 2010-06-20 17:42:39 EDT ---

Patch posted upstream

Comment 2 RHEL Product and Program Management 2010-06-23 18:13:18 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for

Comment 3 Dave Allan 2010-06-29 03:02:33 UTC
libvirt-0_8_1-11_el6 has been built in RHEL-6-candidate with the fix.


Comment 5 weizhang 2010-07-30 11:53:52 UTC
I test with the following steps:

1,  setup a tftp server, edit /etc/xinetd.d/tftp file , set "disable = no" to enable tftp server

2, restart xinetd service /etc/init.d/xinetd restart

3, cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot

4, dowload vmlinuz and initrd.img under the /var/lib/tftpboot folder

 example: http://download.englab.nay.redhat.com/pub/rhel/rel-eng/RHEL6.0-Beta1-4.1/6.0/Server/x86_64/os/images/pxeboot/

5, mkdir /var/lib/tftpboot/pxelinux.cfg

6, create a file name default under the pxelinux.cfg folder

7, edit the file with the contents

DISPLAY boot.txt
LABEL rhel6
        kernel vmlinuz
        append initrd=initrd.img

8, create a new virtual network , the network.xml description is as follows

  <bridge name="virbr1" stp='off' delay='1'/>
  <ip address="" netmask="">
    <tftp root="/var/lib/tftproot" />
      <range start="" end="" />
      <bootp file="pxelinux.0" />

9, define and start network

#virsh net-define network.xml
#virsh net-start netboot
#virsh net-list --all
Name                 State      Autostart
default              active     yes       
netboot              active     no       

#virsh dumpxml pxe2
<domain type='kvm' id='11'>
    <type arch='x86_64' machine='rhel6.0.0'>hvm</type>
    <boot dev='network'/>
  <clock offset='utc'/>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/libvirt/images/pxe.img'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    <interface type='network'>
      <mac address='52:54:00:e6:0b:9e'/>
      <source network='netboot'/>
      <target dev='vnet3'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    <serial type='pty'>
      <source path='/dev/pts/6'/>
      <target port='0'/>
      <alias name='serial0'/>
    <console type='pty' tty='/dev/pts/6'>
      <source path='/dev/pts/6'/>
      <target port='0'/>
      <alias name='serial0'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5903' autoport='yes'/>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
  <seclabel type='dynamic' model='selinux'>

#iptables -I INPUT -i virbr1 -p udp -m udp --dport 69 -j ACCEPT
#virsh start pxe2

screen output shown in the attachment : no bootable device

Is these steps right? If it is right, it seems that there are also some problems.

Comment 6 weizhang 2010-07-30 11:55:10 UTC
Created attachment 435536 [details]
pxe-tftp boot failed

Comment 7 weizhang 2010-07-30 11:56:24 UTC
sorry, forget to list the components version:
[root@redhat ~]# rpm -qa libvirt
[root@redhat ~]# uname -r

Comment 8 Alan Pevec 2010-07-30 17:53:39 UTC
First, for libvirt virtual network, tftp is provided by dnsmasq so keep /etc/xinetd.d/tftp disabled, I guess it even conflicts here, check /var/log/messages.

Also, on F13 I had to have tftproot under /var/lib/dnsmasq/ to get the correct selinux context.

Comment 9 weizhang 2010-08-02 08:09:57 UTC
Hi Alan,
I want to ask where I can get the livecd-iso-to-pxeboot and how to use livecd-iso-to-pxeboot to create a tftp root directory for a LiveCD image? I think I did wrong steps before. I just boot with default tftproot.

Comment 10 Alan Pevec 2010-08-02 08:54:35 UTC
livecd-iso-to-pxeboot is provided by livecd-tools RPMs
Usage: livecd-iso-to-pxeboot <isopath>

Comment 11 weizhang 2010-08-02 10:52:34 UTC
Hi Alan,
Could you please give me a detail steps on testing? I am in rhel6 for testing it. After I set /etc/xinetd.d/tftp disabled, the error is same as before. I don't know what's wrong with my testing steps. Thanks very much

Comment 12 Alan Pevec 2010-08-04 11:59:53 UTC
I have tested on F13 using libvirt-0.8.2-2.fc13
and <tftp root='/var/lib/dnsmasq/tftpboot' /> in "default" network.

First, confirm tftp port 69 is opened:
# virsh net-destroy default
Network default destroyed

# iptables -S INPUT|grep virbr

# virsh net-start default
Network default started

# iptables -S INPUT|grep virbr
-A INPUT -i virbr0 -p udp -m udp --dport 69 -j ACCEPT 
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 

Then generate pxe image (it can be any, not just livecd):
* cd /var/lib/dnsmasq/
* livecd-iso-to-pxeboot /tmp/rhev-hypervisor.iso
 (rhevh is convenient since it's only ~100MB size)
* virsh start VM (with <boot dev='network'/>)
  see gPXE initializing and loading PXELINUX then vmlinuz and initrd...

Comment 15 Jiri Denemark 2010-08-13 11:21:29 UTC
You probably hit bug 594682 in your testing. It works on F13 becuase it contains newer gpxe-roms-qemu package. However, that bug has a known workaround which is covered by bug 623951. Could you retest, once this workaround bug is fixed?

Comment 18 Daniel Veillard 2010-08-15 16:47:03 UTC
libvirt-0.8.1-25.el6 has been built in RHEL-6-candidate with the fix
for issue 623951 so testing should be retried with that version,


Comment 20 Johnny Liu 2010-08-18 04:32:47 UTC
Verify this bug with libvirt-0.8.1-25.el6.x86_64, and PASSED.

1. cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot

2. Dowload vmlinuz and initrd.img under the /var/lib/tftpboot folder, example:

3. mkdir /var/lib/tftpboot/pxelinux.cfg

4. Create a file named "default" under the pxelinux.cfg folder with the following lines.

DISPLAY boot.txt
LABEL rhel6
        kernel vmlinuz
        append initrd=initrd.img

5. Create a new virtual network with the following xml:

  <bridge name="virbr1" stp='off' delay='1'/>
  <ip address="" netmask="">
    <tftp root="/var/lib/tftpboot" />
      <range start="" end="" />
      <bootp file="pxelinux.0" />

6. Define and start network
# virsh net-define network.xml
# virsh net-start netboot

7. Check iptable rules, and dnsmasq process.
# service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    ACCEPT     udp  --             udp dpt:69 
2    ACCEPT     udp  --             udp dpt:53 
3    ACCEPT     tcp  --             tcp dpt:53 
4    ACCEPT     udp  --             udp dpt:67 
5    ACCEPT     tcp  --             tcp dpt:67 

tftp prot - 69 is allowed.

# ps -ef | grep dns |grep netboot
nobody   18370     1  0 19:03 ?        00:00:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/netboot.pid --conf-file=  --listen-address --except-interface lo --dhcp-range, --dhcp-lease-max=253 --dhcp-no-override --enable-tftp --tftp-root /var/lib/tftpboot --dhcp-boot pxelinux.0

The workaround for 623951 is fixed, since "--dhcp-no-override" is seen here.

8. Set selinux to permissive
# setenforce 0

9. Install VM from pxe.
# virt-install --name pxe --ram 1024 --disk path=/var/lib/libvirt/images/pxe.img,size=4 --pxe --network network=netboot

Installation is started successfully.

Comment 21 Johnny Liu 2010-08-18 04:40:40 UTC
This bug is verified successfully, so needinfo flag is removed.

Comment 22 Nan Zhang 2010-09-13 03:04:47 UTC
Verified with libvirt-0.8.1-27.el6.x86_64 & qemu-kvm-

# service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    ACCEPT     udp  --             udp dpt:69 

Comment 23 releng-rhel@redhat.com 2010-11-10 21:05:30 UTC
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.