Description of problem: We need a way to break out of the XML "jail" and provide arbitrary command line options to the DHCP server (dnsmasq) used for the virtual networking in libvirt. Failing that, there needs to be some XML-based way to add arbitrary DHCP options to the DHCP response packet. The various RFCs provide a vast array of options that can be configured to be delivered to clients and all DHCP servers support these either directly or using a simple set of encoding rules. Unfortunately, none of that is exposed in the libvirt XML. If THAT is not acceptable, then can we get direct support in the libvirt XML for at least the most common DHCP options: PXE boot options DNS servers DNS search domains NTP servers NIS servers NIS domain SMTP servers POP server However there are many more options which are less commonly used--but that doesn't mean never. Things like printer servers, web servers, etc. You can see http://linux.die.net/man/5/dhcp-options and/or various RFCs for DHCP for a more complete list. Version-Release number of selected component (if applicable): libvirt-0.9.4-23.el6_2.8.x86_64 libvirt-client-0.9.4-23.el6_2.8.x86_64 How reproducible: Always Steps to Reproduce: Run virsh net-edit default and/or read the libvirt network XML definition and despair! :-)
I certainly understand your frustration if you find yourself looking for a way to set up options not currently provided; there is indeed no way to do it. Question, though: the libvirt DHCP configuration was designed to provide a basic setup that's intended to be not too difficult to configure. The intent is that people who want a basic setup can configure it thorough libvirt, but people who want full control of DHCP can turn off the libvirt configured DHCP server, and libvirt will play nicely with a manually configured DHCP server. Would setting up your DHCP server yourself work for your usecase? Can you tell us more about what you're trying to do?
Well, my goal is to be able to deploy this configuration across a large number of systems. Running "virsh net-define newdefault.xml" with a predefined XML file is simple and easy. Configuring large numbers of systems to not run dnsmasq by default and setting a separate dnsmasq to restart properly when the virtual network restarts seems not-so-simple. At the very least it seems to me that ANY virtual machine would want the DNS and NTP server configurations. The others may be more specific to UNIX Enterprise deployments, I suppose, but DNS is obvious and correct time (NTP) is more and more critical for security reasons these days. Although maybe the virtual machines use the same system clock time as the physical host? Still it would be nice to see NTP, from a testing point of view.
The list of options you'd like to support sound reasonable. Can you provide a list of the exact dnsmasq options that correlate to each (to be sure there is no confusion). Note that libvirt does already support the specification of a single domain suffix. Do you need support for multiple domains, or is one sufficient? As far as a free-for-all "add these to the commandline" option, that's generally frowned on, as it encourages both developers and admins to "take the easy way out", which would eventually lead to an upgrade nightmare if we ever decided to swap out dnsmasq for something different. On the other hand, we do have the precedent of the private qemu namespace, which allows admins to specify extra options on the qemu commandline (at the expense of declaring the guest "tainted", meaning that all warranties are void if some problem occurs.
Hi Laine. Thanks for keeping this on your radar. From the above-linked options page, the DHCP options I think are critical are: option domain-name option domain-name-servers option domain-search option ntp-servers option smtp-server option tftp-server-name option time-servers Plus the PXE netboot options (can't remember if they're available or not); the dnsmasq format for those are: dhcp-option-force=208,f1:00:74:7e <-- this is a static value dhcp-option-force=209,<cfg-file> <-- filename, not pathname dhcp-option-force=210,<boot-dir>/ <-- must end in a "/" The options I think would be nice to have in many situations are: option nis-domain option nis-servers option interface-mtu option routers option subnet-mask If you need me to show the exact dnsmasq syntax I can do that. Note the above syntax for PXE is an example, but you can use names instead of numbers if dnsmasq supports the name. You can get a list of the supported names by running "dnsmasq --help dhcp"
> As far as a free-for-all "add these to the commandline" option, that's > generally frowned on, as it encourages both developers and admins to "take > the easy way out", which would eventually lead to an upgrade nightmare if we > ever decided to swap out dnsmasq for something different. On the other hand, > we do have the precedent of the private qemu namespace, which allows admins > to specify extra options on the qemu commandline (at the expense of > declaring the guest "tainted", meaning that all warranties are void if some > problem occurs. I agree that as a first priority we should look at supporting more desired configuration parameters natively in the XML. Given our precedent with QEMU, and the frequest requests wrt dnsmasq, I think it'd be worth while suppporting a dnsmasq passthrough, in a private XML namespace. Use of such passthrough would of course mark any such network as 'tainted' and be a 'void-your-support' criteria for libvirt in RHEL.
There were patches a couple months ago that would resolve this request, but there were a couple of issues with the design, so it was pulled at the last minute pending further discussion. Here is the continuation of that discussion: https://www.redhat.com/archives/libvir-list/2013-March/msg00166.html https://www.redhat.com/archives/libvir-list/2013-March/msg00822.html Unfortunately I was the only participant in the "discussion" :-), but hopefully that means that the design in those two emails is adequate.
(In reply to Paul Smith from comment #0) <snip> > PXE boot options I'd be interested in this as well, especially option 66 (called next-server by dnsmasq). This would allow me to redirect the guests to an external TFTP server (with a custom boot menu or an automated logic of providing kernel/initrd combinations), used by other hosts without actually bridging the guests into host's network (since the TFTP traffic itself is routable). This has several advantages, like the guests not leaking 52:54 MAC addresses to the network or the ability for them to operate independently on the underlying L2, unlike bridging. (Think: laptop, swithing between ethernet and wifi) All this is of course possible with a custom dhcp server, but it would be nice to make it easier (like "use this piece of network XML"). (In reply to Daniel Berrange from comment #8) > > As far as a free-for-all "add these to the commandline" option, that's > > generally frowned on, as it encourages both developers and admins to "take > > the easy way out", which would eventually lead to an upgrade nightmare if we > > ever decided to swap out dnsmasq for something different. On the other hand, > > we do have the precedent of the private qemu namespace, which allows admins > > to specify extra options on the qemu commandline (at the expense of > > declaring the guest "tainted", meaning that all warranties are void if some > > problem occurs. > > I agree that as a first priority we should look at supporting more desired > configuration parameters natively in the XML. Given our precedent with > QEMU, and the frequest requests wrt dnsmasq, I think it'd be worth while > suppporting a dnsmasq passthrough, in a private XML namespace. Use of such > passthrough would of course mark any such network as 'tainted' and be a > 'void-your-support' criteria for libvirt in RHEL. I'm not familiar with how libvirt chooses to do things, but would a 'type' (or 'driver' / 'backend') option for <dhcp> be a bad idea? Similar to the <domain type='string'> tag. Since, as pointed out in the two emails referenced in comment #11, dhcp option specification and handling depends on the backend, it could be simplified to something like <option code='66' value='1.2.3.4'/> Not caring about value type and passing it with minimal sanitization to dnsmasq would allow cases such as http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2005q2/000231.html to work as well. Though I guess that - at that point - a generic option passtrough is just as "bad".
hi Jaroslav The bug has had rhel-7.2.0+ flag, so it will be fixed with a new build in rhel7.2 , right? thanks.
A bit of a reboot on this topic for me. I've been looking through all the history again. The first ugliness to point out is that there is no accepted authoritative name for any dhcp option. This means that dnsmasq and dhclient/dhcpd use different names for many of them. The names provided by Paul are from dhclient. I'll go through some of those: * option domain-name (RFC2132 option 15 - single domain name) * option domain-name-servers (RFC2132 option 6 - list of IP addresses) * option domain-search (RFC3397 option 119 - list of domain names) Ugh. domain-name and domain-search are nearly the same thing, just that the latter can contain multiple domains to try. This is described detailed in this email: https://www.ietf.org/mail-archive/web/dhcwg/current/msg11447.html Basically what I get from this is that if a domain-name is provided, the dhcp server should put it in either an option 15 or 119 of responses, whichever is requested by the client. Likewise, if a domain-search list is provided and the client requests option 15 but not 119, then the first domain from that list should be sent back to the client. Based on that, try to make life easier for the admin some XML like this (as a subelement of <dhcp> in a libvirt network): <options> <dns> <server address='blah'/> <server address='blah'/> <search domain='blah'/> <search domain='blah'/> </dns> </options> the first server address would be put into option 15 (when requested), and all of them into option 119. One problem with this is that there is no way to force either option to be sent (dnsmasq's "dhcp-option-force" which will cause the option to be added to the response even if the client doesn't ask for it). We could do: <dns force='yes'> but then there would be no way to force sending one of the dns options without sending the other, so I don't like it. * option ntp-servers (RFC2132, option 42 - list of IP addresses) * option smtp-server (RFC2132, option 69 - list of IP addresses) * option time-servers (RFC2132, option 5 - list of IP addresses) The above 3 are all very similar, and could be handled with something like this: <options> <ntp> <server address='blah'/> ... </ntp> <smtp> <server address='blah'/> ... </smtp> <time> <server address='blah'/> ... </time> </options> Again those could have a "force='yes'" added to them. * option tftp-server-name This one is a bit strange, because (in dnsmasq) it could be provided as a part of the dhcp-boot options (the other parts of "dhcp-boot" are already specified in the <bootp> subelement of <dhcp>, but not this one) or as a separate dhcp-option. It's unclear to me if there's a reason to provide both. Also, in dnsmasq this option is called "tftp-server" (option 66), *not* "tftp-server-name". We *could* handle this and other items with something like <tftp> <server name='blah'/> <server address='1.1.1.1'/> <file name='blah'/> </tftp> but that is duplicating information from <bootp>. Aside from any other problems, all of the above suffer from the fact that there isn't any canonical mapping of dhcp option names to numbers, making it confusing to verify what each mnemonic really means. So I think I'll go back to the idea of a single syntax from my long ago email: https://www.redhat.com/archives/libvir-list/2013-March/msg00822.html like this: <option code='x' force='yes|no'> <value address='1.1.1.1'/> ... <value domain='abc.test.com'/> ... <numeric number='123'/> ... </option> So in the example of dns, we would have something like this: <option code='15'> <value domain='example.com'/> </option> <option code='6'> <value address='1.2.3.4'/> <value address='2.3.4.5'/> </option> <option code='115' <value domain='example.com'/> <value domain='test.com'/> </option> This doesn't look as nice, tidy, and purpose-built as the alternatives above (and it requires redundant info for options 15 and 119), but it does assure that there is a 1:1 mapping between our XML and what is possible with DHCP options. The catch (in comparison to the earlier general purpose patch that we considered and then rejected) is that 1) only "known" explicitly supported options will be allowed, and 2) rather than generic string "values", the values for each option will use a list of elements when there is a list of items in the option, and all items will have a specific type, (domain name, ip address, etc), and we will check that the correct type is used for the arguments of each supported option.
*** Bug 1470229 has been marked as a duplicate of this bug. ***
Is this still being implemented? There is also a use case for more DHCP options support in the NFV ecosystem where routers/switches/firewalls/load-balancers/and so on need some parameters to be able to boot, such as: - TFTP server name (66) - TFTP server address (150) - boot file name (67) which can also be used as a config file for some of them - ... However, there is not a common set of expected DHCP options, so there must be a way to differentiate the clients based on some criteria. It is usually done by: - MAC address: this approach is fine but not scalable. - the client ID (61): same remark - the vendor-class ID (60): it is not always populated by the clients but it is a more scalable solution which allows mapping some options to a type of client - the user-class ID (77): same comment Dnsmasq implements this feature with tags mapped to: - MAC address - vendor-class - user-class - subscriber-id relay agent - Client System Architecture - or any other DHCP option All details are at http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html. BTW, the DHCP options numbers are all documented in an official registry maintained by an official authority called IANA, and they are mapped with names defined at https://www.iana.org/assignments/bootp-dhcp-parameters/bootp-dhcp-parameters.xhtml.
There does not even seem to be a way to use dnsmasq default config file (/etc/dnsmasq.conf): - if I destroy and undefine a network, remove the dhcp definition in its backup network xml file, define and start the network then restart libvirtd, the same location for the conf file in /var/lib/libvirt/dnsmasq/ continues to be used. - if I add "<conffile name='/etc/dnsmasq.conf'/>" and remove the <dhcp> </dhcp> definition, the result is the same. Is there another way?
I'm not sure that anyone is working on this. The key points are, that libvirt does not want to do a dumb passthrough through the XML, thus every option needs to be designed into the XML. As of using external configuration file, this is not really supported by libvirt. The input is the XML and only the XML. The config file is then generated. Obviously if you wish to use your own configuration, you can use a libvirt network without dnsmasq or even create your own bridge for custom configurations.
| you can use a libvirt network without dnsmasq But how? Do I need to purge dnsmasq or is there a more subtle way?
If you don't want the dnsmasq instance started by libvirt to respond to dhcp queries, remove the <dhcp> section from the network config. If you also don't want the dnsmasq instance started by libvirt to respond to DNS queries, then add <dns enable='no'/> to the network config. If youo do both of these, then libvirt won't start any dnsmasq instance for that network.
You can add extra dnsmasq options in libvirt on 5.6.0 version or upper. You need to add the dnsmasq namespace like: <network xmlns:dnsmasq='http://libvirt.org/schemas/network/dnsmasq/1.0'> Then you can create a dns section with options: <dnsmasq:options> <dnsmasq:option value='dhcp-option=option6:dns-server,[fd2e:6f44:5dd8:c956::2]'/> </dnsmasq:options>
Thank you for reporting this issue to the libvirt project. Unfortunately we have been unable to resolve this issue due to insufficient maintainer capacity and it will now be closed. This is not a reflection on the possible validity of the issue, merely the lack of resources to investigate and address it, for which we apologise. If you none the less feel the issue is still important, you may choose to report it again at the new project issue tracker https://gitlab.com/libvirt/libvirt/-/issues The project also welcomes contribution from anyone who believes they can provide a solution.