Bug 1262439
Summary: | dhcrelay doesn't forward from VM until restarted | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Nicholas Schuetz <nick> | ||||
Component: | libvirt | Assignee: | Laine Stump <laine> | ||||
Status: | CLOSED NOTABUG | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | unspecified | ||||||
Version: | 7.2 | CC: | dyuan, jpopelka, jsuchane, nick, nschuetz, pjanda, rbalakri, thozza, yafu | ||||
Target Milestone: | rc | ||||||
Target Release: | --- | ||||||
Hardware: | x86_64 | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2016-08-04 17:03:02 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Nicholas Schuetz
2015-09-11 16:47:47 UTC
(In reply to Nicholas Nachefski from comment #0) > dhcrelay doesnt forward requests until it is restarted. When i start my > system, it is enabled and starts, however, does nothing (does not forward > dhcp requests to server). When i 'systemctl restart dhcrelay', then it > starts working. Hmm, I haven't been able to reproduce the issue in VM. > Possibly a problem with it's systemd unit file? ie, dependencies not > structured properly We recently changed After=network.target to Wants=network-online.target After=network-online.target in dhcrelay.service due to bug #1145832 and I have no idea what else could be a culprit. This is still a problem. And, if you search for 'dhcrelay not starting' in google you get a ton of hits about this from other BZ sites. I think this should be looked at a little harder. I'm running F23 (updated) and this is still happening as of today. I think i just found the problem. It's an ordering problem with the systemd unit file. I need this dhcrelay to proxy DHCP request from the VM network to my IPA server (which is also running dhcpd). When dhcrelay starts, virbr0 doesnt exist yet. However, once the box is fully booted, virbr0 *does* exist and when i manually restart dhcrelay it works fine. If you set '-i virbr0' in the unit file (explicitly setting the interface) dhrelay wont start *at all*. Please take a harder look as this is apparently a PITA for everyone. -Nick Does it help if you cp /usr/lib/systemd/system/dhcrelay.service /etc/systemd/system/dhcrelay.service and add Wants=libvirtd.service After=libvirtd.service to [Unit] in /etc/systemd/system/dhcrelay.service ? Sorry for not getting back sooner. This fixed my problem. THANKS! -Nick this fixed the issue on both my RHEL72 and F23 hyper visors. Thanks again! reproduced on RHEL-7.2 x86_64 Server I spoke too soon when i said that this problem was fixed. It is *not* fixed in my lab environment, the problem persists. It's worth noting that if you specify the virbr0 interface for the relay it wont even start at all. -Nick Created attachment 1145038 [details]
systemd-analyze_plot.svg
I can confirm that proposed changes in unit file do not fix problem. Attaching systemd-analyze plot output.
So libvirtd service is considered running even its virtual bridges are not properly configured yet. In case of dhcpd and NetworkManager managed interfaces we have 2 mechanisms how to mitigate such scenario: network-online.target and NM dispatcher script, but none of them can be applied here as the libvirtd's virtual bridges are not NM managed (AFAIK). I'm at my wit's end here so I guess we can only ask libvirt folks if they have any idea how to properly sort the dependencies in a service file if a service needs (during start) the libvirt's virtual bridges being configured. Reassigning to libvirt so the above question gets some attention. Thanks. Laine, is there anything what can libvirt do about this? I guess the daemon returns immediately and does not wait for network initialization finish. It can take several seconds for each libvirt virtual network to start, and some hosts have a *lot* of networks. On top of that, the whole point of libvirt's virtual networks is to have a single point of configuration for a simple bridge+nat+dhcp+dns setup. If you're going to be doing parts of that yourself, you can just as well setup the bridge in the host system configuration (using NM, or directly creating ifcfg files). I don't think it's a good idea to serialize host system startup in order to benefit the small minority of users who want to set up socket listeners on libvirt's networks. So there's nothing that libvirt can (should) do *by default* for this situation. However, I was looking at the example of /usr/lib/systemd/Network-Manager-wait-online.service, and it seems like it would be reasonable to add a similar service that executed a script that would retrieve the device name for a given network, then wait in a loop until that device had an ip address. I'll attach such a script as soon as I'm done typing this comment. Here's a servicefile that I *think* will prevent dhcrelay.service from starting until this new service (I called it libvirt-default-network-wait-online.service) is completed: [Unit] Description=Wait for libvirt's default network to be fully operational Requisite=libvirtd.service After=libvirtd.service # Before=dhcrelay.service # Either uncomment the next above, or add #"After=libvirt-default-network-wait-online.service" # to dhcrelay.service [Service] Type=oneshot ExecStart=/usr/libexec/libvirt-net-wait-online.sh default [Install] # I don't know if this is needed or not :-) # WantedBy=dhcrelay.service I'm no systemd afficianado, but this is what I came up with in about 10 minutes of searching for "systemd network-online.target" and looking at related descriptions and an hour or so of playing around. I'm not 100% if it does what is needed, but it seems to. If you could give it a try I would appreciate it. Possibly we can add a service like this to the package, leaving it disabled until somebody needs it. Petr - how do I generate the "systemd analyze plot"? That looks very useful! ("systemd-analyze plot". Duh :-) Nick - if you can try out my suggestion from Comment 13 and give feedback, possibly we can include an example .service file in the libvirt package. Otherwise I guess this BZ should be closed. No response to my request, so I'm closing this. Re-open if my suggested systemd service file works and you think it should be included in libvirt's examples. |