RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1262439 - dhcrelay doesn't forward from VM until restarted
Summary: dhcrelay doesn't forward from VM until restarted
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.2
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-09-11 16:47 UTC by Nicholas Schuetz
Modified: 2020-04-13 02:25 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-04 17:03:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
systemd-analyze_plot.svg (88.76 KB, application/octet-stream)
2016-04-08 07:29 UTC, Petr Janda
no flags Details

Description Nicholas Schuetz 2015-09-11 16:47:47 UTC
Description of problem:

dhcrelay doesnt forward requests until it is restarted.  When i start my system, it is enabled and starts, however, does nothing (does not forward dhcp requests to server).  When i 'systemctl restart dhcrelay', then it starts working.



[root@hv3 ~]# strace -p 1031
Process 1031 attached
select(22, [4 5 6], [], NULL, NULL
)     = 1 (in [6])
recvfrom(6, "\1\1\6\0Io\263\r\0\33\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0RT\0("..., 1540, 0, {sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, [16]) = 300
select(22, [4 5 6], [], NULL, NULL)     = 1 (in [6])
recvfrom(6, "\1\1\6\0Io\263\r\0*\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0RT\0("..., 1540, 0, {sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, [16]) = 300
select(22, [4 5 6], [], NULL, NULL)     = 1 (in [6])
recvfrom(6, "\1\1\6\0\336wKE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0RT\0("..., 1540, 0, {sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, [16]) = 300
.
.
.
<detached ...>


[root@hv3 ~]# systemctl restart dhcrelay


[root@hv3 ~]# ps -aef |grep dhcrelay
root      2873     1  0 11:39 ?        00:00:00 /usr/sbin/dhcrelay -d --no-pid 192.168.0.254
root      2875  2474  0 11:39 pts/0    00:00:00 grep --color=auto dhcrelay


[root@hv3 ~]# strace -p 2873
Process 2873 attached
select(22, [4 5 6 7 8], [], NULL, NULL) = 3 (in [4 5 8])
recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"\377\377\377\377\377\377RT\0(g\t\10\0E\20\1H\0\0\0\0\200\0219\226\0\0\0\0\377\377"..., 1536}], msg_controllen=36, {cmsg_len=36, cmsg_level=SOL_PACKET, cmsg_type=, ...}, msg_flags=0}, 0) = 342
sendto(3, "<30>Sep 11 11:39:34 dhcrelay: Di"..., 110, MSG_NOSIGNAL, NULL, 0) = 110
write(2, "Discarding packet received on vn"..., 80) = 80
write(2, "\n", 1)                       = 1
recvmsg(5, {msg_name(0)=NULL, msg_iov(1)=
.
.
.
.

write(2, "\n", 1)                       = 1
recvfrom(8, 0\0\3\300\250"..., 1536}], msg_controllen=36, {cmsg_len=36, cmsg_level=SOL_PACKET, cmsg_type=, ...}, msg_flags=0}, 0) = 342
select(22, [4 5 6 7 8], [], NULL, NULL) = 2 (in [6 8])
recvmsg(6, {msg_name(0)=NULL, msg_iov(1)=[{"\0\24\321*G\305\354\250k\371\377\263\10\0E\0\1H\353\244@\0@\21\310\260\300\250\0\376\300\250"..., 1536}], msg_controllen=36, {cmsg_len=36, cmsg_level=SOL_PACKET, cmsg_type=, ...}, msg_flags=0}, 0) = 342
write(5, "RT\0(g\tRT\0\264T\255\10\0E\20\1H\0\0\0\0\200\21\261\274\300\250\3\1\300\250"..., 342) = 342
write(2, "Forwarded BOOTREPLY for 52:54:00"..., 58) = 58
write(2, "\n", 1)                       = 1
recvfrom(8, "\2\1\6\1\366\353<O\0\0\0\0\0\0\0\0\300\250\3\207\0\0\0\0\300\250\3\1RT\0("...,  <detached ...>


[root@hv3 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.1 (Maipo)

[root@hv3 ~]# rpm -qa |grep dhc
dhcp-libs-4.2.5-36.el7.x86_64
dhclient-4.2.5-36.el7.x86_64
dhcp-common-4.2.5-36.el7.x86_64
dhcp-4.2.5-36.el7.x86_64


Possibly a problem with it's systemd unit file?  ie, dependencies not structured properly

Comment 2 Jiri Popelka 2015-09-15 12:20:39 UTC
(In reply to Nicholas Nachefski from comment #0)
> dhcrelay doesnt forward requests until it is restarted.  When i start my
> system, it is enabled and starts, however, does nothing (does not forward
> dhcp requests to server).  When i 'systemctl restart dhcrelay', then it
> starts working.

Hmm, I haven't been able to reproduce the issue in VM.

> Possibly a problem with it's systemd unit file?  ie, dependencies not
> structured properly

We recently changed
After=network.target
to
Wants=network-online.target
After=network-online.target
in dhcrelay.service due to bug #1145832 and I have no idea what else could be a culprit.

Comment 3 Nicholas Schuetz 2016-02-24 20:10:04 UTC
This is still a problem.  And, if you search for 'dhcrelay not starting' in google you get a ton of hits about this from other BZ sites.  I think this should be looked at a little harder.  I'm running F23 (updated) and this is still happening as of today.

I think i just found the problem.  It's an ordering problem with the systemd unit file.  I need this dhcrelay to proxy DHCP request from the VM network to my IPA server (which is also running dhcpd).  When dhcrelay starts, virbr0 doesnt exist yet.  However, once the box is fully booted, virbr0 *does* exist and when i manually restart dhcrelay it works fine.  If you set '-i virbr0' in the unit file (explicitly setting the interface) dhrelay wont start *at all*.

Please take a harder look as this is apparently a PITA for everyone.

-Nick

Comment 4 Jiri Popelka 2016-02-25 08:44:43 UTC
Does it help if you

cp /usr/lib/systemd/system/dhcrelay.service /etc/systemd/system/dhcrelay.service

and add
Wants=libvirtd.service
After=libvirtd.service
to [Unit] in /etc/systemd/system/dhcrelay.service ?

Comment 5 Nicholas Schuetz 2016-03-05 19:17:37 UTC
Sorry for not getting back sooner.  This fixed my problem.

THANKS!

-Nick

Comment 6 Nicholas Schuetz 2016-03-05 19:20:05 UTC
this fixed the issue on both my RHEL72 and F23 hyper visors.  Thanks again!

Comment 7 Petr Janda 2016-04-06 15:48:33 UTC
reproduced on RHEL-7.2 x86_64 Server

Comment 9 Nicholas Schuetz 2016-04-06 15:57:54 UTC
I spoke too soon when i said that this problem was fixed.  It is *not* fixed in my lab environment, the problem persists.  

It's worth noting that if you specify the virbr0 interface for the relay it wont even start at all.

-Nick

Comment 10 Petr Janda 2016-04-08 07:29:22 UTC
Created attachment 1145038 [details]
systemd-analyze_plot.svg

I can confirm that proposed changes in unit file do not fix problem. Attaching systemd-analyze plot  output.

Comment 11 Jiri Popelka 2016-04-08 16:22:48 UTC
So libvirtd service is considered running even its virtual bridges are not properly configured yet.

In case of dhcpd and NetworkManager managed interfaces we have 2 mechanisms how to mitigate such scenario: network-online.target and NM dispatcher script, but none of them can be applied here as the libvirtd's virtual bridges are not NM managed (AFAIK).

I'm at my wit's end here so I guess we can only ask libvirt folks if they have any idea how to properly sort the dependencies in a service file if a service needs (during start) the libvirt's virtual bridges being configured.

Reassigning to libvirt so the above question gets some attention. Thanks.

Comment 12 Jaroslav Suchanek 2016-04-29 12:17:46 UTC
Laine, is there anything what can libvirt do about this? I guess the daemon returns immediately and does not wait for network initialization finish.

Comment 13 Laine Stump 2016-06-01 18:17:17 UTC
It can take several seconds for each libvirt virtual network to start, and some hosts have a *lot* of networks. On top of that, the whole point of libvirt's virtual networks is to have a single point of configuration for a simple bridge+nat+dhcp+dns setup. If you're going to be doing parts of that yourself, you can just as well setup the bridge in the host system configuration (using NM, or directly creating ifcfg files).

I don't think it's a good idea to serialize host system startup in order to benefit the small minority of users who want to set up socket listeners on libvirt's networks. So there's nothing that libvirt can (should) do *by default* for this situation.

However, I was looking at the example of /usr/lib/systemd/Network-Manager-wait-online.service, and it seems like it would be reasonable to add a similar service that executed a script that would retrieve the device name for a given network, then wait in a loop until that device had an ip address. I'll attach such a script as soon as I'm done typing this comment.

Here's a servicefile that I *think* will prevent dhcrelay.service from starting until this new service (I called it libvirt-default-network-wait-online.service) is completed:

[Unit]
Description=Wait for libvirt's default network to be fully operational
Requisite=libvirtd.service
After=libvirtd.service
# Before=dhcrelay.service
# Either uncomment the next above, or add
#"After=libvirt-default-network-wait-online.service"
# to dhcrelay.service

[Service]
Type=oneshot
ExecStart=/usr/libexec/libvirt-net-wait-online.sh default

[Install]
# I don't know if this is needed or not :-)
# WantedBy=dhcrelay.service

I'm no systemd afficianado, but this is what I came up with in about 10 minutes of searching for "systemd network-online.target" and looking at related descriptions and an hour or so of playing around. I'm not 100% if it does what is needed, but it seems to. If you could give it a try I would appreciate it. Possibly we can add a service like this to the package, leaving it disabled until somebody needs it.

Petr - how do I generate the "systemd analyze plot"? That looks very useful!

Comment 14 Laine Stump 2016-06-15 18:19:35 UTC
("systemd-analyze plot". Duh :-)

Comment 15 Laine Stump 2016-06-29 18:38:57 UTC
Nick - if you can try out my suggestion from Comment 13 and give feedback, possibly we can include an example .service file in the libvirt package. Otherwise I guess this BZ should be closed.

Comment 16 Laine Stump 2016-08-04 17:03:02 UTC
No response to my request, so I'm closing this. Re-open if my suggested systemd service file works and you think it should be included in libvirt's examples.


Note You need to log in before you can comment on or make changes to this bug.