Bug 1082458 - lab controller daemons can start before the network is ready
Summary: lab controller daemons can start before the network is ready
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Beaker
Classification: Retired
Component: lab controller
Version: 0.16
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified vote
Target Milestone: ---
Assignee: beaker-dev-list
QA Contact: tools-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-03-31 06:38 UTC by Dan Callaghan
Modified: 2020-10-21 14:20 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-21 14:15:24 UTC


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 978074 0 unspecified CLOSED beaker-proxy services requests before it is authenticated to the server 2021-02-22 00:41:40 UTC

Internal Links: 978074

Description Dan Callaghan 2014-03-31 06:38:31 UTC
Description of problem:
Beaker LC daemons can start before the network is ready.

Version-Release number of selected component (if applicable):
0.16

How reproducible:
unsure

Steps to Reproduce:
unsure

Actual results:
Beaker LC daemons may start before the network link is up and interfaces are ready. One possible symptom is failure to resolve hostnames in beaker-provision, because it read /etc/resolv.conf before it was correctly populated by DHCP. Another possibility is failing to start due to being unable to connect to the remote Beaker server (this is only new in 0.16, in earlier versions the daemon would have started but failed to authenticate).

Expected results:
Beaker LC daemons should only start once network is up and running, so that they can reach the Beaker server and successfully resolve system hostnames.

Additional info:
With systemd we can just set After=network.target for the LC daemons which require network.
On RHEL6 there is probably something we can do with LSB headers or runlevels, I'm not sure.

Comment 1 Dan Callaghan 2014-03-31 06:38:56 UTC
This may also be the cause of bug 978074.

Comment 2 Dan Callaghan 2014-03-31 06:40:23 UTC
(In reply to Dan Callaghan from comment #0)
> One possible symptom is failure to resolve hostnames in
> beaker-provision, because it read /etc/resolv.conf before it was correctly
> populated by DHCP.

Here is a stack trace showing that, submitted by a Beaker user who hit this problem:

Mar 31 13:36:47 pek beaker-provision[2407]: bkr.labcontroller.provision ERROR Error processing command 3354
Mar 31 13:36:47 pek beaker-provision[2407]:  Traceback (most recent call last):
Mar 31 13:36:47 pek beaker-provision[2407]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 114, in handle
Mar 31 13:36:47 pek beaker-provision[2407]:      handle_clear_netboot(command)
Mar 31 13:36:47 pek beaker-provision[2407]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 163, in handle_clear_netboot
Mar 31 13:36:47 pek beaker-provision[2407]:      netboot.clear_all(command['fqdn'])
Mar 31 13:36:47 pek beaker-provision[2407]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/netboot.py", line 418, in clear_all
Mar 31 13:36:47 pek beaker-provision[2407]:      bootloader.clear(fqdn)
Mar 31 13:36:47 pek beaker-provision[2407]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/netboot.py", line 194, in clear_pxelinux
Mar 31 13:36:47 pek beaker-provision[2407]:      basename = pxe_basename(fqdn)
Mar 31 13:36:47 pek beaker-provision[2407]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/netboot.py", line 80, in pxe_basename
Mar 31 13:36:47 pek beaker-provision[2407]:      ipaddr = socket.gethostbyname(fqdn)
Mar 31 13:36:47 pek beaker-provision[2407]:    File "/usr/lib64/python2.6/site-packages/gevent/socket.py", line 623, in gethostbyname
Mar 31 13:36:47 pek beaker-provision[2407]:      return get_hub().resolver.gethostbyname(hostname)
Mar 31 13:36:47 pek beaker-provision[2407]:    File "/usr/lib64/python2.6/site-packages/gevent/resolver_ares.py", line 45, in gethostbyname
Mar 31 13:36:47 pek beaker-provision[2407]:      return self.gethostbyname_ex(hostname, family)[-1][0]
Mar 31 13:36:47 pek beaker-provision[2407]:    File "/usr/lib64/python2.6/site-packages/gevent/resolver_ares.py", line 53, in gethostbyname_ex
Mar 31 13:36:47 pek beaker-provision[2407]:      result = waiter.get()
Mar 31 13:36:47 pek beaker-provision[2407]:    File "/usr/lib64/python2.6/site-packages/gevent/hub.py", line 579, in get
Mar 31 13:36:47 pek beaker-provision[2407]:      return self.hub.switch()
Mar 31 13:36:47 pek beaker-provision[2407]:    File "/usr/lib64/python2.6/site-packages/gevent/hub.py", line 338, in switch
Mar 31 13:36:47 pek beaker-provision[2407]:      return greenlet.switch(self)
Mar 31 13:36:47 pek beaker-provision[2407]:  gaierror: [Errno 11] ARES_ECONNREFUSED: Could not contact DNS servers

In this case restarting beaker-provision fixed the issue (because the network is now up and /etc/resolv.conf is now correct).


Note You need to log in before you can comment on or make changes to this bug.