Description of problem: Beaker LC daemons can start before the network is ready. Version-Release number of selected component (if applicable): 0.16 How reproducible: unsure Steps to Reproduce: unsure Actual results: Beaker LC daemons may start before the network link is up and interfaces are ready. One possible symptom is failure to resolve hostnames in beaker-provision, because it read /etc/resolv.conf before it was correctly populated by DHCP. Another possibility is failing to start due to being unable to connect to the remote Beaker server (this is only new in 0.16, in earlier versions the daemon would have started but failed to authenticate). Expected results: Beaker LC daemons should only start once network is up and running, so that they can reach the Beaker server and successfully resolve system hostnames. Additional info: With systemd we can just set After=network.target for the LC daemons which require network. On RHEL6 there is probably something we can do with LSB headers or runlevels, I'm not sure.
This may also be the cause of bug 978074.
(In reply to Dan Callaghan from comment #0) > One possible symptom is failure to resolve hostnames in > beaker-provision, because it read /etc/resolv.conf before it was correctly > populated by DHCP. Here is a stack trace showing that, submitted by a Beaker user who hit this problem: Mar 31 13:36:47 pek beaker-provision[2407]: bkr.labcontroller.provision ERROR Error processing command 3354 Mar 31 13:36:47 pek beaker-provision[2407]: Traceback (most recent call last): Mar 31 13:36:47 pek beaker-provision[2407]: File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 114, in handle Mar 31 13:36:47 pek beaker-provision[2407]: handle_clear_netboot(command) Mar 31 13:36:47 pek beaker-provision[2407]: File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 163, in handle_clear_netboot Mar 31 13:36:47 pek beaker-provision[2407]: netboot.clear_all(command['fqdn']) Mar 31 13:36:47 pek beaker-provision[2407]: File "/usr/lib/python2.6/site-packages/bkr/labcontroller/netboot.py", line 418, in clear_all Mar 31 13:36:47 pek beaker-provision[2407]: bootloader.clear(fqdn) Mar 31 13:36:47 pek beaker-provision[2407]: File "/usr/lib/python2.6/site-packages/bkr/labcontroller/netboot.py", line 194, in clear_pxelinux Mar 31 13:36:47 pek beaker-provision[2407]: basename = pxe_basename(fqdn) Mar 31 13:36:47 pek beaker-provision[2407]: File "/usr/lib/python2.6/site-packages/bkr/labcontroller/netboot.py", line 80, in pxe_basename Mar 31 13:36:47 pek beaker-provision[2407]: ipaddr = socket.gethostbyname(fqdn) Mar 31 13:36:47 pek beaker-provision[2407]: File "/usr/lib64/python2.6/site-packages/gevent/socket.py", line 623, in gethostbyname Mar 31 13:36:47 pek beaker-provision[2407]: return get_hub().resolver.gethostbyname(hostname) Mar 31 13:36:47 pek beaker-provision[2407]: File "/usr/lib64/python2.6/site-packages/gevent/resolver_ares.py", line 45, in gethostbyname Mar 31 13:36:47 pek beaker-provision[2407]: return self.gethostbyname_ex(hostname, family)[-1][0] Mar 31 13:36:47 pek beaker-provision[2407]: File "/usr/lib64/python2.6/site-packages/gevent/resolver_ares.py", line 53, in gethostbyname_ex Mar 31 13:36:47 pek beaker-provision[2407]: result = waiter.get() Mar 31 13:36:47 pek beaker-provision[2407]: File "/usr/lib64/python2.6/site-packages/gevent/hub.py", line 579, in get Mar 31 13:36:47 pek beaker-provision[2407]: return self.hub.switch() Mar 31 13:36:47 pek beaker-provision[2407]: File "/usr/lib64/python2.6/site-packages/gevent/hub.py", line 338, in switch Mar 31 13:36:47 pek beaker-provision[2407]: return greenlet.switch(self) Mar 31 13:36:47 pek beaker-provision[2407]: gaierror: [Errno 11] ARES_ECONNREFUSED: Could not contact DNS servers In this case restarting beaker-provision fixed the issue (because the network is now up and /etc/resolv.conf is now correct).