Starting with beah 0.7, the daemons all prefer to connect to the lab controller over IPv6 if they can. This allows IPv4 to be turned off without affecting beah's operation, but turning off IPv*6* can now cause problems, as the favoured connection over IPv6 starts failing and beah doesn't fall back to IPv4 in this case. Transparently falling back to IPv4 probably isn't desirable (as it may mask problems with dual stack IPv4/6 support), but a mechanism should be available to force beah to operate in "IPv4 only" mode, allowing this kind of test to be handled without needing to revert back to older versions of beah that don't support IPv6 at all. Until this issue is resolved, the workaround is to specify "beah_rpm=beah-0.6.48" as described in bug 1063090 (this feature is standard in Beaker 0.15.4+, and may be available as a hot fix or configuration change on particular installations).
Beah does already fall back to IPv4 whenever IPv6 fails, except for the following two remaining problems: * beah-beaker-backend only checks for IPv6 connectivity to the LC once at startup: if this succeeds, then it will only attempt IPv6 for the rest of the lifetime of the process. That means if IPv6 is working correctly at first but then stops working (because the task disabled it or misconfigured the network or triggered a kernel bug) beah-beaker-backend will fail to communicate with Beaker. * beah-rhts-task (which executes each task) only checks for IPv6 loopback once at startup: if it successfully binds to ::1, it exports RESULT_SERVER=[::1]:7090 in the task's environment. The rhts-* scripts then connect over IPv6 loopback to talk to beah. However, if IPv6 is subsequently disabled by the task (or the ::1 loopback address is deleted) the RESULT_SERVER environment variable will still be pointing at ::1 which will fail. The task will then become stuck when it tries to report a result or submit a log. The first issue could be solved by making beah-beaker-backend do the try-IPv6-fall-back-to-IPv4 dance for every HTTP request, but that means greatly increased overhead for the IPv4 case. I don't think the second issue can be solved at all, since there's no opportunity to change RESULT_SERVER once it's been exported. All we can do is go back to IPv4 (127.0.0.1) by default, which means it won't support deleting the IPv4 loopback address while the harness is running (which was the original goal of the IPv6 support in beah). The restraint harness uses UNIX domain sockets for this, which avoids this issue altogether.
(In reply to Dan Callaghan from comment #1) > * beah-rhts-task (which executes each task) only checks for IPv6 loopback > once at startup: if it successfully binds to ::1, it exports > RESULT_SERVER=[::1]:7090 in the task's environment. The rhts-* scripts then > connect over IPv6 loopback to talk to beah. However, if IPv6 is subsequently > disabled by the task (or the ::1 loopback address is deleted) the > RESULT_SERVER environment variable will still be pointing at ::1 which will > fail. The task will then become stuck when it tries to report a result or > submit a log. One possible workaround is to update RESULT_SERVER in the task itself, if the task is going to disable IPv6: RESULT_SERVER="${RESULT_SERVER/\[::1\]/127.0.0.1}"
Yeah, I don't believe it's possible to solve this automatically (at least, not in a sensible way) - beah would need to be explicitly told "don't use IPv6 even if its available" as part of the recipe definition. At the moment, the spelling for that is "beah_rpm=beah-0.6.48" in the recipe ks_meta settings, as that is the last version that was IPv4 only. So this issue is really about providing an alternate recipe configuration setting that allows a newer beah version to be used, but still talking to the lab controller over IPv4, and exporting 127.0.0.1 as the result server host.
(In reply to Nick Coghlan from comment #3) > > So this issue is really about providing an alternate recipe configuration > setting that allows a newer beah version to be used, but still talking to > the lab controller over IPv4, and exporting 127.0.0.1 as the result server > host. One approach could be to allow the job submitter to specify a recipe configuration which is then set as an environment variable and available when each of the beah services start. Each of these services will then simply pretend that there is no IPv6 support required (and hence make no attempt to use it) if the environment variable is set.
(In reply to Amit Saha from comment #4) > (In reply to Nick Coghlan from comment #3) > > > > So this issue is really about providing an alternate recipe configuration > > setting that allows a newer beah version to be used, but still talking to > > the lab controller over IPv4, and exporting 127.0.0.1 as the result server > > host. > > One approach could be to allow the job submitter to specify a recipe > configuration which is then set as an environment variable and available > when each of the beah services start. Each of these services will then > simply pretend that there is no IPv6 support required (and hence make no > attempt to use it) if the environment variable is set. This is what I am thinking: If a recipe defines "ipv4only=True" (or something like that) as: <recipe kernel_options="" kernel_options_post="" ks_meta="selinux=disabled" role="None" whiteboard="Nose Tests" ipv4only="True"> .. </recipe> Then, Beah doesn't bother using IPv6 for any communication at all. On the implementation side, we can export it as an env variable when we parse the recipe XML and then look for it at all the places we connect/listen to a network service and proceed appropriately.
(In reply to Amit Saha from comment #5) I would suggest just defining a new ks_meta variable "harness_noipv6", when the variable is defined we add a line like "IPV6_ENABLED=False" when writing /etc/beah_beaker.conf in the kickstart. Then IPV6_ENABLED would just be a normal beah configuration setting (default True). Longer term I think we need a better mechanism for the recipe to pass config to the harness, but I'm not sure what that would look like. I don't think setting everything in the systemwide environment is a good solution because it will lead to a lot of environment pollution as the variables proliferate.
(In reply to Dan Callaghan from comment #6) > (In reply to Amit Saha from comment #5) > > I would suggest just defining a new ks_meta variable "harness_noipv6", when > the variable is defined we add a line like "IPV6_ENABLED=False" when writing > /etc/beah_beaker.conf in the kickstart. Then IPV6_ENABLED would just be a > normal beah configuration setting (default True). Oh, this what you meant the last time we spoke about this. I completely forgot we were creating this file on the test system. Yeah, I think let's attempt to take this approach for now instead of polluting the system environment.
Beah part of the patch: http://gerrit.beaker-project.org/#/c/2993/1 Beaker parts of the patch: - http://gerrit.beaker-project.org/#/c/2992/ - http://gerrit.beaker-project.org/#/c/2931/ (Merged)
To make sure the fix to this bug is clear: If a recipe has ``beah_no_ipv6`` in the ``ksmeta`` variable, Beah will *not* attempt to use IPv6 for any of its network communication. Thus, if a task disables IPv6 during it's execution, it is suggested that the recipe makes use of this new feature. Final patch committed: gerrit.beaker-project.org/#/c/2993/
Job 4201 is "PASS" 2014-04-07 11:56:10,163 beah start_server: INFO ################################ 2014-04-07 11:56:10,163 beah start_server: INFO # Starting a Controller... # 2014-04-07 11:56:10,164 beah start_server: INFO ################################ 2014-04-07 11:56:10,167 beah start_server: INFO Controller: BackendListener listening on 0.0.0.0 port 12432 2014-04-07 11:56:10,168 beah start_server: INFO Controller: BackendListener listening on /var/beah/backend12432.socket 2014-04-07 11:56:10,170 beah start_server: INFO Controller: TaskListener listening on 0.0.0.0 port 12434 2014-04-07 11:56:10,170 beah start_server: INFO Controller: TaskListener listening on /var/beah/task12434.socket 2014-04-07 11:56:10,302 backend make_proxy: INFO Communicating to LC over IPv4 Version: Installing : beah-0.7.3-1.git.7.047496b.el7.noarch 8/10
Beaker 0.16.2 has been released.