Beaker managed virtual guests are occasionally receiving MAC addresses that duplicate other addresses on the network, which then causes conflicts between tests. While the root cause of these problems is not yet completely certain, one likely candidate is the new MAC address management scheme that was adopted in Beaker 0.10 in order to avoid exhausting the DHCP server's pool of IP addresses when the execution of multiple short recipes leads to the creation of a large number of random MAC addresses (this became a much greater concern with the introduction of the oVirt Engine integration in 0.10). One currently known mechanism to trigger this misbehaviour (which is what suggests the MAC address management may be at fault): 1. Run a test with guest recipes. The host machine will be powered off automatically at the end of the test and the MAC addresses of the guests. 2. Run another test with guest recipes in the same lab. This test may be reassigned one or more addresses from the previous recipe. 3. Power the machine from step 1 back on *without* reprovisioning it first. The resurrected guests in from step 3 will then conflict with the running tests from step 2.
Oops, the description in step 1 of the reproducer is incomplete. It should read "... the MAC addresses of the guests will be made available for reuse". Another point to note is that the low layer scheme for assigning MAC addresses is currently known not to be thread safe. It *should* be protected by the high level "only schedule one recipe at a time" behaviour in beakerd, but if we've made a mistake in our assumptions, then parallel execution of that code could potentially trigger this symptom of MAC address conflicts. I'll also note that the use case described in the reproducer *is* one we want to support - test shutdown is deliberately non-destructive so that users can go investigate a system after the fact. That means solutions based on preemptively destroying the guest VMs, or reprovisioning the host machine, are not really appropriate. It may prove necessary to redesign the MAC address pool management such that the MAC address assignment survives until the host machine is reprovisioned (rather than the current behaviour where it is released back to the pool as soon as the guest recipe running on that VM completes).
I don't see why we should support that use case. If a recipe has completed, than we should not guarantee the state of the system beyond the expiration of that recipe. We don't do this with regular systems, so I'm not sure why we should for guest recipes. We have an extendible watchdog for this purpose. Perhaps we could reprovision (or something to make it inoperable, but auto reprovision is familiar) the host on completion to guarantee that the guest cannot be re-used beyond the recipe.
I think it would be enough if at the end of the guest recipe we just re-set the MAC address to some random value (we can probably make libvirt do this by just editing the guest definition and deleting the MAC address in there) and perhaps also set the guests not to autostart. This would be a much less destructive solution. The hard part is how to trigger it. Right now the host recipe typically doesn't wait for the guests, it finishes while the guests are still running, so we can't do it in a normal task on the host. There's also not really any facility to do it as part of returning a system because right now the host is just powered off immediately.
From the recipeset: <guestrecipe guestargs="-r 512 -s 10 --kvm" guestname="testvm1" kernel_options="" kernel_options_post="" ks_meta="skipx" role="None" whiteboard=""> The code which assign MACs to VMs is here: /mnt/tests/distribution/virt/install/get_guest_info.py our part is: for guestrecipe in doc.getElementsByTagName('guestrecipe'): print ' '.join([ guestrecipe.getAttribute('id') or 'RECIPEIDMISSING', guestrecipe.getAttribute('guestname') or 'guestrecipe%s' % guestrecipe.getAttribute('id'), guestrecipe.getAttribute('mac_address') or 'RANDOM', guestrecipe.getAttribute('location') or 'LOCATIONMISSING', guestrecipe.getAttribute('kickstart_url') or 'KSMISSING', guestrecipe.getAttribute('guestargs'), ]) I also tried with this as a workaround: <guestrecipe guestargs="-r 512 -s 10 --kvm" mac_address="52:54:00:11:11:10" guestname="testvm1" kernel_options="" kernel_options_post="" ks_meta="skipx" role="None" whiteboard=""> with result: Job failed schema validation. Please confirm that you want to submit it. I think, DTD for beaker XML should be broken, too. The solution should be to save already used MAC addressed to some DB, check that DB for available MACs when we are deploying new VMs, and let it be in DB while physical machine is still up with same recipeset (reservesys). Random is definitely not enough, here.
(In reply to comment #2) > I don't see why we should support that use case. If a recipe has completed, > than we should not guarantee the state of the system beyond the expiration > of that recipe. We don't do this with regular systems, so I'm not sure why > we should for guest recipes. > > We have an extendible watchdog for this purpose. > > Perhaps we could reprovision (or something to make it inoperable, but auto > reprovision is familiar) the host on completion to guarantee that the guest > cannot be re-used beyond the recipe. Raymond, Until last week we did not have guest logs for Beaker virt testing. So when KVM/Xen testing failed. If the host machine was not re-provisioned for another recipe. We would take the machine and power it on then try and isolate the failure cause. This may happen less now that we have a way to get the guest console logs. But I can see times that this will happen even though we have logs. Is is possible to not reuse this MAC immediately. Would it be possible to keep track of which ones have been used and not reuse them until the entire list has been used. Thanks, Jeff
(In reply to comment #3) > I think it would be enough if at the end of the guest recipe we just re-set > the MAC address to some random value (we can probably make libvirt do this > by just editing the guest definition and deleting the MAC address in there) > and perhaps also set the guests not to autostart. This would be a much less > destructive solution. Just to be sure: when you say "would be enough if at the end of the guest recipe we just re-set the MAC address to some random value " do you mean at the end of the entire XML, or at the end of the specific guestrecipe for that VM? \
Guys, to be clear, let me briefly explain our test case (expectations) used with VMs: 1. submit XML with 1 physical system with 3 VMs 2. run HA test on top of each VM 3. synchronize VMs (we have workaround for that) 4. fence_node <VMx> 5. wait for VMx 6. check if services was relocated, restarted, moved back in cluster The point 4 means, that physical system is still up and running because test are not finished, but VM was rebooted - fenced. If you will change MAC in case of VM reboot, the udev persistent rule will assign different name for eth device -> this breaks cluster. What we need, is to have unchanged configuration (MAC, IPv4, IPv6) for all VMs for a whole physical system lifecycle in one recipeset. In other words, if tests are still running or physical system is in reservesys, the whole VMs environment must be preserved.
(In reply to comment #4) MAC addresses are allocated by the Beaker scheduler according to what is being used by running recipes in the lab at that time. There is no mechanism for users to specify the MAC address as part of their job. The mac_address attribute on <guestrecipe/> appears in the job results XML (bkr job-results) but not when cloning jobs and it is not accepted when submitting jobs. So the schema is correct. get_guest_info.py also gets the MAC address correctly, the 'RANDOM' fallback value will never be used. If you're interested, the scheduler code for allocating MAC addresses is here: http://git.beaker-project.org/cgit/beaker/tree/Server/bkr/server/model.py?h=release-0.11#n6057
(In reply to comment #5) > Is is possible to not reuse this MAC immediately. Would it be possible to > keep track of which ones have been used and not reuse them until the entire > list has been used. The scheduler doesn't have a list of available MAC addresses, instead it just has a base address and it always allocates the lowest unused address starting from that base. The problem is just when "unused" addresses are actually being used by powering up the host after the recipe has finished. We wanted to avoid having fixed-size pools of MAC addresses because then the admins would have to specify the size of the DHCP pool in each lab, and keep it up to date with any changes in the DHCP config. What we have right now doesn't need any configuration.
(In reply to comment #6) > Just to be sure: when you say "would be enough if at the end of the guest > recipe we just re-set the MAC address to some random value " do you mean at > the end of the entire XML, or at the end of the specific guestrecipe for > that VM? Well, I'm not really sure of when exactly it would happen because I still don't have a good way of triggering it. But I guess it would need to be after all the recipes (including guest recipes) in the recipe set have finished and everything is powered down.
(In reply to comment #7) Yes, understood. We won't fiddle with MAC addresses before the end of the recipe set because it breaks the use case you described (and many other similar kinds of tests which I'm sure people are running).
So I finally figured out that the actual most common cause of these duplicate MAC addresses is a simpler (and way nastier) bug than described in comment #0 (powering on machines after Beaker has returned them). Namely, we are re-using MAC addresses as soon as the recipe has finished -- but if the *recipe set* is still running (for example, because the host recipe has /distribution/reservesys) then the guest could still very well be running with the same MAC address. At the very least, we need to only re-use the MAC address once the entire recipe set has finished. We should still also consider some of the ideas above, for mitigating the problem described in comment #0.
I've cloned the problem from comment #12 as bug 905743. The solution for that bug is straightforward and we can do it as a hot fix ASAP. We can leave this bug open for the original problem described in comment #0, which is not urgent because it should only happen rarely, and we don't have a good solution for it yet.
Given that we have at least one site which needs Beaker-assigned MAC addresses to come from as small a range as possible, and our production instance which is more concerned about MAC address conflicts, it might be enough to just have a config option to toggle between two MAC address allocation behaviour: * aggressively reuse MAC addresses (the current behaviour) * never reuse MAC addresses (for example, generate them from base addr + recipe ID)
At the moment, if "base_mac_addr" isn't defined, we default it to '52:54:00:00:00:00' and then use the aggressive MAC address reuse logic Would it be sufficient to say that we'll try to reuse addresses aggressively if a base MAC address is defined, and that otherwise we will try to ensure MAC addresses are always unique? Is there a likely scenario where someone would want to set a minimum address without also wanting to set a maximum address in their lab DHCP server? If this approach sounds reasonable, then we could change _lowest_free_mac to an instance method and use the '52:54:00:00:00:00' + recipe ID approach when base_mac_addr isn't defined at all.
Jan Stancek also pointed out that any system which is set to LeaveOn is *guaranteed* to cause this problem when it runs guest recipes, so this is not a rare problem as I claimed in comment 13.
Possible solution: MAC address allocation logic considers an address to still be reserved even after a recipe finishes, so long as the host system has not been reprovisioned yet. Once it is provisioned again we can assume the MAC address is free to re-use. Hard part is querying whether a system has been reprovisioned yet. We still don't track provisions in a sane way (only configure_netboot commands in the queue).