Description of problem: --- Most likely there is some overt and intermittent bug when using cloud-init with templates & VM Pools. Leveraging cloud-init to set the hostname or password sometimes works and sometimes doesn't. Version-Release number of selected component (if applicable): 3.x & 4.x How reproducible: sometimes, ~ 10-20% Steps to Reproduce: I had random and different results. I can't point out what is wrong because there is no pattern. But I am able to give how I conducted tests. One more fact: at the beginning I was sure it is due to cloud-init and RHEL 7 (systemd) bug: 1417025 "cloud-init tries to run hostnamectl before dbus is up" https://bugzilla.redhat.com/show_bug.cgi?id=1417025 but CU reported he uses RHEL 6 and then I tested + confirmed I experience problems with RHEL 6 templates as well. A brief summary of tests: I decided to strictly follow guide: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/virtual_machine_management_guide/sect-using_cloud-init_to_automate_the_configuration_of_virtual_machines I created VM, installed cloud-init, set cloud-init for VM and after that I created template (as described in section "7.8.3. Using Cloud-Init to Prepare a Template" in above link) After that, I run all tests and everything was set properly (apart from password for 1 case). The most important: for VM Pools everything was correct, even cloud-init settings after edit VM (belonging to VM Pool). --- TESTS --- INFORMATION: CU setup RHEL6 template that makes use of cloud-init to set the hostname - all cloud-init parameters have been configured in the template. During creation VM based on this template, the hostname in the "initial run" tab is set and cloud-init correctly configures the hostname on first boot (the hostname command in the VM returns the correct hostname). When CU creates VM Pool the "initial run" tab shows no hostname and on system first boot the system hostname (in the VM) is not changed by cloud-init (and thus set to localhost.localdomain as is configured in the template) I run 2x tests: in RHV 3.6, and RHV 4.1. I run many minor tests in each environment. Process: 1. create RHEL 6 VM with cloud-init, 2. create a template, according to: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/virtual_machine_management_guide/sect-using_cloud-init_to_automate_the_configuration_of_virtual_machines 3. create VM based on a template which has "Use Cloud-init/Sysprep' in 'Initial Run' tab of New Template/Edit Template: - RHEV 3.6 : hostname set, password not strings /var/run/vdsm/payload/713d22e9-b765-4dd4-b0d7-d2d6c868929d.6c399cc77ccdb928555875ebd95bd6f6.img | grep -e hostname -e password "hostname" : "createdbycloudinit", - RHV 4.1 : hostname set, password not, strings /var/run/vdsm/payload/0dc6a04a-9e60-4448-a211-4aac0cb2a8cc.f338e26081e877a8d1010d87cc1ef6b0.img | grep -e hostname -e password "hostname" : "createdbycloudinit", file=/var/run/vdsm/payload/0dc6a04a-9e60-4448-a211-4aac0cb2a8cc.f338e26081e877a8d1010d87cc1ef6b0.img 4. create VM based on template - using Run Once. - RHEV 3.6: hostname set, password set strings /var/run/vdsm/payload/64fa0e7e-3cb1-456c-824d-48ab443bcb58.24a97fdb76ca51de49963f09516c832c.img | grep -e hostname -e password "hostname" : "createdbycloudinit2", password: 1234abcd2 - RHV 4.1: hostname set, password set strings /var/run/vdsm/payload/cb4e5443-ce05-480f-bb8c-ac4f3b783f97.55708c05d84bf7b703e754037be3f6eb.img | grep -e hostname -e password "hostname" : "createdbycloudinit2", password: 1234abcd2 5 create VM from a template and put new cloud-init hostname + password: - RHEV 3.6: hostname set, password set strings /var/run/vdsm/payload/03c86169-06a0-4322-a87f-4cfd5efdf039.a4b55a4aaa46bff7bc8ceb0c9b686f1a.img | grep -e hostname -e password "hostname" : "createdbycloudinit3", password: 1234abcd3 - RHV 4.1: hostname set, password set strings /var/run/vdsm/payload/b63ce802-03fe-41fc-b9c7-ce4d152ed776.bfc1e10c4fb720a2cdf2959a7e2a5457.img | grep -e hostname -e password "hostname" : "createdbycloudinit3", password: 1234abcd3 6. create VM from VM Pool - 'Use Cloud-Init/Sysprep' in 'Initial Run' tab. By default, VM Pool inherits cloud-init settings from template According to doc: "You can use the Use Cloud-Init/Sysprep options in the Initial Run tab of the New Pool window to specify options for customizing virtual machines taken from that virtual machine pool. This allows you to specify a set of standard settings that will be applied every time a virtual machine is taken from that virtual machine pool. You can inherit or override the options specified for the template on which the virtual machine is based, or specify options for the virtual machine pool itself. " a) test with Initial Run / Cloud-Init/Sysprep options in the tab of the New Pool - RHEV 3.6 hostname set, password set strings /var/run/vdsm/payload/b6d6b45e-a0bf-43c7-b805-848c09332d85.e793ca84375d34b3d1e8d1fe69c0ad44.img | grep -e hostname -e password "hostname" : "createdbycloudinit", password: 1234abcd - RHV 4.1 hostname set, password set (moreover there is no empty fields like you had on screenshot) strings /var/run/vdsm/payload/671783db-a221-442f-9afd-3e903254190b.a5294f620b000c1639b76b75e3e1b296.img | grep -e hostname -e password "hostname" : "createdbycloudinit4", password: 1234abcd4 b) test does it inherit cloud-init settings from template - RHEV 3.6 hostnaem set, password set strings /var/run/vdsm/payload/54843ee8-7438-4397-9e0f-867f58cf9370.adcd40a75595b2a28fb4a707b47cab09.img | grep -e hostname -e password "hostname" : "createdbycloudinit4", password: 1234abcd4 - RHV 4.1 hostname set, password set (moreover there is no empty fields like you had on screenshot) strings /var/run/vdsm/payload/921d0bb4-b974-4efa-aa96-ea16fbc01343.333c27e7d94b910afcc786dedc532cd1.img | grep -e hostname -e password "hostname" : "createdbycloudinit", password: 1234abcd SUMMARY: during tests I didn't hit problem like I hit testing it 1st time, so then I tried: 1. create VM, 2. create template, 3. edit template and set Initial Run / Use Cloud-Init/Sysprep 4. create VM Pool from this template Eventually, I had a different behavior for the same template. I've created 2 different VM Pools and in 1 case there was no payload attached In edit VM / Initial Run / Use Cloud-Init/Sysprep hostname was set. Details: RHV 4.1: hostname set, password set strings /var/run/vdsm/payload/135db379-3b85-4e5f-b1d2-bf0b413e3678.ce65500be789725d09d50f240fda128e.img | grep -e hostname -e password "hostname" : "createdbycloudinit5", password: 12345abcd5 RHEV 3.6: nothing set no payload attached, although edit VM (created from Pool) / Initial Run / Use Cloud-Init/Sysprep has a proper hostname ... One more test with 3.6. strings /var/run/vdsm/payload/b5b0af5b-315f-4448-979e-31515f8a8ad3.d50d90b51c04c4f010ce170541250699.img | grep -e hostname -e password "hostname" : "cratedbycloudinit6", password: 1234abcd6 The results are not consistent at all. I have neither found anything fixed nor reported as bug: 1. in vdsm I haven't found anything new regarding cloud-init. There are only old commits (2013 & 2014): 2. for ovirt-engine nothing has been fixed recently (the latest are from 2014): Actual results: sometimes new hostname or password is not set Expected results: cloud-init works everytime
Do you have cloud-init logs when it failed? The payload?
Just to report on the progress of this issue: - I have succeeded to simulate this issue on 4.1 while creating a VM from template on a template tab (happened only once and only in this flow, in no others) - it is not related to cloud init directly, it is an engine issue because while creating a VM from template, vm init configuration was not properly copied from the template to the VM (in the database) - the payload was correctly created and attached to the VM according to the configuration which was in the DB. Given the configuration for some reason did not contain the password, it was not in the payload. I don't yet know why and if it is a frontend or backend issue - keep investigating.
The issue is that when adding a VM from template with cloud-init configured, the cloud-init password needs to be updated by calling: vmHandler.updateVmInitFromDB(template, false); - nota bene the second param which tells that the password has to be loaded too. For pools I have not found a case where it is not loaded properly but for VMs made from template it happens. But, if the VM is based on the latest template, and the template is updated, the VM will get also the password from the latest template. So, we need to make sure the vmHandler.updateVmInitFromDB(template, false); is properly called always when creating a VM from template with password stored.
Steps to reproduce: 1. Create a linux template with 'Initial Run' > 'Password' set 2. Create a VM based on the template 3. Run the VM 4. Run on host: strings </var/run/vdsm/payload/path-to-cloudinit-disk> | grep password: Check value of printed password. I didn't met any problems with pools. As mentioned in comment 8 VM pools might not work with cloudinit root password if they were based on broken template.
Verified on 4.2.0-0.5.master.el7
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:1488
BZ<2>Jira Resync
sync2jira