Bug 1463597

Summary: [downstream clone - 4.1.3] Issues with automating the configuration of VMs (cloud-init)
Product: Red Hat Enterprise Virtualization Manager Reporter: rhev-integ
Component: ovirt-engineAssignee: jniederm
Status: CLOSED ERRATA QA Contact: Israel Pinto <ipinto>
Severity: medium Docs Contact:
Priority: high    
Version: 4.0.6CC: lsurette, melewis, mgoldboi, michal.skrivanek, mtessun, obockows, rbalakri, Rhev-m-bugs, srevivo, tjelinek, ykaul
Target Milestone: ovirt-4.1.3Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
With this update, an issue where the 'Use already configured password' option was unavailable when creating a new virtual machine from a template that had the cloudinit root password stored has been fixed. This issue meant that the creation of the new virtual machine succeeded but the password was not copied.
Story Points: ---
Clone Of: 1448831 Environment:
Last Closed: 2017-07-06 07:30:42 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1448831    
Bug Blocks:    

Description rhev-integ 2017-06-21 09:47:01 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1448831 +++
======================================================================

Description of problem:
---

Most likely there is some overt and intermittent bug when using cloud-init with templates & VM Pools.
Leveraging cloud-init to set the hostname or password sometimes works and sometimes doesn't. 


Version-Release number of selected component (if applicable):
3.x & 4.x 

How reproducible:
sometimes, ~ 10-20%

Steps to Reproduce:
I had random and different results. I can't point out what is wrong because there is no pattern. But I am able to give how I conducted tests. 
One more fact: at the beginning I was sure it is due to cloud-init and RHEL 7 (systemd) bug:
1417025 "cloud-init tries to run hostnamectl before dbus is up"
https://bugzilla.redhat.com/show_bug.cgi?id=1417025
but CU reported he uses RHEL 6 and then I tested + confirmed I experience problems with RHEL 6 templates as well.

A brief summary of tests: I decided to strictly follow guide:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/virtual_machine_management_guide/sect-using_cloud-init_to_automate_the_configuration_of_virtual_machines

I created VM, installed cloud-init, set cloud-init for VM and after that I created template (as described in section "7.8.3. Using Cloud-Init to Prepare a Template" in above link)
After that, I run all tests and everything was set properly (apart from password for 1 case). The most important: for VM Pools everything was correct, even cloud-init settings after edit VM (belonging to VM Pool).

---
TESTS
---

INFORMATION: 
CU setup RHEL6 template that makes use of cloud-init to set the hostname - all cloud-init parameters have been configured in the template. During creation VM based on this template, the hostname in the "initial run" tab is set and cloud-init correctly configures the hostname on first boot (the hostname command in the VM returns the correct hostname).
When CU creates VM Pool the "initial run" tab shows no hostname and on system first boot the system hostname (in the VM) is not changed by cloud-init (and thus set to localhost.localdomain as is configured in the template)

I run 2x tests: in RHV 3.6, and RHV 4.1. 
I run many minor tests in each environment. 
Process: 

1. create RHEL 6 VM with cloud-init, 

2. create a template, according to: 
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/virtual_machine_management_guide/sect-using_cloud-init_to_automate_the_configuration_of_virtual_machines

3. create VM based on a template which has "Use Cloud-init/Sysprep' in 'Initial Run' tab of New Template/Edit Template:
- RHEV 3.6 : hostname set, password not

strings /var/run/vdsm/payload/713d22e9-b765-4dd4-b0d7-d2d6c868929d.6c399cc77ccdb928555875ebd95bd6f6.img | grep -e hostname -e password
 "hostname" : "createdbycloudinit",

- RHV 4.1 : hostname set, password not, 
strings /var/run/vdsm/payload/0dc6a04a-9e60-4448-a211-4aac0cb2a8cc.f338e26081e877a8d1010d87cc1ef6b0.img | grep -e hostname -e password
  "hostname" : "createdbycloudinit",

file=/var/run/vdsm/payload/0dc6a04a-9e60-4448-a211-4aac0cb2a8cc.f338e26081e877a8d1010d87cc1ef6b0.img

4. create VM based on template - using Run Once. 
- RHEV 3.6: hostname set, password set
strings /var/run/vdsm/payload/64fa0e7e-3cb1-456c-824d-48ab443bcb58.24a97fdb76ca51de49963f09516c832c.img | grep -e hostname -e password
  "hostname" : "createdbycloudinit2",
password: 1234abcd2


- RHV 4.1: hostname set, password set
strings /var/run/vdsm/payload/cb4e5443-ce05-480f-bb8c-ac4f3b783f97.55708c05d84bf7b703e754037be3f6eb.img | grep -e hostname -e password
  "hostname" : "createdbycloudinit2",
password: 1234abcd2

5 create VM from a template and put new cloud-init hostname + password:

- RHEV 3.6: hostname set, password set
strings /var/run/vdsm/payload/03c86169-06a0-4322-a87f-4cfd5efdf039.a4b55a4aaa46bff7bc8ceb0c9b686f1a.img | grep -e hostname -e password
  "hostname" : "createdbycloudinit3",
password: 1234abcd3


- RHV 4.1: hostname set, password set
strings /var/run/vdsm/payload/b63ce802-03fe-41fc-b9c7-ce4d152ed776.bfc1e10c4fb720a2cdf2959a7e2a5457.img | grep -e hostname -e password
  "hostname" : "createdbycloudinit3",
password: 1234abcd3

6. create VM from VM Pool - 'Use Cloud-Init/Sysprep' in 'Initial Run' tab. By default, VM Pool inherits cloud-init settings from template

According to doc: "You can use the Use Cloud-Init/Sysprep options in the Initial Run tab of the New Pool window to specify options for customizing virtual machines taken from that virtual machine pool. This allows you to specify a set of standard settings that will be applied every time a virtual machine is taken from that virtual machine pool. You can inherit or override the options specified for the template on which the virtual machine is based, or specify options for the virtual machine pool itself. "

a) test with Initial Run / Cloud-Init/Sysprep options in the  tab of the New Pool

- RHEV 3.6 hostname set, password set

strings /var/run/vdsm/payload/b6d6b45e-a0bf-43c7-b805-848c09332d85.e793ca84375d34b3d1e8d1fe69c0ad44.img | grep -e hostname -e password
  "hostname" : "createdbycloudinit",
password: 1234abcd

- RHV 4.1 hostname set, password set (moreover there is no empty fields like you had on screenshot)

strings /var/run/vdsm/payload/671783db-a221-442f-9afd-3e903254190b.a5294f620b000c1639b76b75e3e1b296.img | grep -e hostname -e password
  "hostname" : "createdbycloudinit4",
password: 1234abcd4


b) test does it inherit cloud-init settings from template

- RHEV 3.6 hostnaem set, password set

strings /var/run/vdsm/payload/54843ee8-7438-4397-9e0f-867f58cf9370.adcd40a75595b2a28fb4a707b47cab09.img  | grep -e hostname -e password
  "hostname" : "createdbycloudinit4",
password: 1234abcd4

- RHV 4.1 hostname set, password set (moreover there is no empty fields like you had on screenshot)

strings /var/run/vdsm/payload/921d0bb4-b974-4efa-aa96-ea16fbc01343.333c27e7d94b910afcc786dedc532cd1.img | grep -e hostname -e password
  "hostname" : "createdbycloudinit",
password: 1234abcd

SUMMARY:
during tests I didn't hit problem like I hit testing it 1st time, so then I tried:

1. create VM, 
2. create template, 
3. edit template and set Initial Run / Use Cloud-Init/Sysprep
4. create VM Pool from this template

Eventually, I had a different behavior for the same template. I've created 2 different VM Pools and in 1 case there was no payload attached
In edit VM / Initial Run / Use Cloud-Init/Sysprep hostname was set.

Details:

RHV 4.1: hostname set, password set

strings /var/run/vdsm/payload/135db379-3b85-4e5f-b1d2-bf0b413e3678.ce65500be789725d09d50f240fda128e.img | grep -e hostname -e password
  "hostname" : "createdbycloudinit5",
password: 12345abcd5

RHEV 3.6: nothing set
no payload attached, although edit VM (created from Pool) / Initial Run / Use Cloud-Init/Sysprep has a proper hostname

...

One more test with 3.6.

strings /var/run/vdsm/payload/b5b0af5b-315f-4448-979e-31515f8a8ad3.d50d90b51c04c4f010ce170541250699.img | grep -e hostname -e password
  "hostname" : "cratedbycloudinit6",
password: 1234abcd6

The results are not consistent at all.

I have neither found anything fixed nor reported as bug:
1. in vdsm I haven't found anything new regarding cloud-init. There are only old commits (2013 & 2014):
2. for ovirt-engine nothing has been fixed recently (the latest are from 2014):

Actual results:
sometimes new hostname or password is not set

Expected results:
cloud-init works everytime

(Originally by Olimp Bockowski)

Comment 1 rhev-integ 2017-06-21 09:47:11 UTC
Do you have cloud-init logs when it failed? The payload?

(Originally by Yaniv Kaul)

Comment 8 rhev-integ 2017-06-21 09:47:42 UTC
Just to report on the progress of this issue:

- I have succeeded to simulate this issue on 4.1 while creating a VM from template on a template tab (happened only once and only in this flow, in no others)
- it is not related to cloud init directly, it is an engine issue because while creating a VM from template, vm init configuration was not properly copied from the template to the VM (in the database)
- the payload was correctly created and attached to the VM according to the configuration which was in the DB. Given the configuration for some reason did not contain the password, it was not in the payload.

I don't yet know why and if it is a frontend or backend issue - keep investigating.

(Originally by Tomas Jelinek)

Comment 9 rhev-integ 2017-06-21 09:47:48 UTC
The issue is that when adding a VM from template with cloud-init configured, the cloud-init password needs to be updated by calling: vmHandler.updateVmInitFromDB(template, false); - nota bene the second param which tells that the password has to be loaded too.

For pools I have not found a case where it is not loaded properly but for VMs made from template it happens. But, if the VM is based on the latest template, and the template is updated, the VM will get also the password from the latest template.

So, we need to make sure the vmHandler.updateVmInitFromDB(template, false); is properly called always when creating a VM from template with password stored.

(Originally by Tomas Jelinek)

Comment 10 rhev-integ 2017-06-21 09:47:53 UTC
Steps to reproduce:
1. Create a linux template with 'Initial Run' > 'Password' set
2. Create a VM based on the template
3. Run the VM
4. Run on host: strings </var/run/vdsm/payload/path-to-cloudinit-disk> | grep password:

Check value of printed password.

I didn't met any problems with pools. As mentioned in comment 8 VM pools might not work with cloudinit root password if they were based on broken template.

(Originally by Jakub Niedermertl)

Comment 12 Israel Pinto 2017-06-25 11:49:35 UTC
Verify with
Red Hat Virtualization Manager Version: 4.1.3.5-0.1.el7
Steps:
1. Create a linux template with 'Initial Run' > 'Password' set
2. Create a VM based on the template
3. Run the VM
4. Run on host: strings </var/run/vdsm/payload/path-to-cloudinit-disk> | grep 'password:'

output:
1.
[root@puma42 latest]# grep -ir password
user_data:password: '1'
2.
[root@puma42 latest]# strings * 
  "availability_zone" : "nova",
  "hostname" : "new_image_cloud_init",
  "launch_index" : "0",
  "meta" : {
    "role" : "server",
    "dsmode" : "local",
    "essential" : "false"
  },
  "name" : "new_image_cloud_init",
  "uuid" : "7781756f-47a1-4101-9ae6-bc510a8112d2"
#cloud-config
output:
  all: '>> /var/log/cloud-init-output.log'
password: '1'
disable_root: 0
runcmd:
- 'sed -i ''/^datasource_list: /d'' /etc/cloud/cloud.cfg; echo ''datasource_list:
  ["NoCloud", "ConfigDrive"]'' >> /etc/cloud/cloud.cfg'
ssh_pwauth: true
chpasswd:
  expire: false

Comment 15 errata-xmlrpc 2017-07-06 07:30:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1692