Bug 1528253

Summary: cloud-init silently fails setting dumb root password but the ansible code doesn't validate it in advance
Product: [oVirt] ovirt-hosted-engine-setup Reporter: Nikolai Sednev <nsednev>
Component: GeneralAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED WORKSFORME QA Contact: meital avital <mavital>
Severity: high Docs Contact:
Priority: unspecified    
Version: ---CC: bugs, didi, nsednev, stirabos
Target Milestone: ---Flags: nsednev: planning_ack?
nsednev: devel_ack?
nsednev: testing_ack?
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-26 11:26:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1455169    
Attachments:
Description Flags
alma03 logs
none
failed_cloudinit_root_passwd none

Description Nikolai Sednev 2017-12-21 11:07:06 UTC
Created attachment 1370830 [details]
alma03 logs

Description of problem:
I've tried to deploy SHE ansible over NFS and failed with:
[ ERROR ] fatal: [nsednev-he-1.qa.lab.tlv.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Authentication failure.", "unreachable": true}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook


Version-Release number of selected component (if applicable):
ovirt-engine-appliance-4.2-20171220.1.el7.centos.noarch
sanlock-3.5.0-1.el7.x86_64
ovirt-engine-appliance-4.2-20171220.1.el7.centos.noarch
ovirt-engine-sdk-python-3.6.9.2-0.1.20161204.gite99bbd1.el7.centos.noarch
ovirt-imageio-daemon-1.2.1-0.201712201809.git8ed3dbd.el7.centos.noarch
ovirt-setup-lib-1.1.5-0.0.master.20170823122959.git1480342.el7.centos.noarch
ovirt-vmconsole-host-1.0.4-1.el7.noarch
ovirt-host-deploy-1.7.1-0.0.master.20171128145303.gite36ef57.el7.centos.noarch
ovirt-provider-ovn-driver-1.2.2-1.el7.centos.noarch
ovirt-host-dependencies-4.2.0-1.1.master.20171130100954.git90161e9.el7.centos.x86_64
ovirt-hosted-engine-setup-2.2.4-0.0.master.20171220124224.gitbb1831c.el7.centos.noarch
ovirt-release-master-4.2.1-0.0.master.20171220000031.git96e5658.el7.centos.noarch
ovirt-host-4.2.0-1.1.master.20171130100954.git90161e9.el7.centos.x86_64
mom-0.5.11-1.el7.noarch
vdsm-4.20.9-75.gitc7c2d8e.el7.centos.x86_64
ovirt-imageio-common-1.2.1-0.201712201809.git8ed3dbd.el7.centos.noarch
ovirt-vmconsole-1.0.4-1.el7.noarch
libvirt-client-3.2.0-14.el7_4.5.x86_64
ovirt-hosted-engine-ha-2.2.3-0.0.master.20171218181916.20171218181911.git4c22b93.el7.centos.noarch
Linux version 3.10.0-693.15.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Dec 14 05:13:32 EST 2017
Linux 3.10.0-693.15.1.el7.x86_64 #1 SMP Thu Dec 14 05:13:32 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.4 (Maipo)

Deployment details available from here http://pastebin.test.redhat.com/542608

How reproducible:
50%

Steps to Reproduce:
1.Deploy SHE using ovirt-engine-appliance-4.2-20171220.1.el7.centos.noarch over NFS.


Actual results:
Deployment fails.

Expected results:
Deployment should not fail.

Additional info:
Logs from host

Comment 1 Simone Tiraboschi 2017-12-21 11:29:06 UTC
Cloud-init silently fails/refuses to set a dumb password (not sure if it's more a feature or a bug) so ansible cannot then login in the engine VM.

The issue is that the ansible code doesn't validate it in advance.

Comment 2 Yedidyah Bar David 2017-12-24 12:35:32 UTC
Can you please attach /var/log/cloud-init.log from the engine machine?

Tried this once with ovirt-engine-appliance-4.2-20171220.1.el7.centos.noarch and it worked for me, with password '1'.

Comment 3 Nikolai Sednev 2017-12-24 13:28:48 UTC
(In reply to Yedidyah Bar David from comment #2)
> Can you please attach /var/log/cloud-init.log from the engine machine?
> 
> Tried this once with ovirt-engine-appliance-4.2-20171220.1.el7.centos.noarch
> and it worked for me, with password '1'.

I could not establish connectivity with the engine, it inaccessible.

Comment 4 Yedidyah Bar David 2017-12-25 13:47:06 UTC
I tend to close worksforme.

I do not think there is any issue with dumb passwords.

Already tried several times, and while I do run into other issues (networking, currently), in all cases the engine vm was accessible (through either ssh or virsh console) and had the correct dumb password.

Simone - why do you think we had an issue with dumb passwords? What was the "evidence"?

Comment 5 Simone Tiraboschi 2017-12-26 09:44:39 UTC
(In reply to Yedidyah Bar David from comment #4)
> Simone - why do you think we had an issue with dumb passwords? What was the
> "evidence"?

I debugged it a few month ago with Jenny.
She was trying, if I'm not wrong, with 'pass' and chpasswd run by cloud-init was failing.
We noticed that looking at qemu serial console where we found "Failed to set passwords with chpasswd for ['root']".
I still have a screenshot, attaching it.

Comment 6 Simone Tiraboschi 2017-12-26 09:45:34 UTC
Created attachment 1372401 [details]
failed_cloudinit_root_passwd

Comment 7 Yedidyah Bar David 2017-12-26 11:26:54 UTC
Spent a long time and can't see how it should fail in such a case.

cloud-init seems to simply run 'chpasswd'. Running it manually with a file having 'root:pass' works for me.

Perhaps we can run cloud-init with debug mode (no idea how, but checking the code I see it can output stuff). Looked at sources of chpasswd and it does not seem to enforce strong passwords either.

Closing for now. Please reopen if there is more information, including logs etc.