Bug 1540927 - Ansible+otopi deployment: Cockpit fails while attempting to setup hosted engine
Summary: Ansible+otopi deployment: Cockpit fails while attempting to setup hosted engine
Keywords:
Status: CLOSED DUPLICATE of bug 1539560
Alias: None
Product: cockpit-ovirt
Classification: oVirt
Component: Hosted Engine
Version: 0.11.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-4.2.2
: ---
Assignee: Phillip Bailey
QA Contact: Yihui Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-02-01 10:51 UTC by Lev Veyde
Modified: 2018-02-20 14:15 UTC (History)
15 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2018-02-20 14:15:01 UTC
oVirt Team: Integration
Embargoed:
rule-engine: ovirt-4.2?
ylavi: blocker+
cshao: testing_plan_complete?
cshao: testing_ack?


Attachments (Terms of Use)
Ansible output in cockpit (308.14 KB, image/png)
2018-02-09 21:25 UTC, Ryan Barry
no flags Details

Description Lev Veyde 2018-02-01 10:51:27 UTC
Description of problem:
An attempt to setup hosted engine through cockpit (ansible) fails.

Version-Release number of selected component (if applicable):
0.11.7-0.1

How reproducible:
100%

Steps to Reproduce:
1. install latest 4.2-pre node-ng
2. enter cockpit
3. choose the "Hosted Engine Only Deployment" option (default)
4. choose the "Ansible based deployment (Recommended)" option (default)
5. click on the Start button
6. Oops - will appear in red
7. Checking the JavaScript console gives the following error message:

"Failed to read file /var/lib/ovirt-hosted-engine-setup/answers/he-answer.conf. Check that the file exists and is not empty" app.js:25:19159

Actual results:
Installation fails.

Expected results:
Installation should proceed normally.

Additional info:
Latest HE setup RPM (ovirt-hosted-engine-setup.noarch 0:2.2.9-0.0.master.20180131110235.git244bfa3.el7.centos) was installed.

Comment 1 Lev Veyde 2018-02-01 10:54:13 UTC
It seems to fail to install in OTOPI based mode as well.

The same message appears:

12:41:43.856 "Failed to read file /var/lib/ovirt-hosted-engine-setup/answers/he-answer.conf. Check that the file exists and is not empty"1 app.js:25:19159


Also I noted the following message also appears in both cases:
12:38:20.691 "No gdeploy answer files found."1 app.js:25:139345

Comment 2 Ryan Barry 2018-02-09 21:24:54 UTC
Simone -

We're past this now, but where can I find more debugging information? 

I'm not clear on what's going wrong, and there are no logs in /var/log/ovirt-hosted-engine-setup, [/tmp|/var/log]/ovirt-host-deploy, or anywhere else I can look.

ovirtmgmt is up. The engine VM is up, but doesn't respond to pings

[root@nodeng tmp]# ip -4 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
19: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    inet 192.168.124.1/24 brd 192.168.124.255 scope global virbr0
       valid_lft forever preferred_lft forever
22: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    inet 192.168.122.2/24 brd 192.168.122.255 scope global dynamic ovirtmgmt
       valid_lft 3375sec preferred_lft 3375sec
[root@nodeng tmp]# ps -ef | grep qemu
root      6029 29859  0 13:00 pts/1    00:00:00 grep --color=auto qemu
qemu     30315     1 15 11:55 ?        00:10:03 /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu Broadwell,+rtm,+hle,+kvmclock -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 2315b2d9-0ada-4968-9f4c-a9d10bfe6d30 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=off,strict=on -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/tmp/localvmWICzUW/images/342e69be-bc82-4353-a8e1-0a89c9fa50b6/cb855c8f-8fe6-4016-92ce-5b7d4bb53858,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/tmp/localvmWICzUW/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:77:10:13,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-1-HostedEngineLocal/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -vnc 127.0.0.1:0 -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -msg timestamp=on
[root@nodeng tmp]# hosted-engine --vm-status
You must run deploy first

See the attached screenshot.

Comment 3 Ryan Barry 2018-02-09 21:25:29 UTC
Created attachment 1393978 [details]
Ansible output in cockpit

Comment 4 Ryan Barry 2018-02-09 21:26:01 UTC
I wonder if maybe it's not executing the next phase, but I'll check to see what should happen next on the CLI

Comment 5 Simone Tiraboschi 2018-02-09 23:02:15 UTC
(In reply to Ryan Barry from comment #2)
> Simone -
> 
> We're past this now, but where can I find more debugging information? 
> 
> I'm not clear on what's going wrong, and there are no logs in
> /var/log/ovirt-hosted-engine-setup, [/tmp|/var/log]/ovirt-host-deploy, or
> anywhere else I can look.

cockpit on the new flow is directly calling the ansible playbooks skipping all the otopi stuff so otopi will not log because it's not involved at all.

Didi wrote a great logging callback (2_ovirt_logger.py) to get detailed and filtered logs but cockpit has to explicitly whitelist it on ansible-playbook invocation. See here:
https://github.com/oVirt/ovirt-hosted-engine-setup/blob/master/src/ovirt_hosted_engine_setup/ansible_utils.py#L125

(In reply to Ryan Barry from comment #4)
> I wonder if maybe it's not executing the next phase, but I'll check to see
> what should happen next on the CLI

This now is exactly https://bugzilla.redhat.com/show_bug.cgi?id=1540621

All the tasks of the first phase are successfully but the cockpit wizard doesn't move to the next step (create_storage_domain.yml or storage (2) in the screenshot) where you are going to use the engine running on a local VM to let the user customize and validate the storage options before creating a new storage domain on the shared storage.
In the third step (create_target_vm.yml) a new VM will be created on that storage domain, the local engine VM will be shut down and the disk of the local VM will be moved to the shared storage.

Comment 6 Yaniv Lavi 2018-02-14 13:37:26 UTC
Please reopen if still reproducible

Comment 7 Ryan Barry 2018-02-14 14:17:26 UTC
This is definitely still reproducible

Comment 8 Ryan Barry 2018-02-20 14:15:01 UTC

*** This bug has been marked as a duplicate of bug 1539560 ***


Note You need to log in before you can comment on or make changes to this bug.