Bug 1260470

Summary: Failed to setup Hosted Engine on RHEV-H 7.2
Product: Red Hat Enterprise Virtualization Manager Reporter: Ying Cui <ycui>
Component: ovirt-node-plugin-hosted-engineAssignee: Douglas Schilling Landgraf <dougsland>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.6.0CC: cshao, dougsland, fdeutsch, gklein, mgoldboi, nsednev, rbarry, ycui
Target Milestone: ovirt-3.6.0-rc3Keywords: AutomationBlocker, TestBlocker
Target Release: 3.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: node
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-09-23 17:30:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1172511, 1204470, 1208780, 1245143    
Attachments:
Description Flags
varlog.tar.bz2
none
sosreport none

Description Ying Cui 2015-09-07 03:04:56 UTC
Description of problem:
Failed to setup Hosted Engine on RHEV-H 7.2

<snip>
2015-09-07 02:43:59 DEBUG otopi.context context._executeMethod:156 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 146, in _executeMethod
  File "/usr/share/ovirt-hosted-engine-setup/plugins/ovirt-hosted-engine-setup/system/vdsmenv.py", line 140, in _late_setup
  File "/usr/share/ovirt-hosted-engine-setup/plugins/ovirt-hosted-engine-setup/system/vdsmenv.py", line 54, in _connect
  File "/usr/lib/python2.7/site-packages/vdsm/vdscli.py", line 125, in connect
Exception: No permission to read file: /etc/pki/vdsm/keys/vdsmkey.pem
</snip>

# rpm -qa ovirt-node ovirt-hosted-engine-setup vdsm ovirt-node-plugin-hosted-engine
ovirt-node-3.3.0-0.2.20150722git7eba125.el7ev.noarch
ovirt-hosted-engine-setup-1.3.0-0.4.beta.git42eb801.el7ev.noarch
vdsm-4.17.3-1.el7ev.noarch
ovirt-node-plugin-hosted-engine-0.2.0-18.0.el7ev.noarch

# ls -al /etc/pki/vdsm/keys/vdsmkey.pem
ls: cannot access /etc/pki/vdsm/keys/vdsmkey.pem: No such file or directory

# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20150831.0.el7ev)

How reproducible:
100%

Steps to Reproduce:
1. Clean TUI installed rhev-hypervisor-7-7.2-20150831.0 successful
2. Set root password and enable ssh
3. Navigate to Hosted Engine TUI to setup HE
4. Select "PXE Boot Engine VM"
5.  < Start first host setup > 

Actual results:
Failed to setup Hosted Engine.

Expected results:
Can steps Hosted Engine successful.

Comment 1 Ying Cui 2015-09-07 03:12:05 UTC
Created attachment 1070792 [details]
varlog.tar.bz2

Comment 2 Ying Cui 2015-09-07 03:13:05 UTC
Created attachment 1070793 [details]
sosreport

Comment 6 Ryan Barry 2015-09-16 14:12:41 UTC
Does hosted engine work on a normal RHEL 7.2 system? That's pretty far outside where the rhev-h code touches, and we don't normally have rw problems in /etc, either.

Comment 7 Ying Cui 2015-09-16 14:48:11 UTC
(In reply to Ryan Barry from comment #6)
> Does hosted engine work on a normal RHEL 7.2 system? That's pretty far
> outside where the rhev-h code touches, and we don't normally have rw
> problems in /etc, either.

Nikolai, can you help to confirm comment 6 whether or not test HE on RHEL 7.2 yet, and did not see this HE setup issue in RHEL 7.2? Thanks.

Comment 8 Ying Cui 2015-09-16 14:49:41 UTC
(In reply to Ryan Barry from comment #6)
> Does hosted engine work on a normal RHEL 7.2 system? That's pretty far
> outside where the rhev-h code touches, and we don't normally have rw
> problems in /etc, either.

Let's see comment7's reply firstly, I will setup rhel 7.2 later if needed.

Comment 9 Nikolai Sednev 2015-09-16 15:09:54 UTC
(In reply to Ying Cui from comment #7)
> (In reply to Ryan Barry from comment #6)
> > Does hosted engine work on a normal RHEL 7.2 system? That's pretty far
> > outside where the rhev-h code touches, and we don't normally have rw
> > problems in /etc, either.
> 
> Nikolai, can you help to confirm comment 6 whether or not test HE on RHEL
> 7.2 yet, and did not see this HE setup issue in RHEL 7.2? Thanks.

We don't have resources for this at this moment.
If you do, please confirm.

Comment 10 Fabian Deutsch 2015-09-21 12:28:23 UTC
Maybe this is a symptom of the re-appeared bug 1204535 comment 10.

Basically the permissions of the files were wrong.

Comment 11 Douglas Schilling Landgraf 2015-09-23 17:30:19 UTC
Hi Ying,

(In reply to Ying Cui from comment #0)
> Description of problem:
> Failed to setup Hosted Engine on RHEV-H 7.2
> 
> <snip>
> 2015-09-07 02:43:59 DEBUG otopi.context context._executeMethod:156 method
> exception
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/otopi/context.py", line 146, in
> _executeMethod
>   File
> "/usr/share/ovirt-hosted-engine-setup/plugins/ovirt-hosted-engine-setup/
> system/vdsmenv.py", line 140, in _late_setup
>   File
> "/usr/share/ovirt-hosted-engine-setup/plugins/ovirt-hosted-engine-setup/
> system/vdsmenv.py", line 54, in _connect
>   File "/usr/lib/python2.7/site-packages/vdsm/vdscli.py", line 125, in
> connect
> Exception: No permission to read file: /etc/pki/vdsm/keys/vdsmkey.pem
> </snip>
> 
> # rpm -qa ovirt-node ovirt-hosted-engine-setup vdsm
> ovirt-node-plugin-hosted-engine
> ovirt-node-3.3.0-0.2.20150722git7eba125.el7ev.noarch
> ovirt-hosted-engine-setup-1.3.0-0.4.beta.git42eb801.el7ev.noarch
> vdsm-4.17.3-1.el7ev.noarch
> ovirt-node-plugin-hosted-engine-0.2.0-18.0.el7ev.noarch
> 
> # ls -al /etc/pki/vdsm/keys/vdsmkey.pem
> ls: cannot access /etc/pki/vdsm/keys/vdsmkey.pem: No such file or directory
> 
> # cat /etc/redhat-release 
> Red Hat Enterprise Virtualization Hypervisor release 7.2 (20150831.0.el7ev)
> 
> How reproducible:
> 100%
> 
> Steps to Reproduce:
> 1. Clean TUI installed rhev-hypervisor-7-7.2-20150831.0 successful
> 2. Set root password and enable ssh
> 3. Navigate to Hosted Engine TUI to setup HE
> 4. Select "PXE Boot Engine VM"
> 5.  < Start first host setup > 
> 
> Actual results:
> Failed to setup Hosted Engine.
> 
> Expected results:
> Can steps Hosted Engine successful.

I have reproduced the report and it was caused due bz#1259247, vdsm has started but not 100% correctly. As example, /etc/libvirt/libvirtd.conf doesn't contain the "vdsm section" inside. For now, I am closing this bug as duplicate, fell free to re-open.

Some logs indicating bz#1259247

Sep 23 17:28:30 localhost journal: vdsm vds ERROR Exception raised#012Traceback (most recent call last):#012  File "/usr/share/vdsm/vdsm", line 157, in run#012    serve_clients(log)#012  File "/usr/share/vdsm/vdsm", line 94, in serve_clients#012    cif = clientIF.getInstance(irs, log)#012  File "/usr/share/vdsm/clientIF.py", line 192, in getInstance#012  File "/usr/share/vdsm/clientIF.py", line 110, in __init__#012  File "/usr/share/vdsm/virt/secret.py", line 91, in clear#012  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 163, in get#012  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 100, in open_connection#012  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 949, in retry#012  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth#012libvirtError: authentication failed: no agent is available to authenticate

Thanks!

*** This bug has been marked as a duplicate of bug 1259247 ***