Bug 1356743 - HE: Reports Engine Fails to Start - ovirt-engine-reportsd: ERROR run:532 Error: Directory '@OVIRT_REPORTS_JBOSS_HOME@' is required but missing
Summary: HE: Reports Engine Fails to Start - ovirt-engine-reportsd: ERROR run:532 Err...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-dwh
Version: 3.6.7
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Yedidyah Bar David
QA Contact: Pavel Stehlik
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-14 23:52 UTC by Chris Paquin
Modified: 2017-04-10 15:24 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-28 06:09:05 UTC
oVirt Team: Integration
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
engine-setup (6.75 KB, text/plain)
2016-07-14 23:52 UTC, Chris Paquin
no flags Details
d1 (91.09 KB, image/png)
2016-07-22 21:27 UTC, Chris Paquin
no flags Details
d2 (46.51 KB, image/png)
2016-07-22 21:28 UTC, Chris Paquin
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1290073 0 high CLOSED engine-setup should warn users running within hosted engine to set to maintenance 2021-02-22 00:41:40 UTC

Internal Links: 1290073

Description Chris Paquin 2016-07-14 23:52:45 UTC
Created attachment 1179968 [details]
engine-setup

Description of problem: New install of RHEV 3.6.1-latest. Follow instructions to install reports portal, however the reports portal does not start. Attempts to start manually, fail with error.

rhev 2016-07-14 19:34:32,858 ovirt-engine-reportsd: ERROR run:532 Error: Directory '@OVIRT_REPORTS_JBOSS_HOME@' is required but missing


Version-Release number of selected component (if applicable):
3.6.1 Latest

How reproducible:

Attempt to install reports portal on HE rhev-m. 

Steps to Reproduce:
1. yum install rhevm-dwh rhevm-reports -y
2. engine-setup, then reboot
3. Attempt to login to reports portal via webUI. Error "Reports not installed"
4. Attempt to start reports portal manually, fails. service ovirt-engine-reportsd start. 
5. Error in messages file."ERROR run:532 Error: Directory '@OVIRT_REPORTS_JBOSS_HOME@' is required but missing"

Actual results:

Reports portal does not start automatically following install and reboot, due to failed services

Expected results:

According to admin guide, reports portal should be available following install and reboot. Services should not need to be started manually. 

Additional info:

See attached txt file of engine-setup.

Comment 1 Chris Paquin 2016-07-14 23:58:04 UTC
Attempting to start virt-engine-dwhd manually.

#service ovirt-engine-dwhd start


2016-07-14 19:55:15 |Exception in component tJavaFlex_6: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "engine" has been caught
Exception in component tJDBCConnection_3
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "engine"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:289)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:106)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:123)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:28)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:19)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:29)
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:22)
	at org.postgresql.Driver.makeConnection(Driver.java:391)
	at org.postgresql.Driver.connect(Driver.java:265)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:247)
	at routines.system.SharedDBConnection.getConnection(SharedDBConnection.java:63)
	at routines.system.SharedDBConnection.getDBConnection(SharedDBConnection.java:131)
	at ovirt_engine_dwh.sampletimekeepingjob_3_6.SampleTimeKeepingJob.tJDBCConnection_3Process(SampleTimeKeepingJob.java:1615)
	at ovirt_engine_dwh.sampletimekeepingjob_3_6.SampleTimeKeepingJob$2.run(SampleTimeKeepingJob.java:11439)
2016-07-14 19:55:48|OxKRN7|JlfZTp|KG4skj|OVIRT_ENGINE_DWH|SampleTimeKeepingJob|Default|6|Java Exception|tJDBCConnection_3|org.postgresql.util.PSQLException:FATAL: password authentication failed for user "engine"|1
2016-07-14 19:55:55 |Exception in component tJavaFlex_6: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "engine" has been caught

Comment 2 Chris Paquin 2016-07-15 00:13:12 UTC
Found this BZ and attempted to troubleshoot my install

https://access.redhat.com/solutions/291973

However, this file seems to be missing

# egrep '^DWH_DB_USER|^DWH_DB_DATABASE' /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-database.conf 
egrep: /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-database.conf: No such file or directory

I do have this 

/etc/ovirt-engine/engine.conf.d/10-setup-database.conf

however, nothing returns from the command below.

# egrep '^DWH_DB_USER|^DWH_DB_DATABASE' //etc/ovirt-engine/engine.conf.d/10-setup-database.conf

Below are the contents of that file:

[root@rhev ovirt-engine-dwhd.conf.d]# cat //etc/ovirt-engine/engine.conf.d/10-setup-database.conf
ENGINE_DB_HOST="localhost"
ENGINE_DB_PORT="5432"
ENGINE_DB_USER="engine"
ENGINE_DB_PASSWORD="46BVYeD7ue03VEApuCUf2d"
ENGINE_DB_DATABASE="engine"
ENGINE_DB_SECURED="False"
ENGINE_DB_SECURED_VALIDATION="False"
ENGINE_DB_DRIVER="org.postgresql.Driver"
ENGINE_DB_URL="jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory"

Comment 3 Chris Paquin 2016-07-15 00:21:46 UTC
Note I am able to login to the database using the credentials in 10-setup.database.conf

#  psql -h 'localhost' -U engine
Password for user engine: 
psql (8.4.20)
Type "help" for help.

engine-> \list
                                             List of databases
         Name         |        Owner         | Encoding |  Collation  |    Ctype    |   Access privileges   
----------------------+----------------------+----------+-------------+-------------+-----------------------
 engine               | engine               | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 ovirt_engine_history | ovirt_engine_history | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 postgres             | postgres             | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0            | postgres             | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres
                                                                                    : postgres=CTc/postgres
 template1            | postgres             | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres
                                                                                    : postgres=CTc/postgres
(5 rows)

Comment 4 Chris Paquin 2016-07-15 06:05:11 UTC
If I do this:

cp //etc/ovirt-engine/engine.conf.d/10-setup-database.conf /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-database.conf

then

chown ovirt:ovirt /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-database.conf

then

# service ovirt-engine-dwhd restart
Stopping oVirt Engine Data Warehouse:                      [FAILED]
Starting oVirt Engine Data Warehouse:                      [  OK  ]

followed by 

# service ovirt-engine-dwhd status
ovirt-engine-dwhd dead but pid file exists

Error in the logs changes:

root@rhev ovirt-engine-dwh]# tail -40 ovirt-engine-dwhd.log 
2016-07-15 02:03:09|nG3tj7|UcW8Ni|UcW8Ni|OVIRT_ENGINE_DWH|MinimalVersionCheck|Default|6|Java Exception|tJDBCConnection_2|org.postgresql.util.PSQLException:FATAL: Ident authentication failed for user "engine_history"|1
Exception in component tRunJob_2
java.lang.RuntimeException: Child job running failed
	at ovirt_engine_dwh.historyetl_3_6.HistoryETL.tRunJob_2Process(HistoryETL.java:7566)
	at ovirt_engine_dwh.historyetl_3_6.HistoryETL$3.run(HistoryETL.java:11024)
2016-07-15 02:03:09|UcW8Ni|UcW8Ni|UcW8Ni|OVIRT_ENGINE_DWH|HistoryETL|Default|6|Java Exception|tRunJob_2|java.lang.RuntimeException:Child job running failed|1
Exception in component tJDBCConnection_2
org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "engine_history"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:289)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:106)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:123)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:28)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:19)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:29)
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:22)
	at org.postgresql.Driver.makeConnection(Driver.java:391)
	at org.postgresql.Driver.connect(Driver.java:265)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:247)
	at routines.system.SharedDBConnection.getConnection(SharedDBConnection.java:63)
	at routines.system.SharedDBConnection.getDBConnection(SharedDBConnection.java:131)
	at ovirt_engine_dwh.parallelrun_3_6.ParallelRun.tJDBCConnection_2Process(ParallelRun.java:1217)
	at ovirt_engine_dwh.parallelrun_3_6.ParallelRun.tJDBCConnection_1Process(ParallelRun.java:1134)
	at ovirt_engine_dwh.parallelrun_3_6.ParallelRun.tPrejob_1Process(ParallelRun.java:1009)
	at ovirt_engine_dwh.parallelrun_3_6.ParallelRun.runJobInTOS(ParallelRun.java:5624)
	at ovirt_engine_dwh.parallelrun_3_6.ParallelRun.runJob(ParallelRun.java:5303)
	at ovirt_engine_dwh.historyetl_3_6.HistoryETL.tRunJob_4Process(HistoryETL.java:1615)
	at ovirt_engine_dwh.historyetl_3_6.HistoryETL$2.run(HistoryETL.java:10978)
2016-07-15 02:03:09|bMRd9W|UcW8Ni|UcW8Ni|OVIRT_ENGINE_DWH|ParallelRun|Default|6|Java Exception|tJDBCConnection_2|org.postgresql.util.PSQLException:FATAL: Ident authentication failed for user "engine_history"|1
Setting a value for the key "dwhUuidEngine" has failed. Error message: null
Setting a value for the key "dwhHostname" has failed. Error message: null
2016-07-15 02:03:09|jlzgtg|UcW8Ni|UcW8Ni|OVIRT_ENGINE_DWH|ValidateInstallation|Default|5|tDie|tDie_1|2016-07-15 02:03:09| This installation is invalid. There might be a more recent DWH instance on host - null.|5
2016-07-15 02:03:09| This installation is invalid. There might be a more recent DWH instance on host - null.
Exception in component tRunJob_1
java.lang.RuntimeException: Child job running failed
	at ovirt_engine_dwh.historyetl_3_6.HistoryETL.tRunJob_1Process(HistoryETL.java:9011)
	at ovirt_engine_dwh.historyetl_3_6.HistoryETL$5.run(HistoryETL.java:11116)
2016-07-15 02:03:09|UcW8Ni|UcW8Ni|UcW8Ni|OVIRT_ENGINE_DWH|HistoryETL|Default|6|Java Exception|tRunJob_1|java.lang.RuntimeException:Child job running failed|1
2016-07-15 02:03:09|ETL Service Stopped

Comment 7 Yedidyah Bar David 2016-07-17 11:39:58 UTC
Please attach sosreport. Thanks.

Comment 11 JHL 2016-07-19 06:46:19 UTC
I've just found how to execute ovirt-engine-reportsd daemon but it's not working.
It should be tried and please somebody help me!


1. vi /usr/share/ovirt-engine-reports/services/ovirt-engine-reportsd/ovirt-engine-reportsd.conf
===========================================================
18 #OVIRT_REPORTS_JBOSS_HOME="@OVIRT_REPORTS_JBOSS_HOME@" <- comment here(#)
19 OVIRT_REPORTS_JBOSS_HOME="/usr/share/jbossas/"     <--- ADD HERE
===========================================================

2. force to execute the services
===========================================================
# service postgresql start
# service ovirt-engine-dwhd start
# service ovirt-engine-reportsd start
# service ovirt-engine start
===========================================================

3. execute firefox and connect to the engine server, then it doesn't work with this message ;
 "Report is not installed"

please let me know how to do this

Comment 12 Yedidyah Bar David 2016-07-19 09:11:04 UTC
(In reply to JHL from comment #11)
> I've just found how to execute ovirt-engine-reportsd daemon but it's not
> working.
> It should be tried and please somebody help me!

As requested from original requester, please attach sosreport. We can't do much without relevant logs. Also please detail the flow you used. Generally speaking, you should install reports, then run 'engine-setup'. If it finishes successfully, Reports should work.

Comment 13 Chris Paquin 2016-07-22 21:24:58 UTC
Here are the RHEV rpms installed on RHEV-M HE Host. Appears to be 3.6.7 not 3.6.1

[root@rhev setup]# rpm -qa | grep rhev
rhevm-tools-backup-3.6.7.5-0.1.el6.noarch
rhevm-restapi-3.6.7.5-0.1.el6.noarch
rhevm-spice-client-x64-cab-3.6-7.el6.noarch
rhevm-setup-plugins-3.6.5-1.el6ev.noarch
rhevm-setup-plugin-vmconsole-proxy-helper-3.6.7.5-0.1.el6.noarch
rhevm-setup-3.6.7.5-0.1.el6.noarch
rhevm-extensions-api-impl-3.6.7.5-0.1.el6.noarch
rhevm-userportal-3.6.7.5-0.1.el6.noarch
rhevm-setup-base-3.6.7.5-0.1.el6.noarch
rhevm-reports-setup-3.6.5.1-1.el6ev.noarch
rhevm-dependencies-3.6.0-1.el6ev.noarch
redhat-support-plugin-rhev-3.6.0-18.el6.noarch
rhevm-tools-3.6.7.5-0.1.el6.noarch
rhevm-spice-client-x86-cab-3.6-7.el6.noarch
rhevm-guest-agent-common-1.0.11-6.el6ev.noarch
rhevm-image-uploader-3.6.0-1.el6ev.noarch
rhevm-lib-3.6.7.5-0.1.el6.noarch
rhevm-setup-plugin-ovirt-engine-common-3.6.7.5-0.1.el6.noarch
rhevm-dwh-setup-3.6.6-1.el6ev.noarch
rhevm-reports-3.6.5.1-1.el6ev.noarch
rhevm-backend-3.6.7.5-0.1.el6.noarch
rhevm-3.6.7.5-0.1.el6.noarch
rhevm-log-collector-3.6.1-1.el6ev.noarch
rhevm-spice-client-x86-msi-3.6-7.el6.noarch
rhevm-setup-plugin-websocket-proxy-3.6.7.5-0.1.el6.noarch
rhevm-websocket-proxy-3.6.7.5-0.1.el6.noarch
rhevm-branding-rhev-3.6.0-10.el6ev.noarch
rhevm-vmconsole-proxy-helper-3.6.7.5-0.1.el6.noarch
rhevm-doc-3.6.7-1.el6eng.noarch
rhevm-dwh-3.6.6-1.el6ev.noarch
rhevm-webadmin-portal-3.6.7.5-0.1.el6.noarch
rhevm-sdk-python-3.6.7.0-1.el6ev.noarch
rhev-guest-tools-iso-3.6-6.el6ev.noarch
rhevm-dbscripts-3.6.7.5-0.1.el6.noarch
rhevm-spice-client-x64-msi-3.6-7.el6.noarch
rhevm-iso-uploader-3.6.0-1.el6ev.noarch
rhevm-cli-3.6.2.0-1.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-3.6.7.5-0.1.el6.noarch

Comment 14 Chris Paquin 2016-07-22 21:27:00 UTC
SETUP

Perform yum update to ensure that you have latest packages.

# yum -y update

Reboot RHEV-M node.


Install the Data Warehouse and Reporting packages as shown below.

# yum install rhevm-dwh rhevm-reports -y


# engine-setup

See screenshots

Comment 15 Chris Paquin 2016-07-22 21:27:41 UTC
Created attachment 1182961 [details]
d1

Comment 16 Chris Paquin 2016-07-22 21:28:01 UTC
Created attachment 1182962 [details]
d2

Comment 19 Yedidyah Bar David 2016-07-24 10:04:08 UTC
Based on the sos report, it seems that the flow was:
1. Have a 3.6.6 engine set up
2. Install reports
3. engine-setup (which also upgraded the engine to 3.6.7)
In the middle of engine-setup, the engine machine was restarted, also killing engine-setup.

Quite likely, this happened because it is a self-hosted-engine, and the HA daemons were not set to global maintenance mode. See also bug 1290073.

If you have a good backup taken prior to running engine-setup, it's probably best to restore that backup, and then follow the documentation, which says to set global maintenance prior to running engine-setup. If you followed a document that does not, please open a doc bug about that.

If you do not have backups, you can try running 'engine-setup' again (also after setting global maintenance), it might work. If it fails, please attach another sosreport after trying that. Thanks and good luck!

Comment 20 Shirly Radco 2016-07-28 05:53:02 UTC
Chris, Any update on this?

Comment 21 Yedidyah Bar David 2016-07-28 06:09:05 UTC
Closing this bug for now. Please reopen if relevant. If you need further help with a support case, please contact me directly. Thanks.

JHL - please follow the documentation. If you find a bug, please open another one. If you need help, it might be better using irc or the mailing lists - please see [1]. Thanks.

[1] http://www.ovirt.org/community/

Comment 22 Chris Paquin 2017-04-10 15:24:22 UTC
Closed

Comment 23 Chris Paquin 2017-04-10 15:24:33 UTC
Closed


Note You need to log in before you can comment on or make changes to this bug.