Hide Forgot
Description of problem: After upgrading to RHEVM 3.5, a number of end-users have reported getting the following errors in the event log: "ETL service aggregation to hourly tables has encountered an error. Please consult the service log for more details." Version-Release number of selected component (if applicable): How reproducible: By upgrading to RHEVM 3.5 Steps to Reproduce: 1. 2. 3. Actual results: ETL service failing due to NPE Expected results: Should be able to aggregate Additional info: Errors in `/var/log/ovirt-engine/ovirt-engine-dwhd.log` ~~ 2015-02-18 12:05:38|ETL Service Started Exception in component tJDBCInput_1 Exception in component tJDBCInput_10 Exception in component tJDBCInput_11 Exception in component tJDBCInput_13 Exception in component tJDBCInput_2 Exception in component tJDBCInput_5 Exception in component tJDBCInput_3 Exception in component tJDBCInput_4 Exception in component tJDBCInput_9 java.lang.NullPointerException at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.tJDBCInput_1Process(AggregationToHourly.java:2377) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly$2.run(AggregationToHourly.java:25665) java.lang.NullPointerException at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.tJDBCInput_11Process(AggregationToHourly.java:15308) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly$7.run(AggregationToHourly.java:25895) java.lang.NullPointerException at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.tJDBCInput_9Process(AggregationToHourly.java:20681) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly$9.run(AggregationToHourly.java:25987) java.lang.NullPointerException at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.tJDBCInput_13Process(AggregationToHourly.java:18186) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly$8.run(AggregationToHourly.java:25941) java.lang.NullPointerException at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.tJDBCInput_5Process(AggregationToHourly.java:13356) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly$6.run(AggregationToHourly.java:25849) java.lang.NullPointerException at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.tJDBCInput_4Process(AggregationToHourly.java:11256) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly$5.run(AggregationToHourly.java:25803) java.lang.NullPointerException at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.tJDBCInput_3Process(AggregationToHourly.java:8231) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly$4.run(AggregationToHourly.java:25757) java.lang.NullPointerException at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.tJDBCInput_2Process(AggregationToHourly.java:5968) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly$3.run(AggregationToHourly.java:25711) java.lang.NullPointerException at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.tJDBCInput_10Process(AggregationToHourly.java:24091) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly$10.run(AggregationToHourly.java:26033) 2015-02-18 14:00:00|IWpxRl|R75JMm|ujbv9q|OVIRT_ENGINE_DWH|AggregationToHourly|Default|6|Java Exception|tJDBCInput_2|java.lang.NullPointerException:null|1 2015-02-18 14:00:00|IWpxRl|R75JMm|ujbv9q|OVIRT_ENGINE_DWH|AggregationToHourly|Default|6|Java Exception|tJDBCInput_9|java.lang.NullPointerException:null|1 2015-02-18 14:00:00|IWpxRl|R75JMm|ujbv9q|OVIRT_ENGINE_DWH|AggregationToHourly|Default|6|Java Exception|tJDBCInput_10|java.lang.NullPointerException:null|1 Exception in component tJDBCOutput_6 org.postgresql.util.PSQLException: ERROR: transacción abortada, las órdenes serán ignoradas hasta el fin de bloque de transacción at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2094) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1827) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:508) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:384) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:330) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.tRowGenerator_1Process(AggregationToHourly.java:19905) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.tPostjob_1Process(AggregationToHourly.java:19496) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.runJobInTOS(AggregationToHourly.java:26081) at ovirt_engine_dwh.aggregationtohourly_3_5.AggregationToHourly.runJob(AggregationToHourly.java:25464) at ovirt_engine_dwh.hourlytimekeepingjob_3_5.HourlyTimeKeepingJob.tRunJob_1Process(HourlyTimeKeepingJob.java:2541) at ovirt_engine_dwh.hourlytimekeepingjob_3_5.HourlyTimeKeepingJob.tJDBCInput_1Process(HourlyTimeKeepingJob.java:2369) at ovirt_engine_dwh.hourlytimekeepingjob_3_5.HourlyTimeKeepingJob.tJDBCConnection_1Process(HourlyTimeKeepingJob.java:1822) at ovirt_engine_dwh.hourlytimekeepingjob_3_5.HourlyTimeKeepingJob.tJDBCConnection_2Process(HourlyTimeKeepingJob.java:1694) at ovirt_engine_dwh.hourlytimekeepingjob_3_5.HourlyTimeKeepingJob$2.run(HourlyTimeKeepingJob.java:6371) 2015-02-18 14:00:00|IWpxRl|R75JMm|ujbv9q|OVIRT_ENGINE_DWH|AggregationToHourly|Default|6|Java Exception|tJDBCOutput_6|org.postgresql.util.PSQLException:ERROR: transacción abortada, las órdenes serán ignoradas hasta el fin de bloque de transacción|1 Exception in component tRunJob_1 java.lang.RuntimeException: Child job running failed ~~~
Please also provide pg dumps of the engine and history databases: su - su - postgres /usr/bin/pg_dump -U "postgres" -F custom -f "<file>" "engine" /usr/bin/pg_dump -U "postgres" -F custom -f "<file>" "ovirt_engine_history"
*** This bug has been marked as a duplicate of bug 1164125 ***
*** Bug 1222447 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-0404.html