Bug 918152

Summary: [rhevm] - Upgrade - Error Can't start the ovirt-engine-dwhd service (Upgrade from si25.4 to si27.1)
Product: Red Hat Enterprise Virtualization Manager Reporter: Yaniv Lavi <ylavi>
Component: ovirt-engineAssignee: Eli Mesika <emesika>
Status: CLOSED CURRENTRELEASE QA Contact: David Botzer <dbotzer>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.2.0CC: acathrow, bazulay, chetan, dbotzer, dyasny, iheim, italkohe, lpeer, mgoldboi, pstehlik, Rhev-m-bugs, sgrinber, yeylon, ykaul, ylavi, yzaslavs
Target Milestone: ---Keywords: ZStream
Target Release: 3.2.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: infra
Fixed In Version: sf10 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 913384 Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 913384, 922807, 948448    
Attachments:
Description Flags
all-logs-DWHDerror
none
single-upg-log none

Description Yaniv Lavi 2013-03-05 15:32:06 UTC
In order to fully fix this bug for 3.2.0 I need a way to know engine is updating the statistics. Please add a variable to dwh_history_timekeeping with a var_name like lastStatisticsUpdate and update the var_datetime column on every update of a statistics table (of any entity).



Yaniv


+++ This bug was initially created as a clone of Bug #913384 +++

Created attachment 700378 [details]
dwhd servce down

Description of problem:
Cannot Start DWHd service & Receiving Error Can't start the ovirt-engine-dwhd service when upgrading from si25.4 to si27.1 with reports

Version-Release number of selected component (if applicable):
3.1/si27.1 - si25.4

How reproducible:
always

Steps to Reproduce:
1.Install rhevm30 ic158.2 with reports + dwh
2.aggregate data for 2D
3.Upgrade to rhevm31/si24.5 (3.1.1)
  Upgrade reports & dwh & run setup
4.aggregate data for 2D
5.Upgrade to rhevm31/si27.1 (3.1.3)

  
Actual results:
when rhevm-upgrade finishes service DWHd is stopped & I get error:
* Error: Can't start the ovirt-engine-dwhd service 


Expected results:
dwhD service should start & no error should be displayed

Additional info:
3.1/si25.4 upgraded to si27.1

Is it a known issue ? (Can't start the ovirt-engine-dwhd service)

Would you like to proceed? (yes|no): y
Stopping ovirt-engine service...                         [ DONE ]
Stopping DB related services...                          [ DONE ]
Cleaning async tasks...                                  [ DONE ]
Pre-upgrade validations...                               [ DONE ]
Backing Up Database...                                   [ DONE ]
Rename Database...                                       [ DONE ]
Updating rpms (This may take a while
depending on your connection speed to the repository.
To track the installation process run tail -f /var/log/yum.log in
another session).
Note that the transition from package removal to package installation
is expected
to take a significant amount of time. No output will appear in the log
file during
this transition.... [ DONE ]
Updating Database...                                     [ DONE ]
Restore Database name...                                 [ DONE ]
Running post install configuration...                    [ DONE ]
Starting ovirt-engine...                                 [ DONE ]

RHEV Manager upgrade completed successfully!

* Error: Can't start the ovirt-engine-dwhd service

--- Additional comment from David Botzer on 2013-02-21 10:44:26 EST ---

reproduced also on 
3.2/SF8

I upgraded SF7 to SF8 while dwhd service failed to start automatically,
needs to start manually.

--- Additional comment from David Botzer on 2013-02-21 10:46:07 EST ---

Created attachment 700609 [details]
SF7-SF8

--- Additional comment from Yaniv Dary on 2013-02-25 06:45:12 EST ---

Checked the logs the service did start ok nd had return code 0.
The upgrade returned error anyway. Moving to integration to check upgrade.




Yaniv

--- Additional comment from Yaniv Dary on 2013-02-25 07:54:29 EST ---

Please add log of the dwh post upgrade with the error.



Yaniv

--- Additional comment from David Botzer on 2013-02-25 08:11:32 EST ---

which log ?
see all logs in GZ

--- Additional comment from David Botzer on 2013-02-25 08:12:02 EST ---

Created attachment 702344 [details]
si27.1

--- Additional comment from David Botzer on 2013-02-25 08:22:09 EST ---

Created attachment 702355 [details]
SF-7-SF-8

--- Additional comment from Yaniv Dary on 2013-02-25 08:25:27 EST ---

(In reply to comment #7)
> Created attachment 702355 [details]
> SF-7-SF-8

The files are corrupt.

--- Additional comment from Idith Tal-Kohen on 2013-02-28 10:25:50 EST ---

This bug was cloned to the z-stream:
https://bugzilla.redhat.com/show_bug.cgi?id=916643

--- Additional comment from Cheryn Tan on 2013-03-04 01:12:16 EST ---

Hi Moran, you mentioned on virt-program-list that this bug (or its corresponding 3.1.z clone) needs documentation in the release notes. Can you please provide more information on what causes the ovirt-engine-dwhd to fail to start, and the recommended workaround? Thank you!

--- Additional comment from RHEL Product and Program Management on 2013-03-05 05:51:36 EST ---

This bug report previously had all acks and release flag approved.
However since at least one of its acks has been changed, the
release flag has been reset to ? by the bugbot (pm-rhel).  The
ack needs to become approved before the release flag can become
approved again.

Comment 1 Eli Mesika 2013-03-06 14:07:46 UTC
fixed in commit: 0ac55fc

Comment 2 Eli Mesika 2013-03-11 09:38:42 UTC
patch caused deadlock and was reverted.
A different approach will be applied:
Adding _updated field to the following tables:

vds_interface_statistics
vm_interface_statistics
vm_statistics
vds_statistics
disk_image_dynamic
storage_domain_dynamic

This field will be updated with the current time-stamp and will be available for history queries

Comment 3 Eli Mesika 2013-03-11 15:18:08 UTC
fixed in commit : c0c05e5

Comment 5 David Botzer 2013-03-24 08:24:24 UTC
Not Fixed, 3.2/sf9 - sf10
After rhevm-upgrade getting error message & dwhd is down

Starting ovirt-engine service...                     [ DONE ]
RHEV Manager upgrade completed successfully!
* Error: Can't start the ovirt-engine-dwhd service

Not Fixed, 3.2/sf9 - sf10

See logs

Comment 6 David Botzer 2013-03-24 08:25:17 UTC
Created attachment 715437 [details]
all-logs-DWHDerror

Comment 7 David Botzer 2013-03-24 08:25:50 UTC
Created attachment 715438 [details]
single-upg-log

Comment 8 Yaniv Lavi 2013-03-24 08:50:58 UTC
Please first upgrade dwh and then rhevm.
The fix for dwh is in only after the latest version.


Yaniv

Comment 9 David Botzer 2013-03-24 10:11:54 UTC
Fixed 3.2/sf9 - sf10
1. I installed clean sf9, and upgraded dwh package first
---> Package rhevm-dwh.noarch 0:3.2.0-6.el6ev will be updated
---> Package rhevm-dwh.noarch 0:3.2.0-8.el6ev will be an update
2. ran rhevm-dwh-setup
3  rhevm-upgrade
4. No Errors, service running OK,
But messages are not in place...

* Upgrade log available at /var/log/ovirt-engine/ovirt-engine-upgrade_2013_03_24_12_09_15.log
* Perform the following steps to upgrade the history service or the reporting package:
1. Execute: yum update ovirt-engine-reports*
2. Execute: ovirt-engine-dwh-setup
3. Execute: ovirt-engine-reports-setup


Is this going to be the procedure for upgrading dwh & rhevm ??

Comment 10 Eli Mesika 2013-03-24 20:44:03 UTC
(In reply to comment #9)
> Fixed 3.2/sf9 - sf10
> 1. I installed clean sf9, and upgraded dwh package first
> ---> Package rhevm-dwh.noarch 0:3.2.0-6.el6ev will be updated
> ---> Package rhevm-dwh.noarch 0:3.2.0-8.el6ev will be an update
> 2. ran rhevm-dwh-setup
> 3  rhevm-upgrade
> 4. No Errors, service running OK,
> But messages are not in place...
> 
> * Upgrade log available at
> /var/log/ovirt-engine/ovirt-engine-upgrade_2013_03_24_12_09_15.log
> * Perform the following steps to upgrade the history service or the
> reporting package:
> 1. Execute: yum update ovirt-engine-reports*
> 2. Execute: ovirt-engine-dwh-setup
> 3. Execute: ovirt-engine-reports-setup
> 
> 
> Is this going to be the procedure for upgrading dwh & rhevm ??

Question not clear , please elaborate

Comment 11 David Botzer 2013-03-25 07:29:52 UTC
Are we going to instruct customer the following procedure for upgrade:
1. Upgrade dwh
2. run setup
3. yum upgrade rhevm-setup
4. rhevm-upgrade

Comment 12 Moran Goldboim 2013-03-28 09:00:32 UTC
(In reply to comment #11)
> Are we going to instruct customer the following procedure for upgrade:
> 1. Upgrade dwh
> 2. run setup
> 3. yum upgrade rhevm-setup
> 4. rhevm-upgrade

is it different from documentation?

Comment 13 Yaniv Lavi 2013-03-28 12:42:30 UTC
(In reply to comment #12)
> (In reply to comment #11)
> > Are we going to instruct customer the following procedure for upgrade:
> > 1. Upgrade dwh
> > 2. run setup
> > 3. yum upgrade rhevm-setup
> > 4. rhevm-upgrade
> 
> is it different from documentation?

I'm not sue what is in the documentation.
But you can't expect this issue to be resolved, if you don't upgrade the dwh first. Since it is only included in the new package.

Comment 14 Itamar Heim 2013-06-11 08:33:07 UTC
3.2 has been released

Comment 15 Itamar Heim 2013-06-11 08:33:11 UTC
3.2 has been released

Comment 16 Itamar Heim 2013-06-11 08:34:03 UTC
3.2 has been released

Comment 17 Itamar Heim 2013-06-11 08:42:38 UTC
3.2 has been released