Bug 889271 - PRD33 - [RFE] RHEV should log and keep track of the versions installed and upgraded
Summary: PRD33 - [RFE] RHEV should log and keep track of the versions installed and up...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: RFEs
Version: unspecified
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: 3.3.0
Assignee: Yedidyah Bar David
QA Contact: Ilanit Stein
URL:
Whiteboard: integration
Depends On:
Blocks: 977205 978713 978714 1019470
TreeView+ depends on / blocked
 
Reported: 2012-12-20 16:41 UTC by Dan Yasny
Modified: 2015-09-22 13:09 UTC (History)
9 users (show)

Fixed In Version: is5
Doc Type: Enhancement
Doc Text:
The setup scripts of the engine did not keep track of the packages used during setup, upgrade, or cleanup. Conseqeuently it was sometimes hard to analyse and find the root cause for problems if a user did not know the exact version of the packages used. Now, the setup, upgrade, and cleanup scripts have been changed to log an event when started and successfully ended, including the list of packages installed. As a result, when problems are encountered, it is easier to know the exact set of actions taken, whether they succeeded or not, and what versions of packages were installed during these actions.
Clone Of:
: 977205 (view as bug list)
Environment:
Last Closed: 2014-01-21 17:11:57 UTC
oVirt Team: ---
Target Upstream Version:
sgrinber: Triaged+


Attachments (Terms of Use)
logs collected after failure due to zombie tasks (5.14 MB, application/x-xz)
2013-07-24 16:00 UTC, Giuseppe Vallarelli
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2014:0038 0 normal SHIPPED_LIVE Important: Red Hat Enterprise Virtualization Manager 3.3.0 update 2014-01-21 22:03:06 UTC
oVirt gerrit 15863 0 None None None Never
oVirt gerrit 16065 0 None None None Never
oVirt gerrit 16245 0 None None None Never

Description Dan Yasny 2012-12-20 16:41:15 UTC
Description of problem:
Currently we have no way to know what version was originally installed, and then upgraded to the current one. Having this information woul help support track issues that are originating in a specific version-to-verison upgrade scenarios, and also to track unsupported upgrades (from beta to GA for example)

Version-Release number of selected component (if applicable):
3.x and higher

We should be able to see something like this:

First install: 3.1.0 beta2
Update: Dec 1 2012: version: 3.1.1
Update: Jan 1 2013  version: 3.1.2
Update: ...
Version Upgrade: some_date version: 3.2.0

etc

Log collector should gather this information

Comment 7 Yaniv Kaul 2012-12-31 15:41:55 UTC
Where will the information be stored?

Comment 8 Dan Yasny 2013-01-01 16:29:53 UTC
(In reply to comment #7)
> Where will the information be stored?

either a special file, or a db table. file is easier to read and write

Comment 12 Alon Bar-Lev 2013-01-07 09:19:02 UTC
Hello,

I recommend of keeping this information as an option in vdc_options.

Benefits:

1. We already maintain database including surviving upgrades.

2. There is no need to maintain a file, backup/restore/clustering.

3. We already have utility to access vdc_options during setup/upgrade.

Format of option will be:

version@date, version@date

Example:

ovirt-engine-3.1.0-1@2012-12-10, ...

Any objection?

Thanks!

Comment 13 Dan Yasny 2013-01-07 09:23:36 UTC
(In reply to comment #12)
> Hello,
> 
> I recommend of keeping this information as an option in vdc_options.
> 
> Benefits:
> 
> 1. We already maintain database including surviving upgrades.
> 
> 2. There is no need to maintain a file, backup/restore/clustering.
> 
> 3. We already have utility to access vdc_options during setup/upgrade.
> 
> Format of option will be:
> 
> version@date, version@date
> 
> Example:
> 
> ovirt-engine-3.1.0-1@2012-12-10, ...
> 
> Any objection?
> 
> Thanks!


As long as I can see a timeline of upgrades, I'm fine with this

Comment 16 Moran Goldboim 2013-01-14 09:12:24 UTC
due to the lack of proper requirements description, postponing to 3.3

Comment 17 Dan Yasny 2013-01-14 09:15:00 UTC
(In reply to comment #16)
> due to the lack of proper requirements description, postponing to 3.3

See Comment 12 and Comment 13, that's all there is to this request

Comment 19 Moran Goldboim 2013-06-12 12:02:27 UTC
changes will be aggregated to a file indicating all setup/upgrade/cleanup activity been done by our installer.
following #c11 structure should be similar with the addition of operation

format:

date/time        operation      versions installed    
-------------------------------------------------
Jun  9 03:27:37  first install  rhevm-package1.x.y.z
                                rhevm-package2.x.y.z

Jun 10 02:27:37  clean-up       rhevm-cli.x.y.z
                                rhevm-dwh.x.y.z
Jun 10 05:27:15  first install  rhevm-package1.x.y.z

Jun 15 05:27:15  upgrade        rhevm-package3.x.y.z
                                rhevm-packageN.x.y.z

Comment 20 Alon Bar-Lev 2013-06-12 12:15:18 UTC
(In reply to Moran Goldboim from comment #19)
> changes will be aggregated to a file indicating all setup/upgrade/cleanup
> activity been done by our installer.
> following #c11 structure should be similar with the addition of operation
> 
> format:

Moran, best to have parable format.

Anyway, this is rpm specific feature, it won't be available for different distributions.

I also don't think the 'first install' and 'upgrade' are required... a simple setup event with post listing of packages should be sufficient.

I think that a complete rpm -ql is simple and will provide the required information....

ovirt-engine-produce-setup-event.sh --notes="hello world!"
---
#!/bin/sh

NOTES=...

(
echo SETUP-EVENT-BEGIN
echo DATE: $(date -u +%Y%m%d%H%M%S)
echo NOTES: ${NOTES}
rpm -ql | while read package; do
    echo PACKAGE: ${package}
done
echo SETUP-EVENT-END
) >> /var/lib/ovirt-engine/setup-history.txt

Comment 21 Yedidyah Bar David 2013-06-13 14:07:50 UTC
I intend to principally accept Alon's last comment.

This feature is not really relevant to upstream, e.g. on another distribution there won't be a list of rpms etc. The scripts engine-setup/upgrade/cleanup are already in upstream, as part of the packaging for fedora, and I'll patch them there.

Comments are welcome.

Comment 22 Yedidyah Bar David 2013-06-25 14:42:48 UTC
Change 15863 writes to the file /var/lib/ovirt-engine/setup-history.txt,
not in /var/log so that it won't be rotated, two events per each successful run of one of:
engine-setup, engine-cleanup, engine-upgrade, engine-setup-2, engine-cleanup-2.

Each event is a set of lines like the following:
SETUP-EVENT-BEGIN
DATE: 2013-06-25 14:03:42
NOTES: Start of engine-upgrade (monolithic)
ENGINE_CFG: ProductRPMVersion: 3.3.0-0.2.master.20130625092623.fc18 version: general
ENGINE_CFG: VdcVersion: 3.3.0.0 version: general
JBOSS_VER: JBoss Admin Command-line Interface
JBOSS_VER: JBOSS_HOME: /usr/share/jboss-as
JBOSS_VER: JBoss AS release: <connect to the controller and re-run the version command to see the release info>
JBOSS_VER: JAVA_HOME: null
JBOSS_VER: java.version: 1.7.0_19
JBOSS_VER: java.vm.vendor: Oracle Corporation
JBOSS_VER: java.vm.version: 23.7-b01
JBOSS_VER: os.name: Linux
JBOSS_VER: os.version: 3.9.4-200.fc18.x86_64
VIRSH_VER: 1.0.5
UNAME_A: Linux didi-box1 3.9.4-200.fc18.x86_64 #1 SMP Fri May 24 20:10:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
PACKAGE: abattis-cantarell-fonts-0.0.10.1-1.fc18.noarch Wed 09 Jan 2013 07:15:06 PM IST
...
PACKAGE: zlib-1.2.7-9.fc18.x86_64                      Wed 09 Jan 2013 07:14:35 PM IST
SETUP-EVENT-END

One is added during the beginning of the run, one is added in the end. So having
an event with "Start of $something" without one with "End of $something" means that
$something exited unsuccessfully - failed, killed, etc.

NOTES is always "Start of $something" or "End of $something".

$something is one of:
engine-setup (monolithic)
engine-cleanup (monolithic)
engine-upgrade (monolithic)
otopi action setup
otopi action cleanup

When an engine-upgrade-2 based on otopi will be ready, it will add:
otopi action upgrade

The events contain information other than the list of packages,
which, I hope, will also help when diagnosing customer issues.

Change 16065 makes ovirt-log-collector collect this file.

Comment 24 Yedidyah Bar David 2013-07-02 09:32:29 UTC
Added link to change of otopi upgrade

Comment 25 Yedidyah Bar David 2013-07-02 11:31:44 UTC
I have a dilemma regarding what to write exactly in 'NOTES:'.

In 3.2 and before, there was only one set of scripts: engine-setup, engine-cleanup, engine-upgrade.

Now, during development, these ones still exist. There are also new ones: engine-setup-2, engine-cleanup-2. There is no engine-upgrade-2 - upgrades are done by engine-setup-2, which checks if ovirt is already installed and then does upgrade.

The intention is that soon, perhaps before the beta, the old scripts will be removed, and the new ones will be renamed to be like the old ones - engine-setup, engine-cleanup, and probably engine-upgrade as a symlink.

Details in http://www.ovirt.org/Features/Otopi_Infra_Migration .

I was not aware of this, and somehow understood that we'll have both sets also after releasing 3.3. That's why I added specific information on the action done, as can be seen in comment #22. I also have now a pending change, 16245, which adds also one of:
NOTES: Start of otopi action upgrade (from monolithic setup)
NOTES: Start of otopi action upgrade (from otopi setup)

Perhaps this all is completely useless, as we do not intend to have both versions in parallel in production, and do not care much about edge cases (such as users upgrading from one of the current nightly builds to 3.3.0). If so, we can get rid of all the details and have just:
NOTES: Start of setup
NOTES: Start of upgrade
NOTES: Start of cleanup

Having more details naturally has pros and cons. It might ease debugging, it will make parsing more difficult, and currently it make us spent too much time on the exact wording...

Comments/opinions/ideas/etc are welcome, from Lee as well as everyone else.

Comment 26 Alon Bar-Lev 2013-07-02 18:59:49 UTC
(In reply to Yedidyah Bar David from comment #25)
> Perhaps this all is completely useless, as we do not intend to have both
> versions in parallel in production, and do not care much about edge cases
> (such as users upgrading from one of the current nightly builds to 3.3.0).

We do not address nightly build issues at users. Users installs releases or special builds for high priority fix provided by GSS.

Comment 27 Lee Yarwood 2013-07-03 10:11:20 UTC
(In reply to Alon Bar-Lev from comment #26)
> (In reply to Yedidyah Bar David from comment #25)
> > Perhaps this all is completely useless, as we do not intend to have both
> > versions in parallel in production, and do not care much about edge cases
> > (such as users upgrading from one of the current nightly builds to 3.3.0).
> 
> We do not address nightly build issues at users. Users installs releases or
> special builds for high priority fix provided by GSS.

Correct, this shouldn't matter as long as the old scripts are removed before the first public beta.

Lee

Comment 28 Yedidyah Bar David 2013-07-03 13:13:48 UTC
Very well. I now pushed another version of 16245 that merely logs 'setup/upgrade/cleanup' and that's it. Example:

SETUP-EVENT-BEGIN
DATE: 2013-07-03 16:02:07
NOTES: Start of setup
ENGINE_CFG: Connection to the Database failed. Please check that the hostname and port number are correct and that the Database service is up and running.
ENGINE_CFG: Connection to the Database failed. Please check that the hostname and port number are correct and that the Database service is up and running.
JBOSS_VER: JBoss Admin Command-line Interface
JBOSS_VER: JBOSS_HOME: /usr/share/jboss-as
JBOSS_VER: JBoss AS release: <connect to the controller and re-run the version command to see the release info>
JBOSS_VER: JAVA_HOME: null
JBOSS_VER: java.version: 1.7.0_03
JBOSS_VER: java.vm.vendor: Oracle Corporation
JBOSS_VER: java.vm.version: 23.7-b01
JBOSS_VER: os.name: Linux
JBOSS_VER: os.version: 3.9.4-200.fc18.x86_64
VIRSH_VER: 1.0.5
UNAME_A: Linux ovirtebz1.home.local 3.9.4-200.fc18.x86_64 #1 SMP Fri May 24 20:10:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
PACKAGE: aether-1.13.1-4.fc18.noarch                   Thu 13 Jun 2013 11:27:58 AM IDT
...
PACKAGE: zlib-1.2.7-9.fc18.x86_64                      Thu 13 Jun 2013 10:48:15 AM IDT
SETUP-EVENT-END

SETUP-EVENT-BEGIN
DATE: 2013-07-03 16:04:14
NOTES: End of setup
ENGINE_CFG: ProductRPMVersion: 3.3.0-0.2.master.20130703124504.fc18 version: general
ENGINE_CFG: VdcVersion: 3.3.0.0 version: general
JBOSS_VER: JBoss Admin Command-line Interface
JBOSS_VER: JBOSS_HOME: /usr/share/jboss-as
JBOSS_VER: JBoss AS release: <connect to the controller and re-run the version command to see the release info>
JBOSS_VER: JAVA_HOME: null
JBOSS_VER: java.version: 1.7.0_03
JBOSS_VER: java.vm.vendor: Oracle Corporation
JBOSS_VER: java.vm.version: 23.7-b01
JBOSS_VER: os.name: Linux
JBOSS_VER: os.version: 3.9.4-200.fc18.x86_64
VIRSH_VER: 1.0.5
UNAME_A: Linux ovirtebz1.home.local 3.9.4-200.fc18.x86_64 #1 SMP Fri May 24 20:10:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
PACKAGE: aether-1.13.1-4.fc18.noarch                   Thu 13 Jun 2013 11:27:58 AM IDT
...
PACKAGE: zlib-1.2.7-9.fc18.x86_64                      Thu 13 Jun 2013 10:48:15 AM IDT
SETUP-EVENT-END

Comment 30 Ilanit Stein 2013-07-17 10:54:29 UTC
Verified engine-setup, engine cleanup on is5.
Verified engine-upgrade on is4->is5 (using this workaround http://gerrit.ovirt.org/#/c/16664/2/packaging/fedora/setup/engine-upgrade.py, since engine-upgrade failed)

Comment 31 Giuseppe Vallarelli 2013-07-24 15:56:44 UTC
Used ovirt.repo detailed here http://www.ovirt.org/OVirt_3.3_TestDay#oVirt_3.3_New_Features_-_Test_Status_Table

OS: Fedora 19

Steps:

0. sudo yum localinstall http://ovirt.org/releases/ovirt-release-fedora.noarch.rpm

1. Installed ovirt stable version 3.2

2. engine-setup

3. enable nightly repo.

4. yum update

[root@localhost ~]# yum update     
Resolving Dependencies
--> Running transaction check
---> Package ovirt-engine-setup.noarch 0:3.2.2-1.1.fc18 will be updated
---> Package ovirt-engine-setup.noarch 0:3.3.0-0.2.master.20130723211826.git0c5620a.fc19 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

5. engine-setup


--== PACKAGES ==--
         
[ INFO  ] Checking for product upgrade...
          Setup has found packages to be upgraded, do you wish to upgrade them now? (Yes, No) [Yes]: Yes
[ INFO  ] Checking for setup upgrade...

[ ERROR ] Failed to execute stage 'Setup validation': Failed to clear zombie tasks. Please access support in attempt to resolve the problem

Log file:

http://paste.fedoraproject.org/27486/67449113/

10-setup-database.conf content:

http://paste.fedoraproject.org/27489/13746748/

Comment 32 Giuseppe Vallarelli 2013-07-24 16:00:21 UTC
Created attachment 777852 [details]
logs collected after failure due to zombie tasks

Comment 33 Alon Bar-Lev 2013-07-24 16:09:16 UTC
(In reply to Giuseppe Vallarelli from comment #32)
> Created attachment 777852 [details]
> logs collected after failure due to zombie tasks

This happens for different reason, the setup thinks that you are upgrading while some missing files. You can remove /etc/pki/ovirt-engine/ca.pem to start fresh in this state.

Comment 34 Giuseppe Vallarelli 2013-07-24 16:54:43 UTC
Following the same steps but by enabling the beta repo no issues have been found except for "[ ERROR ] Failed to execute stage 'Misc configuration': type object 'NetEnv' has no attribute 'FIREWALLD_DISABLE_SERVICES'"

This error is not present by using the otopi from the nightly repository.
This is how the setup-history.txt looks like:

http://paste.fedoraproject.org/27538/46847111/

Comment 35 Giuseppe Vallarelli 2013-07-29 08:51:00 UTC
It should be noted that no issues has been found related to the specific feature implemented setup-history.txt, it should be noted that a problem has been found when upgrading from stable to nightly due to zombie tasks as previously documented.

Comment 36 Charlie 2013-11-28 00:15:07 UTC
This bug is currently attached to errata RHEA-2013:15231. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes 

Thanks in advance.

Comment 38 errata-xmlrpc 2014-01-21 17:11:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0038.html


Note You need to log in before you can comment on or make changes to this bug.