Bug 1293338 - updating ovirt-engine-extension-aaa-jdbc rpm kills internal profile
Summary: updating ovirt-engine-extension-aaa-jdbc rpm kills internal profile
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine-extension-aaa-jdbc
Classification: oVirt
Component: General
Version: 1.0.4
Hardware: Unspecified
OS: Unspecified
low
high
Target Milestone: ovirt-3.6.2
: 1.0.5
Assignee: Martin Perina
QA Contact: Ondra Machacek
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-12-21 13:08 UTC by Yedidyah Bar David
Modified: 2016-02-18 11:01 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-02-18 11:01:45 UTC
oVirt Team: Infra
rule-engine: ovirt-3.6.z+
mgoldboi: planning_ack+
rule-engine: devel_ack+
pstehlik: testing_ack+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1260573 unspecified CLOSED [AAA] engine-setup don't update package 'ovirt-engine-extension-aaa-jdbc' 2020-10-14 00:28:05 UTC
Red Hat Bugzilla 1293853 high CLOSED Uncaught JS exception selecting unattached hosted_engine storage domain 2020-10-14 00:28:05 UTC
oVirt gerrit 51291 master MERGED packaging: setup: Upgrade aaa-jdbc extension during engine-setup 2020-02-06 20:57:06 UTC
oVirt gerrit 51293 master MERGED doc: Fix 'internal' profile upgrade 2020-02-06 20:57:06 UTC
oVirt gerrit 51363 ovirt-engine-extension-aaa-jdbc-1.0 MERGED doc: Fix 'internal' profile upgrade 2020-02-06 20:57:06 UTC
oVirt gerrit 51495 ovirt-engine-3.6 MERGED packaging: setup: Upgrade aaa-jdbc extension during engine-setup 2020-02-06 20:57:06 UTC
oVirt gerrit 51618 ovirt-engine-3.6.2 MERGED packaging: setup: Upgrade aaa-jdbc extension during engine-setup 2020-02-06 20:57:06 UTC

Internal Links: 1260573 1293853

Description Yedidyah Bar David 2015-12-21 13:08:58 UTC
Description of problem:

$subject.

engine.log has:

Database schema is older than required by currently installed ovirt-engine-extension-aaa-jdbc package version. Please upgrade profile database schema before proceeding (for more info about upgrade please take a look at README.admin file contained in ovirt-engine-extension-aaa-jdbc package

README.admin [1] (found also in /usr/share/doc/ovirt-engine-extension-aaa-jdbc-* in current rpm packaging) says to run 'engine-setup'.

[1] https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extension-aaa-jdbc.git;a=blob_plain;f=README.admin;hb=HEAD

Version-Release number of selected component (if applicable):

All current versions imo

How reproducible:

always

Steps to Reproduce:
1. engine-setup
2. yum update ovirt-engine-extension-aaa-jdbc (to a newer version)
3.

Actual results:

Can't login to internal profile

Expected results:

login to internal profile works

Additional info:

Some possible solutions:

1. If all is needed is access to the db, might as well let the extension update its db during startup if needed. Not sure root access is required, didn't check what the setup plugin does on upgrade.

2. version-lock ovirt-engine-extension-aaa-jdbc and have it updated by engine-setup, similarly to other packages. We can also update it without version-lock, if we do not want to, and still save most cases, assuming we'll usually release newer versions of it in parallel with engine ones.

3. Make it backwards-compatible with previous versions of its db schema.

Comment 1 Oved Ourfali 2015-12-21 13:13:12 UTC
Please attach complete engine and setup logs.

Comment 2 Yedidyah Bar David 2015-12-21 13:23:17 UTC
(In reply to Oved Ourfali from comment #1)
> Please attach complete engine and setup logs.

Not sure why, the above is enough.

Martin already noted [1] that this is working as designed. So this bug is against the design, not a particular "undocumented feature (aka bug)".

Attaching anyway.

[1] http://lists.ovirt.org/pipermail/users/2015-December/036715.html

Comment 4 Oved Ourfali 2015-12-21 13:26:22 UTC
I see.
It is indeed as designed.
One shouldn't run yum update on this package.
If he does, it has implications.

I tend to close it as WONTFIX.
However, I'll wait for Martin to share his thoughts.
Anyhow, it isn't either high severity nor high priority.

Comment 5 Barak Korren 2015-12-21 13:48:05 UTC
Our flow hitting this was:

1. Update repo URLS in order to get oVirt/RHEV updates (RHEV users will have this done for then in the channels, or will do it with Satellite)
2. yum upgrade engine-setup
3 engine-setup (To update engine)
4. yum upgrade (To get all pending updates for the OS)
5. reboot

We got the newer 'ovirt-engine-extension-aaa-jdbc' in #4, we obviously didn't notice because it was one out of hundreds of updated packages.

I don't think this flow is far-fetched or rare. Updating engine and the underlying OS in the same upgrade flow is probably something quite a few Sysadmins will try to do to limit downtime.

Comment 6 Oved Ourfali 2015-12-21 13:55:01 UTC
(In reply to Barak Korren from comment #5)
> Our flow hitting this was:
> 
> 1. Update repo URLS in order to get oVirt/RHEV updates (RHEV users will have
> this done for then in the channels, or will do it with Satellite)
> 2. yum upgrade engine-setup
> 3 engine-setup (To update engine)

Why not yum update ovirt-engine?

> 4. yum upgrade (To get all pending updates for the OS)
> 5. reboot
> 
> We got the newer 'ovirt-engine-extension-aaa-jdbc' in #4, we obviously
> didn't notice because it was one out of hundreds of updated packages.
> 
> I don't think this flow is far-fetched or rare. Updating engine and the
> underlying OS in the same upgrade flow is probably something quite a few
> Sysadmins will try to do to limit downtime.

For some reason Didi created a mail thread rather than discuss it here.
What I explained there is that schema shouldn't just change.
We're still in development so it was probably changed to address an issue, as this is the first version of this extension.

However, I wouldn't expect people just to upgrade it without noticing. It will probably be used by internal only in 99.999% of the cases, and running  engine-setup covers that. It is like you shouldn't just upgrade the engine without running engine-setup afterwards...

Comment 7 Barak Korren 2015-12-21 14:13:47 UTC
> 
> Why not yum update ovirt-engine?
>
This is what the instructions seem to say atm, to let engine-setup do the updates. We asked the intergeneration team multiple times about this before starting.
@didi Should'nt this step also update the extension?

> 
> For some reason Didi created a mail thread rather than discuss it here.
> What I explained there is that schema shouldn't just change.
> We're still in development so it was probably changed to address an issue,
> as this is the first version of this extension.
> 
Well I suppose for a full 3.5->3.6 upgrade (E.g. when not tracking the beta versions) One will get the proper schema the 1st time the extension package is installed.

Comment 8 Yedidyah Bar David 2015-12-21 14:17:42 UTC
(In reply to Barak Korren from comment #7)
> > 
> > Why not yum update ovirt-engine?
> >
> This is what the instructions seem to say atm, to let engine-setup do the
> updates. We asked the intergeneration team multiple times about this before
> starting.
> @didi Should'nt this step also update the extension?

ovirt-engine is version-locked. yum update will not update it.

If we want engine-setup to also update this extension, that's a simple patch. I was under the assumption we do not want that. Doing that will solve many cases of current bug, but there are other solutions.

> 
> > 
> > For some reason Didi created a mail thread rather than discuss it here.
> > What I explained there is that schema shouldn't just change.
> > We're still in development so it was probably changed to address an issue,
> > as this is the first version of this extension.
> > 
> Well I suppose for a full 3.5->3.6 upgrade (E.g. when not tracking the beta
> versions) One will get the proper schema the 1st time the extension package
> is installed.

Indeed. I expect this bug to affect updates after the first upgrade to 3.6. Still, depending on the chosen solution, we might want/need to fix it asap. E.g. if we decide to version-lock it, we need to do that on the upgrade to 3.6.

Comment 9 Oved Ourfali 2015-12-21 14:24:46 UTC
(In reply to Yedidyah Bar David from comment #8)
> (In reply to Barak Korren from comment #7)
> > > 
> > > Why not yum update ovirt-engine?
> > >
> > This is what the instructions seem to say atm, to let engine-setup do the
> > updates. We asked the intergeneration team multiple times about this before
> > starting.
> > @didi Should'nt this step also update the extension?
> 
> ovirt-engine is version-locked. yum update will not update it.
> 
> If we want engine-setup to also update this extension, that's a simple
> patch. I was under the assumption we do not want that. Doing that will solve
> many cases of current bug, but there are other solutions.
> 

So, how do you upgrade the ovirt-engine then?
Why running engine-setup if you the ovirt engine wasn't updated?

> > 
> > > 
> > > For some reason Didi created a mail thread rather than discuss it here.
> > > What I explained there is that schema shouldn't just change.
> > > We're still in development so it was probably changed to address an issue,
> > > as this is the first version of this extension.
> > > 
> > Well I suppose for a full 3.5->3.6 upgrade (E.g. when not tracking the beta
> > versions) One will get the proper schema the 1st time the extension package
> > is installed.
> 
> Indeed. I expect this bug to affect updates after the first upgrade to 3.6.
> Still, depending on the chosen solution, we might want/need to fix it asap.
> E.g. if we decide to version-lock it, we need to do that on the upgrade to
> 3.6.

Comment 10 Yedidyah Bar David 2015-12-21 14:33:14 UTC
(In reply to Oved Ourfali from comment #9)
> (In reply to Yedidyah Bar David from comment #8)
> > (In reply to Barak Korren from comment #7)
> > > > 
> > > > Why not yum update ovirt-engine?
> > > >
> > > This is what the instructions seem to say atm, to let engine-setup do the
> > > updates. We asked the intergeneration team multiple times about this before
> > > starting.
> > > @didi Should'nt this step also update the extension?
> > 
> > ovirt-engine is version-locked. yum update will not update it.
> > 
> > If we want engine-setup to also update this extension, that's a simple
> > patch. I was under the assumption we do not want that. Doing that will solve
> > many cases of current bug, but there are other solutions.
> > 
> 
> So, how do you upgrade the ovirt-engine then?
> Why running engine-setup if you the ovirt engine wasn't updated?

engine-setup does everything - updates both packages and db schema.

If we want to imitate this behavior for the extension, we should just do the same for it as we do for the engine - add it to the list of packages to be version-locked and to the list of packages to be updated.

The discussion in bug 1260573 was implying we do not want to do that - that we want to allow users to decide about updating the extension separately from the engine.

Comment 11 Oved Ourfali 2015-12-21 14:35:36 UTC
(In reply to Yedidyah Bar David from comment #10)
> (In reply to Oved Ourfali from comment #9)
> > (In reply to Yedidyah Bar David from comment #8)
> > > (In reply to Barak Korren from comment #7)
> > > > > 
> > > > > Why not yum update ovirt-engine?
> > > > >
> > > > This is what the instructions seem to say atm, to let engine-setup do the
> > > > updates. We asked the intergeneration team multiple times about this before
> > > > starting.
> > > > @didi Should'nt this step also update the extension?
> > > 
> > > ovirt-engine is version-locked. yum update will not update it.
> > > 
> > > If we want engine-setup to also update this extension, that's a simple
> > > patch. I was under the assumption we do not want that. Doing that will solve
> > > many cases of current bug, but there are other solutions.
> > > 
> > 
> > So, how do you upgrade the ovirt-engine then?
> > Why running engine-setup if you the ovirt engine wasn't updated?
> 
> engine-setup does everything - updates both packages and db schema.
> 
> If we want to imitate this behavior for the extension, we should just do the
> same for it as we do for the engine - add it to the list of packages to be
> version-locked and to the list of packages to be updated.
> 
> The discussion in bug 1260573 was implying we do not want to do that - that
> we want to allow users to decide about updating the extension separately
> from the engine.

The ovirt-engine package requires the extension.
Shouldn't it also update it?
Anyway, I don't see a reason for the setup not to updatge the package.

But, as mentioned earlier, let's wait for Martin's feedback, after the new-years vacation.

Comment 12 Gianluca Cecchi 2015-12-21 14:38:45 UTC
Hello, I was the original guy that noted this problem passing from 3.6.0 to 3.6.1.
The command I run in step 2) of comment#6 actually was 
yum update "ovirt-engine-setup*"

This pulled in only these packages:
Dec 19 13:22:53 Updated: ovirt-engine-lib-3.6.1.3-1.el7.centos.noarch
Dec 19 13:22:54 Updated: ovirt-engine-setup-base-3.6.1.3-1.el7.centos.noarch
Dec 19 13:22:55 Updated: ovirt-engine-setup-plugin-ovirt-engine-common-3.6.1.3-1.el7.centos.noarch
Dec 19 13:22:56 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.6.1.3-1.el7.centos.noarch
Dec 19 13:22:56 Updated: ovirt-engine-setup-3.6.1.3-1.el7.centos.noarch
Dec 19 13:22:57 Updated: ovirt-engine-setup-plugin-vmconsole-proxy-helper-3.6.1.3-1.el7.centos.noarch
Dec 19 13:22:57 Updated: ovirt-engine-setup-plugin-websocket-proxy-3.6.1.3-1.el7.centos.noarch
Dec 19 13:24:30 Updated: iproute-3.10.0-54.el7.x86_64
Dec 19 13:28:21 Updated: ovirt-engine-wildfly-overlay-8.0.4-1.el7.noarch
Dec 19 13:28:38 Updated: ovirt-engine-wildfly-8.2.1-1.el7.x86_64
Dec 19 13:28:39 Updated: ovirt-engine-extensions-api-impl-3.6.1.3-1.el7.centos.noarch
Dec 19 13:28:39 Updated: ovirt-engine-vmconsole-proxy-helper-3.6.1.3-1.el7.centos.noarch
Dec 19 13:28:39 Updated: ovirt-engine-websocket-proxy-3.6.1.3-1.el7.centos.noarch
Dec 19 13:28:51 Updated: ovirt-engine-userportal-3.6.1.3-1.el7.centos.noarch
Dec 19 13:28:52 Updated: ovirt-engine-dbscripts-3.6.1.3-1.el7.centos.noarch
Dec 19 13:28:55 Updated: ovirt-engine-backend-3.6.1.3-1.el7.centos.noarch
Dec 19 13:29:10 Updated: ovirt-engine-webadmin-portal-3.6.1.3-1.el7.centos.noarch
Dec 19 13:29:11 Updated: ovirt-engine-tools-3.6.1.3-1.el7.centos.noarch
Dec 19 13:29:12 Updated: ovirt-engine-restapi-3.6.1.3-1.el7.centos.noarch
Dec 19 13:29:12 Updated: ovirt-engine-3.6.1.3-1.el7.centos.noarch

The following step was reboot the system and verify all access to the engine web admin functionality was ok. 
Then I ran what I thought correct (based on what I've being done since oVirt 3.0) for applying security and bug fixes to the underlying OS too:

systemctl stop ovirt-engine
yum update

and this passed the OS from CentOS 7.1 to CentOS 7.2
and also to the update of other ovirt 3.6 packages that were not updated neither from the yum update "ovirt-engine-setup*" command before nor from the updates triggered by the engine-setup command and between then there are:

Dec 19 13:42:34 Updated: ovirt-host-deploy-1.4.1-1.el7.centos.noarch
Dec 19 13:45:18 Updated: ovirt-host-deploy-java-1.4.1-1.el7.centos.noarch
Dec 19 13:46:31 Updated: ovirt-engine-cli-3.6.0.2-1.el7.centos.noarch
Dec 19 13:46:44 Updated: ovirt-release36-002-2.noarch
Dec 19 13:46:48 Updated: ovirt-engine-extension-aaa-jdbc-1.0.4-1.el7.noarch

But probably the question should be generalized, because actually the update from CentOS 7.1 to 7.2 (and in general for a minor release) carries in also updates of OS components not in ovirt repos, but that could be crucial for oVirt engine functionality, eg in this case:

Dec 19 13:43:13 Updated: httpd-2.4.6-40.el7.centos.x86_64
...
Dec 19 13:43:25 Updated: openldap-2.4.40-8.el7.x86_64
...
Dec 19 13:44:58 Updated: 1:java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64
...
Dec 19 13:45:51 Updated: postgresql-server-9.2.14-1.el7_1.x86_64

For example in my case I stopped the engine before "yum update" of the OS, but I forgot to stop PostgreSQL that runs locally on the engine.
I don't know if an update of postgresql-server package runs a clean stop of the database or could create any problem in general.

For sure I didn't expect a problem caused by an ovirt-engine-xxxx package.
Thanks for your attention
Gianluca

Comment 13 Yedidyah Bar David 2015-12-21 14:47:47 UTC
(In reply to Oved Ourfali from comment #11)
> The ovirt-engine package requires the extension.
> Shouldn't it also update it?

The way yum works, it will not update it.

If you have a package X depending on package Y without a specific version, and you update X, Y will not be updated, even if an update is available. You have to specifically ask to update Y too.

It _will_ update Y (our extension) automatically if we Require a newer version in the spec. We definitely should update the Require line there if needed, but the point in the discussion in bug 1260573 was about the case where it's not: Say engine 3.6.3 can work with both aaa-jdbc 1.0.6 and 1.0.7, and the only change between 1.0.6 and 1.0.7 is some new feature. People that do not care about this feature should not be required to update it.

> Anyway, I don't see a reason for the setup not to updatge the package.

See above.

If we do want to update it, as I said, that's a simple patch.

Still, this will not solve the following flow:

1. some 3.6 version of the engine is released and people upgrade to it

2. Later, a new version of aaa-jdbc is released, but without an update for the engine.

3. yum update

Comment 14 Barak Korren 2015-12-21 14:49:49 UTC
(In reply to Yedidyah Bar David from comment #8)
> (In reply to Barak Korren from comment #7)
> > Well I suppose for a full 3.5->3.6 upgrade (E.g. when not tracking the beta
> > versions) One will get the proper schema the 1st time the extension package
> > is installed.
> 
> Indeed. I expect this bug to affect updates after the first upgrade to 3.6.
> Still, depending on the chosen solution, we might want/need to fix it asap.
> E.g. if we decide to version-lock it, we need to do that on the upgrade to
> 3.6.

It seems I was not very clear and an opposite conclusion to the one I indented was reached.
We hit this bug because our upgrade path was 3.5 -> 3.6.0 -> 3.6.1. The problem occurs on the 3.6.0 to 3.6.1 update. Since 3.6.1 is out, future users doing the update will probably go directly to it from 3.5. 
It seems that in this case this problem will not occur because in this case a recent enough version of 'ovirt-engine-extension-aaa-jdbc' which includes an updated DB schema will be installed during the ovirt-engine upgrade.

So it seems this bug is not as critical at this stage as I initially assumed.

Comment 15 Marina Kalinin 2015-12-21 19:45:27 UTC
Hey y'all. This is quite confusing. But after reading bug 1260573,  I agree, the extension should be upgraded with engine-setup and locked otherwise. It is great that the component can be independent, but if it comes and being installed by engine-setup, it should be also updated with it only. Not otherwise.

Comment 16 Yaniv Kaul 2015-12-27 08:54:07 UTC
Changing severity to High - this is utterly confusing and the consequence of not being able to login is quite bad.

Comment 20 Martin Perina 2016-01-11 08:38:47 UTC
The fix is contained in ovirt-engine-extensions-aaa-jdbc 1.0.5 (updated documentation) and ovirt-engine 3.6.2 (updated engine-setup)

Comment 21 Ondra Machacek 2016-01-21 08:34:25 UTC
ovirt-engine-extension-aaa-jdbc is now upgraded as part of engine-setup execution


Note You need to log in before you can comment on or make changes to this bug.