Bug 1260573

not sure I understand.

if there is a specific requirement then the Requires statement of engine should be updated so that the product update will pull the new jdbc, if change is not mandatory the ovirt-engine-extension-aaa-jdbc should be updated as any other package in system using yum update, it is not version locked.

the extensions and engine should be forward/backward compatible.

(In reply to Alon Bar-Lev from comment #1)
> not sure I understand.
> 
> if there is a specific requirement then the Requires statement of engine
> should be updated so that the product update will pull the new jdbc, if
> change is not mandatory the ovirt-engine-extension-aaa-jdbc should be
> updated as any other package in system using yum update, it is not version
> locked.
> 
> the extensions and engine should be forward/backward compatible.

aaa-jdbc extension is installed (ovirt-engine-backend depends on ovirt-engine-extension-aaa-jdbc package, but no specific version specified) and configured during engine-setup ('internal' domain is provided by aaa-jdbc). So from UX point of view when I execute engine-setup to upgrade engine I would expect also aaa-jdbc to be upgraded.

Currently user have to do this:

1. Execute engine-setup to upgrade engine
2. Execute yum update ovirt-engine-extension-aaa-jdbc to upgrade it
3. Restart engine service to load upgraded package

So if we want aaa-jdbc to be upgraded automatically during engine-setup, we have following options:

1. For each new aaa-jdbc version update Requires in engine spec
     - I really don't like the idea to create additional patch to engine to 
       force update of aaa-jdbc

2. Add ovirt-engine-extension-aaa-jdbc to packages, that should be upgraded, in engine-setup
     - I'd prefer this option

> 2. Add ovirt-engine-extension-aaa-jdbc to packages, that should be upgraded,
> in engine-setup
>      - I'd prefer this option

This option sounds good to me as well.

(In reply to Martin Perina from comment #2)
> (In reply to Alon Bar-Lev from comment #1)
> > not sure I understand.
> > 
> > if there is a specific requirement then the Requires statement of engine
> > should be updated so that the product update will pull the new jdbc, if
> > change is not mandatory the ovirt-engine-extension-aaa-jdbc should be
> > updated as any other package in system using yum update, it is not version
> > locked.
> > 
> > the extensions and engine should be forward/backward compatible.
> 
> aaa-jdbc extension is installed (ovirt-engine-backend depends on
> ovirt-engine-extension-aaa-jdbc package, but no specific version specified)
> and configured during engine-setup ('internal' domain is provided by
> aaa-jdbc). So from UX point of view when I execute engine-setup to upgrade
> engine I would expect also aaa-jdbc to be upgraded.
> 
> Currently user have to do this:
> 

0. yum update (at least the setup package).

If you don't,

> 1. Execute engine-setup to upgrade engine

will tell you that there is a new setup package and abort asking to upgrade it first.

> 2. Execute yum update ovirt-engine-extension-aaa-jdbc to upgrade it
> 3. Restart engine service to load upgraded package
> 
> So if we want aaa-jdbc to be upgraded automatically during engine-setup, we
> have following options:
> 
> 1. For each new aaa-jdbc version update Requires in engine spec
>      - I really don't like the idea to create additional patch to engine to 
>        force update of aaa-jdbc

If it's not really Required, don't patch the engine spec. If it is, better Require the correct version.

> 
> 2. Add ovirt-engine-extension-aaa-jdbc to packages, that should be upgraded,
> in engine-setup

What if there is some weird bug/case (similar things happened in the past) that causes the engine update to be available and upgraded, but somehow updates for ovirt-engine-extension-aaa-jdbc are not found/available/whatever? engine-setup will happily continue and break the engine. If otoh it will require an unavailable version, it will correctly fail.

I checked the documentation again and did upgrade (3.6.x -> 3.6.y where aaa-jdbc package should be also upgraded) as documented:

1. oVirt upgrade steps

     yum update "ovirt-engine-setup*"
     engine-setup

   Result: aaa-jdbc is not upgraded

2. RHEV upgrade steps

      engine-upgrade-check
      yum update rhevm-setup
      engine-setup

   Result: engine-upgrade-check doesn't show aaa-jdbc package upgrade, aaa-jdbc 
           is not upgraded

If aaa-jdbc package is upgraded using 'yum update' when engine service is running, unexpected issues may happen.

We made aaa-jdbc part of engine, when we used it to provide 'internal' domain. So either we need to update documentation that user should update also aaa-jdbc package before running engine-setup (I don't like that options) or update Requires in engine each time aaa-jdbc is updated (I don't like it either, but it's possible to do that) or aaa-jdbc package will be updated same way as engine during engine-setup.

> If aaa-jdbc package is upgraded using 'yum update' when engine service is running, unexpected issues may happen.

why unexpected? this is valid case, all classes are loaded, new ones will be loaded when engine restarted.

there is no reason to force update aaa-jdbc when engine is upgraded or block its upgrade if scheduled by sysadmin.

In addition to what Alon wrote:

(In reply to Martin Perina from comment #5)
> I checked the documentation again and did upgrade (3.6.x -> 3.6.y where
> aaa-jdbc package should be also upgraded) as documented:
> 
> 1. oVirt upgrade steps
> 
>      yum update "ovirt-engine-setup*"
>      engine-setup
> 
>    Result: aaa-jdbc is not upgraded
> 
> 2. RHEV upgrade steps
> 
>       engine-upgrade-check
>       yum update rhevm-setup
>       engine-setup
> 
>    Result: engine-upgrade-check doesn't show aaa-jdbc package upgrade,
> aaa-jdbc 
>            is not upgraded

Same is true for any other dep - java, httpd, etc. We require what we need.

> 
> If aaa-jdbc package is upgraded using 'yum update' when engine service is
> running, unexpected issues may happen.

Did you try that?

I don't know the engine well, but iiuc it has external deps, e.g. slf4j. What happens if these are upgraded?

Anyway, if you are right, it means we also need to version-lock it.

> 
> We made aaa-jdbc part of engine, when we used it to provide 'internal'
> domain. So either we need to update documentation that user should update
> also aaa-jdbc package before running engine-setup

But should he? That's the question.

> (I don't like that
> options) or update Requires in engine each time aaa-jdbc is updated

Each time? What if e.g. a new version of it has only one new feature? Users that want it, can upgrade. Users that want just to update the engine and nothing else, do not have to.

> (I don't
> like it either, but it's possible to do that) or aaa-jdbc package will be
> updated same way as engine during engine-setup.

OK, it seems that I'm the only one that see things differently :-(

Anyway there's one case that we should not forget about: for all aaa-jdbc updates that contains database changes we need to changes Requires in engine spec, because aaa-jdbc database upgrade script is executed during engine-setup. In other cases user can live with older version until yum update and engine service restart is executed.

The aaa-jdbc implementation should be able to use previous schema without applying new features as any decent infra implementation.

This has nothing to do with engine, but well behaved implementation.

# yum update

Should result in working environment, hence the package is updated, the database is not.

Implementation should be usable until such time arrives in which sysadmin refresh schema.

Refreshing schema can be done using jdbc specific tools if only this component is to be updated or by engine setup which is a frontend. It can also be done by the component it self once it starts up, but this requires some extra effort.

The fact that engine is incapable of using previous schema and requires version lock to workaround this issue is a extremely bad implementation, this should not be applied to any new package.

(In reply to Alon Bar-Lev from comment #9)
> The aaa-jdbc implementation should be able to use previous schema without
> applying new features as any decent infra implementation.

No, that's not and never be true. Each application version works with it's specified database schema, so you can't upgrade classes without upgrading the schema.

So, aaa-jdbc update without schema changes can be done by standard yum update. But aaa-jdbc updates with schema changes will be bound to engine updates, otherwise things will get much more complicated for users.

again, you do not understand... let's make it easier...

yum update bugzilla

will make bugzilla not working?

yum update postgresql-server

will make postgres refuse to work?

please try to understand we are managing infra application that should apply same practices.

also please remember that user can install its own instance of aaa-jdbc that engine-setup does not monitor.

(In reply to Alon Bar-Lev from comment #11)
> again, you do not understand... let's make it easier...
> 
> yum update bugzilla
> 
> will make bugzilla not working?

Not sure how exactly it's done in bugzilla, but db changes must be propagated (either in RPM postinstall or during bugzilla startup).

> 
> yum update postgresql-server
> 
> will make postgres refuse to work?

Same as above, any changes in internal structure must be applied before service startup.

> 
> please try to understand we are managing infra application that should apply
> same practices.
> 
> also please remember that user can install its own instance of aaa-jdbc that
> engine-setup does not monitor.

That's true and I forgot about that. So we currently have bug in engine-setup, because aaa-jdbc schema upgrade for 'internal' domain is executed every time, but it should be executed only if aaa-jdbc schema does not exist. When aaa-jdbc schema exists, upgrades should be handled manually:

1. systemctl stop ovirt-engine
2. yum update ovirt-engine-extension-aaa-jdbc
3. For all aaa-jdbc instances (by default only internal) execute schema upgrade using /usr/share/ovirt-engine-extension-aaa-jdbc/dbscripts/schema.sh
4. systemctl start ovirt-engine

Well, 3.6 admin will have to learn a lot of new things.

Read for example:

http://www.postgresql.org/docs/9.2/static/pgupgrade.html

Your component should use existing schema and still operate until upgrade to schema is done. This is why we used key/value tables to simplify the process.

I think that the main use case we'll see for users/customers is using the jdbc extension only for the internal domain.
In addition, I don't expect much changes to the schema looking forward. Maybe only in the first version (3.6), but less and less for others.

We can always add a note to the setup that "if you have additional extension instances you need to run schema update manually".

Adding an additional step to the upgrade process, imo, might have been needed if the main use-case was to have a lot of aaa-jdbc instances, but I think that will be the rare and not the common case.

once again... a component runtime should work its previous [stable] schema while not enabling new features.

this also applies to engine.

after periodic "yum update" system should continue functioning [also after a reboot].

I understood your point.

However, we're not there yet, and not sure how easy it will be to get there.
On the other hand, for the main use-case, the approach I've suggested will solve the issue.

For 3.6 I think we should go with this approach, and going forward to change the aaa-jdbc to be able to support old DB schemas.

*** Bug 1261382 has been marked as a duplicate of this bug. ***

(In reply to Ondra Machacek from bug#1261382 comment#8)
> Btw. you can resolve the issue if you install package
> 'ovirt-engine-extension-aaa-jdbc' just before upgrade, so I believe it's
> kind of duplicate of bug 1260573

Right!
There is an issue indeed, we check for existence of ovirt-engine-extension-aaa-jdbc at customization stage to know if to prompt user.
I think simplest solution is to add dependency for aaa-jdbc into engine setup.
Any other thoughts?

(In reply to Alon Bar-Lev from comment #18)
> (In reply to Ondra Machacek from bug#1261382 comment#8)
> > Btw. you can resolve the issue if you install package
> > 'ovirt-engine-extension-aaa-jdbc' just before upgrade, so I believe it's
> > kind of duplicate of bug 1260573
> 
> Right!
> There is an issue indeed, we check for existence of
> ovirt-engine-extension-aaa-jdbc at customization stage to know if to prompt
> user.
> I think simplest solution is to add dependency for aaa-jdbc into engine
> setup.
> Any other thoughts?

Sorry my bad, I tested upgrade in different way (engine stop, yum update, engine-setup) so it worked fine. I just tried to do the upgrade as documented (yum update "ovirt-engine-setup*", engine-setup) and aaa-jdbc package was installed, but as Alon mention we checked for it before it was installed, so internal domain was configured using legacy internal provider.

Today we are checking for presence of ovirt-aaa-jdbc-tool and if it's present we configure aaa-jdbc provider.

If I understand you correctly you want move aaa-jdbc package dependency from ovirt-engine-backend package to ovirt-engine-setup-plugin-ovirt-engine, right? Wouldn't it be better just to add aaa-jdbc package to the list of updated packages by engine-setup and extend current condition to configure aaa-jdbc or not to form: ovirt-aaa-jdbc-tools exists or aaa-jdbc package is going to be installed?

(In reply to Martin Perina from comment #19)
> If I understand you correctly you want move aaa-jdbc package dependency from
> ovirt-engine-backend package to ovirt-engine-setup-plugin-ovirt-engine,
> right? Wouldn't it be better just to add aaa-jdbc package to the list of
> updated packages by engine-setup and extend current condition to configure
> aaa-jdbc or not to form: ovirt-aaa-jdbc-tools exists or aaa-jdbc package is
> going to be installed?

Or even better change the condition to check if ovirt-aaa-jdbc-tools exists only in devenv, because we know that in RPM in will always exists, because ovirt-engine-backend depends on it.

(In reply to Martin Perina from comment #20)
> (In reply to Martin Perina from comment #19)
> > If I understand you correctly you want move aaa-jdbc package dependency from
> > ovirt-engine-backend package to ovirt-engine-setup-plugin-ovirt-engine,
> > right? Wouldn't it be better just to add aaa-jdbc package to the list of
> > updated packages by engine-setup and extend current condition to configure
> > aaa-jdbc or not to form: ovirt-aaa-jdbc-tools exists or aaa-jdbc package is
> > going to be installed?
> 
> Or even better change the condition to check if ovirt-aaa-jdbc-tools exists
> only in devenv, because we know that in RPM in will always exists, because
> ovirt-engine-backend depends on it.

I wish to avoid devenv specific tests.

Adding dependency to setup-plugin-ovirt-engine for aaa-jdbc will do the trick without any code/logic change.

(In reply to Alon Bar-Lev from comment #21)
> (In reply to Martin Perina from comment #20)
> > (In reply to Martin Perina from comment #19)
> > > If I understand you correctly you want move aaa-jdbc package dependency from
> > > ovirt-engine-backend package to ovirt-engine-setup-plugin-ovirt-engine,
> > > right? Wouldn't it be better just to add aaa-jdbc package to the list of
> > > updated packages by engine-setup and extend current condition to configure
> > > aaa-jdbc or not to form: ovirt-aaa-jdbc-tools exists or aaa-jdbc package is
> > > going to be installed?
> > 
> > Or even better change the condition to check if ovirt-aaa-jdbc-tools exists
> > only in devenv, because we know that in RPM in will always exists, because
> > ovirt-engine-backend depends on it.
> 
> I wish to avoid devenv specific tests.
> 
> Adding dependency to setup-plugin-ovirt-engine for aaa-jdbc will do the
> trick without any code/logic change.

OK, I will prepare the patch

Raising the severity, cause it was confirmed to also be happening on a very realistic upgrade of 3.5->3.6 as mentioned here [1] 

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1261382#c7

(In reply to Gil Klein from comment #23)
> Raising the severity, cause it was confirmed to also be happening on a very
> realistic upgrade of 3.5->3.6 as mentioned here [1] 
> 
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1261382#c7

it is not urgent as there is a simple workaround for the mean time.

perform yum install ovirt-engine-extension-aaa-jdbc before running engine-setup.

In addition, customers or users will have only one version of this package at first. So definitely a blocker for 3.6, but not for the rc build.

*** Bug 1260166 has been marked as a duplicate of this bug. ***

oVirt 3.6.0 has been released on November 4th, 2015 and should fix this issue.
If problems still persist, please open a new BZ and reference this one.