Bug 950479

Created attachment 738457 [details]
engineAIO.log

engine.log just in case.

2013-04-10 10:47:00,462 WARN  [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version
2013-04-10 10:47:00,482 WARN  [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version
2013-04-10 10:47:01,992 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-3) Operation Failed: JBAS014580: Unexpected Error: javax.ejb.EJBException: JBAS014580: Unexpected Error
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:188) [jboss-as-ejb3.jar:7.2.0.Final-redhat-3]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:237) [jboss-as-ejb3.jar:7.2.0.Final-redhat-3]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [jboss-as-ejb3.jar:7.2.0.Final-redhat-3]
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:218) [jboss-as-ejb3.jar:7.2.0.Final-redhat-3]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
    at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.2.0.Final-redhat-3]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
    at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.2.0.Final-redhat-3]
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
    at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.2.0.Final-redhat-3]

Hi Pavel

Please update the hibernate-validator package manually and try again.

Alex - indeed, update to hibernate4-validator-4.3.1-1.Final_redhat_1.1.ep6.el6.4.noarch solved it.

Hi Pavel

Thanks for the testing!

We cannot close the bug yet - we will move it to eap so they can fix their packaging and dependencies. According to Juan Hernandes:

"I see in the logs that the version of JBoss is 7.2, so we are using EAP 6.1. I also see in the logs that the version of hibernate-validator is 4.2.0.Final, but the version of hibernate-validator included in EAP 6.1 should be 4.3.1.Final (at least that is what is included in EAP 6.1 Alpha).

Changes have been made to hibernate-validator to remove the dependency on jtype, so I guess that the dependency removed from hibernate-validator somewhere between 4.2.0.Final and 4.3.1.Final but the hibernate-package hasn't probably been updated."

I think this should be changed to the RPMs component, as I believe it is a packaging issue.

Note that this happened with EAP 6.1, and the relevant message from the attached log is the following:

Caused by: java.lang.NoClassDefFoundError: com/googlecode/jtype/TypeUtils
	at org.hibernate.validator.engine.ValidatorImpl.createIteratorForCascadedValue(ValidatorImpl.java:540) [hibernate-validator.jar:4.2.0.Final-redhat-1]
	at org.hibernate.validator.engine.ValidatorImpl.validateCascadedConstraints(ValidatorImpl.java:476) [hibernate-validator.jar:4.2.0.Final-redhat-1]
	at org.hibernate.validator.engine.ValidatorImpl.validateInContext(ValidatorImpl.java:322) [hibernate-validator.jar:4.2.0.Final-redhat-1]
	at org.hibernate.validator.engine.ValidatorImpl.validate(ValidatorImpl.java:139) [hibernate-validator.jar:4.2.0.Final-redhat-1]

I think this happens because the installation of the EAP 6.1 RPMs didn't force the installation of hibernate-validator 4.3.1 as it should. In my opinion this can be solved adding "Requires: hibernate4-validator >= 4.3.1" to one of the EAP .spec files.

I've added the Requires as above (it was using the same version as the other hibernate4 components, which is no longer true).

However...  if the installation instructions are followed this will not occur.  The presence in the repository of a newer version of hibernate4-validator would cause that package to be updated (as it is verified both by our tests and TPS).

So there is a grave user error happening here and there may be other problems in this installation.  I strongly suggest following the instructions that cause all available updates from the EAP channel (or repo) to be applied.

Hi Fernando,

can you please be a bit more specific?
iirc, we've worked hard so installation instruction won't need any "jboss specific" guidelines by fixing the inconsistencies which existed in the past.

we have an issue if RHEL has RPMs which are newer versions than JBoss ones. we can't disable the rhel channel for yum updates. as far as yum is concerned, RHEL is more updated if they release a newer version of same rpm?
if you don't ever plan to move to the RHEL rpm, why not increase the epoch of your rpm so it will always be newer?

thanks,
   Itamar

(In reply to comment #10)
> Hi Fernando,
> 
> can you please be a bit more specific?
> iirc, we've worked hard so installation instruction won't need any "jboss
> specific" guidelines by fixing the inconsistencies which existed in the past.
> 
> we have an issue if RHEL has RPMs which are newer versions than JBoss ones.
> we can't disable the rhel channel for yum updates. as far as yum is
> concerned, RHEL is more updated if they release a newer version of same rpm?
> if you don't ever plan to move to the RHEL rpm, why not increase the epoch
> of your rpm so it will always be newer?
> 
> thanks,
>    Itamar

hmmm, i got confused with bug 958545 in this comment. please ignore

EAP-6.0.1 instance updated as documented in [1] works as expected. Hibernate-validator is deployed from hibernate4-validator-4.3.1-1.Final_redhat_1.1.ep6.el6.4.noarch.rpm - it's a correct version. This has been true at least from ER4, in ER3 I can't say for sure because we've discovered some differences between ZIP and RPM set in Hibernate, but from my LOG hibernate-validator doesn't look affected.

[1] http://documentation-devel.engineering.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/6.1/html/Installation_Guide/Upgrade_the_JBoss_Enterprise_Application_Platform_6_RPM_Installation.html

When RHEV-M is upgraded we don't run "yum update", but the equivalent to "yum update rhevm". The RHEV-M package requires the following packages, which we expect to transitively bring all the correct versions:

  Requires: jbossas-standalone >= 0:7.1.3
  Requires: jbossas-modules-eap >= 0:7.1.3

It is my understanding that the purpose of the jbossas-modules-eap package is to contain all the module descriptors and the dependencies (including the version number) on the packages that provide the actual .jar files. If these dependencies aren't correct, like in this bug, then our update doesn't work correctly.

Fernando, Pavel, what you are saying is that we need to run "yum update" as part of the RHEV-M update process. This doesn't seem ideal to me, as it would update everything in the machine, maybe including completely unrelated things that the user may not want to update. I believe that it is better to make sure that "yum update jboss-modules-eap" works correctly, and I would go so far as to say that this should be included in the EAP update tests.

Itamar, it is my understanding from comment 9 that currently we can't trust "yum update rhevm", so we need to run "yum update" in our upgrade tool.

I can't say if jbossas-modules-eap brings every required dependency, but I guess no and the reason is not only because some packages (some natives, Apache HTTPd, Javadoc etc.) are only optional. Is this expectation specified somewhere as requirement for this package?

But in current state, it makes sense for me to do dependency to not only jbossas-standalone and jbossas-modules-eap, but to all packages included in jboss-eap6 group ("yum groupinfo jboss-eap6").

Because if your update procedure is aka "yum update rhevm", I'm pretty sure, you can get another EAP instance after update - e.g. what about Apache HTTPd server baked on MW BU side (not on BaseOS side)? In your scenario will be updated as well (in the case it was installed before)? Apache HTTPd is updated in EAP-6.1.0 as well.

If you can strictly update only JBoss EAP, what about --disablerepo/--enablerepo command line switches of yum itself? As possible solution, enable only "jbappplatform-6-i386-server-6-rpm" or "jbappplatform-6-x86_64-server-6-rpm" RHN channel plus (mandatory, because EAP-6.1.0 brings new dependencies to BaseOS as well) BaseOS channel during general can be sufficient.

Huh, typo in last sentence (BZ doesn't support edit of a comment):

...during general "yum update" can be....

(In reply to comment #14)
> I can't say if jbossas-modules-eap brings every required dependency, but I
> guess no and the reason is not only because some packages (some natives,
> Apache HTTPd, Javadoc etc.) are only optional. Is this expectation specified
> somewhere as requirement for this package?
> 
> But in current state, it makes sense for me to do dependency to not only
> jbossas-standalone and jbossas-modules-eap, but to all packages included in
> jboss-eap6 group ("yum groupinfo jboss-eap6").

Till now those dependencies have worked well for us, but I agree that including all the packages in the jboss-eap-6 group is good. So our list should be something like this (for 6.1):

  Requires: jbossas-appclient >= 0:7.2
  Requires: jbossas-bundles  >= 0:7.2
  Requires: jbossas-core >= 0:7.2
  Requires: jbossas-domain >= 0:7.2
  Requires: jbossas-hornetq-native >= 0:7.2
  Requires: jbossas-jbossweb-native >= 0:7.2
  Requires: jbossas-modules-eap >= 0:7.2
  Requires: jbossas-product-eap >= 0:7.2
  Requires: jbossas-standalone >= 0:7.2
  Requires: jbossas-welcome-content-eap >= 0:7.2

However, my understanding from comment 9 is that even if we do this we don't have any guarantee if we do "yum update rhevm", as the only update that is tested is the complete "yum update". The issue in this bug in particular won't be solved by these additional dependencies.

> Because if your update procedure is aka "yum update rhevm", I'm pretty sure,
> you can get another EAP instance after update - e.g. what about Apache HTTPd
> server baked on MW BU side (not on BaseOS side)? In your scenario will be
> updated as well (in the case it was installed before)? Apache HTTPd is
> updated in EAP-6.1.0 as well.

I am not sure I understand. What do yum mean by "get another EAP instance"? And do you mean that the Apache httpd packages from the base channel have to be replaced with the packages from the jboss-eap-6 channel in order to have a supported installation? To be honest, the existence of these packages in the EAP 6 channel is new to me, please elaborate.

> If you can strictly update only JBoss EAP, what about
> --disablerepo/--enablerepo command line switches of yum itself? As possible
> solution, enable only "jbappplatform-6-i386-server-6-rpm" or
> "jbappplatform-6-x86_64-server-6-rpm" RHN channel plus (mandatory, because
> EAP-6.1.0 brings new dependencies to BaseOS as well) BaseOS channel during
> general can be sufficient.

We could use those switches and enable the base, RHEV-M and EAP channels only, but the net result is the same: if we run "yum update" we will be updating things from the base channel that the user may not want to update (kernel, glibc, etc).

(In reply to comment #16)
> However, my understanding from comment 9 is that even if we do this we don't
> have any guarantee if we do "yum update rhevm", as the only update that is
> tested is the complete "yum update". The issue in this bug in particular
> won't be solved by these additional dependencies.

I can't say for sure right now. But you are right, this use-case isn't tested at all, so the result is undefined yet.

> > Because if your update procedure is aka "yum update rhevm", I'm pretty sure,
> > you can get another EAP instance after update - e.g. what about Apache HTTPd
> > server baked on MW BU side (not on BaseOS side)? In your scenario will be
> > updated as well (in the case it was installed before)? Apache HTTPd is
> > updated in EAP-6.1.0 as well.
> 
> I am not sure I understand. What do yum mean by "get another EAP instance"?
> And do you mean that the Apache httpd packages from the base channel have to
> be replaced with the packages from the jboss-eap-6 channel in order to have
> a supported installation? To be honest, the existence of these packages in
> the EAP 6 channel is new to me, please elaborate.

Apache HTTPd is one (not alone) of the optional part available in JBoss channel (version 2.2.22). Unfortunately it is a bit special because it is available in BaseOS channel (version 2.2.15) as well. Fortunately both Apaches are supported use-cases for customer's deployment and are covered by EAP QE.

What I've not so clearly highlight above is the fact - I assume you have installed EAP-6.0.1 plus some additional (not mandatory) part of it - e.g. Apache HTTPd from JBoss EAP channel - your and mine system state is the same before update. Whenever you try to update your EAP-6.0.1 instance like:
- yum update rhevm
- yum groupupdate jboss-eap6

as the result you will get another instance of system than I or anybody else who do only plain 'yum update'. At least, your instance will not have updated Apache HTTPd, but mine will. This can't be fixed by dependencies in .spec file because Apache HTTPd isn't mandatory part of JBoss EAP instance and it isn't necessary to be installed during 'you groupinstall jboss-eap6' command invocation.

> > If you can strictly update only JBoss EAP, what about
> > --disablerepo/--enablerepo command line switches of yum itself? As possible
> > solution, enable only "jbappplatform-6-i386-server-6-rpm" or
> > "jbappplatform-6-x86_64-server-6-rpm" RHN channel plus (mandatory, because
> > EAP-6.1.0 brings new dependencies to BaseOS as well) BaseOS channel during
> > general can be sufficient.
> 
> We could use those switches and enable the base, RHEV-M and EAP channels
> only, but the net result is the same: if we run "yum update" we will be
> updating things from the base channel that the user may not want to update
> (kernel, glibc, etc).

But customer has to have up-to-date system whenever he can try to update his instance. This comes (at least) from [1], but this requirement/prerequisite is there for a long time - so if customer can have EAP-6.1.0, it has to have up-to-date BaseOS as well. Without this prerequisite we can have (and have to support) infinite number of different instance states - impossible.

[1] http://documentation-devel.engineering.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/6.1/html/Installation_Guide/chap-Requirements.html#Installation_Prerequisites_for_JBoss_Enterprise_Application_Platform_6

So the only possible conclusion is that we need to run full "yum update" from our upgrade tool. I created bug 960581 to track that.

jbossas-modules-eap now has
Requires: hibernate4-validator >= 4.3.1

Looks fixed in EAP-6.1.0-ER8, will be verified after RPM re-spin finally.

Verified during EAP-6.1.0-ER8 test cycle, fixed as Fernando describes in comment #22.