Description of problem:
Our RHEV-M 3.1 instance died after RHSA-2012:1592-1 installation (+ update with some other packages like kernel). I did yum update and then "reboot". All web pages produced by RHEV-M are blank from the reboot, 404 is being returned. RHEV-M is unusable.
Logs point to some database issue but there is absolutely nothing related to postgresql in /var/log/messages
Version-Release number of selected component (if applicable):
Steps to Reproduce (in my case):
1. yum update
wget 'https://rhevm.example.com/' --no-check-certificate
2012-12-20 15:31:16 ERROR 404: Not Found.
RHEV-M survived this update :-)
The relevant error message in this case is this one:
2012-12-20 20:10:16,887 ERROR [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015892: Deployment unit processor org.jboss.as.web.deployment.ELExpressionFactoryProcessor@6ca85bcc unexpectedly threw an exception during undeploy phase POST_MODULE of deployment "engine.ear": java.lang.NoClassDefFoundError: org/jboss/el/cache/FactoryFinderCache
That missing class is part of the jboss-el-api_2.2_spec-1.0.2-2.Final_redhat_1.ep6.el6, but is not part of the previous version of the package, and the index file that jboss generates and uses to speed class loading has not been updated:
# ls -l /usr/share/jbossas/modules/javax/el/api/main/jboss-el-api_2.2_spec.jar.index
-rw-r--r--. 1 jbossas jbossas 140 Dec 20 20:33 /usr/share/jbossas/modules/javax/el/api/main/jboss-el-api_2.2_spec.jar.index
# cat /usr/share/jbossas/modules/javax/el/api/main/jboss-el-api_2.2_spec.jar.index
These files are not updated because when the jboss package was updated because they are not part of the package, they are generated when the jboss service is started. But we don't start the jboss service, but the ovirt-engine service, and this runs as user ovirt and has no permission to write to /usr/share/jbossas/modules.
Users that have started the jbossas service will have this same problem.
The workaround to the problem is to remove those index files and restart the engine:
# find /usr/share/jbossas/modules -name '*.jar.index' -delete
# service ovirt-engine restart
To fix this en ovirt-engine we can modify the service script to remove or update these files before starting, when it is still running as root.
The proposed change to solve this issue is here:
The service script will remove the out of date indexes before starting the engine.
Note the behaviour of jboss modules has been changed upstream to skip out of date indexes as well:
According to the comments in that bug the external indexes will probably disappear in the future.
This issue is rhel specific.
# ls -lad /usr/share/jboss-as/modules/
drwxr-xr-x. 12 root root 4096 Aug 2 10:09 /usr/share/jboss-as/modules/
# ls -lad /usr/share/jbossas/modules/
drwxr-xr-x. 13 jboss jboss 4096 Jun 19 2012 /usr/share/jbossas/modules/
jboss service does not run under root, so on fedora it won't create index files while at rhel it will.
Had jboss given group write permission, the solution would have been to add ovirt-engine user to the jboss group.
I think this is actually a jboss packaging bug, which should have removed the indexes when package is updated.
The workaround proposed in comment#5 is acceptable for downstream only, but I suggest opening two issues for jboss:
1. Writable jboss:jboss directory should have g+w.
2. Post install action of jboss should cleanup any cache created.
The jboss process runs with the identity selected by the user, it is just a matter of running /usr/share/jboss-as/bin/standalone.sh directly. That identity can well be root. So this issue is less likely in Fedora, but it can happen as well.
(In reply to comment #8)
> The jboss process runs with the identity selected by the user, it is just a
> matter of running /usr/share/jboss-as/bin/standalone.sh directly. That
> identity can well be root. So this issue is less likely in Fedora, but it
> can happen as well.
Right. There are many deamons out there that if executed as root, then there are issues when executed as the standard user.
In the case you mention, after running the standalone.sh as root, trying to start the standard jboss daemon results with a failure. I am not sure we should solve fundamental wrong design issues of jboss.
In rhel it even worse, as running the standalone.sh as root will create indexes owned by root, which is violation of the permissions of the directory structure, right?
The patch proposed in comment #3 has been modified to create a temporary copy of the modules directory without the indexes and it has been merged upstream:
Also note that removing the indexes that are older than the .jar files is not enough, as the modification date of the files is the date the .rpm package was created, and the .index files can be older than that but still out of date. That is why we need to either remove *all* the indexes or, as we do in the patch, create a temporary copy of the modules directory without them.
To verify this start the engine and make sure that the engine is using the /var/tmp/ovirt-engine/modules directory and that that directory is created every time that the engine starts:
1. With the engine stopped make sure that the /var/tmp/ovirt-engine/modules directory does *not* exist.
2. Start the engine and make sure that it is running with the option "-mp /usr/share/ovirt-engine/modules:/var/tmp/ovirt-engine/modules" (you can use "ps -u ovirt -f" to check that).
If the engine is using /usr/share/jbossas/modules instead of /var/tmp/ovirt-engine/modules then it isn't working.
works in SF17
[root@mp-rhevm32 ~]# service ovirt-engine stop
Stopping engine-service: [ OK ]
[root@mp-rhevm32 ~]# ls -lad /var/tmp/ovirt-engine/modules
ls: cannot access /var/tmp/ovirt-engine/modules: No such file or directory
[root@mp-rhevm32 ~]# service ovirt-engine start
Starting engine-service: [ OK ]
ps -u ovirt -fc | less
UID PID PPID C STIME TTY TIME CMD
ovirt 22810 1 99 16:21 ? 00:00:10 engine-service -server -XX:+TieredCompilation -Xms1g -Xmx1g -XX:PermSize=256m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Djava.util.logging.manager=org.jboss.logmanager -Dlogging.configuration=file:///var/tmp/ovirt-engine/engine-service-logging.properties -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.server.default.config=engine-service -Djboss.home.dir=/usr/share/jbossas -Djboss.server.base.dir=/usr/share/ovirt-engine -Djboss.server.config.dir=/var/tmp/ovirt-engine -Djboss.server.data.dir=/var/lib/ovirt-engine -Djboss.server.log.dir=/var/log/ovirt-engine -Djboss.server.temp.dir=/var/tmp/ovirt-engine -Djboss.controller.temp.dir=/var/tmp/ovirt-engine -jar /usr/share/jbossas/jboss-modules.jar -mp /usr/share/ovirt-engine/modules:/var/tmp/ovirt-engine/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -c engine-service.xml
[root@mp-rhevm32 ~]# ls /var/tmp/ovirt-engine/modules/
3.2 has been released