I'm not exactly sure what is not finding the jvm.dll.  Is it the java service wrapper itself?  On my 64b JDK, there is no JavaHome\bin\client or JavaHome\bin\server, only JavaHome\bin\jre\server.

Was this a "standard" install of the JDK, or customized?

Is this really an upgrade issue or does it happen during a standard install as well?

I think you could set JAVA_HOME=c:\java32b\jre as the workaround, right?

(In reply to Jay Shaughnessy from comment #2)
> I'm not exactly sure what is not finding the jvm.dll.  Is it the java
> service wrapper itself?  
Java service wrapper expects jvm.dll in JavaHome\bin\server\ but "standard" 32b JDK installation has the jvm.dll in JavaHome\bin\client\
It seems that location of jvm.dll differs among 32b/64b, JDK/JRE installations and java service wrapper has a path hardcoded to JavaHome\bin\server\jvm.dll'(this is just my guess).

> On my 64b JDK, there is no JavaHome\bin\client or
> JavaHome\bin\server, only JavaHome\bin\jre\server.
> 
> Was this a "standard" install of the JDK, or customized?
Yes, standard JDK 32b installation
> 
> Is this really an upgrade issue or does it happen during a standard install
> as well?
This issue happens during a standard install as well
> 
> I think you could set JAVA_HOME=c:\java32b\jre as the workaround, right?
Yes

Here are repro steps to be clear:
1- install oracle jdk1.7.0_45 32b to c:\java32b directory
2- set JAVA_HOME=c:\java32b
3- uznip jon-server-3.2.0.ER3.zip
4- enable debug mode for java service wrapper (set wrapper.debug=TRUE in jon-server-3.2.0.ER3\bin\wrapper\rhq-server-wrapper.conf)
5- rhqctl install

Result:
installation fails
See jon-server-3.2.0.ER3\logs\rhq-server-wrapper.log you should find there following error:
Error: missing `server' JVM at `c:\java32b\bin\server\jvm.dll'

OK Filip, thanks for the explanation.  I now understand what you are saying.  The Tanuki Java Service Wrapper is looking for the jvm.dll in different places depending on the architecture of the Java install.

For 64bit Java JSW always looks in the right place (basically JdkHome\jre\bin\server or JreHome\bin\server).

For 32bit Java JSW looks in either JdkHome\bin\server or JreHome\bin\server.

But for 32bit Java the file is actually located in JdkHome\bin\client (or JdkHome\jre\bin\server or JdkHome\jre\bin\client).  So, it is not found for the 32bit JDK.


So, my proposal is to just make this a documentation issue.  The workaround is either to install and use a JRE as opposed to a JDK, or to set JAVA_HOME to JdkHome\jre.

I don't think there is really anything we can do in the code to make this work, unless we build our own variant of JSW.  We are planning to move away from JSW in favor of the JBoss Native Wrapper, I think, anyway.

The only thing we may be able to do code-wise is to perhaps try and detect this in the rhqctl script and abort with a message if we see it.

Asking Larry for an opinion...

From the sound of it, this has always been the case? This is not a regression, correct?

From my perspective a JDK should never be used. However, if we require this for the server then this may make this issue a bit more complicated. I think we could (if not already) document that the JBoss ON agent requires a JRE but in the case of using rhqctl, such a claim would be insufficient.

It is also not clear to me on how or if this is a 32-bit vs. 64-bit issue. I would have expected to run across this issue long ago seeing that we only run on 32-bit JVM for Windows in JBoss ON 3.1 and earlier.

We should probably investigate:J
- JDK vs. JRE requirements for JBoss ON
- Installation structure of client JVM vs. server JVM vs. JDK vs. JRE vs. 32-bit vs. 64-bit
- Tanuki's source code to see how this is being resolved. Most likely, it may be attempting to load the DLL using JNI and a specific search order (current directory, JAVA_HOME, PATH, etc)

Looking at the latest Java7 distros for Windows they seem to conform correctly to this statement from http://www.oracle.com/technetwork/java/javase/jre-7-readme-430162.html:

        On Microsoft Windows platforms, the JDK includes both
        the Java HotSpot(TM) Server VM and Java HotSpot Client VM.
        However, the Java SE Runtime Environment for Microsoft Windows
        platforms includes only the Java HotSpot Client VM. Those wishing
        to use the Java HotSpot Server VM with the Java SE Runtime
        Environment may copy the JDK's jre\bin\server folder to a 
        bin\server directory in the Java SE Runtime Environment. Software
        vendors may redistribute the Java HotSpot Server VM with their
        redistributions of the Java SE Runtime Environment.

The 32bit (1.7.0_45 x86) JRE distro includes:

  jre7\bin\client

The 32bit (1.7.0_45 x86) JDK distro includes 

  jdk1.7.0_45\jre\bin\client
  jdk1.7.0_45\jre\bin\server

  ** It does not include 1.7.0_45\bin\client so I'm not sure how Filip
     reportedly has that directory.

The 64bit (1.7.0_45 x64) JRE distro includes:

  jre7\bin\server

The 64bit (1.7.0_45 x64) JDK distro includes:

  jdk1.7.0_45\jre\bin\server


So, it appears that 64bit arch distros always assume it's a server machine. Only 32bit distros even provide a client jvm.

Now, it should be noted that with regard to our JSW wrapper configurations, the server wrapper configuration (rhq-server-wrapper.conf) specifies the '-server' JVM option.  This means that it requires the server JVM.  Which in turn means, looking at the above, that for a 32bit distribution it requires the JDK.  For 64bit distributions it should need only the JRE.

Furthermore, I was successful in launching the server against the 32bit JDK by setting RHQ_JAVA_HOME=c:\Program Files (x86)\Java\jdk1.7.0_45. So, I'm not sure what is causing Filip to have an issue.

Although, I did discover an issue using the 32bit JDK.  The issue was that the default installation directory on my machine: c:\Program Files (x86)\Java\jdk1.7.0_45, caused a failure in the scripts.  The reason is an annoying Windows thing.  Paths with parentheses have issues when resolved inside of script conditionals, because windows brilliantly uses parens as the conditional's grouping demarcation. (Honestly, could they make things any harder?  Their default paths use spaces, and now also parentheses).

One other note, we should likely add the '-server' option to the storage wrapper config as well.  The agent may benefit as well, but we can't add it without risking problems for existing agent installations when they upgrade.

So, in conclusion, from what I see:
- I can't really replicate Filip's issue. The 32bit JDK is working for me.
- We do require a 32bit JDK for the server due to the -server option.
- We require only the JRE for 64bit (I tried this as well)
- We may want to fix the .bat scripts to handle java home directories with
  parentheses. Although the workaround is pretty easy: install it somewhere
  else.
- I think we should document the 32bit JDK thing. And it would be nice if we
  could explain Filip's issue.  But I don't actually think this is a wrapper
  issue.  I think it's just a -server JVM option issue.

And can you confirm that we didn't change this between 3.1 and 3.2?

We did change it.  It's because we moved to EAP6 as our underlying server and it is invoked via standalone.sh, which specifies -server in its startup.  So, we inherit the requirement.  The server wrapper config on Windows applies the same arguments.

For 64bit systems -server is assumed the java distros supply only the bin/server directory.  For 32bit distros there is still the distinction and only the JDK carries the server JVM support.

When upgrading from 3.1 it will be necessary for anyone using a 32bit JRE to switch to a 32bit JDK (or any 64bit distro, if applicable).

When installing new we'll require the server jvm support.

I still think this can be addresses solely via documentation.  Although, I think we should probably fix the .bat issue such that the 'Program Files (x86)' directory will work.

I've made changes locally and the script fixes seem to work.  I was able to install using RHQ_JAVA_HOME=c:\Program Files (x86)\Java\jdk1.7.0_45 .

But there seems to be one more issue.  When I tried to start the server I noticed that when using the 32bit JDK and the default system account for running the service, that java.io.tmpdir was set to C:\Windows\system32\config\systemprofile\AppData\Local\Temp.  The Local system account doesn't actually have permission to access this directory.  I think it may be necessary to point java.io.tmpdir to another location when using a 32bit JVM.

I finally find out what caused my strange directory structure of JDK installation. Installer installs JDK first and then JRE. I used the same parent directory for both, which resulted in corrupted installation.

I can confirm following:

The 32bit (1.7.0_45 x86) JRE distro includes:

  jre7\bin\client

The 32bit (1.7.0_45 x86) JDK distro includes 

  jdk1.7.0_45\jre\bin\client
  jdk1.7.0_45\jre\bin\server


So this issue was caused by corrupted JDK installation. Thanks Jay for trying this out and sorry for false alarm.

I will retest bug 1023019 to be sure it is not caused by corrupted JDK installation as well.

I have my system now running against the 32bit JDK distro using the default local system user to run the services...


master commit 6d87af2dd0ffaa7ecad1e1ed6d67ac7f0ff45129
Author: Jay Shaughnessy <jshaughn>
Date:   Fri Oct 25 14:17:27 2013 -0400

 The 64bit Java distribution assumes that it will run a '-server' JVM, which
 is required by RHQ (in part because the underlying AS7 also requires it but
 also because it should run as a server).  The 32bit distribution still makes
 a distinction between a client JVM (optimized to be lightweight) and a server
 JVM (optimized to perform).  Only the 32bit JDK supplies the server JVM, so
 for 32 Java bit we require the JDK.

 By default the 32bit distribution installs under 'c:\Program Files (x86)\Java\'.
 Unfortunately, our scripts failed when RHQ_JAVA_HOME was set to this
 directory, because the parentheses cause issues with the script conditionals,
 which (by brilliant M$ design) use parens for grouping.  So, fix #1 in this
 check-in is to enable and delayed expansion of variables in the scripts, in our
 logic determining where the java exe is.

 On Windows we install RHQ into three windows services, one for storage, server
 and agent.  By default, the (non--user) local system account runs the services.
 Unfortunately, when using the 32bit distribution the java.io.tmpdir then
 resolves to a locked-down temp directory under windows/system32, which
 subsequently stops things from working.  The workaround is to use a user
 account via RUN_AS[_ME] or to redefined java.io.tmpdir to be a valid
 directory.  The latter choice is somewhat involved as you need to update
 the property in all three service wrapper configurations.  This was not
 as easily supported as it could be, so fix #2 is to add wrapper.inc files
 for all of the components and beef up some commenting in several of our
 script/conf files.

 Also in this commit:
 - fix the CLI scripts to use the new RHQ_JAVA_HOME and RHQ_JAVA_EXE_FILE_PATH
   properties. This had been previously missed.
 - make the dev version of rhq-server-wrapper.inc apply lower mem settings.

 ToDo: Wiki doco around 32bit distribution.

master commit fdc1cabee3ea72a167f71971a7d4c8f324b8e8db
Author: Jay Shaughnessy <jshaughn>
Date:   Fri Oct 25 17:47:23 2013 -0400

Second commit for this.

Went one step further.  Now for the server and agent we create/set java.io.tmpdir
to InstallDir/temp, if the original java.io.tmpdir is invalid.  We'll generate
a warning about the original setting. We'll only exit if our attempt to
create/set a local temp dir fails.  This helps us succeed out of box with
the known issues using the 32-bit JVm on Windows, as well as getting
around the issue in general.

*** Bug 1023463 has been marked as a duplicate of this bug. ***

Cherry Picked to release/jon3.2.x:

commit 93cbc9c4426363db4d63b4270ba49ba20f7ebe16
    cherry-pick master: fdc1cabee3ea72a167f71971a7d4c8f324b8e8db

commit d34ef2053001dd2ecb7bc17614950f508dfefb60
    cherry-pick master: 6d87af2dd0ffaa7ecad1e1ed6d67ac7f0ff45129

Moving to ON_QA for test with new brew build.

An additional commit for ER6 when trying to run the storage installer with only RHQ_SERVER_JAVA_EXE set...


release/jon3.2.x commit bf15ec1514c7f815123ca20cd48d5feade3ac44f
Author: Jay Shaughnessy <jshaughn>
Date:   Thu Nov 7 09:12:45 2013 -0500

    Respect the legacy settings of RHQ_SERVER_JAVA_EXE_FILE_PATH and
    RHQ_SERVER_JAVA_HOME.

    This is related to Bug 1022620.  The workaround is to use
    RHQ_JAVA_EXE_FILE_PATH.

    Cherry-Pick Master: 6149e1bfbabd05e777885c95c97706ef0a9eab08

There is a problem with the solution of resetting java.io.tmpdir in StartupBean. This value is cached in the JVM and so setting it programmatically is dangerous.  I have come across a case re: file upload where the updated value was not being used.

Set back to ASSIGNED and re-targeting for ER6.

This changes the tmpdir strategy and resolves the FileUpload issues, among possibly other issues as well.  Note that on Windows we now default to a local tmpdir under the install home for server and agent.


release/jon3.2.x commit d9423ea58dd5635d63a987f9f5561fa1100e270b
Author: Jay Shaughnessy <jshaughn>
Date:   Thu Nov 7 13:47:21 2013 -0500

The original work tried to set override java.io.tmpdir at runtime in both
the server (StartupBean) and the agent (AgentMain).  This turns out to
be a bad idea as the override is not guaranteed to be used.  From the docs:
" A different value may be given to this system property when the Java
  virtual machine is invoked, but programmatic changes to this property
  are not guaranteed to have any effect upon the the temporary directory
  used..."

So, the value must be set as a -D and used in the JVM initialization.  The
default value of java.io.tmpdir is primarily an issue on Windows as we
require the 32-bit Java Service Wrapper and therefore a 32-bit JVM (unless
the user provides the 64-bit JSW on their own).

The resolution is now that for the server and agent we automatically
override java.io.tmpdir in the JSW .conf file to be %HOME%/temp.  And now
in the code, at startup, we'll create that directory if necessary.  Any
other accessibility problems with the tmpdir setting will cause an abort
at startup.

Cherry-Pick master: 3d9d48c1a27b6d6f194611597f37424625cbd1ee

Moving to ON_QA as available for testing with new brew build.

Mass moving all of these from ER6 to target milestone ER07 since the ER6 build was bad and QE was halted for the same reason.

All problems except for parenthesis in JAVA_HOME have been resolved. Filed bz 1037601 to track parenthesis problem.

Verified on:
Version :	
3.2.0.ER7
Build Number :	
e8e6401:ff0061d