Bug 899443 (JBEWS-34) - EWS on Windows - Tomcat APR doesn't start with SSL configured when there are garbagge DLL's in WINDOWS\System32
Summary: EWS on Windows - Tomcat APR doesn't start with SSL configured when there are ...
Keywords:
Status: CLOSED EOL
Alias: JBEWS-34
Product: JBoss Enterprise Web Server 1
Classification: JBoss
Component: unspecified
Version: EWS 1.0.2
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: ---
Assignee: Rebecca Newton
QA Contact:
URL: http://jira.jboss.org/jira/browse/JBE...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-06-16 15:49 UTC by Ondřej Žižka
Modified: 2017-08-04 14:55 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Tomcat 6, Windows 2008 64x
Last Closed: 2017-08-04 14:55:03 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker JBEWS-34 0 Critical Closed EWS on Windows - Tomcat APR doesn't start with SSL configured when there are garbagge DLL's in WINDOWS\System32 2017-08-04 14:54:38 UTC
Red Hat Issue Tracker JBQA-4819 0 Blocker Closed Smoke test EWS 1.0.2-CR5 on Windows 2008 2017-08-04 14:54:38 UTC

Description Ondřej Žižka 2011-06-16 15:49:22 UTC
project_key: JBEWS

STR:

1) Install OpenSSL from http://www.openssl.org/related/binaries.html
   *Make sure to do this on a disposable Windows installation* (virtualized)
2) Configure EWS's Tomcat 6 to use SSL, e.g.:

{code}
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               SSLCertificateFile="c:\Program Files\Red Hat\Enterprise Web Server\etc\ssl\certs\localhost.crt"
               SSLCertificateKeyFile="c:\Program Files\Red Hat\Enterprise Web Server\etc\ssl\private\localhost.key"
               clientAuth="false" sslProtocol="TLS" />
{code}

3) Start Tomcat. For simplicity, use startup.bat.
4) With this, Tomcat will show no error anywhere (Event log, console, or tomcat logs), only Tomcat console window flashes, then disappears.


After consultation with Mladen, we revealed that:

A) The error can be only seen if you run `catalina.bat run`, which will show you this:
{code}
Jun 16, 2011 6:06:48 PM org.apache.catalina.core.AprLifecycleListener init
INFO: Loaded APR based Apache Tomcat Native library 1.1.20.
Jun 16, 2011 6:06:48 PM org.apache.catalina.core.AprLifecycleListener init
INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
Jun 16, 2011 6:06:50 PM org.apache.coyote.http11.Http11AprProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080
OPENSSL_Uplink(0000000007A92000,08): no OPENSSL_Applink
{code}

B) It's caused by garbage DLL's in c:\windows\system32 - libeay32.dll, libssl32.dll, ssleay32.dll

For the record (and searchability), after removing libeay32, APR did not load at all and the error was:

{code}
Jun 16, 2011 6:23:47 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080
Jun 16, 2011 6:23:48 PM org.apache.tomcat.util.net.jsse.JSSESocketFactory getStore
SEVERE: Failed to load keystore type JKS with path C:\Users\Administrator/.keystore due to C:\Users\Administrator\.keystore (The system cannot find the file specified)
java.io.FileNotFoundException: C:\Users\Administrator\.keystore (The system cannot find the file specified)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:120)
{code}


This potential gotcha should be documented - users should be advised to check c:\windows\system32 for files overriding EWS's DLL's.
This is especially important because Tomcat doesn't log any error and the poor user is left in chaos.

Maybe EWS could also have a script for this - just something which would list potential troublemakers if present there.

Comment 1 Ondřej Žižka 2011-06-16 20:14:23 UTC
Link: Added: This issue is related to JBQA-4819


Comment 2 Mladen Turk 2011-06-16 21:38:19 UTC
Tomcat cannot display the reason because the only one that OS is giving is LoadLibrary system call failure.
Here is a technical background
http://msdn.microsoft.com/en-us/library/ms682586%28v=vs.85%29.aspx

It means that the SystemDirectory will always win both the PATH (which we modify) and
current directory. Note that item 1 (The directory from which the application loaded) is
the location of java.exe, so we would eventually need to copy our dll's there
which is almost certainly not an option.

Microsoft clearly states that no third party application should ever modify the content
of SystemDirectory or install their own files there.
I'm against adding any sort of check for that in our startup scripts.
However we should add this in our install documentation under some "Potential install problems"
or something like that section.



Comment 3 Ondřej Žižka 2011-06-17 13:05:31 UTC
Link: Removed: This issue is related to JBQA-4819 


Comment 4 Ondřej Žižka 2011-06-17 13:05:42 UTC
Link: Added: This issue relates to JBQA-4819


Comment 5 Ondřej Žižka 2011-06-17 13:26:41 UTC
Release Notes Docs Status: Added: Not Yet Documented
Release Notes Text: Added: **This should go to docs, not Release Notes.** I just don't see a textbox for "just docs". 

Some thirdparty applications add arbitrary libraries to system directory (typically `C:\windows\System32\`), which takes highest precedence when DLL libraries are being looked up. For example, one such is OpenSSL. This behavior is against Microsoft's recomendations.

Similarly, directories configured in the `PATH` environment variable are also searched for DLL files.

In case these libraries happen to have the same name as those used by Tomcat native libraries, they are loaded instead of those distributed with Enterprise Web Server, and cause Tomcat to start without any message logged to Windows Event Log or Tomcat log files. The only way to see any error message is to start using `catalina.bat run`.

Users who encounter such behavior should check the content of the System32 directory and `PATH` directories, and make sure no DLLs libraries conflict with those shipped with Enterprise Web Server; especially `libeay32.dll`, `ssleay32.dll`, and `libssl32.dll`.


Comment 6 Ondřej Žižka 2011-06-17 13:36:01 UTC
Release Notes Text: Removed: **This should go to docs, not Release Notes.** I just don't see a textbox for "just docs". 

Some thirdparty applications add arbitrary libraries to system directory (typically `C:\windows\System32\`), which takes highest precedence when DLL libraries are being looked up. For example, one such is OpenSSL. This behavior is against Microsoft's recomendations.

Similarly, directories configured in the `PATH` environment variable are also searched for DLL files.

In case these libraries happen to have the same name as those used by Tomcat native libraries, they are loaded instead of those distributed with Enterprise Web Server, and cause Tomcat to start without any message logged to Windows Event Log or Tomcat log files. The only way to see any error message is to start using `catalina.bat run`.

Users who encounter such behavior should check the content of the System32 directory and `PATH` directories, and make sure no DLLs libraries conflict with those shipped with Enterprise Web Server; especially `libeay32.dll`, `ssleay32.dll`, and `libssl32.dll`. Added: **This should go to docs, not Release Notes.** I just don't see a textbox for "just docs". 

Some thirdparty applications add arbitrary libraries to system directory (typically `C:\windows\System32\`), which takes highest precedence when DLL libraries are being looked up. For example, one such is OpenSSL. Changing content of the `System32` directory is against Microsoft's recomendations.

Similarly, directories configured in the `PATH` environment variable are also searched for DLL files.

In case these libraries happen to have the same name as those used by Tomcat native libraries, they are loaded instead of those distributed with Enterprise Web Server, and cause Tomcat to start without any message logged to Windows Event Log or Tomcat log files. The only way to see any error message is to start using `catalina.bat run`.

Users who encounter such behavior should check the content of the System32 directory and `PATH` directories, and make sure no DLLs libraries conflict with those shipped with Enterprise Web Server; especially `libeay32.dll`, `ssleay32.dll`, and `libssl32.dll`.


Comment 7 Mladen Turk 2011-06-17 14:17:43 UTC
The release notes text:
" For example, one such is OpenSSL."

This is misleading. Please specify the producer of that package.
1. What you installed is not official OpenSSL distribution
2. This particular package is produced by some individual
   and it will break Subversion, Httpd, our EWS, EAP and
   almost any other package that ships its own ssl libraries.


Comment 8 Mladen Turk 2011-06-17 14:28:32 UTC
BTW, you can just mention that package from
http://www.slproweb.com/products/Win32OpenSSL.html
will cause the EWS to stop function.


Comment 9 Ondřej Žižka 2011-06-17 14:37:48 UTC
Ok, I removed the note, since I think mentioning the package could be considered unfair or such.
Mentioning this jira will be enough if someone looks for that information.
Otherwise fine?

Comment 10 Ondřej Žižka 2011-06-17 14:37:48 UTC
Release Notes Text: Removed: **This should go to docs, not Release Notes.** I just don't see a textbox for "just docs". 

Some thirdparty applications add arbitrary libraries to system directory (typically `C:\windows\System32\`), which takes highest precedence when DLL libraries are being looked up. For example, one such is OpenSSL. Changing content of the `System32` directory is against Microsoft's recomendations.

Similarly, directories configured in the `PATH` environment variable are also searched for DLL files.

In case these libraries happen to have the same name as those used by Tomcat native libraries, they are loaded instead of those distributed with Enterprise Web Server, and cause Tomcat to start without any message logged to Windows Event Log or Tomcat log files. The only way to see any error message is to start using `catalina.bat run`.

Users who encounter such behavior should check the content of the System32 directory and `PATH` directories, and make sure no DLLs libraries conflict with those shipped with Enterprise Web Server; especially `libeay32.dll`, `ssleay32.dll`, and `libssl32.dll`. Added: **This should go to docs, not Release Notes.** I just don't see a textbox for "just docs". 

Some thirdparty applications add arbitrary libraries to system directory (typically `C:\windows\System32\`), which takes highest precedence when DLL libraries are being looked up. Changing content of the `System32` directory is against Microsoft's recomendations.

Similarly, directories configured in the `PATH` environment variable are also searched for DLL files.

In case these libraries happen to have the same name as those used by Tomcat native libraries, they are loaded instead of those distributed with Enterprise Web Server, and cause Tomcat to start without any message logged to Windows Event Log or Tomcat log files. The only way to see any error message is to start using `catalina.bat run`.

Users who encounter such behavior should check the content of the System32 directory and `PATH` directories, and make sure no DLLs libraries conflict with those shipped with Enterprise Web Server; especially `libeay32.dll`, `ssleay32.dll`, and `libssl32.dll`.


Comment 11 Rebecca Newton 2011-06-20 04:31:31 UTC
Hi Ondrej, I think this is RN-worthy so I've put it in there. The only other place it might go is the Install Guide, wdyt? Here is the text:

Some thirdparty applications add libraries to the system directory in Windows. These take precedence over Tomcat when DLL libraries are looked up. This means that if they have the same name as the those used by Tomcat native libraries, they are loaded instead of the libraries distributed with JBoss Enterprise Web Server. Tomcat is therefore started without any messages logged in Windows Event Log, or Tomcat log files. Errors can only be seen by using catalina.bat run.

If this behaviour occurs, check the content of the C:\windows\System32\ directory and PATH directories, and check for DLLs libraries conflicting with those shipped with Enterprise Web Server. In particular, look for libeay32.dil, ssleay32.dll, and libssl32.dll.

Comment 12 Ondřej Žižka 2011-06-22 13:27:33 UTC
Hi Rebecca, the information are correct, only the logic is a bit different.
The matter is, there are two facts:
1) DLL's in system dir break Tomcat (as you wrote)
2) If this happens, Tomcat doesn't start, but doesn't print any visible error about it.
   And the only way to check this error is to start using catalina.bat (which is a non-standard way).

So, could you please reword this in that sense?
{quote}
"Tomcat is therefore started without any messages logged in Windows Event Log, or Tomcat log files. Errors can only be seen by using catalina.bat run."
{quote}

Besides that, it's good. Thanks



Comment 13 Ondřej Žižka 2011-06-22 13:30:19 UTC
And regarding where to put it - it's not affecting particular EWS build or Tomcat version, but general issue/feature of Windows, which may happen no matter how much we try :)  Thus I think it should be in docs in general. I'd put it in Install Guide.

Comment 14 Rebecca Newton 2011-06-23 00:01:55 UTC
Hey Ondrej, thanks for the clarification. Reworded and I've slipped it into the Installation Guide under 4.4 Running Enterprise Web Server (on Windows). It will be available with the new build. Offending sentence reworded below:

"Tomcat therefore does not start, and does not long any error messages about it in Windows Event Log, or Tomcat log files. Errors can only be seen by using catalina.bat run"

Comment 15 Ondřej Žižka 2011-06-23 17:47:40 UTC
Good. Almost :) If you change "long" to "log", I call it done.

Comment 16 Ondřej Žižka 2011-07-19 19:54:56 UTC
Rebecca, pls let's finish this, thx.

Comment 17 Rebecca Newton 2011-07-21 00:53:43 UTC
Okay, I'm updating the stage now. Not sure how I overlooked that! I'll wait for your sign off on JBPAPP-6740 and then update this on the live site. 

Comment 18 Jiri Skrabal 2012-11-13 15:56:35 UTC
Release Notes Docs Status: Removed: Not Yet Documented 
Release Notes Text: Removed: **This should go to docs, not Release Notes.** I just don't see a textbox for "just docs". 

Some thirdparty applications add arbitrary libraries to system directory (typically `C:\windows\System32\`), which takes highest precedence when DLL libraries are being looked up. Changing content of the `System32` directory is against Microsoft's recomendations.

Similarly, directories configured in the `PATH` environment variable are also searched for DLL files.

In case these libraries happen to have the same name as those used by Tomcat native libraries, they are loaded instead of those distributed with Enterprise Web Server, and cause Tomcat to start without any message logged to Windows Event Log or Tomcat log files. The only way to see any error message is to start using `catalina.bat run`.

Users who encounter such behavior should check the content of the System32 directory and `PATH` directories, and make sure no DLLs libraries conflict with those shipped with Enterprise Web Server; especially `libeay32.dll`, `ssleay32.dll`, and `libssl32.dll`. 
Docs QE Status: Removed: NEW 



Note You need to log in before you can comment on or make changes to this bug.