Bug 1104273 - (6.4.z) [QE] JBossWS testsuite failures with JDK 6u75, 7u55 and 8u5 and higher as a consequence to CVE-2014-0458
Summary: (6.4.z) [QE] JBossWS testsuite failures with JDK 6u75, 7u55 and 8u5 and highe...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Web Services
Version: 6.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: CR2
: EAP 6.4.2
Assignee: Rebecca Searls
QA Contact: Rostislav Svoboda
Russell Dickenson
URL:
Whiteboard:
Depends On:
Blocks: 1219165 1234435
TreeView+ depends on / blocked
 
Reported: 2014-06-03 17:10 UTC by Jan Blizňák
Modified: 2017-01-17 10:13 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
RHEL 5/6/7; Fedora 19 x86_64 EAP 6.3.0.ER5 JBossWS 4.3.0.Final (testsuite modified to use EAP productized artifacts where possible)
Last Closed: 2017-01-17 10:13:48 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Logs from runs on different JDKs (2.27 MB, application/zip)
2014-06-03 17:10 UTC, Jan Blizňák
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1087430 0 high CLOSED CVE-2014-0458 OpenJDK: Activation framework default command map caching (JAX-WS, 8025152) 2021-02-22 00:41:40 UTC

Internal Links: 1087430

Description Jan Blizňák 2014-06-03 17:10:57 UTC
Created attachment 901865 [details]
Logs from runs on different JDKs

With update of Oracle JDK to version 1.7.0_55 test errors appeared in org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase, 
these errors didn't appear on JDK 1.7.0_51 and older.

How reproducible:
always

Steps to Reproduce:
1. set JAVA_HOME to JDK 1.7.0_55+
2. start EAP with that JAVA_HOME
  ~/eap/630ER5/jboss-eap-6.3/bin/standalone.sh 
3. run testsuite with that JAVA_HOME from module/testsuite with either option
 a) full testsuite 
mvn -s settings-630er5.xml -Dmaven.repo.local=/tmp/local-repo -Dlog4j.output.dir=target -Djboss.home=~/eap/630ER5/jboss-eap-6.3  -Pjboss720,hudson,testsuite integration-test

 b) tests only from package org.jboss.test.ws.jaxrpc.samples.swa (AttachmentDIITestCase,AttachmentProxyTestCase)
mvn -s settings-630er5.xml -Dmaven.repo.local=/tmp/local-repo -Dlog4j.output.dir=target -Djboss.home=~/eap/630ER5/jboss-eap-6.3  -Pjboss720,hudson,testsuite -Dtest=org.jboss.test.ws.jaxrpc.samples.swa.AttachmentDIITestCase,org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase integration-test 


Actual results:
Tests in error: 
  testSendMimeImageJPEG(org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase): JBWS025264: Call invocation failed; nested exception is: (..)
  testSendMimeTextPlain(org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase): JBWS025263: Call invocation failed with code [Server] because of: ClientAbortException:  java.io.IOException: JBWEB002020: Invalid chunk header; nested exception is: (..)
  testSendMimeApplicationXML(org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase): JBWS025264: Call invocation failed; nested exception is: (..)


Expected results:
 no errors


Additional info:
Several logs from runs with various configurations are attached.

We hit the same errors also on OpenJDK 1.7.0_55(u55-b13), OpenJDK 1.6.0_30 (IcedTea6 1.13.3), IBM SDK 7 Release 1, IBM SDK 6 Service Refresh 16.
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-WS/job/eap-6x-jbossws-testsuite-rhel/34/

When I run only single testcase org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase, no errors occur.
When running with -Dtest=... property, with running all execution, no errors occur when tests are executed in forked mode.
When I run only (AttachmentDIITestCase,AttachmentProxyTestCase) and configure surefire plugin to run these testcases in reverse order (AttachmentProxyTestCase,AttachmentDIITestCase), no errors occur.


Based on these facts it appears that these testcases aren't independant, more precisely AttachmentDIITestCase running first somehow influences AttachmentProxyTestCase.

Comment 1 Jan Blizňák 2014-06-20 16:52:30 UTC
Just noting that issues with these two testcases occur also on 6.2.4.CP with JBossWS 4.2.4.Final on JDKs reported above. 

This time even with failures on AttachmentDIITestCase tests.

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-WS/job/eap-62x-patched-jbossws-testsuite-matrix/21/

Comment 2 Rebecca Searls 2014-06-25 12:45:53 UTC
I've simplified the test case to better identify the issue.  

Disable all test methods except 1 in AttachmentDIITestCase and disable
all test methods except testSendMimeImageJPEG in AttachmentProxyTestCase.
testSendMimeImageJPEG is the method that is failing.  The other test methods in
AttachmentProxyTestCase run in conjuction with AttachmentProxyTestCase without error.
Any of the methods (or all) in AttachmentProxyTestCase can be active and must run before 
AttachmentProxyTestCase to see the failure.

The surefire-reports for AttachmentDIITestCase shows the failure to be, 
   "JBWS025264: Call invocation failed; nested exception is:
    javax.activation.UnsupportedDataTypeException: no object DCH for MIME type image/jpeg" type="java.rmi.RemoteException">

Comment 3 Rostislav Svoboda 2014-06-27 08:19:39 UTC
Can it be JDK issue ?

I found 2 similar cases for JDK 6:
  http://forum.spring.io/forum/osgi-related/dm-server-general/61205-problems-sending-mime-multipart-mails
  http://stackoverflow.com/questions/1969667/send-a-mail-from-java5-and-java6

Both were solved by JDK update.

Comment 4 Rebecca Searls 2014-06-27 14:42:58 UTC
The ref you found may explain the issue.  I also found this ref in the release 
notes of jdk-1.7.0_55.  I've found that this issue is tied to the testsuite side.
The tests run successfully if I exec the testsuit using 1.7.0_51 and run the
server with 1.7.0_55.  I am starting to investigate the testsuit env.

http://www.oracle.com/technetwork/java/javase/7u55-relnotes-2177812.html
The very last item 
  Area: xml/jax-ws
  Synopsis: JAF initialisation in SAAJ clashing with the one in javax.mail

Comment 5 Rebecca Searls 2014-06-30 19:51:54 UTC
This issues appears to be related to the classloader as described in the previous
refers, but there appears to be more to it in our case.  In AttachmentProxyTestCase.testSendMimeImageJPEG 
I added the stmt
      Thread.currentThread().setContextClassLoader(
        com.sun.mail.handlers.image_jpeg.class.getClassLoader());
to allow the classloader to resolve the DCH for MIME type image/jpeg.  This fix works when
running the testsuit in the following 2 ways.

1.  mvn -Djboss720.home=${EAP_PATH}  -Pjboss720  integration-test
 
2.  mvn -Djboss720.home=${EAP_PATH}  -Pjboss720,testsuite  integration-test 

These cmds run the test pair AttachmentDIITestCase and AttachmentProxyTestCase once per run.

It does not work when running the testsuit in this manner.

    mvn -Djboss720.home=${EAP_PATH}  -Pjboss720 -Dtest=org.jboss.test.ws.jaxrpc.samples.swa.AttachmentDIITestCase,org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase integration-test

This cmd causes the test pair AttachmentDIITestCase and AttachmentProxyTestCase to be
run twice in succession per run.  AttachmentDIITestCase and AttachmentProxyTestCase runs successfully
the 1st time but fails with the error the 2nd time.

Can junit and surefire be interfering with the classloader as well.  How might that 
be determined?

Comment 6 Rebecca Searls 2014-07-03 15:05:03 UTC
The ClassLoader is not the issue. The problem is our additions to the 
javax.activation.CommandMap are getting lost.  org.jboss.ws.core.soap.attachment.AttachmentPartImpl
declares a static block in which ContentHandlerRegistry.register() is called on
AttachmentPartImpl init.  ContentHandlerRegistry.register() registers the jbossws
handlers with the classLoader's (tccl) CommandMap.  Between the calls to testSendMimeImageGIF
and testSendMimeImageJPEG in this particular case the ref to the static data is lost.
It is unclear if an new ClassLoader object is created thus loosing the ref or a new
CommandMap is created or what.  The proposed solution is to remove the 
ContentHandlerRegistry.register() stmt from the static block and have it called 
each time a new AttachmentPartImpl is created.  The proposed code change to
jbossws-native-4.2.0.Final (or appropriate version) is

 
-   static
-   {
-      // Load JAF content handlers
-      ContentHandlerRegistry.register();
-   }
-
    public AttachmentPartImpl()
    {
+       // Load JAF content handlers
+       ContentHandlerRegistry.register();
    }
 
    public AttachmentPartImpl(DataHandler handler)
    {
+      this();
       this.dataHandler = handler;
    }

Comment 7 Rebecca Searls 2014-07-03 16:29:11 UTC
Created maintenance branch http://svn.jboss.org/repos/jbossws/stack/native/branches/jbossws-native-4.2.x.   Version changed to 4.2.1-SNAPSHOT.
Above code change checked in (Committed revision 18779)

Comment 10 Rebecca Searls 2014-07-11 13:57:36 UTC
> - is it intentional to invoke ContentHandlerRegistry.register(); every time
>   object is created vs. only once ?
Well a new AttachmentPartImpl is created pre usage.  I have checked in a
revision (Committed revision 18796) related to this.

> - by not using static block initialization there can be threading issue in
> invoked method, I checked ContentHandlerRegistry and there are some risks -
> mainly in registerContentHandler method.

This revision (Committed revision 18796) I think addresses your concerns and the 
issue with the JDK (1.7.0_55+) loosing our static block for ContentHandlerRegistry.
ContentHandlerRegistry is now only called once. Doing this inside a method keeps 
the JDK from loosing the contentHandlers.

Comment 11 Jan Blizňák 2014-08-13 11:46:03 UTC
Having applied changes from revision 18779 on top of jbossws-native-core-4.2.0.Final and replaced dependency for testsuite only and running against EAP 6.3.0.GA, the issues went away.

When applied changes from both 18779 and 18796 for testsuite, the issues are back - failures occur when NOT running in forked mode.

(patching also server doesn't hurt with either version)

Comment 12 Rostislav Svoboda 2014-10-17 06:36:47 UTC
Hi Rebecca, what's the next step with this issue ?

Comment 13 Rebecca Searls 2014-10-17 13:33:50 UTC
(In reply to Rostislav Svoboda from comment #12)
> Hi Rebecca, what's the next step with this issue ?

Do we know if the failure is comming from the same code point?  Is the stacktrace the same or different?

Comment 14 Jan Blizňák 2014-11-12 19:15:16 UTC
(In reply to Rebecca Searls from comment #13)
> Do we know if the failure is comming from the same code point?  Is the
> stacktrace the same or different?

yes, they are the same with applying r18796 (which makes sense as you basically revert your changes from r18779)

Comment 15 Jan Blizňák 2014-11-12 19:21:22 UTC
Here is the summary of what I know:

1. Both AttachmentDIITestCase and AttachmentProxyTestCase use configuration through JBossWSTestSetup with setUp and tearDown methods.
2. In these methods context classloader can be changed (depends on testcase configuration, it IS changed in AttachmentProxyTestCase)
3. with upgrade to JDK7u55+ it looks that the newly created classloader doesn't share handlers for mimetypes from parent because of changes in CommandMap.
Most probably the relevant OpenJDK changeset http://hg.openjdk.java.net/jdk7u/jdk7u/jaxws/rev/a41e0d5e8068
It can be checked for example by adding debug info before and after setting new classloader and before and after restoring original classloader:

org.jboss.wsf.test.JBossWSTestSetup:
protected void setUp() throws Exception
{
  ......
  ClassLoader parent = Thread.currentThread().getContextClassLoader();
  System.out.println("MIMETYPES FOR ORIGINAL CLASSLOADER \n" +Arrays.toString(((MailcapCommandMap) CommandMap.getDefaultCommandMap()).getMimeTypes()));
  originalClassLoader = parent;
  // add client jars to the class loader
  if (!clientJars.isEmpty())
  {
      URL[] urls = new URL[clientJars.size()];
      for (int i = 0; i < clientJars.size(); i++)
      {
	urls[i] = clientJars.get(i);
      }
      URLClassLoader cl = new URLClassLoader(urls, parent);
      Thread.currentThread().setContextClassLoader(cl);
      System.out.println("MIMETYPES FOR MODIFIED CLASSLOADER \n" +Arrays.toString(((MailcapCommandMap) CommandMap.getDefaultCommandMap()).getMimeTypes()));
  }
  ......
}

protected void tearDown() throws Exception
{
  try
  {
     .....
  }
  finally
  {
    System.out.println("MIMETYPES FOR MODIFIED CLASSLOADER BEFORE RESTORING\n" +Arrays.toString(((MailcapCommandMap) CommandMap.getDefaultCommandMap()).getMimeTypes()));
    Thread.currentThread().setContextClassLoader(originalClassLoader);
    System.out.println("MIMETYPES FOR ORIGINAL CLASSLOADER AFTER RESTORING\n" +Arrays.toString(((MailcapCommandMap) CommandMap.getDefaultCommandMap()).getMimeTypes()));
     .....
  }
}

4. running AttachmentDIITestCase first causes that handlers are registered (because static block of AttachmentPartImpl is executed), then they are lost because of new context classloader, and not registered again as the static block won't be executed anymore.
5. the JDK bug related to this: http://bugs.java.com/view_bug.do?bug_id=8043129
6. https://source.jboss.org/changelog/JBossWS?cs=18779 is fixing the test issue by forcing to register handlers every time AttachmentPartImpl is created
7. https://source.jboss.org/changelog/JBossWS?cs=18796 basically behaves the same as unfixed code, so it is definitely not correct
8. javax.activation.* classes are loaded from JDK libs, javax.activation:activation artifact is bundled in EAP mvn repository - if I add that activation-1.1.1-redhat-2.jar to endorsed libs of testsuite, tests will pass even with JDK7u55+


So, we must first decide if this is just the test issue or not (is the scenario of executing sending SOAP messages with attachment with different context classloaders within application lifetime the real use case?)

Possible solutions:
1) as test issue only, we could simply move these testcases to forked execution
2) as test issue only, we could re-register handlers after changing context classloader, for example by calling ContentHandlerRegistry.register() in testcase initialization 
3) we will force usage of defined javax.activation artifact by adding it to endorsed libs before tests start, but it should be forced not only for tests on client side (and what about artifact changes in future?)
4) it is change in JDK and we should deal with it, maybe as Rebecca did in r18779 (but see Rosta's comments)

Alession, could you please comment on this too to choose the proper way?

Comment 16 Alessio Soldano 2014-11-13 09:14:02 UTC
I didn't run any test, just looked at the code and read the comment above, here are my thoughts: if the problem actually comes from the content handler registration being bound to the current context classloader, then the changes in r18796 are clearly not going to work; what that version does it simply deferring the registration to the first time the AttachmentPartImpl is created; further creations of the AttachmentPartImpl won't trigger the registration, even if context classloader is different.

The proper way for fixing this issue should be figured out after having properly isolated the bug. Why is the registration lost when a new TCCL is set? The ContentHandlerRegistry.isRegistered() could then return a valid boolean. The need for actually synchronizing the handler registration depends on what is actually happening with the classloaders under the hood (are there multiple MailcapCommandMap instances or not?).

Comment 17 Rebecca Searls 2014-11-13 13:05:29 UTC
The RH TCK test team also encountered this issue see
https://bugzilla.redhat.com/show_bug.cgi?id=1135992.
Glassfish was having the same issue.  It was resolved by running Glassfish
with JDK 1.7_55- and EAP with JDK 1.7_55+.  This leads me to conclude this
is a JDK issue that is not going to be fixed in a JDK 1.7 release and we
will need to work around it until some future JDK release possible JDK 9 as
noted in http://bugs.java.com/view_bug.do?bug_id=8043129.  It may take
the ability to debug the JDK src to truly find the root cause of this.

Comment 18 Jan Blizňák 2014-11-13 14:14:36 UTC
(In reply to Alessio Soldano from comment #16)
> Why is the registration lost when a new TCCL is set? 

I believe that the change http://hg.openjdk.java.net/jdk7u/jdk7u/jaxws/rev/a41e0d5e8068 is the answer to this - new MailCommandMap is created for not yet registered thread context classloader which is exactly our case

Comment 19 Alessio Soldano 2014-11-13 14:16:24 UTC
Good catch Jan, that really looks the reason for the behavior described here!

Comment 20 Alessio Soldano 2014-11-13 14:18:22 UTC
So, given we now know how the CommandMap is implemented, it should be possible to fix the jbossws-native to also perform the handler registration on a thread context classloader base (we might need to keep track of the classloader on which the handlers have already been registered).

Comment 21 Rostislav Svoboda 2014-11-21 09:00:47 UTC
Changing back to ASSIGNED to reflect current status.

Comment 22 Rebecca Searls 2014-12-03 03:42:44 UTC
Test env:
  jbossws-native-4.2.0.Final:  http://svn.jboss.org/repos/jbossws/stack/native/tags/jbossws-native-4.2.0.Final
  jbossws-cxf-4.3.0.Final:    https://svn.jboss.org/repos/jbossws/stack/cxf/tags/jbossws-cxf-4.3.0.Final
  jboss-as-7.4.1.Final-redhat-SNAPSHOT: https://github.com/jbossas/jboss-eap.git  (branch: 6.3.x-proposed)
  JDK: 1.7.0_55-b13


I've added some debugging as close to the IOException as possible, 
org.jboss.ws.core.soap.attachment.BoundaryDelimitedInputStream after line 141

      System.out.println("##BoundaryDelimitedInputStream");
      for (int i = 0; i < buffer.length; i++) {
        // 10 bytes per line
         System.out.print("(" + i + ":0x" + (String.format("%02X", buffer[i])) + ")");
         if ((i % 10) == 0) {
            System.out.println();
         }
      }
      System.out.println();

This turned up a bit of interesting information. When AttachmentProxyTestCase fails 
[i.e. Invalid chunk header] the (header) byte array length is 296.  When AttachmentProxyTestCase 
succeeds the byte array length is 295.  In addition the run order of AttachmentDIITestCase 
and AttachmentProxyTestCase affects the success or failure of AttachmentProxyTestCase.
When AttachmentProxyTestCase runs before AttachmentDIITestCase, AttachmentProxyTestCase
runs successfully [1]; the header byte array length is 295.  When AttachmentDIITestCase
runs before AttachmentProxyTestCase [2], AttachmentProxyTestCase fails; the header byte 
array length is 296.  When AttachmentProxyTestCase is run as a single test case it
succeeds with a header byte array length is 295 [3].

[1]  mvn -Djboss720.home=<YOUR_JBOSS_HOME>  -Pjboss720  integration-test  -Dtest="org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase,org.jboss.test.ws.jaxrpc.samples.swa.AttachmentXXDIITestCase"

    * [Note: The underlaying testsuite utility puts the tests in alphabetical order.  
             In order to force AttachmentProxyTestCase to run first class AttachmentDIITestCase 
             was renamed to AttachmentXXDIITestCase
      ]

[2]  mvn -Djboss720.home=<YOUR_JBOSS_HOME>  -Pjboss720  integration-test  -Dtest="org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase,org.jboss.test.ws.jaxrpc.samples.swa.AttachmentDIITestCase"

[3]  mvn -Djboss720.home=<YOUR_JBOSS_HOME>  -Pjboss720  integration-test -Dtest="org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase"


I think this implies there is some issue on the client side in prep-ing the header
but I have not been able to find it.

Are there suggestions on how I should proceed to further track down this issue?

Comment 23 Rebecca Searls 2014-12-15 15:59:59 UTC
This problem is caused by the code change made to javax.activation.CommandMap.
As of jdk1.7.0_55 each CommandMap object is tied to an instance of a Thread's classloader.
(see details of code change here, https://mojo.redhat.com/blogs/Mirgation_in_the_Wild/2014/12/15/java7-update-55-invalid-chunk-header-error) 
So Jan's comment in (https://bugzilla.redhat.com/show_bug.cgi?id=1104273#c15),
"4) it is change in JDK and we should deal with it ..." and 
Alessio's advise (https://bugzilla.redhat.com/show_bug.cgi?id=1104273#c20),
".. keep track of the classloader on which the handlers have already been registered",
is the proper solution. 

I will remove the code changes to org.jboss.ws.core.soap.attachment.ContentHandlerRegistry,
as they are not an appropriate solution. I will work with Alessio to find the proper place to
track the thread and classloader in which our handlers are registered.

Comment 24 Rebecca Searls 2014-12-19 16:08:11 UTC
I found tracking the settings by classloader impossible to implement as
a viable generic solution. I am implementing a solution instead of adding a dummy
DataContentHandler with a dummy data flavor to the set of data content handlers
we register and checking for the presents it to determine if a new CommandMap
needs to be configured.

Change "Committed revision 19164."
http://svn.jboss.org/repos/jbossws/stack/native/branches/jbossws-native-4.2.x 
Sending        modules/core/src/main/java/org/jboss/ws/core/soap/attachment/ContentHandlerRegistry.java
Adding         modules/core/src/main/java/org/jboss/ws/core/soap/attachment/DummyDataContentHandler.java

Comment 25 Jan Blizňák 2015-01-06 08:05:19 UTC
I built and tested this version and it looks good, issues are gone with it.

Comment 26 Jan Blizňák 2015-03-03 15:46:30 UTC
Just for reference, adding summary info:

Change in JDK introduced as fix for https://access.redhat.com/security/cve/CVE-2014-0458
Affects all JDKs released after April, 2014.

On Oracle JDK starting with 6u75, 7u55 and 8u5.

Changes in JDK introduced with these changesets:
JDK6: http://hg.openjdk.java.net/jdk6/jdk6/jaxws/rev/fb250b08d453
JDK7: http://hg.openjdk.java.net/jdk7u/jdk7u/jaxws/rev/a41e0d5e8068
JDK8: http://hg.openjdk.java.net/jdk8u/jdk8u/jaxws/rev/2fcd3ddb57a6

Comment 27 Rostislav Svoboda 2015-04-28 10:33:32 UTC
https://source.jboss.org/changelog/JBossWS?cs=19449

Comment 29 Jan Blizňák 2015-06-19 17:29:14 UTC
Fix not included in 6.4.2.CP.CR1

jbossws-native-* modules was not upgraded, this build still contains jbossws-native-4.2.0.Final-redhat-1, which contains this bug.

Expected version is productized build of jbossws-native-4.2.1.Final
http://anonsvn.jboss.org/repos/jbossws/stack/native/tags/jbossws-native-4.2.1.Final/

Comment 30 Carlo de Wolf 2015-06-22 14:19:09 UTC
As per https://bugzilla.redhat.com/show_bug.cgi?id=1104273#c24 and proper component upgrade linked.

Comment 31 Jan Blizňák 2015-06-29 23:26:35 UTC
Verified on 6.4.2.CP.CR2

Comment 32 Petr Penicka 2017-01-17 10:13:48 UTC
Retroactively bulk-closing issues from released EAP 6.4 cumulative patches.


Note You need to log in before you can comment on or make changes to this bug.