Created attachment 901865 [details] Logs from runs on different JDKs With update of Oracle JDK to version 1.7.0_55 test errors appeared in org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase, these errors didn't appear on JDK 1.7.0_51 and older. How reproducible: always Steps to Reproduce: 1. set JAVA_HOME to JDK 1.7.0_55+ 2. start EAP with that JAVA_HOME ~/eap/630ER5/jboss-eap-6.3/bin/standalone.sh 3. run testsuite with that JAVA_HOME from module/testsuite with either option a) full testsuite mvn -s settings-630er5.xml -Dmaven.repo.local=/tmp/local-repo -Dlog4j.output.dir=target -Djboss.home=~/eap/630ER5/jboss-eap-6.3 -Pjboss720,hudson,testsuite integration-test b) tests only from package org.jboss.test.ws.jaxrpc.samples.swa (AttachmentDIITestCase,AttachmentProxyTestCase) mvn -s settings-630er5.xml -Dmaven.repo.local=/tmp/local-repo -Dlog4j.output.dir=target -Djboss.home=~/eap/630ER5/jboss-eap-6.3 -Pjboss720,hudson,testsuite -Dtest=org.jboss.test.ws.jaxrpc.samples.swa.AttachmentDIITestCase,org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase integration-test Actual results: Tests in error: testSendMimeImageJPEG(org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase): JBWS025264: Call invocation failed; nested exception is: (..) testSendMimeTextPlain(org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase): JBWS025263: Call invocation failed with code [Server] because of: ClientAbortException: java.io.IOException: JBWEB002020: Invalid chunk header; nested exception is: (..) testSendMimeApplicationXML(org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase): JBWS025264: Call invocation failed; nested exception is: (..) Expected results: no errors Additional info: Several logs from runs with various configurations are attached. We hit the same errors also on OpenJDK 1.7.0_55(u55-b13), OpenJDK 1.6.0_30 (IcedTea6 1.13.3), IBM SDK 7 Release 1, IBM SDK 6 Service Refresh 16. https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-WS/job/eap-6x-jbossws-testsuite-rhel/34/ When I run only single testcase org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase, no errors occur. When running with -Dtest=... property, with running all execution, no errors occur when tests are executed in forked mode. When I run only (AttachmentDIITestCase,AttachmentProxyTestCase) and configure surefire plugin to run these testcases in reverse order (AttachmentProxyTestCase,AttachmentDIITestCase), no errors occur. Based on these facts it appears that these testcases aren't independant, more precisely AttachmentDIITestCase running first somehow influences AttachmentProxyTestCase.
Just noting that issues with these two testcases occur also on 6.2.4.CP with JBossWS 4.2.4.Final on JDKs reported above. This time even with failures on AttachmentDIITestCase tests. https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-WS/job/eap-62x-patched-jbossws-testsuite-matrix/21/
I've simplified the test case to better identify the issue. Disable all test methods except 1 in AttachmentDIITestCase and disable all test methods except testSendMimeImageJPEG in AttachmentProxyTestCase. testSendMimeImageJPEG is the method that is failing. The other test methods in AttachmentProxyTestCase run in conjuction with AttachmentProxyTestCase without error. Any of the methods (or all) in AttachmentProxyTestCase can be active and must run before AttachmentProxyTestCase to see the failure. The surefire-reports for AttachmentDIITestCase shows the failure to be, "JBWS025264: Call invocation failed; nested exception is: javax.activation.UnsupportedDataTypeException: no object DCH for MIME type image/jpeg" type="java.rmi.RemoteException">
Can it be JDK issue ? I found 2 similar cases for JDK 6: http://forum.spring.io/forum/osgi-related/dm-server-general/61205-problems-sending-mime-multipart-mails http://stackoverflow.com/questions/1969667/send-a-mail-from-java5-and-java6 Both were solved by JDK update.
The ref you found may explain the issue. I also found this ref in the release notes of jdk-1.7.0_55. I've found that this issue is tied to the testsuite side. The tests run successfully if I exec the testsuit using 1.7.0_51 and run the server with 1.7.0_55. I am starting to investigate the testsuit env. http://www.oracle.com/technetwork/java/javase/7u55-relnotes-2177812.html The very last item Area: xml/jax-ws Synopsis: JAF initialisation in SAAJ clashing with the one in javax.mail
This issues appears to be related to the classloader as described in the previous refers, but there appears to be more to it in our case. In AttachmentProxyTestCase.testSendMimeImageJPEG I added the stmt Thread.currentThread().setContextClassLoader( com.sun.mail.handlers.image_jpeg.class.getClassLoader()); to allow the classloader to resolve the DCH for MIME type image/jpeg. This fix works when running the testsuit in the following 2 ways. 1. mvn -Djboss720.home=${EAP_PATH} -Pjboss720 integration-test 2. mvn -Djboss720.home=${EAP_PATH} -Pjboss720,testsuite integration-test These cmds run the test pair AttachmentDIITestCase and AttachmentProxyTestCase once per run. It does not work when running the testsuit in this manner. mvn -Djboss720.home=${EAP_PATH} -Pjboss720 -Dtest=org.jboss.test.ws.jaxrpc.samples.swa.AttachmentDIITestCase,org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase integration-test This cmd causes the test pair AttachmentDIITestCase and AttachmentProxyTestCase to be run twice in succession per run. AttachmentDIITestCase and AttachmentProxyTestCase runs successfully the 1st time but fails with the error the 2nd time. Can junit and surefire be interfering with the classloader as well. How might that be determined?
The ClassLoader is not the issue. The problem is our additions to the javax.activation.CommandMap are getting lost. org.jboss.ws.core.soap.attachment.AttachmentPartImpl declares a static block in which ContentHandlerRegistry.register() is called on AttachmentPartImpl init. ContentHandlerRegistry.register() registers the jbossws handlers with the classLoader's (tccl) CommandMap. Between the calls to testSendMimeImageGIF and testSendMimeImageJPEG in this particular case the ref to the static data is lost. It is unclear if an new ClassLoader object is created thus loosing the ref or a new CommandMap is created or what. The proposed solution is to remove the ContentHandlerRegistry.register() stmt from the static block and have it called each time a new AttachmentPartImpl is created. The proposed code change to jbossws-native-4.2.0.Final (or appropriate version) is - static - { - // Load JAF content handlers - ContentHandlerRegistry.register(); - } - public AttachmentPartImpl() { + // Load JAF content handlers + ContentHandlerRegistry.register(); } public AttachmentPartImpl(DataHandler handler) { + this(); this.dataHandler = handler; }
Created maintenance branch http://svn.jboss.org/repos/jbossws/stack/native/branches/jbossws-native-4.2.x. Version changed to 4.2.1-SNAPSHOT. Above code change checked in (Committed revision 18779)
> - is it intentional to invoke ContentHandlerRegistry.register(); every time > object is created vs. only once ? Well a new AttachmentPartImpl is created pre usage. I have checked in a revision (Committed revision 18796) related to this. > - by not using static block initialization there can be threading issue in > invoked method, I checked ContentHandlerRegistry and there are some risks - > mainly in registerContentHandler method. This revision (Committed revision 18796) I think addresses your concerns and the issue with the JDK (1.7.0_55+) loosing our static block for ContentHandlerRegistry. ContentHandlerRegistry is now only called once. Doing this inside a method keeps the JDK from loosing the contentHandlers.
Having applied changes from revision 18779 on top of jbossws-native-core-4.2.0.Final and replaced dependency for testsuite only and running against EAP 6.3.0.GA, the issues went away. When applied changes from both 18779 and 18796 for testsuite, the issues are back - failures occur when NOT running in forked mode. (patching also server doesn't hurt with either version)
Hi Rebecca, what's the next step with this issue ?
(In reply to Rostislav Svoboda from comment #12) > Hi Rebecca, what's the next step with this issue ? Do we know if the failure is comming from the same code point? Is the stacktrace the same or different?
(In reply to Rebecca Searls from comment #13) > Do we know if the failure is comming from the same code point? Is the > stacktrace the same or different? yes, they are the same with applying r18796 (which makes sense as you basically revert your changes from r18779)
Here is the summary of what I know: 1. Both AttachmentDIITestCase and AttachmentProxyTestCase use configuration through JBossWSTestSetup with setUp and tearDown methods. 2. In these methods context classloader can be changed (depends on testcase configuration, it IS changed in AttachmentProxyTestCase) 3. with upgrade to JDK7u55+ it looks that the newly created classloader doesn't share handlers for mimetypes from parent because of changes in CommandMap. Most probably the relevant OpenJDK changeset http://hg.openjdk.java.net/jdk7u/jdk7u/jaxws/rev/a41e0d5e8068 It can be checked for example by adding debug info before and after setting new classloader and before and after restoring original classloader: org.jboss.wsf.test.JBossWSTestSetup: protected void setUp() throws Exception { ...... ClassLoader parent = Thread.currentThread().getContextClassLoader(); System.out.println("MIMETYPES FOR ORIGINAL CLASSLOADER \n" +Arrays.toString(((MailcapCommandMap) CommandMap.getDefaultCommandMap()).getMimeTypes())); originalClassLoader = parent; // add client jars to the class loader if (!clientJars.isEmpty()) { URL[] urls = new URL[clientJars.size()]; for (int i = 0; i < clientJars.size(); i++) { urls[i] = clientJars.get(i); } URLClassLoader cl = new URLClassLoader(urls, parent); Thread.currentThread().setContextClassLoader(cl); System.out.println("MIMETYPES FOR MODIFIED CLASSLOADER \n" +Arrays.toString(((MailcapCommandMap) CommandMap.getDefaultCommandMap()).getMimeTypes())); } ...... } protected void tearDown() throws Exception { try { ..... } finally { System.out.println("MIMETYPES FOR MODIFIED CLASSLOADER BEFORE RESTORING\n" +Arrays.toString(((MailcapCommandMap) CommandMap.getDefaultCommandMap()).getMimeTypes())); Thread.currentThread().setContextClassLoader(originalClassLoader); System.out.println("MIMETYPES FOR ORIGINAL CLASSLOADER AFTER RESTORING\n" +Arrays.toString(((MailcapCommandMap) CommandMap.getDefaultCommandMap()).getMimeTypes())); ..... } } 4. running AttachmentDIITestCase first causes that handlers are registered (because static block of AttachmentPartImpl is executed), then they are lost because of new context classloader, and not registered again as the static block won't be executed anymore. 5. the JDK bug related to this: http://bugs.java.com/view_bug.do?bug_id=8043129 6. https://source.jboss.org/changelog/JBossWS?cs=18779 is fixing the test issue by forcing to register handlers every time AttachmentPartImpl is created 7. https://source.jboss.org/changelog/JBossWS?cs=18796 basically behaves the same as unfixed code, so it is definitely not correct 8. javax.activation.* classes are loaded from JDK libs, javax.activation:activation artifact is bundled in EAP mvn repository - if I add that activation-1.1.1-redhat-2.jar to endorsed libs of testsuite, tests will pass even with JDK7u55+ So, we must first decide if this is just the test issue or not (is the scenario of executing sending SOAP messages with attachment with different context classloaders within application lifetime the real use case?) Possible solutions: 1) as test issue only, we could simply move these testcases to forked execution 2) as test issue only, we could re-register handlers after changing context classloader, for example by calling ContentHandlerRegistry.register() in testcase initialization 3) we will force usage of defined javax.activation artifact by adding it to endorsed libs before tests start, but it should be forced not only for tests on client side (and what about artifact changes in future?) 4) it is change in JDK and we should deal with it, maybe as Rebecca did in r18779 (but see Rosta's comments) Alession, could you please comment on this too to choose the proper way?
I didn't run any test, just looked at the code and read the comment above, here are my thoughts: if the problem actually comes from the content handler registration being bound to the current context classloader, then the changes in r18796 are clearly not going to work; what that version does it simply deferring the registration to the first time the AttachmentPartImpl is created; further creations of the AttachmentPartImpl won't trigger the registration, even if context classloader is different. The proper way for fixing this issue should be figured out after having properly isolated the bug. Why is the registration lost when a new TCCL is set? The ContentHandlerRegistry.isRegistered() could then return a valid boolean. The need for actually synchronizing the handler registration depends on what is actually happening with the classloaders under the hood (are there multiple MailcapCommandMap instances or not?).
The RH TCK test team also encountered this issue see https://bugzilla.redhat.com/show_bug.cgi?id=1135992. Glassfish was having the same issue. It was resolved by running Glassfish with JDK 1.7_55- and EAP with JDK 1.7_55+. This leads me to conclude this is a JDK issue that is not going to be fixed in a JDK 1.7 release and we will need to work around it until some future JDK release possible JDK 9 as noted in http://bugs.java.com/view_bug.do?bug_id=8043129. It may take the ability to debug the JDK src to truly find the root cause of this.
(In reply to Alessio Soldano from comment #16) > Why is the registration lost when a new TCCL is set? I believe that the change http://hg.openjdk.java.net/jdk7u/jdk7u/jaxws/rev/a41e0d5e8068 is the answer to this - new MailCommandMap is created for not yet registered thread context classloader which is exactly our case
Good catch Jan, that really looks the reason for the behavior described here!
So, given we now know how the CommandMap is implemented, it should be possible to fix the jbossws-native to also perform the handler registration on a thread context classloader base (we might need to keep track of the classloader on which the handlers have already been registered).
Changing back to ASSIGNED to reflect current status.
Test env: jbossws-native-4.2.0.Final: http://svn.jboss.org/repos/jbossws/stack/native/tags/jbossws-native-4.2.0.Final jbossws-cxf-4.3.0.Final: https://svn.jboss.org/repos/jbossws/stack/cxf/tags/jbossws-cxf-4.3.0.Final jboss-as-7.4.1.Final-redhat-SNAPSHOT: https://github.com/jbossas/jboss-eap.git (branch: 6.3.x-proposed) JDK: 1.7.0_55-b13 I've added some debugging as close to the IOException as possible, org.jboss.ws.core.soap.attachment.BoundaryDelimitedInputStream after line 141 System.out.println("##BoundaryDelimitedInputStream"); for (int i = 0; i < buffer.length; i++) { // 10 bytes per line System.out.print("(" + i + ":0x" + (String.format("%02X", buffer[i])) + ")"); if ((i % 10) == 0) { System.out.println(); } } System.out.println(); This turned up a bit of interesting information. When AttachmentProxyTestCase fails [i.e. Invalid chunk header] the (header) byte array length is 296. When AttachmentProxyTestCase succeeds the byte array length is 295. In addition the run order of AttachmentDIITestCase and AttachmentProxyTestCase affects the success or failure of AttachmentProxyTestCase. When AttachmentProxyTestCase runs before AttachmentDIITestCase, AttachmentProxyTestCase runs successfully [1]; the header byte array length is 295. When AttachmentDIITestCase runs before AttachmentProxyTestCase [2], AttachmentProxyTestCase fails; the header byte array length is 296. When AttachmentProxyTestCase is run as a single test case it succeeds with a header byte array length is 295 [3]. [1] mvn -Djboss720.home=<YOUR_JBOSS_HOME> -Pjboss720 integration-test -Dtest="org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase,org.jboss.test.ws.jaxrpc.samples.swa.AttachmentXXDIITestCase" * [Note: The underlaying testsuite utility puts the tests in alphabetical order. In order to force AttachmentProxyTestCase to run first class AttachmentDIITestCase was renamed to AttachmentXXDIITestCase ] [2] mvn -Djboss720.home=<YOUR_JBOSS_HOME> -Pjboss720 integration-test -Dtest="org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase,org.jboss.test.ws.jaxrpc.samples.swa.AttachmentDIITestCase" [3] mvn -Djboss720.home=<YOUR_JBOSS_HOME> -Pjboss720 integration-test -Dtest="org.jboss.test.ws.jaxrpc.samples.swa.AttachmentProxyTestCase" I think this implies there is some issue on the client side in prep-ing the header but I have not been able to find it. Are there suggestions on how I should proceed to further track down this issue?
This problem is caused by the code change made to javax.activation.CommandMap. As of jdk1.7.0_55 each CommandMap object is tied to an instance of a Thread's classloader. (see details of code change here, https://mojo.redhat.com/blogs/Mirgation_in_the_Wild/2014/12/15/java7-update-55-invalid-chunk-header-error) So Jan's comment in (https://bugzilla.redhat.com/show_bug.cgi?id=1104273#c15), "4) it is change in JDK and we should deal with it ..." and Alessio's advise (https://bugzilla.redhat.com/show_bug.cgi?id=1104273#c20), ".. keep track of the classloader on which the handlers have already been registered", is the proper solution. I will remove the code changes to org.jboss.ws.core.soap.attachment.ContentHandlerRegistry, as they are not an appropriate solution. I will work with Alessio to find the proper place to track the thread and classloader in which our handlers are registered.
I found tracking the settings by classloader impossible to implement as a viable generic solution. I am implementing a solution instead of adding a dummy DataContentHandler with a dummy data flavor to the set of data content handlers we register and checking for the presents it to determine if a new CommandMap needs to be configured. Change "Committed revision 19164." http://svn.jboss.org/repos/jbossws/stack/native/branches/jbossws-native-4.2.x Sending modules/core/src/main/java/org/jboss/ws/core/soap/attachment/ContentHandlerRegistry.java Adding modules/core/src/main/java/org/jboss/ws/core/soap/attachment/DummyDataContentHandler.java
I built and tested this version and it looks good, issues are gone with it.
Just for reference, adding summary info: Change in JDK introduced as fix for https://access.redhat.com/security/cve/CVE-2014-0458 Affects all JDKs released after April, 2014. On Oracle JDK starting with 6u75, 7u55 and 8u5. Changes in JDK introduced with these changesets: JDK6: http://hg.openjdk.java.net/jdk6/jdk6/jaxws/rev/fb250b08d453 JDK7: http://hg.openjdk.java.net/jdk7u/jdk7u/jaxws/rev/a41e0d5e8068 JDK8: http://hg.openjdk.java.net/jdk8u/jdk8u/jaxws/rev/2fcd3ddb57a6
https://source.jboss.org/changelog/JBossWS?cs=19449
Fix not included in 6.4.2.CP.CR1 jbossws-native-* modules was not upgraded, this build still contains jbossws-native-4.2.0.Final-redhat-1, which contains this bug. Expected version is productized build of jbossws-native-4.2.1.Final http://anonsvn.jboss.org/repos/jbossws/stack/native/tags/jbossws-native-4.2.1.Final/
As per https://bugzilla.redhat.com/show_bug.cgi?id=1104273#c24 and proper component upgrade linked.
Verified on 6.4.2.CP.CR2
Retroactively bulk-closing issues from released EAP 6.4 cumulative patches.