### Description of problem: The following NullPointerException rarely happens in Web Service application. This phenomenon occurs when Full GC happens before calling CachedContextAndSchemasInternal#getContext() in a constructor of org.apache.cxf.common.jaxb.JAXBContextCache$CachedContextAndSchemas. Because "WeakReference<JAXBContext> context" inside org.apache.cxf.common.jaxb.JAXBContextCache$CachedContextAndSchemasInternal is cleared at the Full GC and CachedContextAndSchemasInternal#getContext() returns null. As far as I tested, this issue can not be reproducible with OpenJDK/Oracle JDK but it was reproducible with IBM JDK. ~~~ java.lang.NullPointerException at org.apache.cxf.common.jaxb.JAXBUtils.createJAXBContextProxy(JAXBUtils.java:1425) at org.apache.cxf.jaxb.JAXBDataBinding.checkForJAXBAnnotations(JAXBDataBinding.java:433) at org.apache.cxf.jaxb.JAXBDataBinding.justCheckForJAXBAnnotations(JAXBDataBinding.java:427) at org.apache.cxf.jaxb.JAXBDataBinding.initialize(JAXBDataBinding.java:363) at org.apache.cxf.service.factory.AbstractServiceFactoryBean.initializeDataBindings(AbstractServiceFactoryBean.java:86) at org.apache.cxf.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL(ReflectionServiceFactoryBean.java:446) at org.apache.cxf.service.factory.ReflectionServiceFactoryBean.initializeServiceModel(ReflectionServiceFactoryBean.java:548) at org.apache.cxf.service.factory.ReflectionServiceFactoryBean.create(ReflectionServiceFactoryBean.java:265) at org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServiceFactoryBean.java:215) at org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpoint(AbstractWSDLBasedEndpointFactory.java:102) at org.apache.cxf.frontend.ClientFactoryBean.create(ClientFactoryBean.java:91) at org.apache.cxf.frontend.ClientProxyFactoryBean.create(ClientProxyFactoryBean.java:157) at org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create(JaxWsProxyFactoryBean.java:142) at org.apache.cxf.jaxws.ServiceImpl.createPort(ServiceImpl.java:478) at org.jboss.wsf.stack.cxf.client.ProviderImpl$JBossWSServiceImpl.createPort(ProviderImpl.java:558) at org.apache.cxf.jaxws.ServiceImpl.getPort(ServiceImpl.java:329) at org.apache.cxf.jaxws.ServiceImpl.getPort(ServiceImpl.java:324) at javax.xml.ws.Service.getPort(Service.java:168) [jboss-jaxws-api_2.2_spec-2.0.2.Final-redhat-1.jar:2.0.2.Final-redhat-1] at org.jboss.as.quickstarts.wshelloworld.Client.<init>(Client.java:44) [classes:] at org.jboss.as.quickstarts.wshelloworld.WSClientServlet.doGet(WSClientServlet.java:97) [classes:] at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1] at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.6.Final-redhat-2.jar:7.5.6.Final-redhat-2] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.12.Final-redhat-1.jar:7.5.12.Final-redhat-1] at java.lang.Thread.run(Thread.java:785) [vm.jar:1.8.0] ~~~ ### Version-Release number of selected component (if applicable): CXF 2.7.18, CXF 2.7.18.SP1 The above stack trace is from EAP 6.4.6 (CXF 2.7.18). Same issue is reproducible with the latest EAP 6.4.10 (CXF 2.7.18.SP1). ### How reproducible: You can reproduce this issue anytime when you use byteman to invoke Full GC (System.gc()) before calling CachedContextAndSchemasInternal#getContext() in a constructor of org.apache.cxf.common.jaxb.JAXBContextCache$CachedContextAndSchemas. ### Steps to Reproduce: 0. Extract the attached reproducer cxf-npe-reproducer.tar.gz 1. Build and deploy two applications - helloworld-ws : Hello World JAX-WS Web service Application (based on JBoss EAP QuickStarts) - servlet-wsclient : Web Services Client on Servlet 2. Place the following byteman rule as /tmp/byteman-script.btm You can use byteman-script.btm in the cxf-npe-reproducer.tar.gz ~~~ RULE org.apache.cxf.common.jaxb.JAXBContextCache$CachedContextAndSchemas_init_ENTRY CLASS org.apache.cxf.common.jaxb.JAXBContextCache$CachedContextAndSchemas METHOD <init> AT ENTRY IF TRUE DO System.gc(); ENDRULE ~~~ 3. Add the following to $JBOSS_HOME/bin/standalone.conf Note: Make sure to tune BYTEMAN_HOME to a byteman install location ~~~ BYTEMAN_RULE=/tmp/byteman-script.btm BYTEMAN_HOME=/path/to/your/byteman BYTEMAN_OPTS="-javaagent:$BYTEMAN_HOME/lib/byteman.jar=listener:false,boot:$BYTEMAN_HOME/lib/byteman.jar" if [ "x$BYTEMAN_RULE" != "x" ]; then BYTEMAN_OPTS="${BYTEMAN_OPTS},script:$BYTEMAN_RULE" fi BYTEMAN_OPTS="$BYTEMAN_OPTS -Dorg.jboss.byteman.transform.all -Dorg.jboss.byteman.debug" # BYTEMAN_OPTS="$BYTEMAN_OPTS -Djboss.modules.system.pkgs=org.jboss.byteman" JAVA_OPTS="$BYTEMAN_OPTS $JAVA_OPTS" ~~~ 4. Set IBM JDK 7 or IBM JDK 8 as JAVA_HOME. For example: export JAVA_HOME=/usr/lib/jvm/java-1.7.0-ibm.x86_64" or export JAVA_HOME=/usr/lib/jvm/java-1.8.0-ibm.x86_64" 5. Start JBoss EAP 6 ./bin/standalone.sh 6. Access Web Services Client Servlet curl -v http://localhost:8080/jboss-servlet-wsclient/client ### Actual results: Application fails with NPE ### Expected results: Application succeeds without NPE ### Additional info: A similar issue was reported in https://issues.apache.org/jira/browse/CXF-4947 but this does not look a perfect fix.
Created attachment 1210446 [details] cxf-npe-reproducer.tar.gz
Sorry, I forgot to mention the version of JBoss which a customer originally reported. This issue was originally reported by the customer using JBoss EAP 6.4.3 in the support case 01704042.
I got an update from the customer. They said this NPE occurred on OpenJDK in their environment although I have not yet reproduced this on OpenJDK in my environment. Anyway I think the attached reproducer can be used to reproduce this issue on IBM JDK reliably. Please utilize it to investigate a root cause and how to fix it. FYI: Here's the version information they are using: ~~~ # rpm -qa | grep java java-1.7.0-openjdk-devel-1.7.0.95-2.6.4.0.el6_7.x86_64 tzdata-java-2014g-1.el6.noarch java-1.7.0-openjdk-1.7.0.95-2.6.4.0.el6_7.x86_64 libvirt-java-0.4.9-1.el6.noarch libvirt-java-devel-0.4.9-1.el6.noarch java-1.7.0-openjdk-debuginfo-1.7.0.95-2.6.4.0.el6_7.x86_64 ~~~ ~~~ # java -version java version "1.7.0_95" OpenJDK Runtime Environment (rhel-2.6.4.0.el6_7-x86_64 u95-b00) OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode) ~~~
Created attachment 1213467 [details] fix suggestion
Verified with EAP 6.4.14.CP.CR1 using the provided reproducer.
Released with EAP 6.4.14 on March 14 (ZIPs) and March 22 (RPMs).