started a server and had users simultaneously access a published cms article. I got this error: -*-*-*-*-*- Begin Error Report -*-*-*-*-*- -*-*-*- ACS Error Report Code: 127.0.0.1:261e668:f9421a33e7 -*-*-*- -*-*-*- Message 1: com.caucho.jsp.JspLineException: Illegal access to a locked com.arsdigita.bebop.SimpleContainer -*-*-*- -*-*-*- Message 2: java.lang.IllegalStateException: Illegal access to a locked com.arsdigita.bebop.SimpleContainer -*-*-*- -*-*-*- Section: HTTP headers -*-*-*- Accept: text/html, image/gif, image/jpeg, */* Accept-Language: en-us Connection: Keep-Alive Host: 172.16.65.125 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32) -*-*-*- Section: CCM User -*-*-*- Party not logged in -*-*-*- Section: Stack trace -*-*-*- java.lang.IllegalStateException: Illegal access to a locked com.arsdigita.bebop.SimpleContainer at com.arsdigita.util.Assert.assertTrue(Assert.java:374) at com.arsdigita.util.Assert.assertNotLocked(Assert.java:521) at com.arsdigita.bebop.SimpleContainer.add(SimpleContainer.java:92) at com.arsdigita.bebop.Page.add(Page.java:324) at com.arsdigita.ui.SimplePage.add(SimplePage.java:140) at com.arsdigita.bebop.jsp.DefineContainer.addComponent(DefineContainer.java:77) at com.arsdigita.bebop.jsp.DefineComponent.doStartTag(DefineComponent.java:52) at com.arsdigita.bebop.jsp.DefineComponentImpl.doStartTag(DefineComponentImpl.java:53) at _packages._content_22dsection._templates._default._item__jsp._jspService(_item__jsp.java:60) at com.caucho.jsp.JavaPage.service(JavaPage.java:75) at com.caucho.jsp.Page.subservice(Page.java:497) at com.caucho.server.http.FilterChainPage.doFilter(FilterChainPage.java:182) at com.caucho.server.http.Invocation.service(Invocation.java:315) at com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java:225) at com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java:106) at com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java:83) at com.arsdigita.dispatcher.DispatcherHelper.forwardHelper(DispatcherHelper.java:190) at com.arsdigita.dispatcher.DispatcherHelper.forwardRequestByPath(DispatcherHelper.java:219) at com.arsdigita.dispatcher.DispatcherHelper.forwardRequestByPath(DispatcherHelper.java:232) at com.arsdigita.cms.dispatcher.ContentItemDispatcher$1.excurse(ContentItemDispatcher.java:111) at com.arsdigita.cms.CMSExcursion$1.excurse(CMSExcursion.java:80) at com.arsdigita.kernel.KernelExcursion.run(KernelExcursion.java:57) at com.arsdigita.cms.CMSExcursion.run(CMSExcursion.java:75) at com.arsdigita.cms.dispatcher.ContentItemDispatcher.dispatch(ContentItemDispatcher.java:115) at com.arsdigita.cms.ContentSectionServlet.serveItem(ContentSectionServlet.java:200) at com.arsdigita.cms.ContentSectionServlet.doService(ContentSectionServlet.java:124) at com.arsdigita.web.BaseApplicationServlet$1.excurse(BaseApplicationServlet.java:111) at com.arsdigita.kernel.KernelExcursion.run(KernelExcursion.java:57) at com.arsdigita.web.BaseApplicationServlet.doService(BaseApplicationServlet.java:105) at com.arsdigita.web.BaseServlet$1.excurse(BaseServlet.java:167) at com.arsdigita.kernel.KernelExcursion.run(KernelExcursion.java:57) at com.arsdigita.web.BaseServlet.internalService(BaseServlet.java:142) at com.arsdigita.web.BaseServlet.doGet(BaseServlet.java:252) at javax.servlet.http.HttpServlet.service(HttpServlet.java:126) at javax.servlet.http.HttpServlet.service(HttpServlet.java:103) at com.caucho.server.http.FilterChainServlet.doFilter(FilterChainServlet.java:96) at com.caucho.server.http.Invocation.service(Invocation.java:315) at com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java:225) at com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java:106) at com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java:83) at com.arsdigita.web.BaseDispatcher.forward(BaseDispatcher.java:227) at com.arsdigita.web.BaseDispatcher.forward(BaseDispatcher.java:211) at com.arsdigita.web.BaseDispatcher.dispatch(BaseDispatcher.java:135) at com.arsdigita.web.DispatcherServlet.doService(DispatcherServlet.java:123) at com.arsdigita.web.BaseServlet$1.excurse(BaseServlet.java:167) at com.arsdigita.kernel.KernelExcursion.run(KernelExcursion.java:57) at com.arsdigita.web.BaseServlet.internalService(BaseServlet.java:142) at com.arsdigita.web.BaseServlet.doGet(BaseServlet.java:252) at javax.servlet.http.HttpServlet.service(HttpServlet.java:126) at javax.servlet.http.HttpServlet.service(HttpServlet.java:103) at com.caucho.server.http.FilterChainServlet.doFilter(FilterChainServlet.java:96) at com.caucho.server.http.Invocation.service(Invocation.java:315) at com.caucho.server.http.HttpRequest.handleRequest(HttpRequest.java:246) at com.caucho.server.http.HttpRequest.handleConnection(HttpRequest.java:163) at com.caucho.server.TcpConnection.run(TcpConnection.java:139) at java.lang.Thread.run(Thread.java:513) -*-*-*- Section: Servlet attributes -*-*-*- com.arsdigita.cms.dispatcher.item: [com.arsdigita.cms.contenttypes.Article:{id=41316}] com.arsdigita.dispatcher.RequestContext: com.arsdigita.sitenode.SiteNodeRequestContext@20f1666a com.arsdigita.dispatcher.RequestValue: {com.arsdigita.dispatcher.RequestValue@3fcba676=null} com.arsdigita.kernel.ACSObject:41316: [com.arsdigita.cms.contenttypes.Article:{id=41316}] com.arsdigita.web.BaseApplicationServlet.application_id: 66 com.arsdigita.web.BaseDispatcher.dispatched: true com.arsdigita.web.BaseServlet.request_url: /ccm/content/Article_scaling_2004.jsp -*-*-*- Section: Request summary -*-*-*- Context path: Request URI: /__ccm__/servlet/content-section/Article_scaling_2004.jsp Query string: app-id=66 Method: GET Remote user: null -*-*-*- Section: System properties -*-*-*- ccm.home: /var/ccm-devel/web/root/cms-dev com.arsdigita.util.Assert.enabled: true file.encoding: ISO-8859-1 file.encoding.pkg: sun.io file.separator: / invokedviajava: java.assistive: ON java.awt.fonts: java.awt.graphicsenv: sun.awt.X11GraphicsEnvironment java.awt.printerjob: sun.awt.motif.PSPrinterJob java.class.path: :/opt/apps/oracle/product/9.2.0/jdbc/lib/classes12.zip:/opt/apps/oracle/product/9.2.0/jdbc/lib/classes12.zip:/opt/resin/2.1.8/lib/resin.jar:/opt/resin/2.1.8/lib/jdbc2_0-stdext.jar:/opt/resin/2.1.8/lib/jta-spec1_0_1.jar:/opt/resin/2.1.8/lib/jndi.jar:/opt/resin/2.1.8/lib/dom.jar:/opt/resin/2.1.8/lib/sax.jar:/opt/resin/2.1.8/lib/jaxp.jar:/opt/resin/2.1.8/lib/jmx.jar:/opt/resin/2.1.8/lib/webutil.jar:/opt/resin/2.1.8/lib/jdbc-mysql.jar:/opt/IBMJava2-131/lib/tools.jar:/opt/IBMJava2-131/jre/lib/rt.jar:/opt/resin/2.1.8/lib/jsdk23.jar java.class.version: 46.0 java.compiler: jitc java.ext.dirs: /opt/IBMJava2-131/jre/lib/ext:/opt/IBMJava2-131/lib/ext:/usr/share/ccm-tools/lib/security java.fullversion: J2RE 1.3.1 IBM build cxia32131-20030618 (JIT enabled: jitc) java.home: /opt/IBMJava2-131/jre java.io.tmpdir: /tmp java.library.path: /opt/IBMJava2-131/jre/bin:/opt/IBMJava2-131/jre/bin/classic:/opt/apps/oracle/product/9.2.0/lib:/opt/apps/oracle/product/9.2.0/ctx/lib:/usr/lib:/usr/local/lib:/opt/apps/oracle/product/9.2.0/JRE/lib/linux/native_threads:/opt/resin/2.1.8/libexec:/usr/lib java.naming.factory.initial: com.caucho.naming.InitialContextFactoryImpl java.naming.factory.url.pkgs: com.caucho.naming java.runtime.name: Java(TM) 2 Runtime Environment, Standard Edition java.runtime.version: 1.3.1 java.specification.name: Java Platform API Specification java.specification.vendor: Sun Microsystems Inc. java.specification.version: 1.3 java.vendor: IBM Corporation java.vendor.url: http://www.ibm.com/ java.vendor.url.bug: java.version: 1.3.1 java.vm.info: J2RE 1.3.1 IBM build cxia32131-20030618 (JIT enabled: jitc) java.vm.name: Classic VM java.vm.specification.name: Java Virtual Machine Specification java.vm.specification.vendor: Sun Microsystems Inc. java.vm.specification.version: 1.0 java.vm.vendor: IBM Corporation java.vm.version: 1.3.1 javax.xml.parsers.DocumentBuilderFactory: org.apache.xerces.jaxp.DocumentBuilderFactoryImpl javax.xml.parsers.SAXParserFactory: org.apache.xerces.jaxp.SAXParserFactoryImpl javax.xml.transform.TransformerFactory: com.icl.saxon.TransformerFactoryImpl line.separator: log4j.configuration: file:///var/ccm-devel/web/root/cms-dev/conf/log4j.properties org.xml.sax.driver: com.caucho.xml.Xml os.arch: x86 os.name: Linux os.version: 2.4.9-e.27smp path.separator: : resin.home: /opt/resin/2.1.8 sun.boot.class.path: /opt/IBMJava2-131/jre/lib/rt.jar:/opt/IBMJava2-131/jre/lib/i18n.jar:/opt/IBMJava2-131/jre/classes sun.boot.library.path: /opt/IBMJava2-131/jre/bin sun.io.unicode.encoding: UnicodeLittle user.dir: /var/local/resin/2.1.8 user.home: /root user.language: en user.name: root user.region: US user.timezone: America/New_York -*-*-*-*-*- End Error Report -*-*-*-*-*-
Vaguely reminiscent of this thread: https://www.redhat.com/archives/redhat-ccm-list/2003-December/msg00003.html
com.arsdigita.bebop.jsp.DefinePage is lacking any thread safety whatsoever. We fixed this in London 5.2 & I was sure we had the fix merged with mainline, but evidently not. p4 31469 needs to be merged with 5.2, 6.0, dev.
Dan, What's the multiplicity of DefinePage instances per .jsp file? If a .jsp file has a single <define:page> tag, does this mean there will be a single DefinePage instance per JVM (webapp context)? One DefinePage instance per worker thread? Or something else entirely? I'm trying to form a clear picture of how this stuff works. In the olden days, the assumption was, you instantiate each page once per webapp context. This was achieved, I think, by having a single instance of BebopMapDispatcher (or some such) per webapp context. Do DefinePage and friends have roughly the same multiplicity implications as BebopMapDispatcher and friends? Pardon my ignorance.
There is usually a single <define:page> per JSP file. I imagine the servlet container will only compile a JSP file once. The problem is that the <define:page> tag (because it is just part of the JSP body) will be *run* on every request - the first time through it actually creates the bebop page, the subsequent times it basically just does the generate XML bit. The race condition is that under load several threads may request a page between the JSP being compiled & the calling on lock() on the bebop page. The fix I did is basically constructs a string that uniquely identifies the JSP file within the servlet. THis is based on DispatcherHelper.getCurrentResourcePath ((HttpServletRequest)pageContext.getRequest()); there is a 1-1 cardinality between this string & constructing a bebop page, so it then takes a lock on this string. This sounds complicated, but it ensures that the global lock is held for the minimum amount of time, so multiple *different* JSP files can be constructed pretty concurrently. Since implemneting this, basically all concurrency errors relating to JSPs have completely dissappeared from our production server logfs, so I wouldn't be inclined to try another approach unless this is proven inaquate in further production testing.
Ah, yes, I remember now http://post-office.corp.redhat.com/archives/ccm-engineering-list/2003-May/msg00107.html
merged on trunk @39143
Closing old tickets