Bug 111500

Summary: server not stable under load
Product: [Retired] Red Hat Enterprise CMS Reporter: Bryan Che <bche>
Component: otherAssignee: Daniel Berrangé <berrange>
Status: CLOSED CURRENTRELEASE QA Contact: Jon Orris <jorris>
Severity: medium Docs Contact:
Priority: medium    
Version: nightlyCC: vnasardinov
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2003-12-09 21:53:37 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 106481    
Attachments:
Description Flags
spawn 100 threads each trying to open a gazillion file handles none

Description Bryan Che 2003-12-04 17:26:55 UTC 
CMS is not stable serving published articles under load.  Have tried a
couple tests so far, and each has run into problems.  On the first
test, I got a bunch of 500 errors, with the web pages returning:

java.lang.NoClassDefFoundError: com/arsdigita/sitenode/ServletErrorReport
at
com.arsdigita.web.BaseServlet.internalService(BaseServlet.java(Compiled
Code))
at com.arsdigita.web.BaseServlet.doGet(BaseServlet.java(Compiled Code))
at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))
at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))
at
com.caucho.server.http.FilterChainServlet.doFilter(FilterChainServlet.java(Compiled
Code))
at com.caucho.server.http.Invocation.service(Invocation.java(Compiled
Code))
at
com.caucho.server.http.HttpRequest.handleRequest(HttpRequest.java(Compiled
Code))
at
com.caucho.server.http.HttpRequest.handleConnection(HttpRequest.java(Compiled
Code))
at com.caucho.server.TcpConnection.run(TcpConnection.java(Compiled Code))
at java.lang.Thread.run(Thread.java(Compiled Code))



On the second test, the server stopped serving the ccm webapp so that
any url under /ccm/ gave a 404.  

In both cases, there weren't any ccm error reports.
Comment 1 Bryan Che 2003-12-04 20:35:00 UTC 
This time, I got error reports.  The server ran stably for around
16,000 page views before these errors started showing up:

-*-*-*-*-*- Begin Error Report -*-*-*-*-*-
-*-*-*- ACS Error Report Code: 127.0.0.1:66b3c3ae:f942e5d5d6 -*-*-*-
-*-*-*- Message 1: com.caucho.jsp.JspLineException: no path to XSL
stylesheet found; try modifying
/WEB-INF/resources/stylesheet-paths.txt -*-*-*-
-*-*-*- Message 2: java.lang.RuntimeException: no path to XSL
stylesheet found; try modifying
/WEB-INF/resources/stylesheet-paths.txt -*-*-*-

-*-*-*- Section: HTTP headers -*-*-*-
Accept: text/html, image/gif, image/jpeg, */*
Accept-Language: en-us
Connection: Keep-Alive
Host: 172.16.65.125
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)

-*-*-*- Section: CCM User -*-*-*-
Party not logged in

-*-*-*- Section: Stack trace -*-*-*-
java.lang.RuntimeException: no path to XSL stylesheet found; try
modifying /WEB-INF/resources/stylesheet-paths.txt
	at
com.arsdigita.templating.PatternStylesheetResolver.resolve(PatternStylesheetResolver.java(Compiled
Code))
	at
com.arsdigita.templating.Templating.getTemplate(Templating.java(Inlined
Compiled Code))
	at
com.arsdigita.bebop.page.PageTransformer.servePage(PageTransformer.java(Compiled
Code))
	at
com.arsdigita.bebop.page.PageTransformer.servePage(PageTransformer.java(Compiled
Code))
	at
com.arsdigita.bebop.jsp.ShowAll.servePageWithPresentationManager(ShowAll.java(Compiled
Code))
	at com.arsdigita.bebop.jsp.ShowAll.doEndTag(ShowAll.java(Inlined
Compiled Code))
	at
_packages._content_22dsection._templates._default._item__jsp._jspService(_item__jsp.java(Compiled
Code))
	at com.caucho.jsp.JavaPage.service(JavaPage.java(Compiled Code))
	at com.caucho.jsp.Page.subservice(Page.java(Compiled Code))
	at
com.caucho.server.http.FilterChainPage.doFilter(FilterChainPage.java(Compiled
Code))
	at com.caucho.server.http.Invocation.service(Invocation.java(Compiled
Code))
	at
com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java(Compiled
Code))
	at
com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java(Compiled
Code))
	at
com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java(Compiled
Code))
	at
com.arsdigita.dispatcher.DispatcherHelper.forwardHelper(DispatcherHelper.java(Compiled
Code))
	at
com.arsdigita.dispatcher.DispatcherHelper.forwardRequestByPath(DispatcherHelper.java(Inlined
Compiled Code))
	at
com.arsdigita.dispatcher.DispatcherHelper.forwardRequestByPath(DispatcherHelper.java(Inlined
Compiled Code))
	at
com.arsdigita.cms.dispatcher.ContentItemDispatcher$1.excurse(ContentItemDispatcher.java(Compiled
Code))
	at
com.arsdigita.cms.CMSExcursion$1.excurse(CMSExcursion.java(Compiled Code))
	at
com.arsdigita.kernel.KernelExcursion.run(KernelExcursion.java(Compiled
Code))
	at com.arsdigita.cms.CMSExcursion.run(CMSExcursion.java(Compiled Code))
	at
com.arsdigita.cms.dispatcher.ContentItemDispatcher.dispatch(ContentItemDispatcher.java(Inlined
Compiled Code))
	at
com.arsdigita.cms.ContentSectionServlet.serveItem(ContentSectionServlet.java(Compiled
Code))
	at
com.arsdigita.cms.ContentSectionServlet.doService(ContentSectionServlet.java(Compiled
Code))
	at
com.arsdigita.web.BaseApplicationServlet$1.excurse(BaseApplicationServlet.java(Compiled
Code))
	at
com.arsdigita.kernel.KernelExcursion.run(KernelExcursion.java(Compiled
Code))
	at
com.arsdigita.web.BaseApplicationServlet.doService(BaseApplicationServlet.java(Compiled
Code))
	at com.arsdigita.web.BaseServlet$1.excurse(BaseServlet.java(Compiled
Code))
	at
com.arsdigita.kernel.KernelExcursion.run(KernelExcursion.java(Compiled
Code))
	at
com.arsdigita.web.BaseServlet.internalService(BaseServlet.java(Compiled
Code))
	at com.arsdigita.web.BaseServlet.doGet(BaseServlet.java(Compiled Code))
	at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled
Code))
	at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled
Code))
	at
com.caucho.server.http.FilterChainServlet.doFilter(FilterChainServlet.java(Compiled
Code))
	at com.caucho.server.http.Invocation.service(Invocation.java(Compiled
Code))
	at
com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java(Compiled
Code))
	at
com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java(Compiled
Code))
	at
com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java(Compiled
Code))
	at
com.arsdigita.web.BaseDispatcher.forward(BaseDispatcher.java(Compiled
Code))
	at
com.arsdigita.web.BaseDispatcher.forward(BaseDispatcher.java(Compiled
Code))
	at
com.arsdigita.web.BaseDispatcher.dispatch(BaseDispatcher.java(Compiled
Code))
	at
com.arsdigita.web.DispatcherServlet.doService(DispatcherServlet.java(Compiled
Code))
	at com.arsdigita.web.BaseServlet$1.excurse(BaseServlet.java(Compiled
Code))
	at
com.arsdigita.kernel.KernelExcursion.run(KernelExcursion.java(Compiled
Code))
	at
com.arsdigita.web.BaseServlet.internalService(BaseServlet.java(Compiled
Code))
	at com.arsdigita.web.BaseServlet.doGet(BaseServlet.java(Compiled Code))
	at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled
Code))
	at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled
Code))
	at
com.caucho.server.http.FilterChainServlet.doFilter(FilterChainServlet.java(Compiled
Code))
	at com.caucho.server.http.Invocation.service(Invocation.java(Compiled
Code))
	at
com.caucho.server.http.HttpRequest.handleRequest(HttpRequest.java(Compiled
Code))
	at
com.caucho.server.http.HttpRequest.handleConnection(HttpRequest.java(Compiled
Code))
	at com.caucho.server.TcpConnection.run(TcpConnection.java:139)
	at java.lang.Thread.run(Thread.java:513)


-*-*-*- Section: Servlet attributes -*-*-*-
com.arsdigita.bebop.PageState: com.arsdigita.bebop.PageState@514a43b3 = {
m_page = com.arsdigita.ui.SimplePage@39d1028d,
m_request = com.caucho.server.http.DispatchRequest@57f242c8,
m_response = com.caucho.server.http.HttpResponse@57be02a0,
m_pageState = com.arsdigita.bebop.FormData@514843b3 = {
m_parameterDataValues = {bbp.5.sel={null, []}, bbp.e={null, []},
g11n.enc={UTF-8, []}, bbp.v={null, []}, bbp.i={null, []}, bbp.s={null,
[]}},
m_formErrors = null,
m_model = com.arsdigita.bebop.FormModel@28ac28d,
m_locale = en_US,
m_isTransformed = true,
m_isValid = true,
m_isSubmission = true
},
m_attributes = null,
,
m_grabbingComponent = null,
m_invisible = {}
}
com.arsdigita.bebop.RequestLocal:
{com.arsdigita.bebop.RequestLocal@39cc428d=null,
com.arsdigita.bebop.Page$1@7215028d=com.arsdigita.bebop.Label@39cf428d}
com.arsdigita.cms.dispatcher.item:
[com.arsdigita.cms.contenttypes.Article:{id=31296}]
com.arsdigita.dispatcher.RequestContext:
com.arsdigita.sitenode.SiteNodeRequestContext@602082c3
com.arsdigita.dispatcher.RequestValue:
{com.arsdigita.dispatcher.RequestValue@42fc29f=null}
com.arsdigita.kernel.ACSObject:31296:
[com.arsdigita.cms.contenttypes.Article:{id=31296}]
com.arsdigita.web.BaseApplicationServlet.application_id: 66
com.arsdigita.web.BaseDispatcher.dispatched: true
com.arsdigita.web.BaseServlet.request_url:
/ccm/content/Article_scaling_1503.jsp
com.arsdigita.xml.Document: <?xml version="1.0"
encoding="UTF-8"?><bebop:page
xmlns:bebop="http://www.arsdigita.com/bebop/1.0" application="content"
class="simplePage"
id="itemPage"><bebop:title>CMS</bebop:title><ui:siteBanner
xmlns:ui="http://www.arsdigita.com/ui/1.0" admin=""
hostname="poweredge1650-000" metadata.tag="bottom"
sitename="poweredge1650-000"/><ui:userBanner
xmlns:ui="http://www.arsdigita.com/ui/1.0" greeting="Welcome "
helpLabel="Help " loginURL="/ccm/register/"
logoutURL="/ccm/register/logout" metadata.tag="top" portalLabel="My
Portal " signoutLabel="Sign out "
workspaceURL="/ccm/pvt/"/><cms:contentPanel
xmlns:cms="http://www.arsdigita.com/cms/1.0" id="itemXML"><cms:item
oid="[com.arsdigita.cms.contenttypes.Article:{id=31296}]"><objectType>com.arsdigita.cms.contenttypes.Article</objectType><type
oid="[com.arsdigita.cms.ContentType:{id=193}]"><label>Article</label><description>An
article type with
image</description></type><name>Article_scaling_1503</name><language>en</language><title>Article_scaling_1503</title><launchDate>Dec
1, 2003 1:51 PM</launchDate><textAsset
oid="[com.arsdigita.cms.TextAsset:{id=31298}]"><content>0 All work and
no play makes Jack a dull boy.  0 All work and no play makes Jack a
dull boy.  0 All work and no play makes Jack a dull boy.  0 All work
and no play makes Jack a dull boy.  0 All work and no play makes Jack
a dull boy.  0 All work and no play makes Jack a dull boy.  0 All work
and no play makes Jack a dull boy.  0 All work and no play makes Jack
a dull boy.  0 All work and no play makes Jack a dull boy.  0 All work
and no play makes Jack a dull boy.  0 All work and no play makes Jack
a dull boy.  0 All work and no play makes Jack a dull boy.  0 All work
and no play makes Jack a dull boy.  0 All work and no play makes Jack
a dull boy.  0 All work and no play makes Jack a dull boy.  0 All work
and no play makes Jack a dull boy.  0 All work and no play makes Jack
a dull boy.  0 All work and no play makes Jack a dull boy.  0 All work
and no play makes Jack a dull boy.  0 All work and no play makes Jack
a dull boy.  </content></textAsset><imageCaptions
oid="[com.arsdigita.cms.ArticleImageAssociation:{id=31299}]"><caption>This
is an image</caption><imageAsset
oid="[com.arsdigita.cms.ReusableImageAsset:{id=31300}]"><id>31300</id><mimeType
oid="[com.arsdigita.cms.MimeType:{mimeType=image/gif}]"><mimeType>image/gif</mimeType><label>GIF
image</label><fileExtension>gif</fileExtension></mimeType><width>121</width><height>45</height></imageAsset></imageCaptions><lead>All
work and no play...</lead></cms:item></cms:contentPanel></bebop:page>

-*-*-*- Section: Request summary -*-*-*-
Context path: 
Request URI: /__ccm__/servlet/content-section/Article_scaling_1503.jsp
Query string: app-id=66
Method: GET
Remote user: null

-*-*-*- Section: System properties -*-*-*-
ccm.home: /var/ccm-devel/web/root/cms-dev
com.arsdigita.util.Assert.enabled: true
file.encoding: ISO-8859-1
file.encoding.pkg: sun.io
file.separator: /
invokedviajava: 
java.assistive: ON
java.awt.fonts: 
java.awt.graphicsenv: sun.awt.X11GraphicsEnvironment
java.awt.printerjob: sun.awt.motif.PSPrinterJob
java.class.path:
:/opt/apps/oracle/product/9.2.0/jdbc/lib/classes12.zip:/opt/apps/oracle/product/9.2.0/jdbc/lib/classes12.zip:/opt/resin/2.1.8/lib/resin.jar:/opt/resin/2.1.8/lib/jdbc2_0-stdext.jar:/opt/resin/2.1.8/lib/jta-spec1_0_1.jar:/opt/resin/2.1.8/lib/jndi.jar:/opt/resin/2.1.8/lib/dom.jar:/opt/resin/2.1.8/lib/sax.jar:/opt/resin/2.1.8/lib/jaxp.jar:/opt/resin/2.1.8/lib/jmx.jar:/opt/resin/2.1.8/lib/webutil.jar:/opt/resin/2.1.8/lib/jdbc-mysql.jar:/opt/IBMJava2-131/lib/tools.jar:/opt/IBMJava2-131/jre/lib/rt.jar:/opt/resin/2.1.8/lib/jsdk23.jar
java.class.version: 46.0
java.compiler: jitc
java.ext.dirs:
/opt/IBMJava2-131/jre/lib/ext:/opt/IBMJava2-131/lib/ext:/usr/share/ccm-tools/lib/security
java.fullversion: J2RE 1.3.1 IBM build cxia32131-20030618 (JIT
enabled: jitc)
java.home: /opt/IBMJava2-131/jre
java.io.tmpdir: /tmp
java.library.path:
/opt/IBMJava2-131/jre/bin:/opt/IBMJava2-131/jre/bin/classic:/opt/apps/oracle/product/9.2.0/lib:/opt/apps/oracle/product/9.2.0/ctx/lib:/usr/lib:/usr/local/lib:/opt/apps/oracle/product/9.2.0/JRE/lib/linux/native_threads:/opt/resin/2.1.8/libexec:/usr/lib
java.naming.factory.initial: com.caucho.naming.InitialContextFactoryImpl
java.naming.factory.url.pkgs: com.caucho.naming
java.runtime.name: Java(TM) 2 Runtime Environment, Standard Edition
java.runtime.version: 1.3.1
java.specification.name: Java Platform API Specification
java.specification.vendor: Sun Microsystems Inc.
java.specification.version: 1.3
java.vendor: IBM Corporation
java.vendor.url: http://www.ibm.com/
java.vendor.url.bug: 
java.version: 1.3.1
java.vm.info: J2RE 1.3.1 IBM build cxia32131-20030618 (JIT enabled: jitc)
java.vm.name: Classic VM
java.vm.specification.name: Java Virtual Machine Specification
java.vm.specification.vendor: Sun Microsystems Inc.
java.vm.specification.version: 1.0
java.vm.vendor: IBM Corporation
java.vm.version: 1.3.1
javax.xml.parsers.DocumentBuilderFactory:
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
javax.xml.parsers.SAXParserFactory:
org.apache.xerces.jaxp.SAXParserFactoryImpl
javax.xml.transform.TransformerFactory:
com.icl.saxon.TransformerFactoryImpl
line.separator: 

log4j.configuration:
file:///var/ccm-devel/web/root/cms-dev/conf/log4j.properties
org.xml.sax.driver: com.caucho.xml.Xml
os.arch: x86
os.name: Linux
os.version: 2.4.9-e.27smp
path.separator: :
resin.home: /opt/resin/2.1.8
sun.boot.class.path:
/opt/IBMJava2-131/jre/lib/rt.jar:/opt/IBMJava2-131/jre/lib/i18n.jar:/opt/IBMJava2-131/jre/classes
sun.boot.library.path: /opt/IBMJava2-131/jre/bin
sun.io.unicode.encoding: UnicodeLittle
user.dir: /var/local/resin/2.1.8
user.home: /root
user.language: en
user.name: root
user.region: US
user.timezone: America/New_York
-*-*-*-*-*- End Error Report -*-*-*-*-*-
Comment 2 Bryan Che 2003-12-08 18:21:37 UTC 
After running the server on RHEL3 with the Sun 1.4.2 JVM, I was able
to get a better stack trace with line numbers:

-*-*-*-*-*- Begin Error Report -*-*-*-*-*-
-*-*-*- ACS Error Report Code: 192.168.37.22:2e1973:f956fa8db9 -*-*-*-
-*-*-*- Message 1: com.caucho.jsp.JspLineException: no path to XSL
stylesheet found; try modifying
/WEB-INF/resources/stylesheet-paths.txt -*-*-*-
-*-*-*- Message 2: java.lang.RuntimeException: no path to XSL
stylesheet found; try modifying
/WEB-INF/resources/stylesheet-paths.txt -*-*-*-

-*-*-*- Section: CCM User -*-*-*-
Party not logged in

-*-*-*- Section: System properties -*-*-*-
ccm.home: /var/ccm-devel/web/root/cms-dev
com.arsdigita.util.Assert.enabled: true
file.encoding: UTF-8
file.encoding.pkg: sun.io
file.separator: /
java.awt.graphicsenv: sun.awt.X11GraphicsEnvironment
java.awt.printerjob: sun.print.PSPrinterJob
java.class.path:
:/opt/apps/oracle/product/9.2.0/jdbc/lib/classes12.zip:/opt/apps/oracle/product/9.2.0/jdbc/lib/classes12.zip:/opt/resin/2.1.8/lib/resin.jar:/opt/resin/2.1.8/lib/jdbc2_0-stdext.jar:/opt/resin/2.1.8/lib/jta-spec1_0_1.jar:/opt/resin/2.1.8/lib/jndi.jar:/opt/resin/2.1.8/lib/dom.jar:/opt/resin/2.1.8/lib/sax.jar:/opt/resin/2.1.8/lib/jaxp.jar:/opt/resin/2.1.8/lib/jmx.jar:/opt/resin/2.1.8/lib/webutil.jar:/opt/resin/2.1.8/lib/jdbc-mysql.jar:/opt/j2sdk1.4.2_02/lib/tools.jar:/opt/j2sdk1.4.2_02/jre/lib/rt.jar:/opt/resin/2.1.8/lib/jsdk23.jar
java.class.version: 48.0
java.endorsed.dirs: /var/local/j2sdk1.4.2_02/jre/lib/endorsed
java.ext.dirs:
/opt/j2sdk1.4.2_02/jre/lib/ext:/opt/j2sdk1.4.2_02/lib/ext:/usr/share/ccm-tools/lib/security
java.home: /var/local/j2sdk1.4.2_02/jre
java.io.tmpdir: /tmp
java.library.path:
/var/local/j2sdk1.4.2_02/jre/lib/i386/client:/var/local/j2sdk1.4.2_02/jre/lib/i386:/var/local/j2sdk1.4.2_02/jre/../lib/i386:/opt/apps/oracle/product/9.2.0/lib:/opt/apps/oracle/product/9.2.0/ctx/lib:/usr/lib:/usr/local/lib:/opt/apps/oracle/product/9.2.0/JRE/lib/linux/native_threads:/opt/resin/2.1.8/libexec
java.naming.factory.initial: com.caucho.naming.InitialContextFactoryImpl
java.naming.factory.url.pkgs: com.caucho.naming
java.runtime.name: Java(TM) 2 Runtime Environment, Standard Edition
java.runtime.version: 1.4.2_02-b03
java.specification.name: Java Platform API Specification
java.specification.vendor: Sun Microsystems Inc.
java.specification.version: 1.4
java.util.prefs.PreferencesFactory:
java.util.prefs.FileSystemPreferencesFactory
java.vendor: Sun Microsystems Inc.
java.vendor.url: http://java.sun.com/
java.vendor.url.bug: http://java.sun.com/cgi-bin/bugreport.cgi
java.version: 1.4.2_02
java.vm.info: mixed mode
java.vm.name: Java HotSpot(TM) Client VM
java.vm.specification.name: Java Virtual Machine Specification
java.vm.specification.vendor: Sun Microsystems Inc.
java.vm.specification.version: 1.0
java.vm.vendor: Sun Microsystems Inc.
java.vm.version: 1.4.2_02-b03
javax.xml.parsers.DocumentBuilderFactory:
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
javax.xml.parsers.SAXParserFactory:
org.apache.xerces.jaxp.SAXParserFactoryImpl
javax.xml.transform.TransformerFactory:
com.icl.saxon.TransformerFactoryImpl
line.separator: 

log4j.configuration:
file:///var/ccm-devel/web/root/cms-dev/conf/log4j.properties
org.xml.sax.driver: com.caucho.xml.Xml
os.arch: i386
os.name: Linux
os.version: 2.4.21-4.ELsmp
path.separator: :
resin.home: /opt/resin/2.1.8
sun.arch.data.model: 32
sun.boot.class.path:
/var/local/j2sdk1.4.2_02/jre/lib/rt.jar:/var/local/j2sdk1.4.2_02/jre/lib/i18n.jar:/var/local/j2sdk1.4.2_02/jre/lib/sunrsasign.jar:/var/local/j2sdk1.4.2_02/jre/lib/jsse.jar:/var/local/j2sdk1.4.2_02/jre/lib/jce.jar:/var/local/j2sdk1.4.2_02/jre/lib/charsets.jar:/var/local/j2sdk1.4.2_02/jre/classes
sun.boot.library.path: /var/local/j2sdk1.4.2_02/jre/lib/i386
sun.cpu.endian: little
sun.cpu.isalist: 
sun.io.unicode.encoding: UnicodeLittle
sun.java2d.fontpath: 
sun.os.patch.level: unknown
user.country: US
user.dir: /var/local/resin/2.1.8
user.home: /root
user.language: en
user.name: root
user.timezone: America/New_York

-*-*-*- Section: Stack trace -*-*-*-
java.lang.RuntimeException: no path to XSL stylesheet found; try
modifying /WEB-INF/resources/stylesheet-paths.txt
	at
com.arsdigita.templating.PatternStylesheetResolver.resolve(PatternStylesheetResolver.java:251)
	at com.arsdigita.templating.Templating.getTemplate(Templating.java:188)
	at
com.arsdigita.bebop.page.PageTransformer.servePage(PageTransformer.java:182)
	at
com.arsdigita.bebop.page.PageTransformer.servePage(PageTransformer.java:143)
	at
com.arsdigita.bebop.jsp.ShowAll.servePageWithPresentationManager(ShowAll.java:116)
	at com.arsdigita.bebop.jsp.ShowAll.doEndTag(ShowAll.java:80)
	at
_packages._content_22dsection._templates._default._item__jsp._jspService(_item__jsp.java:95)
	at com.caucho.jsp.JavaPage.service(JavaPage.java:75)
	at com.caucho.jsp.Page.subservice(Page.java:497)
	at
com.caucho.server.http.FilterChainPage.doFilter(FilterChainPage.java:182)
	at com.caucho.server.http.Invocation.service(Invocation.java:315)
	at
com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java:225)
	at
com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java:106)
	at
com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java:83)
	at
com.arsdigita.dispatcher.DispatcherHelper.forwardHelper(DispatcherHelper.java:193)
	at
com.arsdigita.dispatcher.DispatcherHelper.forwardRequestByPath(DispatcherHelper.java:222)
	at
com.arsdigita.dispatcher.DispatcherHelper.forwardRequestByPath(DispatcherHelper.java:235)
	at
com.arsdigita.cms.dispatcher.ContentItemDispatcher$1.excurse(ContentItemDispatcher.java:111)
	at com.arsdigita.cms.CMSExcursion$1.excurse(CMSExcursion.java:80)
	at com.arsdigita.kernel.KernelExcursion.run(KernelExcursion.java:57)
	at com.arsdigita.cms.CMSExcursion.run(CMSExcursion.java:75)
	at
com.arsdigita.cms.dispatcher.ContentItemDispatcher.dispatch(ContentItemDispatcher.java:115)
	at
com.arsdigita.cms.ContentSectionServlet.serveItem(ContentSectionServlet.java:200)
	at
com.arsdigita.cms.ContentSectionServlet.doService(ContentSectionServlet.java:124)
	at
com.arsdigita.web.BaseApplicationServlet$1.excurse(BaseApplicationServlet.java:111)
	at com.arsdigita.kernel.KernelExcursion.run(KernelExcursion.java:57)
	at
com.arsdigita.web.BaseApplicationServlet.doService(BaseApplicationServlet.java:105)
	at com.arsdigita.web.BaseServlet$1.excurse(BaseServlet.java:167)
	at com.arsdigita.kernel.KernelExcursion.run(KernelExcursion.java:57)
	at com.arsdigita.web.BaseServlet.internalService(BaseServlet.java:142)
	at com.arsdigita.web.BaseServlet.doGet(BaseServlet.java:252)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:126)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:103)
	at
com.caucho.server.http.FilterChainServlet.doFilter(FilterChainServlet.java:96)
	at com.caucho.server.http.Invocation.service(Invocation.java:315)
	at
com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java:225)
	at
com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java:106)
	at
com.caucho.server.http.QRequestDispatcher.forward(QRequestDispatcher.java:83)
	at com.arsdigita.web.BaseDispatcher.forward(BaseDispatcher.java:227)
	at com.arsdigita.web.BaseDispatcher.forward(BaseDispatcher.java:211)
	at com.arsdigita.web.BaseDispatcher.dispatch(BaseDispatcher.java:135)
	at
com.arsdigita.web.DispatcherServlet.doService(DispatcherServlet.java:123)
	at com.arsdigita.web.BaseServlet$1.excurse(BaseServlet.java:167)
	at com.arsdigita.kernel.KernelExcursion.run(KernelExcursion.java:57)
	at com.arsdigita.web.BaseServlet.internalService(BaseServlet.java:142)
	at com.arsdigita.web.BaseServlet.doGet(BaseServlet.java:252)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:126)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:103)
	at
com.caucho.server.http.FilterChainServlet.doFilter(FilterChainServlet.java:96)
	at com.caucho.server.http.Invocation.service(Invocation.java:315)
	at com.caucho.server.http.HttpRequest.handleRequest(HttpRequest.java:246)
	at
com.caucho.server.http.HttpRequest.handleConnection(HttpRequest.java:163)
	at com.caucho.server.TcpConnection.run(TcpConnection.java:139)
	at java.lang.Thread.run(Thread.java:534)


-*-*-*- Section: HTTP headers -*-*-*-
Accept: text/html, image/gif, image/jpeg, */*
Accept-Language: en-us
Connection: Keep-Alive
Host: 172.16.65.125
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)

-*-*-*- Section: Servlet attributes -*-*-*-
com.arsdigita.bebop.PageState: com.arsdigita.bebop.PageState@1783ca4 = {
m_page = com.arsdigita.ui.SimplePage@c9d6c3,
m_request = com.caucho.server.http.DispatchRequest@d561e6,
m_response = com.caucho.server.http.HttpResponse@10e5d1c,
m_pageState = com.arsdigita.bebop.FormData@305ea = {
m_parameterDataValues = {g11n.enc={UTF-8, []}, bbp.5.sel={null, []},
bbp.i={null, []}, bbp.v={null, []}, bbp.e={null, []}, bbp.s={null, []}},
m_formErrors = null,
m_model = com.arsdigita.bebop.FormModel@16dd75b,
m_locale = en_US,
m_isTransformed = true,
m_isValid = true,
m_isSubmission = true
},
m_attributes = null,
,
m_grabbingComponent = null,
m_invisible = {}
}
com.arsdigita.bebop.RequestLocal:
{com.arsdigita.bebop.Page$1@130cd4a=com.arsdigita.bebop.Label@55a154,
com.arsdigita.bebop.RequestLocal@f5d08e=null}
com.arsdigita.cms.dispatcher.item:
[com.arsdigita.cms.contenttypes.Article:{id=31299}]
com.arsdigita.dispatcher.RequestContext:
com.arsdigita.sitenode.SiteNodeRequestContext@c3721a
com.arsdigita.dispatcher.RequestValue:
{com.arsdigita.dispatcher.RequestValue@77b77b=null}
com.arsdigita.kernel.ACSObject:31299:
[com.arsdigita.cms.contenttypes.Article:{id=31299}]
com.arsdigita.web.BaseApplicationServlet.application_id: 66
com.arsdigita.web.BaseDispatcher.dispatched: true
com.arsdigita.web.BaseServlet.request_url:
/ccm/content/Article_scaling_1503.jsp
com.arsdigita.xml.Document: <?xml version="1.0"
encoding="UTF-8"?><bebop:page
xmlns:bebop="http://www.arsdigita.com/bebop/1.0" application="content"
class="simplePage"
id="itemPage"><bebop:title>CMS</bebop:title><ui:userBanner
xmlns:ui="http://www.arsdigita.com/ui/1.0" greeting="Welcome "
helpLabel="Help " loginURL="/ccm/register/"
logoutURL="/ccm/register/logout" metadata.tag="top" portalLabel="My
Portal " signoutLabel="Sign out "
workspaceURL="/ccm/pvt/"/><ui:siteBanner
xmlns:ui="http://www.arsdigita.com/ui/1.0" admin=""
hostname="poweredge1650-001" metadata.tag="bottom"
sitename="poweredge1650-001"/><cms:contentPanel
xmlns:cms="http://www.arsdigita.com/cms/1.0" id="itemXML"><cms:item
oid="[com.arsdigita.cms.contenttypes.Article:{id=31299}]"><objectType>com.arsdigita.cms.contenttypes.Article</objectType><type
oid="[com.arsdigita.cms.ContentType:{id=193}]"><label>Article</label><description>An
article type with
image</description></type><name>Article_scaling_1503</name><language>en</language><title>Article_scaling_1503</title><launchDate>Dec
5, 2003 6:13 PM</launchDate><textAsset
oid="[com.arsdigita.cms.TextAsset:{id=31301}]"><content>0 All work and
no play makes Jack a dull boy.  0 All work and no play makes Jack a
dull boy.  0 All work and no play makes Jack a dull boy.  0 All work
and no play makes Jack a dull boy.  0 All work and no play makes Jack
a dull boy.  0 All work and no play makes Jack a dull boy.  0 All work
and no play makes Jack a dull boy.  0 All work and no play makes Jack
a dull boy.  0 All work and no play makes Jack a dull boy.  0 All work
and no play makes Jack a dull boy.  0 All work and no play makes Jack
a dull boy.  0 All work and no play makes Jack a dull boy.  0 All work
and no play makes Jack a dull boy.  0 All work and no play makes Jack
a dull boy.  0 All work and no play makes Jack a dull boy.  0 All work
and no play makes Jack a dull boy.  0 All work and no play makes Jack
a dull boy.  0 All work and no play makes Jack a dull boy.  0 All work
and no play makes Jack a dull boy.  0 All work and no play makes Jack
a dull boy.  </content></textAsset><imageCaptions
oid="[com.arsdigita.cms.ArticleImageAssociation:{id=31302}]"><caption>This
is an image</caption><imageAsset
oid="[com.arsdigita.cms.ReusableImageAsset:{id=31303}]"><id>31303</id><mimeType
oid="[com.arsdigita.cms.MimeType:{mimeType=image/gif}]"><mimeType>image/gif</mimeType><label>GIF
image</label><fileExtension>gif</fileExtension></mimeType><width>121</width><height>45</height></imageAsset></imageCaptions><lead>All
work and no play...</lead></cms:item></cms:contentPanel></bebop:page>

-*-*-*- Section: Request summary -*-*-*-
Context path: 
Request URI: /__ccm__/servlet/content-section/Article_scaling_1503.jsp
Query string: app-id=66
Method: GET
Remote user: null
-*-*-*-*-*- End Error Report -*-*-*-*-*-
Comment 3 Bryan Che 2003-12-08 20:59:30 UTC 
After doing more investigation, I have found that the server is
failing in PatternStyleSheetResolver.java line 237.  Under load, the
method call xfrmedURL.openStream() is returning null even though it
shouldn't.  I added some logging, and the xfrmedURL is a valid file in
the filesystem.   Also, when not under load, the
xfrmedURL.openStream() check works fine.

Because this check is failing, PatternStyleSheetResolver.resolve()
never finds a stylesheet and gets to line 251, where it throws an
exception
Comment 4 Richard Li 2003-12-08 21:03:40 UTC 
Dan: can you take a look?
Comment 5 Vadim Nasardinov 2003-12-08 22:02:33 UTC 
We are leaking filehandles in the following code:

$ p4 print //core-platform/dev/src/com/arsdigita/templating/PatternStylesheetResolver.java#10 |
 grep -v //core | nl -ba - | head -n 248 | tail -n 14 
   235	            try {
   236	                if (xfrmedURL != null &&
   237	                    xfrmedURL.openStream() != null) {
   238	                    return origURL;
   239	                }
   240	            } catch (FileNotFoundException ex) {
   241	                if (s_log.isDebugEnabled()) {
   242	                    s_log.debug("File not found " + resource, ex);
   243	                }
   244	                // fall through & try next pattern
   245	            } catch (IOException ex) {
   246	                throw new UncheckedWrapperException(
   247	                    "cannot open stream " + resource, ex);
   248	            }


xfrmedURL.openStream() opens a file handle that is never closed in our
code.  It is eventually auto-closed when the offending objects are
GC-ed.  This may not be the cause of this particular bug, but it is
certainly a bug that merits a prompt fix.
Comment 6 Daniel Berrangé 2003-12-09 14:23:37 UTC 
I've examined the code for java.net.URL, java.net.URLConnection and
sun.www.net.protocols.file.Handler/FileURLConnection and (in the JDK
that I'm using - IBM 1.3.1) there does not appear to be any code path
that could lead to openStream returning null for file:/// URLs. This
leads me to two thoughts:

  * Your server is still checking http:// URLs (but your change 38514
should have fixed that bug)
  * Your JDK has a different impl of FileURLConnection that does have
a code path returning null.

In any case, I think the first cause of action is to fix the file
handle leak Vadim identified & re-try with this fix & p4 38514. I'll
commit a fix for this shortly.

NB, (aside from the code for processing file:// urls) java.net.URL &
its associated classes have a very nasty habit of catching & droppping
exceptions & then returning nulls :-( So it is conceivable that a 'out
of file handles' exception is getting swallowed somewhere.
Comment 7 Vadim Nasardinov 2003-12-09 14:37:12 UTC 
Created attachment 96420 [details]
spawn 100 threads each trying to open a gazillion file handles

With Sun's jdk1.3.1_09, I get the following exception when running the
attached test:

java.io.FileNotFoundException: /tmp/scratch.txt (Too many open files)
	at java.io.FileInputStream.open(Native Method)
	at java.io.FileInputStream.<init>(FileInputStream.java:91)
	at java.io.FileInputStream.<init>(FileInputStream.java:54)
	at
sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:64)
	at
sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:128)

	at java.net.URL.openStream(URL.java:793)
	at UrlOpenStream.run(UrlOpenStream.java:23)
	at java.lang.Thread.run(Thread.java:479)


Could be educational to check what Bryan's JRE makes of it.
Comment 8 Bryan Che 2003-12-09 15:06:32 UTC 
I see the same exception as Vadim with the Sun 1.4.2_02 JRE
Comment 9 Daniel Berrangé 2003-12-09 18:18:07 UTC 
Resource leak fixed in p4 38631
Comment 10 Bryan Che 2003-12-09 21:53:37 UTC 
Tested with 38631 to over 100,000 page views with no problems.