tag:blogger.com,1999:blog-7864604315847802409.post-63313090411747527332008-05-05T18:16:00.001+02:002008-05-06T10:31:10.218+02:00Tomcat is probably <a href="http://blog.springsource.com/main/2007/12/24/is-it-a-tomcat-or-the-elephant-in-the-room/">one of the more widely used pieces of open source infrastructure</a> and it's a great boon to our productivity. Sometimes, though, a deep-rooted problem can sneak up on you and drag you into a sleep-deprived night of frantic web searching.<br /><br />One day, I happened to see our production JVM restart, for no apparent reason. Intrigued, I looked into it a little bit more and realised that since <a href="http://www.javablackbelt.com/NewsletterPreview.wwa?newsletterId=5819858">the release of JBB v3</a> last December, it had been restarting roughly 6 times per month. After an OutOfMemory, <a href="http://blogs.sun.com/alanb/entry/heap_dumps_are_back_with">the JVM dumps its heap into an hprof file</a>, so there were a number of these files on the server, often over 1GB each!<br /><br />I downloaded an hprof file and tried to open it with <a href="https://hat.dev.java.net/">jhat</a>, provided with the default Java 6 distro. jhat creates a website for you to visualise the heap [1]. Unfortunately, I couldn't get jhat to work, probably because <a href="http://blogs.sun.com/alanb/entry/heap_dumps_are_back_with#comment-1174977909000">JavaScript support is not yet implemented in MacOSX's JDK 1.6</a>. I got around this problem by reading the hprof with <a href="http://yourkit.com/">YourKit Java Profiler 7.0</a> and quickly saw that vast amounts of memory were being taken up by several huge char arrays (click on the image for a larger version).<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://www.javablackbelt.com/blog/uploaded_images/yourkit_sc-743134.jpg"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="http://www.javablackbelt.com/blog/uploaded_images/yourkit_sc-743123.jpg" alt="" border="0" /></a><br /><br />As you can see from the screenshot, 115 instances of <a href="http://tomcat.apache.org/tomcat-6.0-doc/api/org/apache/jasper/runtime/BodyContentImpl.html">org.apache.jasper.runtime.BodyContentImpl</a> were holding on to over 560 million bytes worth of objects, keeping them from being garbage-collected and eating up inordinate amounts of RAM. So, what's BodyContentImpl actually doing?<br /><br />Tomcat maintains a pool of PageContext instances, which in turn have an array of BodyContentImpls. Each BodyContentImpl has an array with the text from the tag's body. This array has a default size of 512 that is hard-coded as org.apache.jasper.Constants.DEFAULT_TAG_BUFFER_SIZE. Once the body of a custom tag gets bigger than that, the array grows and is never reset to its original size. With concurrent users, several such arrays might be created, so if a few popular pages use big custom tags, you can end up with a number of gigantic, un-garbage-collectable char arrays. In our case, this happened because in v3 we started decorating large blocks of JSP code with SiteMesh custom tags.<br /><br />One solution would be to set the environment variable org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER to true, so that arrays over 512 bytes are released. Another would be to recompile Tomcat with a tag buffer size more suited to our needs. So both choices are extreme: either you maintain huge arrays for ever, or you're constantly creating and garbage collecting arrays, as 512 bytes is a fairly low limit for custom tags. We settled on giving the JVM more RAM to handle the extra garbage collection.<br /><br />Of course, <a href="https://issues.apache.org/bugzilla/show_bug.cgi?id=37793">others</a> have <a href="http://jira.atlassian.com/browse/JRA-10145">experienced</a> the same problem<a href="https://issues.apache.org/bugzilla/show_bug.cgi?id=37793"></a>. What's the word from the Tomcat camp? Well, Remy Maucherat, Tomcat committer, <a href="https://issues.apache.org/bugzilla/show_bug.cgi?id=37793#c8">considers that</a>: <blockquote>Using sensibly written software helps. It should be obvious reading the API that using large body tags is going to be a huge problem.<br /></blockquote>Please draw your own conclusions.<br /><ol><li><span style="font-size:85%;">By the way, jhat loads the whole hprof file into memory, so you may need to use the -J-mx512m flag if you're having OutOfMemory thrown.</span></li></ol>Aymeric Levauxhttp://www.blogger.com/profile/15554948182907260874noreply@blogger.com