情况:Web应用部署在tomcat 下,突然cpu 100%,查看是java进程引起的!
项目构架:Struts2 +spring +hibrenate .
以下是用jstack 打出的堆栈信息!求大虾指教。
个人感觉是tomcat 使用了非线程安全的java.text.SimpleDateFormat引起的死锁。
也可能是服务网络瓶颈,引起的内存溢出!!
Heap
def new generation total 52032K, used 75K [0x00000000eb400000, 0x00000000eec70000, 0x00000000f0750000)
1
eden space 46272K, 0% used [0x00000000eb400000, 0x00000000eb412e68, 0x00000000ee130000)
from space 5760K, 0% used [0x00000000ee130000, 0x00000000ee130000, 0x00000000ee6d0000)
to space 5760K, 0% used [0x00000000ee6d0000, 0x00000000ee6d0000, 0x00000000eec70000)
tenured generation total 115372K, used 67519K [0x00000000f0750000, 0x00000000f77fb000, 0x00000000fae00000)
the space 115372K, 58% used [0x00000000f0750000, 0x00000000f493fef0, 0x00000000f4940000, 0x00000000f77fb000)
compacting perm gen total 83968K, used 83967K [0x00000000fae00000, 0x0000000100000000, 0x0000000100000000)
the space 83968K, 99% used [0x00000000fae00000, 0x00000000ffffff78, 0x0000000100000000, 0x0000000100000000)
No shared spaces configured.
2014-10-15 18:35:10
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.60-b09 mixed mode):
"http-bio-8080-exec-19" daemon prio=10 tid=0x00007f5990133800 nid=0x2fac waiting on condition [0x00007f59b433c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f36e4b48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"http-bio-8080-exec-18" daemon prio=10 tid=0x00007f5990133000 nid=0x2fab waiting on condition [0x00007f59b443d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f36e4b48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"http-bio-8080-exec-17" daemon prio=10 tid=0x00007f5990002800 nid=0x2faa waiting on condition [0x00007f59b4c53000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f36e4b48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"http-bio-8080-exec-16" daemon prio=10 tid=0x00007f5990002000 nid=0x2fa9 waiting on condition [0x00007f59b4d54000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f36e4b48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"http-bio-8080-exec-15" daemon prio=10 tid=0x00007f5990008800 nid=0x2fa8 waiting on condition [0x00007f59b454c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f36e4b48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"http-bio-8080-exec-10" daemon prio=10 tid=0x00007f599094b000 nid=0x1b2b runnable [0x00007f59b494c000]
java.lang.Thread.State: RUNNABLE
at freemarker.core.FMParserTokenManager.jjMoveNfa_0(FMParserTokenManager.java:3138)
at freemarker.core.FMParserTokenManager.jjMoveStringLiteralDfa0_0(FMParserTokenManager.java:595)
at freemarker.core.FMParserTokenManager.getNextToken(FMParserTokenManager.java:6366)
at freemarker.core.FMParser.jj_ntk(FMParser.java:4618)
at freemarker.core.FMParser.OptionalBlock(FMParser.java:2746)
at freemarker.core.FMParser.If(FMParser.java:1202)
at freemarker.core.FMParser.FreemarkerDirective(FMParser.java:2377)
at freemarker.core.FMParser.Content(FMParser.java:2623)
at freemarker.core.FMParser.OptionalBlock(FMParser.java:2791)
at freemarker.core.FMParser.If(FMParser.java:1177)
at freemarker.core.FMParser.FreemarkerDirective(FMParser.java:2377)
at freemarker.core.FMParser.Content(FMParser.java:2623)
at freemarker.core.FMParser.OptionalBlock(FMParser.java:2791)
at freemarker.core.FMParser.UnifiedMacroTransform(FMParser.java:1977)
at freemarker.core.FMParser.FreemarkerDirective(FMParser.java:2404)
at freemarker.core.FMParser.Content(FMParser.java:2623)
at freemarker.core.FMParser.OptionalBlock(FMParser.java:2791)
at freemarker.core.FMParser.Root(FMParser.java:2963)
at freemarker.template.Template.<init>(Template.java:171)
at freemarker.cache.TemplateCache.loadTemplate(TemplateCache.java:447)
at freemarker.cache.TemplateCache.getTemplate(TemplateCache.java:360)
at freemarker.cache.TemplateCache.getTemplate(TemplateCache.java:235)
at freemarker.template.Configuration.getTemplate(Configuration.java:578)
at freemarker.template.Configuration.getTemplate(Configuration.java:543)
at org.apache.struts2.components.template.FreemarkerTemplateEngine.renderTemplate(FreemarkerTemplateEngine.java:97)
at org.apache.struts2.components.UIBean.mergeTemplate(UIBean.java:593)
at org.apache.struts2.components.UIBean.end(UIBean.java:547)
at org.apache.struts2.views.jsp.ComponentTagSupport.doEndTag(ComponentTagSupport.java:42)
at org.apache.jsp.WEB_002dINF.lottery.order.list_jsp._jspx_meth_s_005fselect_005f2(list_jsp.java:560)
at org.apache.jsp.WEB_002dINF.lottery.order.list_jsp._jspService(list_jsp.java:323)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:411)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338)
at org.apache.struts2.dispatcher.ServletDispatcherResult.doExecute(ServletDispatcherResult.java:164)
at org.apache.struts2.dispatcher.StrutsResultSupport.execute(StrutsResultSupport.java:186)
at com.opensymphony.xwork2.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:371)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:275)
at org.apache.struts2.interceptor.DeprecationInterceptor.intercept(DeprecationInterceptor.java:41)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:256)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:167)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:265)
at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:68)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:138)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:254)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:254)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:191)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:73)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:91)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:252)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:100)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:141)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:145)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:139)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:193)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:189)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:54)
at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:562)
at org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:77)
at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:99)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.cx.tm.service.SecurityFilter.doFilter(SecurityFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
- locked <0x00000000f4901e80> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"http-bio-8080-exec-8" daemon prio=10 tid=0x00007f5994008800 nid=0x1b29 waiting on condition [0x00007f59b4b52000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f36e4b48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"http-bio-8080-exec-5" daemon prio=10 tid=0x00007f5990406000 nid=0x1b1f waiting on condition [0x00007f59b4f56000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f36e4b48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"http-bio-8080-exec-4" daemon prio=10 tid=0x00007f59905a2800 nid=0x1ace waiting on condition [0x00007f59b5057000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f36e4b48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"http-bio-8080-exec-3" daemon prio=10 tid=0x00007f5990a34800 nid=0x1acd waiting on condition [0x00007f59b5157000]
java.lang.Thread.State: RUNNABLE
at sun.util.resources.OpenListResourceBundle.loadLookup(OpenListResourceBundle.java:122)
at sun.util.resources.OpenListResourceBundle.loadLookupTablesIfNecessary(OpenListResourceBundle.java:113)
at sun.util.resources.OpenListResourceBundle.handleGetObject(OpenListResourceBundle.java:74)
at sun.util.resources.TimeZoneNamesBundle.handleGetObject(TimeZoneNamesBundle.java:75)
at java.util.ResourceBundle.getObject(ResourceBundle.java:389)
at java.util.ResourceBundle.getStringArray(ResourceBundle.java:372)
at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:118)
at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:99)
at java.util.TimeZone.getDisplayNames(TimeZone.java:458)
at java.util.TimeZone.getDisplayName(TimeZone.java:409)
at java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:1213)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:978)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:948)
at java.text.DateFormat.format(DateFormat.java:336)
at org.apache.tomcat.util.http.FastHttpDateFormat.getCurrentDate(FastHttpDateFormat.java:114)
- locked <0x00000000f33cc0c0> (a java.text.SimpleDateFormat)
at org.apache.coyote.http11.AbstractHttp11Processor.prepareResponse(AbstractHttp11Processor.java:1501)
at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:773)
at org.apache.coyote.Response.action(Response.java:174)
at org.apache.coyote.Response.sendHeaders(Response.java:365)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:352)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:309)
at org.apache.catalina.connector.Response.finishResponse(Response.java:510)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:435)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
- locked <0x00000000f4928230> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"ajp-bio-8009-AsyncTimeout" daemon prio=10 tid=0x00007f59b8209800 nid=0x1abe waiting on condition [0x00007f59b5259000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run(JIoEndpoint.java:151)
at java.lang.Thread.run(Thread.java:745)
"ajp-bio-8009-Acceptor-0" daemon prio=10 tid=0x00007f59b8208800 nid=0x1abd runnable [0x00007f59b535a000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:219)
at java.lang.Thread.run(Thread.java:745)
"http-bio-8080-AsyncTimeout" daemon prio=10 tid=0x00007f59b8011800 nid=0x1abc waiting on condition [0x00007f59b545b000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run(JIoEndpoint.java:151)
at java.lang.Thread.run(Thread.java:745)
"http-bio-8080-Acceptor-0" daemon prio=10 tid=0x00007f59b800f800 nid=0x1abb runnable [0x00007f59b555c000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:219)
at java.lang.Thread.run(Thread.java:745)
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00007f59b800d000 nid=0x1aba waiting on condition [0x00007f59b6804000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1508)
at java.lang.Thread.run(Thread.java:745)
"Timer-0" daemon prio=10 tid=0x00007f599c217800 nid=0x1ab3 in Object.wait() [0x00007f59b565d000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f2bb4b28> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000f2bb4b28> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"org.springframework.scheduling.quartz.SchedulerFactoryBean#0_QuartzSchedulerThread" prio=10 tid=0x00007f599c216000 nid=0x1ab2 in Object.wait() [0x00007f59b575e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:411)
- locked <0x00000000f2bb4c20> (a java.lang.Object)
"org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10" prio=10 tid=0x00007f599d6fc000 nid=0x1ab1 in Object.wait() [0x00007f59b585f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0x00000000f2bb4c30> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9" prio=10 tid=0x00007f599d6fa000 nid=0x1ab0 in Object.wait() [0x00007f59b5960000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0x00000000f2bb4ca8> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-8" prio=10 tid=0x00007f599c229000 nid=0x1aaf in Object.wait() [0x00007f59b5a61000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0x00000000f2bb4d20> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-7" prio=10 tid=0x00007f599c227000 nid=0x1aae in Object.wait() [0x00007f59b5b62000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0x00000000f2bb4d98> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6" prio=10 tid=0x00007f599c225000 nid=0x1aad in Object.wait() [0x00007f59b5c63000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0x00000000f2bb4e10> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-5" prio=10 tid=0x00007f599c223000 nid=0x1aac in Object.wait() [0x00007f59b5d64000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0x00000000f2bb4e88> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-4" prio=10 tid=0x00007f599dd9b800 nid=0x1aab in Object.wait() [0x00007f59b5e65000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0x00000000f2bb4f00> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-3" prio=10 tid=0x00007f599dd99800 nid=0x1aaa in Object.wait() [0x00007f59b6099000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0x00000000f2bb4f78> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2" prio=10 tid=0x00007f599dd9e000 nid=0x1aa9 in Object.wait() [0x00007f59b619a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0x00000000f2bb4ff0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1" prio=10 tid=0x00007f599dd9c800 nid=0x1aa8 in Object.wait() [0x00007f59b629b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0x00000000f2bb5068> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
"Abandoned connection cleanup thread" daemon prio=10 tid=0x00007f599c0a0000 nid=0x1aa7 in Object.wait() [0x00007f59b639c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000000f227a160> (a java.lang.ref.ReferenceQueue$Lock)
at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
"MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00007f599dfc1000 nid=0x1a9a in Object.wait() [0x00007f59b6517000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f1308f88> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000f1308f88> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Thread-2" daemon prio=10 tid=0x00007f599d13c800 nid=0x1a92 runnable [0x00007f59b6618000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPoll.epollWait(Native Method)
at sun.nio.ch.EPollPort$EventHandlerTask.poll(EPollPort.java:194)
at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:268)
at java.lang.Thread.run(Thread.java:745)
"GC Daemon" daemon prio=10 tid=0x00007f59b81ff000 nid=0x1a85 in Object.wait() [0x00007f59b6f75000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0920428> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:117)
- locked <0x00000000f0920428> (a sun.misc.GC$LatencyLock)
"Service Thread" daemon prio=10 tid=0x00007f59b808b000 nid=0x1a7d runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007f59b8088800 nid=0x1a7c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007f59b8086000 nid=0x1a7b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f59b8084800 nid=0x1a7a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007f59b8065000 nid=0x1a79 in Object.wait() [0x00007f59b7efd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000000f07baac8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" daemon prio=10 tid=0x00007f59b8063000 nid=0x1a78 in Object.wait() [0x00007f59b7ffe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000000f07bab70> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x00007f59b8009000 nid=0x1a76 runnable [0x00007f59c1c5a000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:451)
at org.apache.catalina.startup.Catalina.await(Catalina.java:777)
at org.apache.catalina.startup.Catalina.start(Catalina.java:723)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:321)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:455)
"VM Thread" prio=10 tid=0x00007f59b805f000 nid=0x1a77 runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f59b8096000 nid=0x1a7e waiting on condition
JNI global references: 146
好长的异常信息.
看是不是有某个死循环,不断新建了进程什么的,可以把可疑代码注释了来排查。
代码问题。代码里面有死锁。死锁导致逻辑线程池线程全部被占用。消息无法得到处理堆积起来,内存满了,cpu也满了。
FMParser是个什么玩意,这里面的代码涉及到锁的检查一下。