Our Glassfish 3.12 went to 100% last night, which locked the users out of the system (there are only two). I did a dump of the Glassfish log (below) and there are ton of locks, although none of them used more than 1 second of CPU. 100% CPU usage seems to be a common problem with Glassfish 3.1.2, but I haven't found a clear solution yet. Any ideas welcome.
./asadmin generate-jvm-report --type=thread --user admin Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 12 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0 Total CPU time for this thread: 0 seconds 79,691,017 nanoseconds. User-level CPU time for this thread: 0 seconds 79,691,017 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] -------------------------------------------------------------------------------- Thread Execution Information: ----------------------- Thread "Thread-12" thread-id: 24 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.atomic.AtomicInteger@4cc6afaa at: java.lang.Object.wait(Native Method) at: com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119) at: com.sun.grizzly.Controller.start(Controller.java:869) at: com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285) at: com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123) at: com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212) at: com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142) at: com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342) Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 0 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 946 Total CPU time for this thread: 0 seconds 17,073,572 nanoseconds. User-level CPU time for this thread: 0 seconds 17,073,572 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] -------------------------------------------------------------------------------- Thread Execution Information: ----------------------- Thread "Grizzly-kernel-thread(1)" thread-id: 22 thread-state: RUNNABLE Running in native at: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at: sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at: sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at: com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513) at: com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190) at: com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132) at: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at: java.lang.Thread.run(Thread.java:745) Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 1 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0 Total CPU time for this thread: 0 seconds 55,218,828 nanoseconds. User-level CPU time for this thread: 0 seconds 55,218,828 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] -------------------------------------------------------------------------------- Thread Execution Information: ----------------------- Thread "Grizzly-kernel-thread(1)" thread-id: 20 thread-state: RUNNABLE Running in native at: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at: sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at: sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at: com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513) at: com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190) at: com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132) at: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at: java.lang.Thread.run(Thread.java:745) Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 3 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0 Total CPU time for this thread: 0 seconds 60,298,518 nanoseconds. User-level CPU time for this thread: 0 seconds 60,298,518 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] -------------------------------------------------------------------------------- Thread Execution Information: ----------------------- Thread "Thread-7" thread-id: 19 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.atomic.AtomicInteger@836e947 at: java.lang.Object.wait(Native Method) at: com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119) at: com.sun.grizzly.Controller.start(Controller.java:869) at: com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285) at: com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123) at: com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212) at: com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142) at: com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342) Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 0 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 946 Total CPU time for this thread: 0 seconds 21,716,796 nanoseconds. User-level CPU time for this thread: 0 seconds 21,716,796 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] -------------------------------------------------------------------------------- Thread Execution Information: ----------------------- Thread "Thread-4" thread-id: 16 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.atomic.AtomicInteger@4d0d3887 at: java.lang.Object.wait(Native Method) at: com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119) at: com.sun.grizzly.Controller.start(Controller.java:869) at: com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285) at: com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123) at: com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212) at: com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142) at: com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342) Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 2 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 946 Total CPU time for this thread: 0 seconds 58,339,427 nanoseconds. User-level CPU time for this thread: 0 seconds 58,339,427 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] -------------------------------------------------------------------------------- Thread Execution Information: ----------------------- Thread "Thread-2" thread-id: 14 thread-state: WAITING Waiting on lock: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1efa4e96 at: sun.misc.Unsafe.park(Native Method) at: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at: java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374) at: com.sun.enterprise.server.logging.GFFileHandler.log(GFFileHandler.java:665) at: com.sun.enterprise.server.logging.GFFileHandler$1.run(GFFileHandler.java:179) Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 0 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 78 Total CPU time for this thread: 0 seconds 68,044,526 nanoseconds. User-level CPU time for this thread: 0 seconds 68,044,526 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] -------------------------------------------------------------------------------- Thread Execution Information: ----------------------- Thread "pool-1-thread-1" thread-id: 13 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@433d69bd at: sun.misc.Unsafe.park(Native Method) at: java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at: java.lang.Thread.run(Thread.java:745) Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 0 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 1,892 Total CPU time for this thread: 0 seconds 210,072,724 nanoseconds. User-level CPU time for this thread: 0 seconds 210,072,724 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] -------------------------------------------------------------------------------- Thread Execution Information: ----------------------- Thread "FelixStartLevel" thread-id: 12 thread-state: WAITING Waiting on lock: java.util.ArrayList@74e919c7 at: java.lang.Object.wait(Native Method) at: java.lang.Object.wait(Object.java:503) at: org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:272) at: java.lang.Thread.run(Thread.java:745) Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 42 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 42 Total CPU time for this thread: 0 seconds 688,900,829 nanoseconds. User-level CPU time for this thread: 0 seconds 688,900,829 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] -------------------------------------------------------------------------------- Thread Execution Information: ----------------------- Thread "FelixDispatchQueue" thread-id: 11 thread-state: WAITING Waiting on lock: java.util.ArrayList@739b93c7 at: java.lang.Object.wait(Native Method) at: java.lang.Object.wait(Object.java:503) at: org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:1063) at: org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:54) at: org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:101) at: java.lang.Thread.run(Thread.java:745) Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 232 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 217 Total CPU time for this thread: 0 seconds 117,033,893 nanoseconds. User-level CPU time for this thread: 0 seconds 117,033,893 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] -------------------------------------------------------------------------------- Thread Execution Information: ----------------------- Thread "Signal Dispatcher" thread-id: 4 thread-state: RUNNABLE Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 0 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0 Total CPU time for this thread: 0 seconds 70,212 nanoseconds. User-level CPU time for this thread: 0 seconds 70,212 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] -------------------------------------------------------------------------------- Thread Execution Information: ----------------------- Thread "Finalizer" thread-id: 3 thread-state: WAITING Waiting on lock: java.lang.ref.ReferenceQueue$Lock@6b03291c at: java.lang.Object.wait(Native Method) at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at: java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 20 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 19 Total CPU time for this thread: 0 seconds 52,757,964 nanoseconds. User-level CPU time for this thread: 0 seconds 52,757,964 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] -------------------------------------------------------------------------------- Thread Execution Information: ----------------------- Thread "Reference Handler" thread-id: 2 thread-state: WAITING Waiting on lock: java.lang.ref.Reference$Lock@1c802427 at: java.lang.Object.wait(Native Method) at: java.lang.Object.wait(Object.java:503) at: java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) Thread Synchronization Statistics: ----------------------- Number of times this thread was blocked (to enter/reenter a Monitor): 18 Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 18 Total CPU time for this thread: 0 seconds 39,254,957 nanoseconds. User-level CPU time for this thread: 0 seconds 39,254,957 nanoseconds. Object Monitors currently held or requested by this thread: [] Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: [] No deadlock found 2 Answers
Answers 1
Something you might find useful:
glassfish full gc once an hour
https://docs.oracle.com/cd/E18930_01/html/821-2431/abeic.html
Answers 2
I found in the log an OutOfMemory our due to garbage collection. I increased some of the JVM parameters in the Glassfish server and set it to "server" instead of "client" and it seems to be ok.
0 comments:
Post a Comment