I have configured tomcat with the following configurations:
<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" maxThreads="500" maxConnections="20000" acceptCount="150" etc... /> It works fine most of the time, but on peak times, when I have much more requests than usual, it takes too long to respond. Sometimes above 15 seconds and in rare cases timeOut. It may look okay, as maxThreads=500 and I have several thousand requests, however, on Server Status I see:
Max threads: 500 Current thread count: 17 Current thread busy: 1 Keep alive sockets count: 1
The max number of currentThreadCount I have seen so far was 27. If there are so many connections, shouldn't tomcat create more threads (up to 500) to respond faster?
So, what am I doing wrong? What am I missing? I have 2 core CPU (max usage during peak hours ~10%) and 2GB of RAM (max usage 60%).
Short info about web app: normally, each user makes at least 2 requests per session: static JSON response and 1 database query. In peak time I have 15-20k active users, but I don't know how many requests per second do I get. However, slow responses start from 5k active users.
I also increased max-active connections on app properties, with no change on performance, my current application.properties:
spring.jpa.hibernate.ddl-auto=update spring.datasource.driverClassName=com.mysql.jdbc.Driver spring.datasource.url=jdbc:mysql://localhost:3306/database_name spring.datasource.username=$username$ spring.datasource.password=$password$ spring.datasource.tomcat.max-active=200 spring.datasource.tomcat.max-wait=10000 spring.datasource.tomcat.max-idle=50 spring.datasource.tomcat.min-idle=10 spring.datasource.tomcat.initial-size=10 I am adding database query below. Results of the query later added into another object and returned as ResponseBody.
@Query("select new ObjectClass(s.id, s.a, s.b, s.c") from TableName s " + "where s.x > :param order by id desc") List<ObjectClass> getObjects(@Param("param") long param); CPU usage doesn't grow, RAM is almost half-free, if I am having too many requests, shouldn't I have overloaded on the server? Instead, I just get slow response time. Therefore, I think I have a configuration problem which I want to resolve.
-Xms512M -Xmx1024M
The app that hangs on peak time:
Active sessions: 3243 Session count: 475330 Max active sessions: 4685 Rejected session creations: 0 Expired sessions: 472105 Longest session alive time: 7457 s Average session alive time: 9 s Processing time: 3177 ms JSPs loaded: 0 JSPs reloaded: 0
P.S: same numbers for AJP connector, maxThreads=500 and acceptCount="150".
6 Answers
Answers 1
I'm afraid the question does not currently have enough information for anything other than guesses. The low CPU usage indicates that your java process is waiting for something, which could be anything from obtaining a connection to the database, waiting for the result of a query, or anything else. I would start by looking at what is causing the wait, before trying to fix it. One way to do that is to run
jstack <pid> (where <pid> would be your process' pid) during peak. It will list a stack trace for each thread. You may be able to spot the problem from that, or you can paste it into the question and perhaps the community can help you out. Good luck with your tuning!
Answers 2
You can allow as many threads as you want, but if the number of queries increases, then the response time of the RDBMS will deteriorate, which is probably your root cause.
You need to determine where he bottleneck is. Create a dummy page and issue requests to it like a maniac from several computers. If the dummy page responds in time, then your problem is loosely related if at all to connection number and much more to your database. It is highly probable that this is the case.
Take a look at your database, make sure your schema is in normal form. Also, if you search frequently by some columns, make sure you create the correct indexes. Take a look at your queries and observe whether they are unnecessarily slow. If so, optimize them. Cache some data which does not change too frequently and reuse it.
Answers 3
If there are so many connections, shouldn't tomcat create more threads (up to 500) to respond faster?
=> As per Tomcat8 docs, if more simultaneous requests are received than can be handled by the currently available request processing threads, additional threads will be created up to the configured maximum (the value of the maxThreads attribute). If still more simultaneous requests are received, they are stacked up inside the server socket created by the Connector, up to the configured maximum (the value of the acceptCount attribute).
So your tomcat must be creating the threads as required. Also, Tomcat 8 by default works in NIO mode meaning one thread can serve multiple requests. You can confirm that behavior by starting monitoring tool like "jvisualvm" during your load test.
Live threads: This shows the current number of live/active threads including both daemon and non-daemon threads(Currently running).
Live Peak: This gives the peak count of live threads since the Java virtual machine started or peak was reset.
Daemon Threads: This gives the current number of live daemon threads.
Total Threads: This gives the total number of threads created and also started since the Java virtual machine started.
So, what am I doing wrong? What am I missing? I have 2 core CPU (max usage during peak hours ~10%) and 2GB of RAM (max usage 60%).CPU usage doesn't grow, RAM is almost half-free if I am having too many requests, shouldn't I have overloaded on the server? Instead, I just get slow response time.
=>IMO, threads are blocking while fetching data from DB. It could be due to the poor performance of query during load times. I would suggest enabling "hibernate.show_sql" capture the SQL. Check the execution plan of the SQL, ensure that indexes are being applied. You can also check the performance of query during load time, by executing it on SQL client.
Answers 4
When using spring the default is Tomcat. How ever you can use Netty or Undertow or Jetty for better performance. Please also remember that despite having a 2 core CPU you don't really have 500 threads.
How ever the above answer to actually simulate how your application reacts to traffic is probably the best way to go. If using relation database remember that writes can be even ten times slower than reads (you can see some interesting statistics on that in Cassandra documentation). If using hibernate you may want to look for n+1 problem too. Best way to do that: write an integration test, log sql sent to database. If your test sends 51 instead of one query there you have it.
Answers 5
please check IO, It is not necessarily the case that the top number of the thread is a reason for more speed. If, for example, your IO is high, it can be a solid reason
Answers 6
In these kind of bugs , First of all we should identify where is problem . Here giving a plan of action for deugging these type of issues :
For example in your case
Requests come from user to tomcat , then it will give it to your application .
First of all , check where is issue i.e. there can be issues in following places :
- Your any application api or all api started taking time , but tomcat threads are free .
- Your tomcat threads are not free and processing of these each thread is taking time , so latency occurs .
- You database starts taking time .
- As , you are quering database , there may be case that more data is being loaded into your app and some java gc issues started occurs.
So, in first case , please check your application logs and if logs are not there please put the logs and check , if any of your application is taking time (Logs Never Lies )
In Second Case , check your tomcat logs that what is condition there .
In third case , please check your database logs , that queries is taking more time or not .
In fourth case , you can monitor your java health monitoring , there are many tools in market like jfr, jcisualvm etc ..
Also , Your question has not enough explanation , So, please answer following
- what is sample structure of your application ?
- What you do to bring back your application not normal state , for example , restarting solves your issue or not . I am asking this , beacuse if you need to restart it , then there may be deadlock , in case of it , you need to take jstack and analyse it .
- How much XMX is given to Applcation .
- Is your databse server and Applicaion server is on same machine ? Beacuse , there may be some io problem in peak time on some machine , so we need to check both .
So , please identify first where is problem , then we can proceed further , how we can identify and solve the issue .
Thanx .
0 comments:
Post a Comment