Java应用常见性能问题及排查方法－高CPU by 刘长炯 beansoft@126.com github: beansoftapp Objectives After completing this lesson, you should be able to: • Identify high CPU usage • Learn the steps to identify high CPU usage threads • Describe resolution steps for the causes of this pattern 6-2 Road Map • High CPU Usage Pattern – Review: Processes, Threads, and CPU Usage – Symptoms of High CPU Usage • • 6-3 High CPU Usage Investigation Troubleshooting Checklist Review: Processes and Threads • A Process: – Is identified by an operating system process ID (PID) – May be single or multithreaded • A Thread: – Is a subtask of a process – Will use resources—for example, it will use the CPU independently of other threads – May map to a PID – May be identified by a lightweight process ID Process X (LWPID), which relates to the parent process LWPID: 1 Thread ID: 127 6-4 2 129 3 133 Review: Application Server A Application Server instance is: • An instance of a JVM that runs as a server • A multithreaded process, where threads include: – Listen threads, for receiving incoming requests – Socket Reader threads, for routing incoming requests – Execute threads, for running application and server code Server Reader threads 6-5 Execute threads CPU Usage A machine’s CPUs process code on behalf of: • Multiple processes on the machine • Multiple threads within a process App Server Multiple threads 6-6 Process Y Process X Single thread Multiple threads High CPU Usage • • • 6-7 Occurs when one process or thread uses an unexpectedly high percentage of the CPU May deprive or starve other processes or threads of CPU power for performing their required processing Should be monitored periodically High CPU Usage Symptoms • Symptoms of high CPU: – The response time to users is slow. – App Server machine becomes unusually slow. – Requests or operations start to time out. • Monitor CPU utilization with the following tools: – On Windows: Task Manager or Perfmon – On UNIX: vmstat <interval> – On Mac: top or Activity Monitor 6-8 Section Summary In this section, you should have learned about: • Processes and threads • CPU usage • How high CPU usage might be detected 6-9 Quiz Multiple threads can execute within a single process. 1. True 2. False 6 - 10 Road Map • • High CPU Usage Pattern High CPU Usage Investigation – Investigating on Various Platforms: — — — • 6 - 11 Linux Windows Mac Troubleshooting Checklist Overview of Investigation To determine which Server threads cause high CPU usage: 1. Take thread dumps when high utilization occurs – Note: Not all the ways to take thread dumps produce suitable output. 2. Locate the high usage thread in the server thread dump – This will vary depending on the platform. – This may involve several steps. 3. Repeat the preceding steps to ensure that you have established a pattern and identified the correct thread 4. Use the thread dumps to investigate the specific cause of the high CPU usage 6 - 12 Investigating: On Linux On Linux, each thread appears as a separate lightweight process. When high CPU usage occurs in Linux, repeat these steps to snapshot the server’s activity: Use ps aux –L to determine which threads are using the CPU. Take a server thread dump, using: kill -3 <javapid>. Then use the output as follows: Get the LWPID of the thread with the highest CPU usage. Verify that this thread is Application Server and not some other rogue process. Convert the high usage LWPID to hexadecimal. Find the hexadecimal LWPID in the server thread dumps. Determine what the thread was doing to cause high CPU usage. 6 - 13 Example: Linux ps and Thread Dump Example of ps output: USER PID LWP %CPU NLWP %MEM VSZ RSS TTY STAT START TIME COMMAND ... weblogic 29251 29322 0.0 32 38.8 808192 200536 Sl+ 11:53 0:00 /opt/bea/jdk150_06/java ... 6 - 14 ? Example: Linux ps and Thread Dump And if LWPID 29322 = 0x728a Scan for the hexadecimal LWPID and Example of kill -3 output: check what the thread is doing. . . . "weblogic.store.WseeFileStore" daemon prio=1 tid=0x08271f00 nid=0x728a waiting on condition [0x88f51000..0x88f51db0] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSuppo rt.java:118) at java.util.concurrent.locks.AbstractQueuedSynchronizer $ConditionObject.await(AbstractQueuedSynchronizer.jav a:1767) at java.util.concurrent.LinkedBlockingQueue.take (LinkedBlockingQueue.java:359) . . . 6 - 15 Using a Script on Linux To capture data rapidl